Bridges is slightly different from Carbonate
Getting access to Bridges
File transfer to Bridges
Filespaces on Bridges
Submitting jobs on Bridges
Why move to Bridges
Genome assembly requires large memory and most likely many of you have already been running genome assembly using the high memory nodes with < 250GB. In some cases, 500GB is not enough, and most have seen a similar error from your job script.
In this job script, I am running MUSCLE- Multiple Sequence Alignment programs with 3 cores and assigned 50 GB memory. The below message shows that job has already used up 50GB and exited with an error-suggesting this job requires more memory. The solution is to increase the vmem option in my job script to maybe, 100GB or 250 GB. In the case you have run out of 500GB memory, there is no other available higher memory node on Carbonate.
NCGAS partners with Pittsburg Supercomputing Center (PSC)to provide researchers with access to Bridges that provides large memory nodes with 3TB and 12TB. To make this transition easy, this guide helps you get started on Bridges.
Bridges is slightly different from Carbonate
Bridges is hosted by Pittsburg Supercomputing Center (PSC) and is part of XSEDE resources. For information on bridges system configuration go to this page.
- Different computers, different file systems, so the data is not shared between the two computers. For example, logging into Carbonate, you can access the files on dc2/scratch, and files in your Karst directory (/N/u/username/Karst). The file system is shared between the computers, but not with Bridges. These file paths “N/dc2/scratch/username” doesn’t exist in Bridges. This means the files need to be moved from Mason/Carbonate to Bridges. More on how to do this is explained later
- Job schedulers are different, so the job scripts are written for IU computers: Carbonate needs to be altered for Bridges. Carbonate uses a job scheduler called TORQUE and Bridges uses another called SLURM. More on how the job script needs to translated can be found here or use the awesome power of Google.
Getting access to Bridges
Go through the next couple of steps to get started on Bridges.
1. Allocation
Here are you allocation options:
(i) get added to NCGAS allocation -a trial allocation then you can get your own if this resource is optimal for you. Remember this is shared allocation between a set of researchers trying out this resources.
To get into NCGAS allocation, go to and click on link “Request allocation” for PSC Bridges. This will take you to an XSEDE login page. An XSEDE username is necessary to login to Bridges. Sign up, if you don’t already have a username.
Once you have XSEDE username, fill in this survey. After submitting the survey, we will receive an email and add you to our allocation.
(ii). In the case you have an entire project where you require large memory nodes, we recommend getting your own allocation – More information on this can be found here.
How do you confirm you have Bridges allocation?
Through either of the above options, once you have requested for allocation, an email will be sent when your account is made on bridges. Also, you can also see this resource added to your XSEDE User portal, under “My Projects” section or Bridges will be listed here.
2. Logging in to PSC Bridges
Request to set a password via email instructions from PSC. Go to here, change the password, get the security code to email. Finally setup a new password. DONE! For more information.
Hostname: bridges.psc.edu.
port: 2222
username: PSC username (from email),
password: to set password PSC password
You are in!
Optional: Setting up ssh keys
- Go to here, to the section “Authenticating using a public-private key pair”.
- Login with PSC credentials, add the rsa- public key
- Once the key is added wait for a confirmation email, you should receive a mail within 24 hours, requesting confirmation.
Then you can add the private key to your login and forget your password happily 😊.
Note: Logging in with a key, changes in one aspect, public key authentication is intentionally disabled for GSI SSH (port 2222)
Hostname: bridges.psc.edu.
port: 22,
username: PSC username (from email),
SSH-authentication- include your key
File transfer to Bridges
Transfer data from Carbonate to Bridges via Globus connect
A detailed explanation of setting up Globus Connect can be found in our blog post and more information can also be found here
Once you have setup Globus account – Go to globus app.
Then go to endpoints list, both Bridges, and dc2 already have Globus endpoints setup, all the user needs to do is activate the endpoint, which is what is being explained below.
Select “search all” then type “psc#bridges-xsede” in search.
Select”PSC Bridges with XSEDE authentication”, provide XSEDE authentication
In the below case, I have already activated which is why it says “expires in 10 days” under credentials and “ready” under status.
If you haven’t already, you need to activate dc2 endpoint, similarly. Go to endpoints, search for “iu#dc2”. Select and activate with IU login ID and password. Once its setup, under “Endpoints” you should be able to see them listed with status as “ready”.
In the below case I have three endpoints, Bridges, dc2(IU cluster) and my laptop.
To transfer files, Click on “Transfer Files”. Select the endpoints and transfer files
Once the data is transferred successfully, an email will be sent. Then you can log in to Bridges and start the analysis.
A really helpful hack
Once you are logged in to Bridges, run the command below to symlink your scratch space to your home directory as “scratch” folder:
ln -s $(echo $SCRATCH) scratch
The reason for this step is that in Bridges scratch space allows long-term storage and it doesn’t have an easy file path to remember. If you have a symlinked scratch folder in your home directory when you log in to the PSC Bridges endpoint on Globus, your home directory will have a scratch folder and you can transfer to the long-term storage directly.
Alternatives to data movement via ftp,sftp, scp, rsync
Alternatives to data transfer can include ftp, or sftp which is also possible, more information on that can be found in bridges documents or here.
File spaces on Bridges
Before you start an analysis, another bit of useful information is the different file spaces on Bridges and in comparison with Carbonate to help.
| File spaces | Carbonate | Bridges |
|
Home directory Disk Space Storage |
$HOME or /N/u/username/Carbonate/ 100GB Long-term |
$HOME or /home/username 10GB Short-term |
|
Scratch space Disk Space Storage |
/N/dc2/scratch/username Fluctuates Short term, unaccessed files are purged every 60 days |
$SCRATCH or /pylon5/chargeid/username Based on the proposal approved not purged. Long-term storage |
More information on different file systems on Bridges, click here.
Submitting jobs on Bridges
Some of you may have already been submitting jobs on Carbonate that uses Torque job schedulers. The difference is that Bridges uses a different job scheduler called SLURM. For more information on TORQUE, refer here and SLURM, refer here.
Why do you need to know this? Different job schedulers = Difference in job scripts.
If you are interested in taking on the challenge and translate your job scripts yourself, here is how to translate between TORQUE to SLURM and vice-versa, here is a guide to help you.
In most cases, genome assembly is where the transition is required from Carbonate to Bridges for more memory. So here is an example of job script for a large memery (LM) queue and for more information, click here.
Batch submission
Job script for LM partition – large memory partition
#!/bin/bash
#SBATCH –mail-type=ALL
#SBATCH –mail-user=”$EMAIL”
#SBATCH –job-name=”$JOBNAME”
#SBATCH –nodes=1
#SBATCH –ntasks=1
#SBATCH -p LM
#SBATCH -t 5:00:00
#SBATCH –mem=2000GB
#echo commands to stdout
set -x
#move to pylon5 filespace
cd $SCRATCH
#copy input file from your home space
cp -r $HOME/example_files/fq #if the file is not already in your scratch space
#Set up an environment
module load spades
#command
spades.py -1 right.fq -2 left.fq –only-assembler -o test
Submitting job script
Replace jobname.sh with the actual job script name.
sbatch -p LM – t 10:00:00 –mem 2000GB jobname.sh #for LM partition memory must be mentioned
Job submission commands translation
| Commands | Carbonate | Bridges |
| Submit jobs | qsub | sbatch |
| Status of jobs | qstat -u “username”qstat -i “jobID” | squeue -u “username”squeue -i “jobID” |
| Delete a job | qdel -i “jobID” | scancel -i “jobID” |