Bridges is slightly different from Carbonate
Getting access to Bridges
File transfer to Bridges
Filespaces on Bridges
Submitting jobs on Bridges
Why move to Bridges
Some analyses, like genome assembly, require large memory. Some users have already run genome assembly using our high memory nodes with < 250GB. However, in some cases, 500GB is not enough and may result in an error from the job handler.
In the job script below, I am running MUSCLE (Multiple Sequence Alignment programs) with 3 cores and assigned it 50 GB memory. The message below shows that job used up 50GB and exited with an error, suggesting this job requires more memory. One solution is to increase the vmem option in the job script to 100GB, 250 GB, or even 500GB. If the case you have run out of 500GB, there are no higher memory nodes on Carbonate.
NCGAS partners with Pittsburg Supercomputing Center (PSC) to provide researchers with access to Bridges, a supercomputer that provides large memory nodes with 3TB and 12TB. This guide is designed to make the transition from Carbonate to Bridges easy.
Bridges is slightly different from Carbonate
Bridges is hosted by Pittsburg Supercomputing Center (PSC) and is part of XSEDE resources. Please see here for information on Bridges system configuration.
- Different computers, different file systems, so the data is not shared between the two computers. For example, the Slate file system is shared between computers, but not with Bridges. The file path “/N/slate/$USER” doesn’t exist in Bridges. This means your files need to be moved from Carbonate to Bridges. More on how to do this later.
- Job schedulers are different, so the job scripts are written for IU computers: Carbonate needs to be altered for Bridges. Carbonate uses a job scheduler called TORQUE and Bridges uses another called SLURM. More information on converting your job script can be found here, or use the awesome power of Google.
Getting access to Bridges
The steps to get started on Bridges are described below.
1. Allocation
Here are you allocation options:
(i) get added to an NCGAS trial allocation, then get your own allocation if this resource is optimal for you. Remember, this is a shared allocation between a set of researchers trying out this resource.
To join an NCGAS allocation, go to http://ncgas.org/hps.php and click on ‘Request Bridges allocation’. This will take you to an XSEDE login page. An XSEDE username is necessary to login to Bridges. Sign up if you don’t already have a username.
Fill out this survey once you have an XSEDE username. NCGAS will receive an email when you’re added to our allocation.
(ii) In the case you have an entire project that requires large memory nodes, we recommend getting your own allocation. More information on this can be found here.
How do you confirm you have a Bridges allocation?
With either of the above options, you’ll receive an email when your Bridges allocation is active. You will also see this resource added to your XSEDE User Portal here under the “My Projects” section.
2. Logging in to PSC Bridges
Request to set a password by following PSC’s emailed instructions. Change your password here. Finally, setup a new password. DONE!
Hostname: bridges.psc.edu.
port: 2222
username: PSC username (from email),
password: to set password PSC password
You are in!
Optional: Setting up ssh keys
- Go here to the section titled “Authenticating using a public-private key pair”.
- Login with your PSC credentials, then add your rsa-public key.
- You should receive an email confirming the key is added within 24h.
Then you can add the private key to your login and happily forget your password. 😊
Note: Logging in with a public key authentication is intentionally disabled for GSI SSH (port 2222)
Hostname: bridges.psc.edu.
port: 22,
username: PSC username (from email),
SSH-authentication- include your key
File transfer to Bridges
Transfer data from Carbonate to Bridges via Globus connect
A detailed explanation of setting up Globus Connect can be found in our blog post. More information can also be found here and here.
Once you set up a Globus account you can access the Globus Web App at https://globus.iu.edu/.
Select ‘ENDPOINTS’ from the menu on the left-hand side of the web app. Bridges and Slate already have endpoints configured, but you will need to activate them following the steps described below.
Type “psc#bridges-xsede” in the search bar at the top of the Endpoints menu.
Select ‘PSC Bridges with XSEDE authentication’ to see details for the endpoint. Click ‘Activate’, then provide your XSEDE username and password .
Once your endpoint is successfully activated the icon next to it will appear green.
You may also need to activate your Slate endpoint. Go to endpoints and search for “IURT-Slate”. Select and activate the endpoint with your IU login ID and password.
To transfer files, select ‘FILE MANAGER’ from the menu. Select the endpoints from the right and left panels, then click ‘Start’ to begin the file transfer.
In the example below, I am transferring the file named ‘test_BDD_CEM_fst.weir.fst’ from my Slate account to my Bridges home directory.
You will receive an email once the data transfer is complete.
A really helpful hack
Once you are logged in to Bridges, run the command below to symlink your scratch space to your home directory as “scratch” folder:
ln -s $(echo $SCRATCH) scratch
The reason for this step is that scratch space on Bridges allows for long-term storage, but it doesn’t have an easy file path to remember. A symlinked scratch folder in your home directory allows you to directly transfer files to your scratch space using the PSC Bridges endpoint on Globus.
Alternatives to data movement via ftp,sftp, scp, rsync
Alternatives to data transfer can include ftp, or sftp which is also possible, more information on that can be found here.
File spaces on Bridges
Before you start an analysis, another bit of useful information is the different file spaces on Bridges and in comparison with Carbonate to help.
| File spaces | Carbonate | Bridges |
|
Home directory Disk Space Storage |
$HOME or /N/u/username/Carbonate/ 100GB Long-term |
$HOME or /home/username 10GB Short-term |
|
Scratch space Disk Space Storage |
/N/slate/$USER initial quota 800 GB Long-term |
$SCRATCH or /pylon5/chargeid/username Based on the proposal approved Long-term |
Click here for more information on Bridges’ different file systems.
Submitting jobs on Bridges
Some of you may have already been submitting jobs on Carbonate that uses Torque job schedulers. The difference is that Bridges uses a different job scheduler called SLURM. For more information on TORQUE, refer here and SLURM, refer here.
Why do you need to know this? Different job schedulers = Difference in job scripts.
If you are interested in taking on the challenge to translate your job scripts, here is a guide to help you translate between TORQUE to SLURM and vice-versa.
For most users, genome assembly is where the transition from Carbonate to Bridges for more memory is required. Here is an example of a job script for a large memory (LM) queue. Click here for more information.
Batch submission
Job script for LM partition – large memory partition
#!/bin/bash
#SBATCH –mail-type=ALL
#SBATCH –mail-user=”$EMAIL”
#SBATCH –job-name=”$JOBNAME”
#SBATCH –nodes=1
#SBATCH –ntasks=1
#SBATCH -p LM
#SBATCH -t 5:00:00
#SBATCH –mem=2000GB
#echo commands to stdout
set -x
#move to pylon5 filespace
cd $SCRATCH
#copy input file from your home space
cp -r $HOME/example_files/fq #if the file is not already in your scratch space
#Set up an environment
module load spades
#command
spades.py -1 right.fq -2 left.fq –only-assembler -o test
Submitting job script
Replace jobname.sh with the actual job script name.
sbatch -p LM – t 10:00:00 –mem 2000GB jobname.sh #for LM partition memory must be mentioned
Job submission commands translation
| Commands | Carbonate | Bridges |
| Submit jobs | qsub | sbatch |
| Status of jobs | qstat -u “username”qstat -i “jobID” | squeue -u “username”squeue -i “jobID” |
| Delete a job | qdel -i “jobID” | scancel -i “jobID” |