Submitting jobs is a necessary part of working on a cluster. We will be talking about this quite a bit in the near future, going from the basics to building in checks on the output and making pipelines in a mini series on job management. First up, submitting jobs!
NOTE: This is written specifically for the IU clusters, however, much of this is generally applicable with slight changes to syntax. IU runs torque as a job handler, and any machine with the same job handler will have similar syntax. Other job handlers, such as Grid Engine, slurm, etc. will have the same parts, but they may look slightly different.
Job Queues
You can run jobs without having to be signed in or wait on them. They are submitted to the “queue”. Any job that will take more than 20 minutes must be submitted to the queue (there is one for Mason and one for Karst, just depends on where you are logged in).
Job Submission
In order to submit to the que, you need to make a job file. They work kind of like an executable.ba would – they run the code in the file. However, additional information is needed in the header. It looks like this (all #comments are added):
#PBS -k oe #keep output and error
#PBS -m abe #mail me when job : a – abort, b – begins, e – ends
#PBS -M <your email>
#PBS -N <name of job>
#PBS -l nodes=1:ppn=1,vmem=16gb,walltime=2:00:00 #See note below
#The –l flag requires you to set how long and what resources you are requesting. Please see https://kb.iu.edu/d/bdkd for information on this setting.
#Load modules required
module load java/1.7.0_40
module load fastqc
#you must enter the directory where your data is, with an absolute path
cd /N/dc2/scratch/ss93/Analysis
#call the program as you would normally
fastqc NewData.fq
Managing your jobs
Submitting a job
qsub <JobFile>
What is the status of my job (for example if you are me/ss93)?
qstat -u ss93
NOTE: q – waiting in the queue, r – running, c – complete
Delete a job (the number is listed when you use qstat
qdel 60123
What’s going on with my job?
Your job will create a file, entitled whatever you used in the -N flag above (i.e. RunTrimmerMagna1) followed by a .o and the job number (i.e. RunTrimmer.o23453). You will also see a .e file of the same sort (i.e. RunTrimmer.e23453). These are your STDOUT and STDERR outputs, respectively. They will give you the output on the job.
Tips
- Try running commands before putting them in a job file. This ensures they will run correctly. You can always stop them with ctrl c.
- Create a blank job file and just copy it to another file name when you are creating a new job submission
- Have some sort of naming paradigm for jobs. I call all mine RunWhatever. Some people give them a .job extension. It doesn’t matter what you do, but it helps when you are looking for them in your folders.
- You can run multiple things in parallel if you put an ‘&’ at the end of each command and then ‘wait’ at the end of the file. This will run all commands in the background at the same time, and wait until all background jobs are done before terminating. If you forget ‘wait’ the job will launch, push everything to the background, run out of code to run, and terminate. If you use &, you must use ‘wait’!
- MAKE YOUR JOBS FLEXIBLE – you will be running them multiple times. If you set it up so that the input is at the top as a variable, you can quickly swap inputs when adjusting the analysis to different data, parameters, etc. Below is an example of how to do this:
…
#you must enter the directory where your data is, with an absolute path
cd /N/dc2/scratch/ss93/Analysis
#Set up input/parameters
right=NewData_1.fq #EASY TO SWAP WITHOUT HAVING TO CHANGE THE CALL
left=NewData_2.fq
output=/N/dc2/projects/myproj/trinity_out
memory=200G
#call the program as you would normally
trinity –seqType fq –max_memory $memory –left $left –right $right –CPU 6
…