Errors

If you are running into an error and not sure what it means, ask for help! We have a blog walking through a few steps you can take on, where to look for errors and how to read them. Even after finding the error it might not make sense, especially if it’s just printed out code. Reading these errors and debugging them takes practice, so it’s best not to overthink and reach out for help sooner rather than later. 

Initial steps

Stepping back a little, the most common errors are that the command was incorrect; incorrect flags or parameters, cannot find input files, or requested resources are too low. 

For these errors, take a look at the job script and check if the script is correct. To help debug these errors, here is a blog post to help. In case the resources requested are incorrect, see here for information pertaining to each of IU’s supercomputers. You can also use the HPC Everywhere site to help write the job script for IU machines (HPC Everywhere found here). 

If you are still unsure or can’t figure out the error, email us at help@ncgas.org, we are happy to help! In the email, make sure to include the following information:

• The program you are running with version
• The compute cluster you are running the program on
• The job script, job error logs (both the .e and.o files, and any other outputs you have handy)
• If you can easily share the input files you used, that would be a big help for us too!

This will help us and the support team narrow down the error and run some tests themselves before sending you the fix. In some cases, the error may be unique to your dataset, and they will require you to give them access to your data securely. 

Data access to help debug the errors

In some cases, the problem may be unique to the dataset the user is running, so we will need access to the dataset. We have encountered common misconceptions from users:

These directories are owned by you, with permissions to access, read or write only to YOU, unless you have changed the permissions. If you know how to do this then you can probably skip this blog since you know what we will be talking about. If you are interested in learning how to read and change permissions here is a blogpost to help.  

More explanation on why we don’t have access to your datasets 

For example, let’s say the data you are trying to share is in the /N/dc2/projects directory. Go into the project directory and run the command “ls -lah”.

For example, in /N/dc2/projects/ncgas/, running the command “ls –lah” gives the following output:

drwxr-xr-x  324 root    root    20K    Jan  22   09:43   ..
drwxrwsr-x+ 15  ss93    ncgas   4.0K   Nov  1    17:22   example_files
drwxrwsr-x+ 17  bhnala  ncgas   4.0K   Sep  14   09:39   genomes
drwxrwsr-x+ 4   ss93    ncgas   4.0K   Jul  6    2018    scripts

The format for the above lines is defined below, and more information is available here. 

Permissions     number of links    owner     group    size     date-time     directory/filename

In your specific project directory, it’s likely that you see a different group name that you belong to. How do you know what groups you are part of? Run the command “id”.

For example, if I run the command “id” my output is the following, 
uid=1616918(bhnala) gid=236(uits) groups=236(uits),1135(ncgas)

This means I am part of the groups bhnala, uits, and ncgas, and I do not have access to other directories that belong to other groups unless they have permissions open to everyone. 

Giving us access to the data

Here are a couple of ways to give us access:

• Copy your datasets over to “/N/dc2/projects/ncgas/Userdata/”. We have changed the permissions so everyone can access this directory. Once you copy the dataset here, we will copy it over to our local directory to help debug the error.  If you copied over the files, no permission needs to be changed. But do note, anybody can access this directory! Just to confirm though, after copying the files make sure to run “ls -lah” and confirm the file has the permissions for group “ncgas” to read and write (-rw-rw—-), or all users to read and write (-rw-rw-rw-). If you are not sure how to change permissions, here is a blogpost to help. 
• Alternatively, if you are dealing with a really large file, copying over the file will take a long time. Instead, you can symlink the file to the /N/dc2/projects/ncgas/Userdata/. Run the command,

ln -s file.fasta /N/dc2/projects/ncgas/Userdata/

However, the permissions still need to be changed for us to copy the data over, or read the symlinked data. How do you change the permissions?Let’s say I am working in the Tetrahymena project directory and would like to give the NCGAS team access to these files through symlinking. First symlink the file from my project directory to ncgas project space:

ln -s file.fasta /N/dc2/projects/ncgas/Userdata/. 

• Next, change the permissions – start with the project directory itself, so the NCGAS team can enter this project directory:

chmod -R o+x Tetrahymena

The e(x)ecute permission allows access to the directory, but since we aren’t changing the (r)ead or (w)rite permission, NCGAS can’t read or write to this directory, just enter it. 

Then give permissions to the directory that contains the file.fasta, “/N/dc2/projects/Tetrahymena/input_files/” by running the command below. The command gives NCGAS the permissions to enter “/N/dc2/projects/Tetrahymena/input_files/”, and read the files in this directory. Since this directory does have file.fasta that I am trying to share, NCGAS team can now see the file and copy it over as necessary. 

chmod -R o+rx input_files

Permissions is another topic that takes some practice getting used. For more information on file permissions, click here, or contact us. 

• Depending on the sensitivity of the data or the amount of help requested, the other alternative is letting us join the project group itself so that we can access all the files directly in the project folder itself without having to change permissions. 
• We can also make special directories in our scratch folders for your files if your data can’t be otherwise shared. Just request this when contacting us and we can use Access Control Lists (ACLs) on IU machines to fine-tune access.

Once we have the permissions to the file, we can start debugging without trouble and will get back to you as soon as we figure out the issue.

Hope this helps and if you have any questions, email us at help@ncgas.org.