Note: This blog post has been updated on August 22nd since the previous blog post had some commands that were part of the legacy CLI, which has been deprecated as of August 1st, for more information on that go here.
Jetstream is an XSEDE, cloud computing that provides access to preconfigured virtual machines (VM) with root access. This is an NSF funded project between Indiana University and Texas Advanced Computing Center (TACC), for more information on Jetstream go here.
Interested in using Jetstream for your research here is a tutorial on “Getting started on Jetstream“.
File transfer to/from Jetstream
File transfer to a Jetstream instance can be done in several ways. For small files, such as a simple two-line text file, copy pasting the content is probably just as quick as transferring the file. When you get to larger or multiple files, such as a directory (folder) or bigger file, you can transfer it using:
- WinSCP , FileZilla (windows users) or
- CyberDuck (windows and OS).
Using these commands to transferring files and managing data is also covered here.
However, when you have really large files, such as raw sequence data, it can be a bit tedious to transfer files via command line, not to mention problematic if your internet connection drops out. In these cases, Globus is a favorable alternative. Globus allows you to transfer gigabytes or terabytes of data securely and quickly.
Before you begin
Jetstream being an XSEDE resource and already has a globus subscription but endpoints are NOT already setup on the preconfigured virtual machines. The user must set up a globus endpoint themselves.
This is a tutorial walking through setting up globus endpoint on the Jetstream virtual machine. To setup globus on your laptop/computer or transfer files between cluster <-> computer, go to here.
Setting up Globus endpoint Jetstream VM
These are basically the instructions for the set up of Globus on any Linux machine. However, an example on Jetstream with screen captures is provided to assist newer users.
Hopefully, you should already have an account on Globus – if not go to here, and have a Jetstream virtual machine setup, if not go here. In the following instructions, lines in red are commands, the portions in purple indicate that you need fill in the information. I have also attached screenshots of how a screen will look if the command ran successfully for the first few commands, to gain some confidence that you are doing it right.
Log into the VM (terminal in WebDesktop, Webshell or ssh- your preference). You can access the WebDesktop (red) and Webshell (black) with the following links when you click on an active image in your Jetstream Atmosphere Portal. In this case, you will want to use the Web Shell.
You can also run these commands via putty (Windows), ssh command on terminal (LINUX/MAC), you can log in to your VM using your username, IP address (can be found under Jetstream instance details), your key, port 22. This can make things a bit easier if you are comfortable using terminals. For more information here.
If you are working on the Webshell, the main weirdness you will run into is using copy or paste from your computer to Webshell and vice-versa. Here is the trick, on the WebShell:
Windows (shift+ctl+alt)/Mac (shift+ctrl+cmd) -> This brings up a sidebar, with a box for use as a common clipboard.
- To paste from your computer to the VM, you will have to paste the text here. Then, exit the sidebar with shift+ctl+cmd, and paste into the shell with shift+ctl+alt/shift+ctrl+v(mac).
- To paste from the VM to your computer, highlight the text, bring up the sidebar, and make sure the text is in the common clipboard, then paste as normal.
- To paste from the VM to the VM, you can just highlight the text and paste as normal (not going across computers).
Once you are in a terminal of your choice, follow these steps:
- In your home directory, or wherever you install files, download globus connect using this command: wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
- Unzip the globus connect file downloaded:
tar -xvzf globusconnectpersonal-latest.tgz
Goto https://app.globus.org/endpoints on your laptop or on WebDesktop, login to Globus using your account. This page should list all your endpoints on your globus account, in the figure below you can see my endpoints listed.
- Once you click on the “Create new endpoint”, you will be taken to a new page, click on “Globus Connect Personal”.
- Fill in the “Display name” with what you would like this endpoint to called and select on “Generate Setup Key”. Copy the setup key, you will need this to setup Globus Endpoint on the Jetstream VM.
- Back to the terminal, go to the globusconnect directory. In our case, we installed in the home directory. Also make note the version may have changed since the blog post.
cd ~/globusconnectpersonal-2.3.6
- To link the endpoint you created above to the jetstream VM, run the below command, replacing the Setup key with the copied setup key (look at the above image if you are not sure what Setup key is)
./globusconnect -setup Setup_key
./globusconnect -start &
Globus connect endpoint is now setup and files can be transferred to and from Jetstream to the local computer or other places with Globus endpoints already set up. (Ex: /N/dc2/scratch/username- if you have access to IU cluster.
- Transfer files using globus connect online https://app.globus.org/file-manager
- To stop globusconnect connection in Jetstream terminal, type
./globusconnect -stop
DONE! You are all set!
Globus Command Line Interface (globus-cli)
If you are comfortable with the command line you may have already noticed that the above globusconnect personal is great to link already generated endpoints (using setup key), start and stop globus and that’s it.
If you prefer to create endpoints using the command line as well, you need to download globus-cli, first. To do that use the following tutorial https://docs.globus.org/cli/
Common Questions
I have logged in to globus but can’t find IU#dc2 endpoint to transfer files to IU computers?
Here is the trick you need to login to globus using your google account, which means,
- you need to have your google account (username@gmail.com), xsede account (username@xsede.org), IU ID (username@iu.edu) or other accounts all listed under accounts.
- Now logout and log back in using your Google credentials
- Search for the endpoint IU#dc2 and you should be able to connect to this endpoint.
If you have any questions mail us at help@ncgas.org