External Transfer

We currently offer two ways to upload/download data from/to CSCS:

• The CSCS globus online endpoint (recommended)
• The globus-url-copy command line tool (deprecated)

Globus Online Endpoint (recommended)

The recommended way to transfer data externally occurs via the CSCS globus-online endpoint (www.globus.org). Please follow the official get started documentation to login (in case you don't have an organisation account, you can just use the option "Sign in with Google"):

Step 2 and 3

• Use the file manager to search for an endpoint typing "CSCS"
• Please make sure that the login page belongs to the cscs.ch domain (shown in the URL)
• The CSCS endpoint requires authentication, therefore use your CSCS credentials to log in

Step 4

• if you want to transfer the data to another endpoint, just search for it and transfer the data
• if you want to download the data to your local system, you will need the Globus Connect Personal client: the client will turn your local system into an endpoint, so you will be able to select it and transfer the data
• download the client from the link provided above and follow the instructions

For more information about Globus Connect Personal, please read the official Frequently Asked Questions.

Globus-Url-Copy (deprecated)

The globus-url-copy client is still working, but it is now deprecated. Therefore, we suggest to use the recommended Globus Online endpoints whenever possible.

GridFTP client setup and authentication

Any user can install GridFTP on a local workstation following the instructions provided by the grid community forum. The GridFTP client is already available on the CSCS front-end system Ela (ela.cscs.ch) and it is accessible only via SSH keys. Therefore you need first to create the SSH keys on Ela (we recommend to NOT set an empty passphrase) and then place the public key within the authorized_keys file in the CSCS home directory ($HOME), as follows:

$ pwd
 /users/<username>
$ ssh-keygen
 …
$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys

Then you should be able to log in to Ela with the SSH key just created and to use gridftp.cscs.ch for your data transfer. For a file transfer between CSCS (using gridftp.cscs.ch) and a remote site, also the remote endpoint has to support GridFTP with SSH authentication.

The GridFTP data transfer can be started with the GridFTP command globus-url-copy on the front-end system Ela (ela.cscs.ch). Therefore to transfer a file from CSCS to a remote site, you need first to log in to Ela and then run the following command:

$ globus-url-copy \
   sshftp://<cscs_username>@gridftp.cscs.ch/<path>/<filename> \ 
   sshftp://<remote_username>@<remote_server>/<remote_path>/<remote_filename

Please change the variables in between angle brackets (<...>) according to your needs. There are several options to optimize the transfer speed, for instance the number of parallel streams: you can type the command man globus-url-copy for the full list of options. Please try first with the default, since in most cases you'll not need to change them. Below you can find an example of a transfer which opens four parallel streams, using the option -p 4: 

$ globus-url-copy -p 4 \
   sshftp://<cscs_username>@gridftp.cscs.ch/<path>/<filename> \
   sshftp://<remote_username>@<remote_server>/<remote_path>/<remote_filename>

The GridFTP server at CSCS has access to the following filesystems:

/project
/store
/scratch/shared
/scratch/snx3000
/capstor/scratch

Therefore the <path> variable of the examples above can access any of them, even if the command is executed on Ela. In case the remote site doesn’t provide a GridFTP server with SSH authentication you can pull data from CSCS provided that the remote site provides a node where:

• you can login
• the destination file system is mounted
• you can use the command globus-url-copy

From that node you have to run a command like the following:

$ globus-url-copy \ 
   sshftp://<cscs_username>@gridftp.cscs.ch/<path>/<filename> \ 
   file:///<local_path>/<local_filename>

Troubleshooting

error: an end-of-file was reached

This is the most common error that can occur and there are multiple reasons for this error. Please try to run the command below to start debugging:

$ globus-url-copy -list sshftp://<cscs_username>@gridftp.cscs.ch/

The command should return just a list of the accessible directories:

project/
scratch/
store/
...

Please check the following list of possible outcomes:

1. if you do not get the folders listed above, then you might have a problem with your ssh keys. Please check again if the public SSH key of your local system is regularly stored in the authorized_keys in ELA

2. if your SSH keys are fine and you get the output above, then perhaps you are looking for the wrong path. Please use the option --list to check one by one every sub-directory until you find the error. Do not forget that the directory might exist, but with the wrong access permissions, therefore you might not be able to read its content

3. if you are moving data between two gridftp servers, the error originate from the external server. Please use the option --list to list the target directory of the external server: if an error is returned, please contact your local support

If none of the above solves your issue, please contact us with a request for support.