Skip to content

Moving data on and off the cluster


The recommended way to move your data on and off the cluster is by using rsync.


Rsync is a fast and versatile file copying tool, it is most useful for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

Basic examples

# Copy a local directory to your cluster home directory
rsync -avz --partial example_local_dir

# Copy the contents of a local directory to your cluster home directory
rsync -avz --partial example_local_dir/

# Copy a local directory to a specific directory
rsync -avz --partial example_local_dir/

# Copy a remote directory to current local directory
rsync -avz --partial .

# Copy a local directory to a different local directory
rsync -av --partial /data/home/abc123/source /data/example/destination

The switches in use here are:

-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
    -r, --recursive             recurse into directories
    -l, --links                 copy symlinks as symlinks
    -p, --perms                 preserve permissions
    -t, --times                 preserve modification times
    -g, --group                 preserve group
    -o, --owner                 preserve owner (super-user only)
    -D                          same as --devices --specials
        --devices               preserve device files (super-user only)
        --specials              preserve special files
--partial                   keep partially transferred files
-v, --verbose               increase verbosity
-z, --compress              compress file data during the transfer

Dry run

Sometimes an rsync command line can get complicated, using -n or --dry-run will allow you to test your command without actually affecting any data.

Example job

Large transfers should be run as a job e.g.

#$ -cwd
#$ -j y
#$ -pe smp 1
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G

rsync -av --partial <source> <destination>

Using SCP

SCP (secure copy) can be used to copy individual files over ssh, although unlike rsync, resuming of file copying is not supported. If your connection is interrupted, you will have to repeat the upload.

# Basic Copy
scp example_file

# Copy to specific directory
scp example_file

# Copy whole directory
scp -r example_directory

Using SFTP

SFTP (Secure File Transfer) can be used to interactively transfer files over ssh.

Command line

$ sftp
sftp> ls
example_remote_file1  example_remote_file_2
sftp> lls
example_local_file1  example_local_file_2
sftp> get example_remote_file1
Fetching /data/home/abc123/example_remote_file1 to example_remote_file1
sftp> put example_local_file1
Uploading example_local_file1 to /data/home/abc123/example_local_file1

Further commands are available via the help command or the man pages.

GUI - FileZilla

For a GUI on Windows, Mac OS or Linux we suggest FileZilla.

Adding a site connection

To connect to the cluster:

  • From the File menu open the Site Manager.
  • Click the New Site button and name the connection Apocrita.
  • In the Host box put
  • Set the Protocol to SFTP.
  • Set the Logon Type to Normal
  • Enter your Apocrita username in the User field e.g.: abc123.
  • Enter your Apocrita login password in the Password field.


Importing a private key into FileZilla

Your private key can be presented via the SSH agent, or you can import the key into FileZilla:

  • Click Edit, then Settings.
  • In the Connection section, click SFTP.
  • Click Add key file... to import an existing private key into FileZilla.
  • Browse to the relevant OpenSSH private key.
  • Select Yes when asked if you would like to convert it into a supported format.
  • Provide a filename for the converted key file e.g. filezilla-apocrita-key.ppk, then click Save to import it.


Finally, you can return to the Site Manager and click Connect on the Apocrita site you created.

FileZilla security

If you ask FileZilla to save passwords for you, it is recommended that you protect passwords with a master password. This can be found under Edit-> Settings -> Interface.

Adding bookmarks to commonly used folders

You can optionally set up bookmarks under the Apocrita site, to jump to specific folders, such as scratch, or shared storage:

  • From the File menu open the Site Manager.
  • Select the Apocrita site.
  • Press the New Bookmark button.
  • Give the bookmark an appropriate name (e.g. YOURSHARE-NAME).
  • Enter the path to the desired folder in the Remote Directory box e.g. /data/YOURSHARE-NAME.

Selecting the bookmark and clicking Connect will open the folder.

Using Mobaxterm on Windows

Mobaxterm can use rsync or the gui to download/upload files.

Mobaxterm - RSYNC

Mobaxterm is bundled with a command-line rsync tool that functions identically to the one described above. Be sure to use full paths as you may experience issues with mobaxterm incorrectly interpreting shortcuts.

# On QMUL-managed computers /drives/g should point to your Windows home folder.
rsync -avz /drives/g

Mobaxterm - GUI

Login to Apocrita as per Logging in. The left sidebar should then display a list of files on the remote server.
Files can be downloaded by right clicking and selecting 'Download'.


Files can be uploaded by clicking the upload button mobaup_screen at the top of the sidebar.

Alternatively files can be drag-and-dropped from file explorer.


Aspera - ASCP

Aspera's ascp is a high-speed file transfer application, commonly used for the download of genome data and other large datasets.

To load the ascp binary into your PATH, run command: module load aspera.

Usage: ascp [OPTION] SRC... DEST
          SRC to DEST, or multiple SRC to DEST dir
          SRC, DEST format: [[user@]host:]PATH

Transfer rate limit

By default, ascp will utilise all available bandwidth, and impact other cluster users. To avoid this, please set the max transfer rate to 300Mbit/s by passing the -l 300M switch.