Skip to content

Backups and snapshots


Backups

The research data is backed up nightly to tape and kept for 3 months, for the purpose of disaster recovery. If you would like to recover old or lost files within the last 3 months, raise a support ticket with us with a list of filenames and last known access times.

However, we take nightly snapshots of the whole file system, and if the file exists on an overnight snapshot, you can recover it yourself quickly as shown in the snapshots section.

What is backed up:

  • User home directories /data/home
  • Shared project storage /data/

What is NOT backed up

  • auto deleting scratch space/data/scratch

The backup schedule consists of daily incremental backups which are retained for 3 months. This is currently handled by Spectrum Protect (previously TSM) to build a full backup. The tapes are kept geographically separate to our cluster.


Snapshots

Snapshots are taken daily and hold a copy of the entire storage at the point they were taken. All files exist in the snapshots with exactly the same permissions as they did in the original, which means you can recover files from snapshots yourself. These snapshots are kept for a week (3 days in the case of scratch space /data/scratch).

Snapshot structure

Snapshots can be found under any directory shared between nodes in the cluster (scratch, home or group storage). They are found in a directory called .snapshots. They are normally taken at 23:00 each day and kept for 7 days; scratch snapshots are only kept for 3 days. These locations contain a dated directory for each available snapshot.

These directories are named by date (any additional ones are there for administration purposes, such as taking tape backups). Below these directories is the same structure as the original directory. Hence the snapshot of the the file lost_file can be found at .snapshots/YYYY-M-D/lost_file. Directories are also mirrored at higher levels so the entire directory structures can be recovered under /data/.snapshots

These directories are read-only; they can not be changed.

A full list of available snapshots can be gathered with mmlssnapshot home for group shares and home directories, and mmlssnapshot gpfsScratch for scratch:

$ mmlssnapshot home
Snapshots in file system home:
Directory                SnapId    Status  Created
2018-6-20                1344      Valid   Wed Jun 20 23:02:22 2018
2018-6-21                1346      Valid   Thu Jun 21 23:00:02 2018
2018-6-22                1348      Valid   Fri Jun 22 23:00:05 2018
2018-6-23                1350      Valid   Sat Jun 23 23:00:02 2018
2018-6-24                1352      Valid   Sun Jun 24 23:00:02 2018
2018-6-25                1354      Valid   Mon Jun 25 23:00:01 2018
2018-6-26                1356      Valid   Tue Jun 26 23:00:02 2018

Recovering files from snapshots

To recover a file from the snapshot, simply copy it somewhere outside the .snapshots directory. For example:

cp .snapshots/<date>/lost_file recovered_data/