Backups and snapshots¶
The research data is backed up nightly to tape and kept for 3 months, for the purpose of disaster recovery. If you would like to recover old or lost files within the last 3 months, raise a support ticket with us with a list of filenames and last known access times.
However, we take nightly snapshots of the whole file system, and if the file exists on an overnight snapshot, you can recover it yourself quickly as shown in the snapshots section.
What is backed up:
- User home directories
- Shared project storage
What is NOT backed up
- auto deleting scratch space
The backup schedule consists of daily incremental backups which are retained for 3 months. This is currently handled by Spectrum Protect (previously TSM) to build a full backup. The tapes are kept geographically separate to our cluster.
Snapshots are taken daily and hold a copy of the entire storage at the point
they were taken. All files exist in the snapshots with exactly the same
permissions as they did in the original, which means you can recover files from
snapshots yourself. These snapshots are kept for a week
(3 days in the case of scratch space
Snapshots can be found under any directory shared between nodes in
the cluster (scratch, home or group storage). They are found in a
.snapshots. They are normally taken at 23:00 each
day and kept for 7 days; scratch snapshots are only kept for 3 days.
These locations contain a dated directory for each available snapshot.
These directories are named by date (any additional ones are there for
administration purposes, such as taking tape backups). Below these
directories is the same structure as the original directory. Hence the
snapshot of the the file
lost_file can be found at
.snapshots/YYYY-M-D/lost_file. Directories are also mirrored at higher
levels so the entire directory structures can be recovered under
These directories are read-only; they can not be changed.
A full list of available snapshots can be gathered with
mmlssnapshot home for group shares and home directories, and
mmlssnapshot gpfsScratch for scratch:
$ mmlssnapshot home Snapshots in file system home: Directory SnapId Status Created 2018-6-20 1344 Valid Wed Jun 20 23:02:22 2018 2018-6-21 1346 Valid Thu Jun 21 23:00:02 2018 2018-6-22 1348 Valid Fri Jun 22 23:00:05 2018 2018-6-23 1350 Valid Sat Jun 23 23:00:02 2018 2018-6-24 1352 Valid Sun Jun 24 23:00:02 2018 2018-6-25 1354 Valid Mon Jun 25 23:00:01 2018 2018-6-26 1356 Valid Tue Jun 26 23:00:02 2018
Recovering files from snapshots¶
To recover a file from the snapshot, simply copy it somewhere outside the
.snapshots directory. For example:
cp .snapshots/<date>/lost_file recovered_data/