Backups and snapshots¶
The research data is backed up nightly to tape and kept for 3 months, for the purpose of disaster recovery. If you would like to recover old or lost files within the last 3 months, raise a support ticket with us with a list of filenames and last known access times.
However, we take nightly snapshots of the whole file system, and if the file exists on an overnight snapshot, you can recover it yourself quickly as shown in the snapshots section.
What is backed up:
- User home directories
- Shared project storage
What is NOT backed up
- auto deleting scratch space
The backup schedule consists of daily incremental backups which are retained for 3 months. This is currently handled by Spectrum Protect (previously TSM) to build a full backup. The tapes are kept geographically separate to our cluster.
Snapshots are taken daily and hold a copy of the entire storage at the point
they were taken. All files exist in the snapshots with exactly the same
permissions as they did in the original, which means you can recover files from
snapshots yourself. These snapshots are kept for a week
(3 days in the case of scratch space
Snapshots can be found under any directory shared between nodes in
the cluster (scratch, home or group storage). They are found in a
.snapshots. They are normally taken at 23:00 each
day and kept for 7 days; scratch snapshots are only kept for 3 days.
These locations contain a dated directory for each available snapshot.
These directories are named by date (any additional ones are there for
administration purposes, such as taking tape backups). Below these
directories is the same structure as the original directory. Hence the
snapshot of the the file
lost_file can be found at
.snapshots/YYYY-M-D/lost_file. Directories are also mirrored at higher
levels so the entire directory structures can be recovered under
These directories are read-only; they can not be changed.
A full list of available snapshots can be gathered with
mmlssnapshot gpfsWork for group shares and home directories, and
mmlssnapshot gpfsScratch for scratch.
$ mmlssnapshot gpfsWork Snapshots in file system gpfsWork: Directory SnapId Status Created 2023-2-3 509 Valid Fri Feb 3 23:00:02 2023 2023-2-4 511 Valid Sat Feb 4 23:00:04 2023 2023-2-5 513 Valid Sun Feb 5 23:00:04 2023 2023-2-6 515 Valid Mon Feb 6 23:00:01 2023 2023-2-7 517 Valid Tue Feb 7 23:00:02 2023 2023-2-8 519 Valid Wed Feb 8 23:00:03 2023 2023-2-9 521 Valid Thu Feb 9 23:00:05 2023
Recovering files from snapshots¶
To recover a file from the snapshot, simply copy it somewhere outside the
.snapshots directory. For example:
cp .snapshots/<date>/lost_file recovered_data/