Skip to content

Backups and snapshots

Backups

The research data is backed up nightly to tape and kept for 3 months, for the purpose of disaster recovery. If you would like to recover old or lost files within the last 3 months, raise a support ticket with us with a list of filenames and last known access times.

However, we take nightly snapshots of the whole file system, and if the file exists on an overnight snapshot, you can recover it yourself quickly as shown in the snapshots section.

What is backed up:

  • User home directories /data/home
  • Shared project storage /data/

What is NOT backed up

  • auto deleting scratch space/data/scratch

The backup schedule consists of daily incremental backups which are retained for 3 months. This is currently handled by Spectrum Protect (previously TSM) to build a full backup. The tapes are kept geographically separate to our cluster.

Snapshots

Snapshots are taken daily and hold a copy of the entire storage at the point they were taken. All files exist in the snapshots with exactly the same permissions as they did in the original, which means you can recover files from snapshots yourself. These snapshots are kept for a week (3 days in the case of scratch space /data/scratch).

Snapshot structure

Snapshots can be found under any directory shared between nodes in the cluster (scratch, home or group storage). They are found in a directory called .snapshots. They are normally taken at 23:00 each day and kept for 7 days; scratch snapshots are only kept for 3 days. These locations contain a dated directory for each available snapshot.

These directories are named by date (any additional ones are there for administration purposes, such as taking tape backups). Below these directories is the same structure as the original directory. Hence the snapshot of the the file lost_file can be found at .snapshots/YYYY-M-D/lost_file. Directories are also mirrored at higher levels so the entire directory structures can be recovered under /data/.snapshots

These directories are read-only; they can not be changed.

A full list of available snapshots can be gathered with mmlssnapshot gpfsWork for group shares and home directories, and mmlssnapshot gpfsScratch for scratch.

$ mmlssnapshot gpfsWork
Snapshots in file system gpfsWork:
Directory                SnapId    Status  Created
2023-2-3                 509       Valid   Fri Feb  3 23:00:02 2023
2023-2-4                 511       Valid   Sat Feb  4 23:00:04 2023
2023-2-5                 513       Valid   Sun Feb  5 23:00:04 2023
2023-2-6                 515       Valid   Mon Feb  6 23:00:01 2023
2023-2-7                 517       Valid   Tue Feb  7 23:00:02 2023
2023-2-8                 519       Valid   Wed Feb  8 23:00:03 2023
2023-2-9                 521       Valid   Thu Feb  9 23:00:05 2023

Recovering files from snapshots

To recover a file from the snapshot, simply copy it somewhere outside the .snapshots directory. For example:

cp .snapshots/<date>/lost_file recovered_data/