Quotas are used to keep track of storage space that is being used on per-user or per-group basis. Quotas are measured both in terms of number of files used and total size of files. File sizes are measured in terms of the number of blocks used.
Finding out current usage¶
When you are logged in to Apocrita, you can see which storage
you have access to, and how much you have used, using the
This will list your home directory, scratch storage and any shared project storage you have access to.
$ qmquota -s Disk quotas for abc123 Space Files Location Use% Used Free Quota Limit Use% Used Free Quota Limit /data/Shared-Lab-Storage 0.00% 32K 1T 1T 1.1T - 3 - - - /data/home/abc123 19% 92.1G 40G 50G 55G - 20667 - - - /data/scratch/abc123 0.00% 0 300G 300G 500G 0.00% 4 999996 - 1000000
Several options are available to modify the output similar to those for the
$ qmquota -h Usage: qmquota [OPTIONS] -s, --human-readable : display numbers in human friendly units (MB, GB...) -c, --colour : Highlights storage which is over the quota -h, --help : This small usage guide -A, --nfs-all : display quota for all NFS mountpoints -v, --verbose : Include where no storage is allocated -l, --local-only : Do not query network filesystems
You will receive emails when you exceed scratch or home directory quotas.
You will also receive emails if you are the designated group contact or data owner for shared storage.
Group contacts are recorded in the README file in the top directory. If this needs to be altered, please contact email@example.com
Good practice for managing your storage¶
Storage space is counted in blocks of 128KB, the smallest unit of allocation in GPFS. As a result, space usage may seem larger than the sum of files used.
- Tar large directories of files when not in use. You can de-compress and re-compress when you need them, even within your jobs. Use bzip2 for maximum compression.
- We provide a database server running PostgreSQL and MariaDB for accessing/writing small amounts of data. This is more efficient than writing lots of small files.
- Try to avoid having multiple processes writing to the same file. e.g.
collecting logs from an application that's running on lots of nodes.
Store each with a different name e.g. labelled by their process id (PID)