Memory

Virtual Memory vs. Resident Set Size

Memory usage can be broadly simplified into two values, Virtual Memory (VMEM) which a program believes it has and Resident Set Size (RSS) which is the actual amount of memory it uses.

In order to make better use of physical memory, the operating system doesn't actually give a program the memory it requests until the program uses it. This means multiple applications can request memory and until they actually use it they won't affect the available RAM. Once data is written to the memory it is actually in use and part of the RSS - the size of the memory set that is resident in RAM.

The diagram below shows the memory usage for an example program. VMEM is much larger than the RSS which is in turn slightly larger than the actual RAM usage. Shared libraries (for example glibc make up the rest of the RSS. These common software libraries are only loaded into RAM once and then used by multiple applications. However these libraries are still counted in each applications RSS regardless of the number of applications using them.

Virtual Memory and Resident Set Size can be seen in top as VIRT and RES:

  PID USER        VIRT    RES S %CPU %MEM     TIME+ COMMAND
12786 abc123     95532  62788 R 48.0  0.0  59:14.78 rsync
44858 root     34.336g 5.760g S  8.8  1.1   2256:19 mmfsd

In qacct, they are shown as maxvmem and ru_maxrss:

maxvmem      23.828G
ru_maxrss    22808828

Example program

This example program shows the difference between VMEM and RSS. The program initially requests 1GiB of RAM with malloc(size);. At this point Although the memory has been granted its not actually in use and therefore is not taking up space in physical RAM.

The program then fills the space with data which causes the requested memory to actually take up space in RAM.

The source code for this application is below and it can be compiled and run with the following commands:

gcc memory_demo.c -o memory_demo
./memory_demo

#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

/* Get vmem and rss usage from /proc/<pid>/statm */
static int mem_used(pid_t pid, unsigned long* vmem, unsigned long* rss) {
    FILE* file;
    char path[40];
    unsigned int page_size;

    snprintf(path, 40, "/proc/%ld/statm", (long) pid);
    file = fopen(path, "r");
    // vmem and rss are the first values in the file
    fscanf(file, "%lu %lu", vmem, rss);
    // values in statm are in pages so to get bytes we need to know page size
    page_size = (unsigned) getpagesize();
    *vmem = *vmem * page_size;
    *rss = *rss * page_size;

    fclose(file);
    return 0;
}

int main(int argc, char **argv) {
    unsigned char *address;
    char input;
    size_t size = 1024*1024*1024;  // 1 GiB
    unsigned long i;
    unsigned long vmem = 0;
    unsigned long rss = 0;
    pid_t pid;

    pid = getpid();
    printf("Pid: %ld\n", (long) pid);

    mem_used(pid, &vmem, &rss);
    printf("VMEM: %lu RSS: %lu\n", vmem, rss);

    address = malloc(size);
    printf("Allocated %d Bytes of memory\n", (int) size);

    mem_used(pid, &vmem, &rss);
    printf("VMEM: %lu RSS: %lu\n", vmem, rss);

    printf("Press any key to continue");
    scanf("%c", &input);

    printf("Filling memory with data...");
    fflush(stdout);  // Flush output so message appears before end of loop
    for (i = 0; i < size; i++) {
        *(address + i) = 123;
    }

    mem_used(pid, &vmem, &rss);
    printf("\nVMEM: %lu RSS: %lu\n", vmem, rss);

    printf("Press any key to continue");
    scanf("%c", &input);

    free(address);
    return 0;
}

Job requests

Job memory is handled via the h_vmem resource. Whilst this is named vmem it is actually linked to Resident Set Size and only jobs that exceed the requested amount of RSS will be killed.