Skip to content

Gromacs

Gromacs is a molecular dynamics package mainly designed for simulations of proteins, lipids and nucleic acids.

Gromacs is available as a module on Apocrita.

Versions

Gromacs on Apocrita is available with three different installation types:

All Gromacs modules for CPU are installed with Intel MPI libraries.

AVX2 support required

Gromacs will run only on nodes which support the AVX2 cpu instruction set. The -l avx2 complex is required in serial job scripts.

Usage

A typical Gromacs workload consists of two stages:

Firstly grompp is used to prepare a file that contains the parameters for the simulation. The grompp process does not use MPI and hence must be executed in a serial job.

The second stage is to use mdrun to take this file as input and run the simulation. If you only intend to use a single compute node to run the entire workload you can therefore include both the grompp and mdrun tasks in a single job script. However, if you wish to carry out the simulation making use of parallel processing via MPI then you will need a separate job script for your mdrun job.

You may wish to investigate queuing dependent jobs.

To run the required version of Gromacs, simply load the gromacs/<version>-<type> module, substituting version and type with the actual version and type desired.

Core Usage

To ensure that Gromacs uses the correct number of cores, the -ntomp=${NSLOTS} option should be used to spawn one OpenMP thread per core. (Serial jobs)

For parallel jobs use mpirun -np ${NSLOTS} (see Parallel example) For more information see Parallelisation schemes.

Intel MPI

This build of Gromacs is installed with Intel MPI parallelisation to run parallel simulations.

To run the required version of Gromacs with Intel MPI support, simply load the gromacs/<version>-intelmpi module:

module load gromacs/<version>-intelmpi
gmx -h

Example jobs can be found below

Plumed

This Gromacs build uses Intel MPI with the plumed plugin, which provides extra molecular dynamics features and a wide variety of free energy methods.

To run the required version of Gromacs with plumed support, simply load the gromacs/<version>-plumed module:

module load gromacs/<version>-plumed
gmx -h

This build of Gromacs adds the -plumed option to the gmx binary. To check if this option exists, run:

gmx mdrun -h |grep plumed

You should get this output:

Command line:
  gmx mdrun -h

 [-multidir [<dir> [...]]] [-plumed [<.dat>]] [-membed [<.dat>]]
 -plumed [<.dat>] (plumed.dat) (Opt.)

An example Gromacs with plumed support can be found here.

GPU

This build of Gromacs provides GPU support; simulation performances with a high calculation intensity on a GPU node.

GPU job submission is required

Gromacs with GPU support requires a GPU node. Information on how to submit to GPU nodes is available here.

To run the required version of Gromacs with GPU support, simply load the gromacs/<version>-gpu module:

module load gromacs/<version>-gpu

An example Gromacs job script requesting a GPU node can be found here.

Example jobs

Gromacs job submission examples of each type:

Intel MPI jobs

Serial job

Here is an example job running on 4 cores and 4GB of memory on a node which supports the AVX2 cpu instruction set:

#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
#$ -l avx2

module load gromacs/<version>-intelmpi

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -s example.tpr -deffnm example

When using mpirun (for example, when running multi-simulations) on a single node, add the following to your job script after loading the gromacs module:

export I_MPI_HYDRA_BOOTSTRAP=fork

Parallel job

Stage 1 - prepare .tpr file - must be done serially:

Here is an example job running on 4 cores and 4GB of memory on a node which supports the AVX2 cpu instruction set:

#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
#$ -l avx2

module load gromacs/<version>-intelmpi

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr

Stage 2 - mdrun is Intel MPI aware so let's use 2 nodes:

Here is an example job running on 96 cores across 2 ddy nodes with MPI:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe parallel 96
#$ -l infiniband=ddy-i
#$ -l h_rt=240:0:0

module load gromacs/<version>-intelmpi

mpirun -np ${NSLOTS} gmx mdrun -v -deffnm example -s example.tpr

Plumed job

Here is an example job running on 4 cores and 4G of memory on a node which supports the AVX2 cpu instruction set:

#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
#$ -l avx2

module load gromacs/<version>-plumed

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -plumed example.dat -s example.tpr -ntomp ${NSLOTS}

GPU job

Here is an example job running on 1 GPU:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 8
#$ -l h_rt=240:0:0
#$ -l h_vmem=11G
#$ -l gpu=1

module load gromacs/<version>-gpu

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -deffnm example

Parallelisation schemes

Gromacs provides considerable flexibility with respect to how it can be configured. A command of the form: mpirun -np M gmx mdrun -ntomp N... will launch M MPI processes with N OpenMP threads each. However you can omit -ntomp N in this case and Gromacs will spawn 1 thread per MPI process.

To make this command work well with the scheduler, we suggest:

mpirun -np ${NSLOTS} gmx mdrun...`

In this case ${NSLOTS} will expand to the number of cores you have requested in your job script. That way you won't waste resources by requesting more cores than Gromacs will actually use.

By default, Gromacs carries out particle-particle (PP) and particle mesh Ewald (PME) calculations one after another within the same process. However this can slow things down a lot since PME calculations depend on global communication and they may spend time waiting for other nodes to become available.

If a job uses more than 8 processes, then mdrun will attempt to designate dedicated nodes for PME estimating the optimal amount. However, you can override this behaviour:

mpirun -np NP_tot mdrun_mpi -npme NP_pme -ntomp NT`

This will launch NP_tot processes, with NP_pme dedicated to PME and NT threads for each MPI process. Note that for the moment we recommend ensuring that you use the environment variable ${NSLOTS} to provide NP_tot as this will ensure that your job distributes the workload properly across all cores when using multiple nodes.

References