Skip to content

Gromacs

Gromacs is a molecular dynamics package mainly designed for simulations of proteins, lipids and nucleic acids.

Gromacs is available as a module on Apocrita.

A typical gromacs workload consists of two stages:

Firstly grompp is used to prepare a file that contains the parameters for the simulation. The grompp process does not use MPI and hence must be executed in a serial job. The second stage is to use mdrun to take this file as input and run the simulation. If you only intend to use a single compute node to run the entire workload you can therefore include both the grompp and mdrun tasks in a single job script. However, if you wish to carry out the simulation making use of parallel processing via MPI then you will need a separate job script for your mdrun job. You may wish to investigate queuing dependent jobs.

Usage

Gromacs on Apocrita is available with 3 different installation types:

To load the desired Gromacs module, execute:

module load gromacs/<version>-<type>
gmx

Core Usage

To ensure that Gromacs uses the correct number of cores, the -ntomp=${NSLOTS} option should be used to spawn one OpenMP thread per core. (Serial jobs)

For parallel jobs use mpirun -np ${NSLOTS} (see Parallel example) For more information see Parallelisation schemes.

Running on CPU

All Gromacs modules for CPU are installed with Intel MPI libraries.

Special node type required

Gromacs will run only on nxv or sdv nodes, -l node_type=nxv or -l node_type=sdv is required in the job script.

Intel MPI

This build of Gromacs is installed with Intel MPI parallelisation to run parallel simulations.

Loading a specific version of Gromacs with Intel MPI support:

module load gromacs/<version>-intelmpi
gmx

Example jobs can be found below

plumed

This Gromacs build uses Intel MPI with the plumed plugin, which provides extra molecular dynamics features and a wide variety of free energy methods.

Loading the specific version of Gromacs with plumed support:

module load gromacs/<version>-plumed
gmx

This build of Gromacs adds the -plumed option to the gmx binary. To check if this option exists, run:

gmx mdrun -h |grep plumed

You should get this output:

Command line:
  gmx mdrun -h

 [-multidir [<dir> [...]]] [-plumed [<.dat>]] [-membed [<.dat>]]
 -plumed [<.dat>] (plumed.dat) (Opt.)

An example Gromacs with plumed support can be found here.

Running on GPU

This build of Gromacs provides GPU support; simulation performances with a high calculation intensity on a GPU node.

GPU job submission is required

Gromacs with GPU support requires at least one GPU node. Information on how to submit to GPU nodes is available here.

Loading a specific version of Gromacs with GPU support:

module load gromacs/<version>-gpu

An example Gromacs job script requesting a GPU node can be found here.

Example jobs

Gromacs job submission examples of each type:

Intelmpi

Serial job

Here is an example of a serial Gromacs job running on the nxv node type with 4 cores:

#!/bin/sh
#$ -cwd
#$ -pe smp 4
#$ -l h_vmem=1G
#$ -l node_type=nxv
#$ -l h_rt=1:0:0

module load gromacs/<version>-intelmpi

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -s example.tpr -deffnm example

When using mpirun (for example, when running multi-simulations) on a single node, add the following to your job script after loading the gromacs module:

export I_MPI_HYDRA_BOOTSTRAP=fork

Parallel job

Here is an example of a parallel Gromacs job running on 2 nxv nodes with 64 cores:

Stage 1 - prepare .tpr file - must be done serially:

#!/bin/sh
#$ -cwd
#$ -pe smp 4
#$ -l h_vmem=1G
#$ -l node_type=nxv
#$ -l h_rt=1:0:0

module load gromacs/<version>-intelmpi

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr

Stage 2 - mdrun is Intel MPI aware so let's use 2 nodes:

#!/bin/sh
#$ -cwd
#$ -pe parallel 64
#$ -l infiniband=nxv
#$ -l h_rt=1:0:0

module load gromacs/<version>-intelmpi

mpirun -np ${NSLOTS} gmx mdrun -v -deffnm example -s example.tpr

Plumed job

Here is an example of a Gromacs with plumed support job running on the nxv node type with 32 cores:

#!/bin/sh
#$ -cwd
#$ -pe smp 32
#$ -l h_vmem=1G
#$ -l node_type=nxv
#$ -l h_rt=1:0:0

module load gromacs/<version>-plumed

gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -plumed example.dat -s example.tpr -ntomp ${NSLOTS}

GPU job

Here is an example of a Gromacs job running on 1 GPU card using half a GPU node:

#!/bin/sh
#$ -cwd
#$ -pe smp 16
#$ -l h_vmem=7G
#$ -l gpu=1
#$ -l h_rt=1:0:0

module load gromacs/<version>-gpu
export CUDA_VISIBLE_DEVICES=${SGE_HGR_gpu// /,}
gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -deffnm example

Note that the export CUDA_VISIBLE_DEVICES=${SGE_HGR_gpu// /,} line is extremely important to ensure correct resource use.

Parallelisation Schemes

Gromacs provides considerable flexibility with respect to how it can be configured. A command of the form: mpirun -np M gmx mdrun -ntomp N... will launch M MPI processes with N OpenMP threads each. However you can omit -ntomp N in this case and Gromacs will spawn 1 thread per MPI process.

In order to make this command work well with the scheduler we suggest:

mpirun -np ${NSLOTS} gmx mdrun...

In this case ${NSLOTS} will expand to the number of cores you have requested in your jobscript. That way you won't waste resources by requesting more cores than Gromacs will actually use.

By default, Gromacs carries out particle-particle (PP) and particle mesh Ewald (PME) calculations one after another within the same process. However this can slow things down a lot since PME calculations depend on global communication and they may spend time waiting for other nodes to become available.

If a job uses more than 16 processes, then mdrun will attempt to designate dedicated nodes for PME estimating the optimal amount. However, you can override this behaviour:

mpirun -np NP_tot mdrun_mpi -npme NP_pme -ntomp NT

This will launch NP_tot processes, with NP_pme dedicated to PME and NT threads for each MPI process. Note that for the moment we recommend ensuring that you use the environment variable ${NSLOTS} to provide NP_tot as this will ensure that your job distributes the workload properly across all cores when using multiple nodes.

References