Gromacs¶
Gromacs is a molecular dynamics package mainly designed for simulations of proteins, lipids and nucleic acids.
Gromacs is available as a module on Apocrita.
Versions¶
Gromacs on Apocrita is available with three different installation types:
All Gromacs modules for CPU are installed with Intel MPI libraries.
Usage¶
A typical Gromacs workload consists of two stages:
Firstly grompp is used to prepare a file that contains the parameters for
the simulation. The grompp process does not use MPI and hence must be executed
in a serial job.
The second stage is to use mdrun to take this file as input and run the
simulation. If you only intend to use a single compute node to run the entire
workload you can therefore include both the grompp and mdrun tasks in a
single job script. However, if you wish to carry out the simulation making use
of parallel processing via MPI then you will need a separate job script for
your mdrun job.
You may wish to investigate queuing dependent jobs.
To run the required version of Gromacs, simply load the
gromacs/<version>-<type> module, substituting version and type with
the actual version and type desired.
Core Usage
To ensure that Gromacs uses the correct number of cores, the
-ntomp=${NSLOTS} option should be used to spawn one OpenMP
thread per core. (Serial jobs)
For parallel jobs use mpirun -np ${NSLOTS} (see Parallel example)
For more information see
Parallelisation schemes.
Intel MPI¶
This build of Gromacs is installed with Intel MPI parallelisation to run parallel simulations.
To run the required version of Gromacs with Intel MPI support, simply load the
gromacs/<version>-intelmpi module:
module load gromacs/<version>-intelmpi
gmx -h
Example jobs can be found below
Plumed¶
This Gromacs build uses Intel MPI with the plumed plugin, which provides extra molecular dynamics features and a wide variety of free energy methods.
To run the required version of Gromacs with plumed support, simply load the
gromacs/<version>-plumed module:
module load gromacs/<version>-plumed
gmx -h
This build of Gromacs adds the -plumed option to the gmx binary.
To check if this option exists, run:
gmx mdrun -h |grep plumed
You should get this output:
Command line:
gmx mdrun -h
[-multidir [<dir> [...]]] [-plumed [<.dat>]] [-membed [<.dat>]]
-plumed [<.dat>] (plumed.dat) (Opt.)
An example Gromacs with plumed support can be found here.
GPU¶
This build of Gromacs provides GPU support; simulation performances with a high calculation intensity on a GPU node.
GPU job submission is required
Gromacs with GPU support requires a GPU node. Information on how to submit to GPU nodes is available here.
To run the required version of Gromacs with GPU support, simply load the
gromacs/<version>-gpu module:
module load gromacs/<version>-gpu
An example Gromacs job script requesting a GPU node can be found here.
Example jobs¶
Gromacs job submission examples of each type:
Intel MPI jobs¶
Serial job¶
Here is an example job running on 4 cores and 4GB of memory:
#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load gromacs/<version>-intelmpi
gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -s example.tpr -deffnm example
When using mpirun (for example, when running
multi-simulations)
on a single node, add the following to your job script after loading the
gromacs module:
export I_MPI_HYDRA_BOOTSTRAP=fork
Parallel job¶
Stage 1 - prepare .tpr file - must be done serially:
Here is an example job running on 4 cores and 4GB of memory:
#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load gromacs/<version>-intelmpi
gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
Stage 2 - mdrun is Intel MPI aware so let's use 2 nodes:
Here is an example job running on 96 cores across 2 ddy nodes with MPI:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe parallel 96
#$ -l infiniband=ddy-i
#$ -l h_rt=240:0:0
module load gromacs/<version>-intelmpi
mpirun -np ${NSLOTS} gmx mdrun -v -deffnm example -s example.tpr
Plumed job¶
Here is an example job running on 4 cores and 4G of memory:
#!/bin/bash
#$ -cwd
#$ -pe smp 4
#$ -l h_rt=1:0:0
#$ -l h_vmem=1G
module load gromacs/<version>-plumed
gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -plumed example.dat -s example.tpr -ntomp ${NSLOTS}
GPU job¶
Here is an example job running on 1 GPU:
#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe smp 12
#$ -l h_rt=240:0:0
#$ -l h_vmem=7.5G
#$ -l gpu=1
module load gromacs/<version>-gpu
gmx grompp -f example.mdp -c example.gro -p example.top -o example.tpr
gmx mdrun -ntomp ${NSLOTS} -v -deffnm example
Parallelisation schemes¶
Gromacs provides considerable flexibility with respect to how it can be
configured. A command of the form: mpirun -np M gmx mdrun -ntomp N...
will launch M MPI processes with N OpenMP threads each.
However you can omit -ntomp N in this case and Gromacs will
spawn 1 thread per MPI process.
To make this command work well with the scheduler, we suggest:
mpirun -np ${NSLOTS} gmx mdrun...`
In this case ${NSLOTS} will expand to the number of cores you have
requested in your job script. That way you won't waste resources by requesting
more cores than Gromacs will actually use.
By default, Gromacs carries out particle-particle (PP) and particle mesh Ewald (PME) calculations one after another within the same process. However this can slow things down a lot since PME calculations depend on global communication and they may spend time waiting for other nodes to become available.
If a job uses more than 8 processes, then mdrun will attempt to designate
dedicated nodes for PME estimating the optimal amount. However, you can
override this behaviour:
mpirun -np NP_tot mdrun_mpi -npme NP_pme -ntomp NT`
This will launch NP_tot processes, with NP_pme dedicated to PME and
NT threads for each MPI process. Note that for the moment we recommend
ensuring that you use the environment variable ${NSLOTS} to provide
NP_tot as this will ensure that your job distributes the workload
properly across all cores when using multiple nodes.