GROMACS (GROningen Machine for Chemical Simulations) is a versatile and widely-used open source package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.

It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.

Key Features

1. Molecular Dynamics Simulations: GROMACS performs classical MD simulations, which compute the trajectories of atoms based on Newton's laws of motion. It integrates the equations of motion to simulate the behavior of molecular systems, capturing their dynamic properties and conformational changes.
2. Force Fields: GROMACS supports a wide range of force fields, including CHARMM, AMBER, OPLS-AA, and GROMOS, which describe the potential energy function and force interactions between atoms. These force fields provide accurate descriptions of the molecular interactions, allowing researchers to study various biological processes and molecular systems.
3. Parallelization and Performance: GROMACS is designed for high-performance computing (HPC) and can efficiently utilize parallel architectures, such as multi-core CPUs and GPUs. It employs domain decomposition methods and advanced parallelization techniques to distribute the computational workload across multiple computing resources, enabling fast and efficient simulations.
4. Analysis and Visualization: GROMACS offers a suite of analysis tools to extract and analyze data from MD simulations. It provides functionalities for computing properties such as energy, temperature, pressure, radial distribution functions, and free energy landscapes. GROMACS also supports visualization tools, allowing users to visualize and analyze the trajectories of molecular systems.
5. User-Friendly Interface: GROMACS provides a command-line interface (CLI) and a set of well-documented input and control files, making it accessible to both novice and expert users. It offers flexibility in defining system parameters, simulation conditions, and analysis options through easily modifiable input files.
6. Integration with Other Software: GROMACS can be integrated with other software packages and tools to perform advanced analysis and extend its capabilities. It supports interoperability with visualization tools like VMD and PyMOL, analysis packages like GROMACS Analysis Tools (GROMACS Tools) and MDAnalysis, and scripting languages such as Python, allowing users to leverage a wide range of complementary tools.

Licensing Terms and Conditions

GROMACS is a joint effort, with contributions from developers around the world: users agree to acknowledge use of GROMACS in any reports or publications of results obtained with the Software (see GROMACS Homepage for details).

ALPS (GH200)

Setup

On ALPS, we provide pre-built user environments containing GROMACS alongside all the required dependencies for the GH200 hardware setup. To access the gmx_mpi executable, we do the following:

uenv image pull gromacs/VERSION:TAG
uenv start gromacs/VERSION:TAG

uenv view gromacs # load the gromacs view
gmx_mpi --version # check GROMACS version

The images also provide two alternative views, namely plumed and develop .

After starting the pulled image using uenv start ... , one may do the following see the available views.

$ uenv status
/user-environment:gromacs-gh200
  GPU-optimised GROMACS with and without PLUMED, and the toolchain to build your own GROMACS.
  modules: no modules available
  views:
    develop
    gromacs
    plumed

The develop view has all the required dependencies or GROMACS without the program itself. This is meant for those users who want to use a customized variant of GROMACS for their simulation which they build from source. This view makes it convenient for users as it provides the required compilers (GCC 12), CMake, CUDA, hwloc, Cray MPICH, among many others which their GROMACS can use during build and installation. Users must enable this view each time they want to use their custom GROMACS installation.

The plumed view contains GROMACS 2022.5 (older version) with PLUMED 2.9.0. This is due to the compatibility requirements of PLUMED. CSCS will periodically update these user environment images to feature newer versions as they are made available.

The gromacs view contains the newest GROMACS 2024.1 that has been configured and tested for the highest performance on the Grace-Hopper nodes.

How to Run

To start a job, 2 bash scripts are required: a standard SLURM submission script, and a wrapper to start the CUDA MPS daemon (in order to have multiple MPI ranks per GPU).

The CUDA MPS wrapper here: Oversubscription of GPU cards

The wrapper script above needs to be made executable with chmod +x mps-wrapper.sh. 

The SLURM submission script can be adapted from the template below to use the application and the mps-wrapper.sh in conjunction.

#!/bin/bash

#SBATCH --job-name="JOB NAME"
#SBATCH --nodes=1             # number of GH200 nodes with each node having 4 CPU+GPU
#SBATCH --ntasks-per-node=8   # 8 MPI ranks per node
#SBATCH --cpus-per-task 32    # 32 OMP threads per MPI rank
#SBATCH --account=ACCOUNT
#SBATCH --hint=nomultithread  

export MPICH_GPU_SUPPORT_ENABLED=1
export FI_CXI_RX_MATCH_MODE=software

export GMX_GPU_DD_COMMS=true
export GMX_GPU_PME_PP_COMMS=true
export GMX_FORCE_UPDATE_DEFAULT_GPU=true
export GMX_ENABLE_DIRECT_GPU_COMM=1
export GMX_FORCE_GPU_AWARE_MPI=1

srun ./mps-wrapper.sh gmx_mpi mdrun -s input.tpr -ntomp 32 -bonded gpu -nb gpu -pme gpu -pin on -v -noconfout -dlb yes -nstlist 300 -gpu_id 0123 -npme 1 -nsteps 10000 -update gpu

This can be run using sbatch launch.sbatch on the login node with the user environment loaded.

This submission script is only representative. Users must run their input files with a range of parameters to find an optimal set for the production runs. Some hints for this exploration below:

Scaling

Benchmarking done with large MD simulation problems of 1.4 million and 3 million atom systems in order to fully saturate the GPUs from the HECBioSim Benchmark Suite.

In addition, the STMV (~1 million atom) benchmark that NVIDIA publishes on its website was also tested for comparison. 

The STMV test case is a fairly large problem size, with constraints operating only on a smaller set of atoms (h-bonds) which allows the update step to also take place on GPUs. This makes the simulation almost fully GPU resident with the key performance intensive bits namely the long-range forces (PME), short-range non-bonded forces (NB) and bonded forces all running on the GPU. On a single node, this leads to the following scaling on GROMACS 2024.1.

STMV - Single Node - Multiple Ranks

The other benchmark cases from HECBioSim simulates a pair of proteins (hEGFR Dimers/Tetramers of 1IVO and 1NQL) with a large lipid membrane. This also involves a fairly large number of charged ions which increases the proportion of PME in the total compute workload. For these simulations, constraints are applicable on all atoms, which effectively prevents the update from happening in the GPU, thus negatively impacting scaling due large host-to-device data transfers and key computations happening on the CPU. These show the following scaling characteristics on GROMACS 2024.1:

1.4m Atom System - Single Node - Multiple Ranks

Total number of atoms = 1,403,182

Protein atoms = 43,498  Lipid atoms = 235,304  Water atoms = 1,123,392  Ions = 986

3m Atom System - Single Node - Multiple Ranks

Total number of atoms = 2,997,924

Protein atoms = 86,996  Lipid atoms = 867,784  Water atoms = 2,041,230  Ions = 1,914

Piz Daint

Setup

You can see a list of the available versions of the program installed on the machine after loading the gpu or multicore modulefile. In the examples below we use the daint-gpu modulefile:

module load daint-gpu
module avail GROMACS

The following module command will load the environment of the default version of the program:

module load GROMACS

You can either type this command every time you intend to use the program within a new session, or you can automatically load it by including it in your shell configuration file.

The following module commands will print the environment variables set by loading the program and a help message:

module show GROMACS
module help GROMACS

How to Run

GROMACS performance depends on several factors, such as the usage (or the lack) of GPUs, the number of MPI tasks and OpenMP, the loading balance algorithm, the ratio between the number of Particle-Particle (PP) ranks and Particle-Mesh-Ewald (PME) ranks, the type of simulation being performed, force field used and of course the simulated system. For a complete set of GROMACS options, please refer to GROMACS documentation.

The following job script is just an example and asks for 64 nodes, using 12 MPI tasks per node and 1 OpenMP threads per MPI task. Please try other mdrun flags in order to see if they make your simulation run faster. Examples of such flags are -npme, -dlb, -ntomp. If you use more MPI tasks per node you will have less memory per MPI task. If you use multiple MPI tasks per node, you need to set CRAY_CUDA_MPS=1 to enable the tasks to access the GPU device on each node at the same time.

#!/bin/bash -l
#
# GROMACS on Piz Daint: 64 nodes, 12 MPI tasks per node, 1 OpenMP threads per task no hyperthreading (--ntasks-per-core=1)
#
#SBATCH --job-name=gromacs
#SBATCH --account="<project>"
#SBATCH --time=01:00:00
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=12
#SBATCH --ntasks-per-core=1
#SBATCH --cpus-per-task=1
#SBATCH --hint=nomultithread
#SBATCH --constraint=gpu
#========================================
# load modules and run simulation
module load daint-gpu
module load GROMACS
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export CRAY_CUDA_MPS=1
srun gmx_mpi mdrun -s input.tpr -ntomp 1

Please replace the string <project> with the ID of the active project that will be charged for the allocation.

Scaling on Piz Daint

We provide a GROMACS scaling example simulating the NPT dynamics of Protein embedded in a lipid membrane. The input file can be downloaded here.

We run the scaling jobs with the constraint gpu on the Cray XC50, using 12 MPI tasks per node and 1 OpenMP threads. The ns/day of each job is retrieved from the performance section of GROMACS' output file; the relative speed-up is computed taking the longest runtime as a reference value. We reach the ~50% limit in parallel efficiency running on 8 nodes in this particular example.

The scaling data are reported in the table below:

Strong scaling results are plotted against ideal scaling as follows:

GROMACS and PLUMED

We don't provide a pre-built version of GROMACS patched with PLUMED, but this can easily be built by using our EasyBuild installation. Please follow the instructions from the EasyBuild user page.

In our production repository of EasyBuild recipes on GitHub, you will find examples for building GROMACS patched against PLUMED under easybuild/easyconfigs/g/GROMACS. Note that you can customize these recipes to build different (mutually compatible) combinations of GROMACS and PLUMED versions.

Further Documentation

GROMACS Homepage

GROMACS Manual