Run GROMACS Simulations on HPC Clusters

This guide shows how to configure, submit, and manage GROMACS simulations on HPC clusters using PolyzyMD’s self-resubmitting SLURM workflow.

Prerequisites

A working config.yaml validated with polyzymd validate
GROMACS installed on your cluster (via module load or container)
Familiarity with your cluster’s SLURM partitions and GPU types
PolyzyMD installed with pixi (see Install PolyzyMD with pixi)

All commands below assume you prefix with pixi run -e build or have activated the environment with pixi shell -e build.

Quick Start: CPU

Add a minimal gromacs: block to your config.yaml and submit:

# config.yaml (add this block alongside your existing config)
gromacs:
  module_load: "module load gcc/11.2.0 gromacs/2024.2"
  ntmpi: 1
  ntomp: 8

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset aa100 \
    --replicates 1-3

This generates self-resubmitting SLURM scripts that run EM, equilibration, and production with checkpoint-based restart.

Build-only GROMACS export

Use polyzymd build --format gromacs when you only want PolyzyMD to construct and parameterize the system:

pixi run -e build polyzymd build -c config.yaml --format gromacs

The core handoff is the .gro coordinate file, .top topology file, and component .itp parameter files. PolyzyMD may also generate MDP defaults and a run script as conveniences from the validated config.yaml, but those files are not required. You may use the generated defaults as a starting point or replace them and continue entirely in your own GROMACS workflow.

Use polyzymd run --engine gromacs instead when you want PolyzyMD to build and execute the full local GROMACS workflow.

Quick Start: GPU

For GPU-accelerated runs, set gpu: true and use the thread-MPI gmx binary (not gmx_mpi):

# config.yaml
gromacs:
  gpu: true
  gpus: 1
  gmx_binary: "gmx"
  ntmpi: 1
  ntomp: 12
  module_load: "module load gcc/11.2.0 gromacs/2024.2"
  mdrun_flags: "-nb gpu -pme gpu -bonded gpu -update gpu -pin on"

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset blanca-shirts \
    --constraint "A40" \
    --replicates 1-3

Important

Use --constraint to pin your job to a compatible GPU architecture. GROMACS uses ahead-of-time compiled CUDA kernels, so a binary compiled for one GPU type may not run on another. OpenMM does not need this because it JIT-compiles kernels at launch.

GROMACS Flag Glossary

These flags are passed to gmx mdrun via the mdrun_flags, mdrun_flags_equilibration, and mdrun_flags_production config fields.

Parallelism flags

Flag	Description
`-ntmpi N`	Thread-MPI ranks. Used by thread-MPI builds (`gmx`). Set via `gromacs.ntmpi` in config.
`-ntomp N`	OpenMP threads per rank. Set via `gromacs.ntomp` in config.
`-npme N`	Dedicated PME ranks. Useful with multi-GPU runs (e.g., `-npme 1` with 3 GPUs).

GPU offload flags

Flag	Description	Safe for EM?
`-nb gpu`	Offload nonbonded forces to GPU	Yes
`-pme gpu`	Offload PME electrostatics to GPU	No
`-bonded gpu`	Offload bonded forces to GPU	No
`-update gpu`	Offload integration/constraints to GPU	No

Performance flags

Flag	Description
`-pin on`	Pin threads to CPU cores (recommended for performance)
`-pinstride N`	Stride between pinned threads
`-dlb yes\|auto`	Dynamic load balancing
`-gpu_id NNN`	Explicit GPU device IDs (e.g., `012` for 3 GPUs)

Energy minimization restrictions

Warning

Only -nb gpu is safe during energy minimization. GROMACS will fail with “Non-dynamical integrator” if -pme gpu, -bonded gpu, or -update gpu are used during EM.

PolyzyMD handles this automatically — it strips unsafe GPU flags from the mdrun command during EM stages. You do not need separate EM-specific flag configuration.

Understanding MPI vs OpenMP Parallelism in GROMACS

GROMACS supports two parallelism models that are easy to confuse:

Concept	Thread-MPI	Real MPI
Binary	`gmx`	`gmx_mpi`
Launch	Direct (`gmx mdrun -ntmpi N`)	Via launcher (`mpirun -np N gmx_mpi mdrun`)
GPU support	Yes (CUDA thread-MPI)	Varies by build (often GPU-disabled)
Multi-node	No (single node only)	Yes
Config field	`gromacs.ntmpi`	`gromacs.ntmpi` + `gromacs.mpi_launcher_flags`

Rule of thumb

ntmpi × ntomp = total CPU cores allocated

GPU runs typically use thread-MPI with 1 MPI rank and many OpenMP threads:

gromacs:
  gmx_binary: "gmx"       # thread-MPI build
  ntmpi: 1                 # 1 rank = 1 GPU
  ntomp: 12                # 12 OpenMP threads

CPU runs can use either model. For multi-node runs, use real MPI:

gromacs:
  gmx_binary: "gmx_mpi"   # real MPI build
  ntmpi: 8                 # 8 MPI ranks
  ntomp: 1                 # 1 OpenMP thread per rank

Tip

On most clusters, the gmx binary (thread-MPI) is the best choice for single-node GPU runs. Use gmx_mpi only when you need multi-node scaling.

Complete `gromacs:` Config Reference

All fields in the gromacs: block of your config.yaml:

Field	Type	Default	Description
`gmx_binary`	`str \| null`	`null`	GROMACS binary path or name. Resolved via config > `$GMX_BIN` > PATH if null.
`mdrun_flags`	`str`	`""`	Extra flags for `gmx mdrun` (applied to all stages).
`mdrun_flags_equilibration`	`str \| null`	`null`	Override `mdrun_flags` for equilibration only. Falls back to `mdrun_flags` if null.
`mdrun_flags_production`	`str \| null`	`null`	Override `mdrun_flags` for production only. Falls back to `mdrun_flags` if null.
`grompp_flags`	`str`	`"-maxwarn 1"`	Extra flags for `gmx grompp`.
`command_prefix`	`str \| null`	`null`	Prefix prepended to all GROMACS commands (e.g., Singularity wrapper).
`mpi_launcher_flags`	`str`	`""`	Extra flags for the MPI launcher (`mpirun`). Only used with real-MPI binaries.
`module_load`	`str \| null`	`null`	Module load command inserted into SLURM scripts verbatim.
`env_exports`	`dict[str, str]`	`{}`	Environment variables exported before GROMACS commands.
`setup_commands`	`list[str]`	`[]`	Shell commands run after `module_load` and before GROMACS.
`ntmpi`	`int`	`1`	MPI ranks (`-ntmpi`). Also sets SLURM `--ntasks` (unless `slurm_ntasks` overrides).
`slurm_ntasks`	`int \| null`	`null`	Override SLURM `--ntasks` independently of `-ntmpi`. For multi-node MPI+GPU workflows.
`ntomp`	`int`	`8`	OpenMP threads per rank (`-ntomp`). Sets SLURM `--cpus-per-task`.
`gpu`	`bool`	`false`	Request GPU via SLURM. When false, `--gres=gpu` is omitted entirely.
`gpus`	`int`	`1`	Number of GPUs to request when `gpu` is true.
`memory`	`str`	`"16G"`	SLURM `--mem` allocation for GROMACS jobs.

Stage-specific mdrun flags

Use mdrun_flags_equilibration and mdrun_flags_production to apply different flags during different simulation phases. When set to null (default), they fall back to mdrun_flags.

gromacs:
  # Applied to all stages by default
  mdrun_flags: "-pin on"

  # Override for equilibration only (more conservative)
  mdrun_flags_equilibration: "-pin on -dlb yes"

  # Override for production only (full GPU offload)
  mdrun_flags_production: "-pin on -dlb auto -nb gpu -pme gpu -bonded gpu -update gpu"

command_prefix vs mpi_launcher_flags

These two fields serve different purposes. Use one approach, not both:

command_prefix — wraps all GROMACS commands with a prefix. Use for containers or site-specific launchers:

gromacs:
  command_prefix: "singularity exec --rocm --bind $PWD /path/to/gromacs.sif"

mpi_launcher_flags — passes extra flags to the mpirun launcher that PolyzyMD generates for real-MPI binaries:

gromacs:
  gmx_binary: "gmx_mpi"
  mpi_launcher_flags: "-genv I_MPI_FABRICS shm:tcp"
  # Result: mpirun -genv I_MPI_FABRICS shm:tcp gmx_mpi mdrun ...

Note

When command_prefix is set with a real-MPI binary (gmx_mpi), PolyzyMD skips automatic mpirun wrapping to avoid double-launching. Any mpi_launcher_flags are ignored (a warning is emitted).

env_exports

Environment variables exported in the SLURM script before GROMACS commands. Keys must be valid shell variable names ([A-Za-z_][A-Za-z0-9_]*).

gromacs:
  env_exports:
    GMX_GPU_DD_COMMS: "true"
    GMX_GPU_PME_PP_COMMS: "true"
    GMX_FORCE_UPDATE_DEFAULT_GPU: "true"
    OMP_PLACES: "cores"
    OMP_PROC_BIND: "close"

setup_commands

Shell commands run in order after module_load and env_exports, before any GROMACS commands:

gromacs:
  setup_commands:
    - "ulimit -s unlimited"
    - "source /opt/gromacs-2024/bin/GMXRC"

SBATCH to PolyzyMD YAML Mapping

If you are translating an existing SLURM batch script to PolyzyMD, use this table to find where each directive goes:

SBATCH Directive	PolyzyMD Equivalent	Location
`--partition`	`--partition` CLI or `--preset`	CLI override
`--qos`	`--qos` CLI	CLI override
`--time`	`--time-limit` CLI	CLI override
`--gres=gpu:TYPE:N`	`gromacs.gpu` + `gromacs.gpus` + `--gpu-type` CLI	Config + CLI
`--constraint`	`--constraint` CLI	CLI override
`--nodelist`	`--nodelist` CLI	CLI override
`--ntasks`	`gromacs.slurm_ntasks` or `gromacs.ntmpi`	Config
`--cpus-per-task`	`gromacs.ntomp`	Config
`--mem`	`gromacs.memory` or `--memory` CLI	Config + CLI
`--mail-user`	`--email` CLI	CLI override
`--account`	`--account` CLI	CLI override
`module load ...`	`gromacs.module_load`	Config
`export VAR=value`	`gromacs.env_exports`	Config
Setup commands	`gromacs.setup_commands`	Config
`mpirun` flags	`gromacs.mpi_launcher_flags`	Config
`singularity exec`	`gromacs.command_prefix`	Config

Cluster Recipes

Copy-pasteable configurations for common cluster setups. Each recipe shows the gromacs: YAML block and the CLI submit command.

Alpine AA100 (NVIDIA A100, 3 GPUs, real MPI)

Multi-GPU run using real MPI (gmx_mpi) with Intel compiler stack.

gromacs:
  gmx_binary: "gmx_mpi"
  gpu: true
  gpus: 3
  ntmpi: 3
  ntomp: 4
  memory: "64G"
  module_load: "module load intel/2022.1.2 impi/2021.5.0 gromacs/2023.3"
  mpi_launcher_flags: "-np 3"
  env_exports:
    GMX_GPU_DD_COMMS: "true"
    GMX_GPU_PME_PP_COMMS: "true"
    GMX_FORCE_UPDATE_DEFAULT_GPU: "true"
  mdrun_flags: "-pme gpu -nb gpu -bonded gpu -npme 1 -gpu_id 012 -ntomp 4"

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset aa100 \
    --replicates 1-3

Note

If your existing workflow uses mpirun -np 3 gmx_mpi mdrun ... with custom GMXRC sourcing or Plumed integration, translate those into the mpi_launcher_flags and setup_commands config fields.

Alpine AMI100 (AMD MI100, Singularity container)

AMD GPU run using a ROCm container with Singularity.

gromacs:
  gmx_binary: "gmx"
  command_prefix: "singularity exec --rocm --bind $PWD /projects/shared/gromacs-rocm.sif"
  gpu: true
  gpus: 3
  ntmpi: 3
  ntomp: 3
  slurm_ntasks: 16
  memory: "64G"
  module_load: "module load singularity"
  mdrun_flags: "-pme gpu -nb gpu -bonded gpu"

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset aa100 \
    --gpu-type mi100 \
    --replicates 1-3

Note

slurm_ntasks: 16 decouples the SLURM task count from the GROMACS thread-MPI rank count (ntmpi: 3). This is needed when the scheduler requires more tasks than GROMACS MPI ranks (e.g., for container overhead or multi-GPU resource allocation).

Alpine Amilan (CPU-only)

CPU-only run with direct MPI.

gromacs:
  gmx_binary: "gmx_mpi"
  ntmpi: 8
  ntomp: 1
  memory: "16G"
  module_load: "module load gcc/11.2.0 openmpi/4.1.1 gromacs/2024.2"
  mdrun_flags: "-ntomp 8"

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset aa100 \
    --partition amilan \
    --time-limit 24:00:00 \
    --replicates 1-3

Note

CPU-only runs do not need gpu: true or --constraint. The --partition CLI flag overrides the preset’s default partition.

Blanca GPU (preemptable, Intel MPI, A40/A100)

Multi-GPU run on Blanca’s preemptable QoS with Intel MPI fabric settings.

gromacs:
  gmx_binary: "gmx_mpi"
  gpu: true
  gpus: 3
  ntmpi: 3
  ntomp: 4
  memory: "64G"
  module_load: "module load intel/2022.1.2 impi/2021.5.0 gromacs/2023.3"
  mpi_launcher_flags: "-np 3 -genv I_MPI_FABRICS shm:tcp"
  env_exports:
    GMX_GPU_DD_COMMS: "true"
    GMX_GPU_PME_PP_COMMS: "true"
    GMX_FORCE_UPDATE_DEFAULT_GPU: "true"
  mdrun_flags: "-pme gpu -nb gpu -bonded gpu -npme 1 -gpu_id 012 -ntomp 4"

ml slurm/blanca  # required to see Blanca partitions

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset blanca-shirts \
    --constraint "A40|A100" \
    --email you@university.edu \
    --replicates 1-3

Important

Blanca uses qos=preemptable, meaning jobs can be killed by the node owner. The generated SLURM scripts handle this gracefully — see Preemption resilience below.

You must run module load slurm/blanca before sbatch to see Blanca partitions.

Dedicated GPU Node (nodelist pinning, A40)

Pin a job to a specific GPU node using --nodelist. Useful for lab-owned or reserved nodes where you know the hardware.

gromacs:
  gmx_binary: "gmx_mpi"
  gpu: true
  gpus: 3
  ntmpi: 3
  ntomp: 4
  memory: "64G"
  module_load: "module load intel/2022.1.2 impi/2021.5.0 gromacs/2023.3"
  mpi_launcher_flags: "-np 3 -genv I_MPI_FABRICS shm:tcp"
  env_exports:
    GMX_GPU_DD_COMMS: "true"
    GMX_GPU_PME_PP_COMMS: "true"
    GMX_FORCE_UPDATE_DEFAULT_GPU: "true"
  mdrun_flags: "-pme gpu -nb gpu -bonded gpu -npme 1 -gpu_id 012 -ntomp 4"

ml slurm/blanca

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset blanca-shirts \
    --constraint "A40" \
    --nodelist "gpu-node-001" \
    --email you@university.edu \
    --replicates 1-3

Note

--nodelist pins the job to a specific node. This is useful when you have dedicated or reserved nodes with known GPU hardware. Replace gpu-node-001 with your actual node hostname.

CPU-Only with Architecture Constraint (cascadelake)

CPU-only run on Blanca with CPU architecture pinning.

gromacs:
  gmx_binary: "gmx_mpi"
  ntmpi: 8
  ntomp: 1
  memory: "16G"
  module_load: "module load gcc/11.2.0 openmpi/4.1.1 gromacs/2024.2"
  mdrun_flags: "-ntomp 8"

ml slurm/blanca

pixi run -e build polyzymd submit \
    -c config.yaml \
    --engine gromacs \
    --preset blanca-shirts \
    --constraint "cascadelake" \
    --time-limit 7-00:00:00 \
    --replicates 1-3

Note

CPU constraints like cascadelake ensure the job runs on nodes with a compatible instruction set. A 7-day wall time (7-00:00:00) is common for long production runs on condo partitions.

GPU Constraints and Preemption

Why GPU constraints matter

GROMACS uses ahead-of-time compiled CUDA kernels (unlike OpenMM, which JIT-compiles at launch). A binary compiled for one GPU architecture may crash on a different one. If your cluster has mixed GPU types, always use --constraint:

--constraint "A40"              # single GPU type
--constraint "A40|A100"         # either type (OR)
--constraint "avx2&rh8"         # feature AND (CPU + OS flags)

This maps directly to #SBATCH --constraint in the generated script.

Preemption resilience

GROMACS SLURM scripts trap SIGTERM (the signal SLURM sends before preempting a job). When the trap fires:

The script forwards SIGTERM to gmx mdrun
GROMACS flushes a .cpt checkpoint file
The script waits for GROMACS to exit
The script resubmits itself via sbatch

Combined with --constraint, this ensures resumed jobs land on compatible GPU hardware. This is especially important on clusters with qos=preemptable (e.g., Blanca).

The -maxh flag is automatically set so GROMACS exits cleanly before the SLURM wall-time limit.

Note

Stopping a job permanently. A plain scancel <job_id> sends SIGTERM, which triggers the resubmission logic described above — the job will restart automatically. To cancel a job without resubmission, send SIGKILL instead:

scancel --signal=KILL <job_id>

This bypasses the trap entirely so no checkpoint flush or resubmission occurs. See also: need to stop a job permanently in the general SLURM guide.

Recovering Preempted Jobs

If automatic resubmission fails (e.g., sbatch was unavailable, or the job was killed without a grace period), use polyzymd recover:

Check recovery status

pixi run -e build polyzymd recover \
    -c config.yaml -r 1 --engine gromacs

This shows per-stage progress without submitting anything.

Submit a recovery job

pixi run -e build polyzymd recover \
    -c config.yaml -r 1 \
    --engine gromacs \
    --submit \
    --preset blanca-shirts \
    --constraint "A40" \
    --email you@university.edu

How checkpoint resume works

Stage	Checkpoint	Resume behavior
Energy minimization	`em.cpt`	Resumes from last EM step
Equilibration stage N	`eq_XX.cpt`	Resumes from last equilibration step
Production	`state.cpt`	Resumes from last production checkpoint

Completed stages (those with a .gro output file) are automatically skipped on resubmission. Partially completed stages resume from their checkpoint.

Dry-run recovery preview

pixi run -e build polyzymd recover \
    -c config.yaml -r 1 \
    --engine gromacs \
    --submit \
    --dry-run

GROMACS Output Files

When a GROMACS job completes, files are located in {projects_dir}/replicate_{N}/gromacs/:

gromacs/
├── {system}.gro              # Initial coordinates
├── {system}.top              # Topology (includes all molecule types)
├── *.itp                     # Molecule parameters (one per component)
│
├── em.mdp                    # Energy minimization parameters
├── eq_01_heating.mdp         # Heating stage
├── eq_02_free_equilibration.mdp
├── prod.mdp                  # Production MD parameters
│
├── run_{system}_gromacs.sh   # Generated shell script
│
├── em.tpr, em.gro, em.edr   # Energy minimization outputs
├── eq_01.*, eq_02.*          # Equilibration outputs
│
├── prod.tpr                  # Production run input
├── prod.xtc                  # Production trajectory
├── prod.edr                  # Production energies
├── prod.gro                  # Final coordinates
├── state.cpt                 # Checkpoint for restart
│
├── prod_nojump.xtc           # Trajectory with PBC jumps removed
└── prod_centered.xtc         # Centered trajectory for visualization

Position restraints are appended as #ifdef POSRES_* blocks inside the molecule .itp files. MDP files use -DPOSRES_PROTEIN, -DPOSRES_POLYMER, etc. to activate them during equilibration stages.

Troubleshooting

“GROMACS executable not found”

Cause: gmx command not in PATH after module loading.

Fix: Check your gromacs.module_load field. List prerequisites (compiler, MPI) before the GROMACS module:

gromacs:
  module_load: "module load gcc/11.2.0 openmpi/4.1.1 gromacs/2024.2"

“Non-dynamical integrator” error during EM

Cause: GPU offload flags incompatible with energy minimization.

Fix: This should not happen — PolyzyMD automatically strips unsafe GPU flags during EM. If it does, check that you are using PolyzyMD v1.3.0 or later.

“grompp warnings about charge groups”

Cause: OpenFF generates systems without charge groups.

Fix: This is expected and safe. The grompp_flags: "-maxwarn 1" default suppresses it.

“Fatal error: Number of atoms does not match”

Cause: Topology/coordinate mismatch from an interrupted build.

Fix: Rebuild from scratch:

rm -rf replicate_*/gromacs/
pixi run -e build polyzymd build -c config.yaml --format gromacs

Job dies with OOM

Fix: Increase gromacs.memory in config or use --memory CLI override:

polyzymd submit -c config.yaml --engine gromacs --memory 64G ...

Trajectory has broken molecules

Fix: Use the post-processed trajectories:

prod_nojump.xtc — molecules don’t jump across PBC boundaries
prod_centered.xtc — system centered for visualization

Run GROMACS Simulations on HPC Clusters

Quick Start: CPU

Build-only GROMACS export

Quick Start: GPU

GROMACS Flag Glossary

Parallelism flags

GPU offload flags

Performance flags

Energy minimization restrictions

Understanding MPI vs OpenMP Parallelism in GROMACS

Rule of thumb

Complete gromacs: Config Reference

Stage-specific mdrun flags

command_prefix vs mpi_launcher_flags

env_exports

setup_commands

SBATCH to PolyzyMD YAML Mapping

Cluster Recipes

Alpine AA100 (NVIDIA A100, 3 GPUs, real MPI)

Alpine AMI100 (AMD MI100, Singularity container)

Alpine Amilan (CPU-only)

Blanca GPU (preemptable, Intel MPI, A40/A100)

Dedicated GPU Node (nodelist pinning, A40)

CPU-Only with Architecture Constraint (cascadelake)

GPU Constraints and Preemption

Why GPU constraints matter

Preemption resilience

Recovering Preempted Jobs

Check recovery status

Submit a recovery job

How checkpoint resume works

Dry-run recovery preview

GROMACS Output Files

Troubleshooting

“GROMACS executable not found”

“Non-dynamical integrator” error during EM

“grompp warnings about charge groups”

“Fatal error: Number of atoms does not match”

Job dies with OOM

Trajectory has broken molecules

See Also

Complete `gromacs:` Config Reference