OpenMPI and Slurm Integration on Debian Clusters

Integrating OpenMPI with the Slurm Workload Manager on a Debian cluster enables efficient execution and resource management for high-performance computing (HPC) applications. This integration relies on native process management interfaces—primarily PMIx—to allow Slurm to allocate hardware resources, launch processes, and manage synchronization while OpenMPI handles high-speed, inter-process communication across Debian-based compute nodes.

Core Architecture and Communication

The integration between OpenMPI and Slurm hinges on the Process Management Interface (PMI), specifically PMIx (PMI Exascale). When a job is submitted, Slurm handles node allocation, security validation, and task distribution. Rather than spawning independent processes via SSH, OpenMPI utilizes Slurm's underlying infrastructure to launch and synchronize ranks.

Debian provides native binary packages for both platforms. OpenMPI packages in Debian (openmpi-bin, libopenmpi-dev) are built with PMI/PMIx support enabled, allowing them to hook directly into the Slurm daemon (slurmd) running on each node.

Package Installation on Debian

To establish the environment across all nodes in the cluster, install the necessary Slurm and OpenMPI packages via the Debian apt repository:

On the control node:

sudo apt update
sudo apt install slurmctld slurm-client munge

On the compute nodes:

sudo apt update
sudo apt install slurmd slurm-client openmpi-bin libopenmpi-dev libpmix-dev munge

Authentication across the cluster must be synchronized using munge. The /etc/munge/munge.key file must be identical across all nodes and owned by the munge user.

Configuring Slurm for OpenMPI

To ensure Slurm knows which process manager to use by default, modify /etc/slurm/slurm.conf. Add or update the MpiDefault configuration directive:

MpiDefault=pmix

Alternatively, MpiDefault=pmi2 can be used if older applications require it, but PMIx provides the lowest latency and deepest integration with modern OpenMPI releases. Once updated, restart the Slurm daemons across the cluster:

sudo systemctl restart slurmctld   # On the controller
sudo systemctl restart slurmd      # On compute nodes

Job Execution Workflows

There are two primary methods to launch OpenMPI jobs inside a Slurm-managed Debian environment:

Because OpenMPI leverages PMIx, you can invoke your compiled MPI executable directly with Slurm's srun command. This bypasses the need for OpenMPI's mpirun wrapper, allowing Slurm to have direct accounting and resource control over every individual rank.

#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --partition=compute

srun --mpi=pmix ./my_mpi_program

2. Using mpirun Inside Slurm

If your workflow requires mpirun (or mpiexec), OpenMPI automatically detects the Slurm environment through variables such as SLURM_JOB_ID, SLURM_NNODES, and SLURM_NODELIST. It queries Slurm for the allocated hosts and distributes the processes without requiring a separate hostfile:

#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --partition=compute

mpirun ./my_mpi_program

High-Speed Fabric and Network Considerations

On Debian clusters utilizing high-speed fabrics such as InfiniBand, OpenMPI relies on the Unified Communication X (UCX) framework or Libfabric. Ensure the necessary user-space drivers are installed:

sudo apt install libibverbs1 ibverbs-providers librdmacm1

OpenMPI automatically discovers InfiniBand or RoCE interfaces when running under Slurm allocations, binding memory and CPU affinity based on the topological constraints Slurm assigns to the job allocation.