MPI

OpenMPI and MVAPICH are provided as alternate Message Passing Interface (MPI) implementations on Bede.

OpenMPI is the main supported MPI on bede.

We commit to the following convention for all MPIs we provide as modules:

• The wrapper to compile C programs is called mpicc

• The wrapper to compile C++ programs is called mpicxx

• The wrapper to compile Fortran programs is called mpif90

CUDA-enabled MPI is available through OpenMPI, when a cuda module is loaded alongside openmpi, I.e.

ppc64leaarch64

module load gcc cuda openmpi

OpenMPI is provided by the openmpi module(s):

ppc64leppc64le

module load openmpi

module load openmpi/4.0.5

ppc64le partitions only

MVAPICH2 is only provided on ppc64le partitions/nodes (gpu, infer, test).

MVAPICH2 is provided by the mvapich2 module(s):

ppc64leaarch64

module load mvapich2

module load mvapich2/2.3.5-2

Note

There are a number of issues with OpenMPI 4 and the one-sided MPI communication features added by the MPI-3 standard. These features are typically useful when combined with GPUs, due to the asynchronous nature of the CUDA and OpenCL programming models.

For codes that require these features, we currently recommend using the mvapich2 module on ppc64le nodes/partitions.

We also offer the mvapich2-gdr/2.3.6 module on ppc64le nodes/partitions. This is a version of MVAPICH2 that is specifically designed for machines like Bede, providing optimised communications directly between GPUs - even when housed in different compute nodes.

Unlike the openmpi and mvapich2 modules, mvapich2-gdr does not adapt itself to the currently loaded compiler and CUDA modules. This version of the software was built using GCC 8.4.1 and CUDA 11.3.

ppc64leaarch64

module load mvapich2-gdr/2.3.6 gcc/8.4.0 cuda/11.3.1

Further information can be found on the MVAPICH2-GDR pages.