Assign different number of openmp threads to each mpi process

Question

Assume that I have a code that runs on 384 MPI processes (24 compute nodes with 16 cores per compute node) and use the following simple script to submit my job to a job queue

#!/bin/bash
#PBS -S /bin/bash
#PBS -l nodes=24:ppn=16
#PBS -l walltime=01:00:00

cd $PBS_O_WORKDIR
module load openmpi
mpirun mycode > output_file

Is the following scenario possible: I need to assign one more node with 16 cores to do some specific calculations using 'openmp' and updates the rest of the 384 processes at some point with the results of the computations. So now I have 384 MPI processes with one thread running sequentially on each and one MPI process with 16 openmp threads.

Is it possible to accomplish this by OMP_NUM_THREADS and mpirun or any other tools?

I appreciate any suggestions

Thank you

Sina

Hristo Iliev Hristo Iliev · Accepted Answer · 2013-07-02T12:12:21

You could request 25 nodes with 16 ppns and then force only 385 MPI processes:

#PBS -l nodes=25:ppn=16
...
mpirun -np 384 mycode : -np 1 -x OMP_NUM_THREADS=16 mycode > output_file

This utilises the MPMD launch mode of Open MPI with different launch configurations separated by colons. Since by default ranks are populated sequentially over node slots, the first 384 ranks will span exactly 24 nodes, then the additional rank will get started on the very last node. For it the OMP_NUM_THREADS environment variable will get set to 16 therefore enabling 16 OpenMP threads. If the OpenMP program is a different executable, just substitute its name in the second launch configuration, e.g.:

mpirun -np 384 mycode : -np 1 -x OMP_NUM_THREADS=16 myompcode > output_file

Assign different number of openmp threads to each mpi process

1 Answers