Launching OpenMPI/pthread apps with slurm

1
votes

On Cray computers such as an XE6, when launching a hybrid MPI/pthreads application via aprun there is a depth parameter which indicates the number of threads each process can spawn. For example,

aprun -N2 -n12 -d5

Each process can spawn 5 threads which the OS will distribute.

Is there a similar option when launching OpenMPI/pthread applications with Slurm's srun? The machine is a generic HP cluster with nehalem processors and IB interconnect. Does it matter if thread support level is only MPI_THREAD_FUNNELED?

2
mpiopenmpi

2 Answers

4
votes

This is the script I use to launch a mixed MPI-OpenMP job. Here n is the number of nodes and t the number of threads.

sbatch <<EOF
#!/bin/bash
#SBATCH --job-name=whatever
#SBATCH --threads-per-core=1
#SBATCH --nodes=$n
#SBATCH --cpus-per-task=$t
#SBATCH --time=48:00:00
#SBATCH --mail-type=END
#SBATCH [email protected]
#SBATCH --output=whatever.o%j

. /etc/profile.d/modules.sh
module load gcc
module unload openmpi
module load mvapich2

export OMP_NUM_THREADS=$t 
export LD_LIBRARY_PATH=/apps/eiger/Intel-CPP-11.1/mkl/lib/em64t:${LD_LIBRARY_PATH}

mpiexec -np $n myexe

EOF

Hope it helps

0
votes

You typically select the number of MPI processes with --ntasks and the number of threads per process with --cpu-per-task. If you request --ntasks=2 and --ncpus-per-task=4, then slurm will allocate 8 cpus either on one node, or on two nodes, four cores each, depending on resource availability and cluster configuration.

If you specify --nodes instead of --ntasks, Slurm will allocate one process per node, as if you choose --ntask-per-node=1.