How to assign multiple cores of a single node to single job/process in MPI cluster?

Question

I have an MPI programme which I want to run on 30 nodes (each node has having 32 cores). How can I assign all cores of a node to a single job/process?

I am using slots to restrict the no of jobs for a particular node. node001 slots=1 max_slots=20 node002 slots=1 max_slots=20

Is there any parameter I can use to achieve this?

Thanks in advance.

Welcome to stackoverflow! If you are using a job manager like Slurm, the command sbatch features an option --exclusive which might be helpful, though other users may cry. For PBS'sqsub, there is the option -n corresponding to node-exclusive. Is there a job manager on your cluster? If you are alone on the cluster and if you are running mpirun directly, using different hostfiles, specifying maxslots could do the trick. Look at the --bynode option of mpirun . — francis
I am not using any job manager, I am running it with plain mpirun and hostfile. I want one slot for one node so I mentioned "slots=1" in hostfile. I want this single slot to use all the CPUs in that node. If we use --bynode option it distributes the jobs in a round robin fashion. Is there any option to allocate CPUs/Resources explicitly ? Thank you. — Uma Maheswara Rao Pinninti

Quelqu'un Quelqu'un · Accepted Answer · 2017-10-12T11:46:11

With openmpi, you can use the option --rankfile to explicitly set the ranks.

The syntax of the file can be found here : https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php^

Here is a very simple MPI+OpenMP program :

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sched.h>
#include <mpi.h>
#include <omp.h>

void main(int argc, char** argv)
{
    MPI_Init(&argc, &argv);
    unsigned cpu;
    unsigned node;

    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    #pragma omp parallel
    {
        printf("[%d:%d] %d\n", rank, omp_get_thread_num(), sched_getcpu());
    }
    MPI_Finalize();
}

Which prints [MPI_rank:OMP_rank] cpu for each OpenMP thread.

The basic format for rankfiles is :

rank <rank>=<host> slot=<slot>:<cores>

With this basic rankfile (Host=Marvin, 2cpu on one slot):

>cat ./rankfile
rank 0=Marvin slot=0:0
rank 1=Marvin slot=0:0
rank 2=Marvin slot=0:0
rank 3=Marvin slot=0:1
rank 4=Marvin slot=0:0-1
rank 5=Marvin slot=0:0

These are my prints :

>mpirun -n 6 --rankfile ./rankfile ./main
[0:0] 0
[1:0] 0
[2:0] 0
[3:0] 1
[4:0] 1
[4:1] 0
[5:0] 0

I didn't set OMP_NUM_THREADS environment variable in order to let OpenMP detect how many cores are available for each rank.

Hope this may help you

How to assign multiple cores of a single node to single job/process in MPI cluster?

1 Answers