How to determine MPI rank/process number local to a socket/node

Question

Say, I run a parallel program using MPI. Execution command

mpirun -n 8 -npernode 2 <prg>

launches 8 processes in total. That is 2 processes per node and 4 nodes in total. (OpenMPI 1.5). Where a node comprises 1 CPU (dual core) and network interconnect between nodes is InfiniBand.

Now, the rank number (or process number) can be determined with

int myrank;
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

This returns a number between 0 and 7.

But, How can I determine the node number (in this case a number between 0 and 3) and the process number within a node (number between 0 and 1)?

Answer 1

I believe you can achieve that with MPI-3 in this manner:

MPI_Comm shmcomm;
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0,
                    MPI_INFO_NULL, &shmcomm);
int shmrank;
MPI_Comm_rank(shmcomm, &shmrank);

Answer 2

It depends on the MPI implementation - and there is no standard for this particular problem.

Open MPI has some environment variables that can help. OMPI_COMM_WORLD_LOCAL_RANK will give you the local rank within a node - ie. this is the process number which you are looking for. A call to getenv will therefore answer your problem - but this is not portable to other MPI implementations.

See this for the (short) list of variables in OpenMPI.

I don't know of a corresponding "node number".

Answer 3

This exact problem is discussed on Markus Wittmann's Blog, MPI Node-Local Rank determination.

There, three strategies are suggested:

1. A naive, portable solution employs MPI_Get_processor_name or gethostname to create an unique identifier for the node and performs an MPI_Alltoall on it. [...]
2. [Method 2] relies on MPI_Comm_split, which provides an easy way to split a communicator into subgroups (sub-communicators). [...]
3. Shared memory can be utilized, if available. [...]

For some working code (presumably LGPL licensed?), Wittmann links to MpiNodeRank.cpp from the APSM library.

Answer 4

Alternatively you can use

int MPI_Get_processor_name( char *name, int *resultlen )

to retreive node name, then use it as color in

int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm)

This is not as simple as MPI_Comm_split_type, however it offers a bit more freedom to split your comunicator the way you want.