Non-homogeneous distribution of processes among nodes using OpenMPI

Question

I am running my executable with OpenMPI on a cluster using the SLURM resource managing software. I would like to find a way to specify how many and which processes should be assigned to each of the nodes, where the number of processes might be different for each node.

An example to clarify what I am looking for: Suppose I want to run 7 processes on 3 nodes. Then I want to be able to say something like: node 1 should run the process with rank n, node 2 and 3 should each run 3 of the remaining processes.

I do not care which physical node is node 1, as all the nodes are equal on the cluster I am using. Also I do not know a priori which nodes I will get assigned by SLURM, so I cannot hard-code the nodes' names in a hostfile. An example in the OpenMPI documentation that I found would define the hostfile like this for my example:

aa slots=1
bb slots=3
cc slots=3

but I have two problems with this approach:

I do not know a priori the names aa, bb, cc of the nodes.
Even if I knew them, the process on node aa does not necessarily have the right rank.

Open MPI supports mapping of ranks to hosts via the use of rankfile. See this FAQ item for an example. You'll have to parse the hostfile provided by SLURM and generate a corresponding rankfile. — Hristo Iliev
Oh, thanks, perfect, I have missed that. It is difficult to find something if you do not know what you are looking for. — Johanna

Johanna Johanna · Accepted Answer · 2014-02-27T10:30:28

Thanks to Hristo Iliev's comment, I found the solution for the example stated in the question:

#!/bin/bash 

HOSTFILE=./myHostfile
RANKFILE=./myRankfile

# Write the names of the nodes allocated by SLURM to a file
scontrol show hostname ${SLURM_NODELIST} > $HOSTFILE

# Number of processes
numProcs=7

# Number of nodes
numNodes=${SLURM_JOB_NUM_NODES}

# Counts the number of processes already assigned
count=0

while read p; do
  # Write the node names to a rank file
  if [ $count == 0 ]
  then
    echo "rank $count=$p slot=0-7" > $RANKFILE
    let count=$count+1
    let numNodes=$numNodes-1 # Number of nodes that are still available
  else
    # Compute the number of processes that should be assigned to this node
    # by dividing the number of processes that still need to be assigned by 
    # the number of nodes that are still available. (This automatically "floor"s the result.)
    let numProcsNode=($numProcs-$count)/$numNodes
    for i in `seq 1 $numProcsNode`
    do
        echo "rank $count=$p slot=0-7" >> $RANKFILE
        let count=$count+1
    done
    let numNodes=$numNodes-1 # Number of nodes that are still available
  fi
done < $HOSTFILE

mpirun --display-map -np $numProcs -rf $RANKFILE hostname

It is a bit ugly though. And probably "slot=0-7" should not have the "7" hard-coded.

Non-homogeneous distribution of processes among nodes using OpenMPI

1 Answers