MPI hostfile under PBS

1
votes

I am trying to run an MPI job on a cluster under PBS resource management. The cluster guide says I shouldn't have to worry about passing anything to mpiexec, as PBS should take care of that. For jobs on a single node, this is true and the job runs perfectly.

When I submit jobs requiring more than one node, the job exits saying it can't recognise the hosts. I included a routine in my PBS script to parse the $PBS_NODEFILE, and reconstruct a hosts file with the proper DNS suffix. PBS now recognises the hosts.

Now comes the troubling part: the hosts file I generate, isn't getting passed properly to mpiexec. See below for the hosts file I pass, and the output from the MPI process.

My hosts file:

cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-2-2.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk
cx1-25-3-1.cx1.hpc.ic.ac.uk

Output from the MPI process:

Host   : "cx1-25-2-2.cx1.hpc.ic.ac.uk"
PID    : 32752
nProcs : 24
Slaves : 
23
(
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32753"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32754"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32755"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32756"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32757"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32758"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32759"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32760"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32761"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32762"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32763"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32764"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32765"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32766"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.32767"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.316"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.319"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.320"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.321"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.322"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.323"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.324"
"cx1-25-2-2.cx1.hpc.ic.ac.uk.325"
)

Should the list of processes be identical to the host file? Why doesn't mpiexec accept the host file?

The implementation is OpenMPI 1.6.0, and a MWE of my PBS script follows:

#!/bin/sh
#PBS -l walltime=40:00:00
#PBS -l select=2:ncpus=12:mpiprocs=24:mem=4gb

module load openfoam/2.3.0 libscotch
pbsdsh2 cp -r $INPUT_DIR $TMPDIR/ 2>&1

# setting up hosts file
sed -n 1~24p $PBS_NODEFILE > hosts_buffer
for ii in `cat hosts_buffer`; do echo ${ii}.cx1.hpc.ic.ac.uk slots=12; done > hosts
nprocs=24;

# execution
mpiexec --hostfile hosts -np $nprocs $SOLVER 2>&1
2
mpicluster-computinghpcpbs
Please include your pbs script as well - Dmitri Chubarov
Which MPI implementation do you use and is it compiled with so-called tight integration with PBS? Open MPI can be compiled with support for tm (the PBS API) and when so it can use the tm interface to both obtain information about the host list and to launch processes on remote nodes. To check if you MPI is Open MPI just issue: mpicc --showme:version and it should print the version of Open MPI. - Hristo Iliev
MPI implementation is OpenMPI 1.6.0. - I didn't run mpicc --showme:version because it won't run if not in a PBS environment and the queues are very long, but I loaded the module and saw $MPI_ARCH_PATH and $MPI_LIBS pointing to the location for OpenMPI 1.6.0. As for the compile flag, a quick ompi_info | grep tm showed MCA ras: tm (MCA v2.0, API v2.0, Component v1.6) MCA plm: tm (MCA v2.0, API v2.0, Component v1.6) MCA ess: tm (MCA v2.0, API v2.0, Component v1.6) - linuxpirates

2 Answers

0
votes

I think you need to skip 12 lines

sed -n 1~12p $PBS_NODEFILE > hosts_buffer

or

uniq $PBS_NODEFILE > hosts_buffer

And also I notice your host file has only 23 lines.

You could also try it this way:

cd  $PBS_O_WORKDIR
mpiexec -hostfile $PBS_NODEFILE  -np  `wc -l < $PBS_NODEFILE` $SOLVER 2>&1
0
votes

I think the problem is related to your PBS directives.

Try changing from :

    #PBS -l select=2:ncpus=12:mpiprocs=24:mem=4gb

to :

    #PBS -l select=2:ncpus=12:mpiprocs=12:mem=4gb

This way, you request PBS to spawn 12 processes for each node instead of 24 processes previously. I don't think you need to regenerate the host file. Just run the code as :

    mpiexec -hostfile $PBS_NODEFILE -np 24 $SOLVER 2>&1

That should do it, hopefully.