migrate from pure OpenMP to the hyprid with MPI

Question

I am conducting a global maximization for a rather daunting high dimensional problem. To make my life easier, my building-block program was ported to OpenMP and ran smoothly fine.

The main program actually consists of 4 building-block programs, each working under different setting. And the real task of mine is to feed the main program with a long list of parameter combinations. My preliminary thinking for overcoming this challenge is to divide the list into 10 smaller parts in parallelism.

Suppose the computing capacity I have is a high performance cluster on which a node has 8 cores (or 16 threads). My question is: is it correct that I simply use the usual MPI routines like MPI_INIT and its pals to complete the extension of my program from OpenMP to the hybrid with MPI? Is it correct that I simply specify the following in my PBS script:

#!/bin/bash -l
#PBS -l nodes=40:ppn=8
...
export OMP_NUM_THREADS=16
...

Or else do I need to think deeper by using alternative routine like MPI_INIT_THREAD to have my work done?

=============[edited June 24, 2014]

Here is the PBS file I finally figured out for my multi-threaded MPI program (without overlapping communication across OMP and MPI). My program works in this way: one multi-threaded MPI process is executed per node. Each node fully forks workload to all the threads that are physically associated with it. In addition, since I am also using Intel MKL and Intel MPI, I made corresponding adjustment in the PBS script below.

1 #!/bin/bash -l
2 #PBS -l walltime=01:00:00,nodes=32:ppn=8,pmem=2000mb
3 export OMP_NUM_THREADS=8
4 cd $PBS_O_WORKDIR 
5 mpirun -perhost 1 -np 32 -hostfile "$PBS_NODEFILE" \
6   -env I_MPI_PIN_DOMAIN omp \
7   -env KMP_AFFINITY compact ./main

Besides, be sure to add -mt_mpi into the compiler flags to correctly enable the support of Intel MKL.

Be aware, if you have 8 cores, using 16 threads might not be a good idea. See novell.com/coolsolutions/feature/637.html — Lyndon White
...be also aware when running on more than one node, that your export OMP_NUM_THREADS line is likely only being executed on the first node, and others may use only the default number of threads. If you want to change that environment variable on all threads, you may have to either put the line in your .bashrc (or whatever) file, or have the executable you launch be a script that first sets your environment variables, and then runs the real executable. — Jonathan Dursi

Wesley Bland Wesley Bland · Accepted Answer · 2014-06-18T13:47:10

It's true that it's not required to do anything special here in terms of MPI as long as you're never calling MPI functions in a parallel section. If you are going to do that, you need to use MPI_INIT_THREAD and provide the level of thread safety that you require.

In reality, you should probably do this anyway. If you're not going to be doing multiple MPI calls in parallel, then you can get by with MPI_THREAD_FUNNELED, otherwise, you probably need MPI_THREAD_MULTIPLE.

migrate from pure OpenMP to the hyprid with MPI

1 Answers