unwind loop in python for data parallelism, has to work in MPI environment

Question

I need to execute a for loop that acts on many data points, each one independently of any other data point.

for x in range(1,10000000) some_procedure(x)

and i need to make it run faster, by exploiting data parallelism, but I have to work with MPI.

I am using mpi4py but am completely new to this. Which one is supposed to work faster in theory between option a and b, and why:

a. in foo.py:

from mpi4py import MPI

sendbuf=[]
root=0
comm = MPI.COMM_WORLD
if comm.rank==0:
    sendbuf= range(0,10000000)

v=comm.scatter(sendbuf,root) 
some_procedure(v)

and later do the following:

mpiexec -np 10000000 python foo.py

b. or from mpi4py import MPI
import sys client_script = 'some_procedure.py' comm = MPI.COMM_SELF.Spawn(sys.executable, args=[client_script], maxprocs=10000000)

Many thanks beforehand!!

Whatever you do don't start 10^7 processes unless you have a machine with 10^7 processors. Generally speaking, especially while you are becoming familiar with MPI, do not start more processes than you have processors (or cores) to use. — High Performance Mark

Wesley Bland Wesley Bland · Accepted Answer · 2013-07-16T17:27:37

I think you're misunderstanding what MPI is for. MPI is the message passing interface that lets processes communicate. It doesn't take regular code and automatically parallelize it, nor does it do any sort of lightweight execution like threads that you can heavily overload on your system. As @High Performance Mark said, (until you really know what you're doing) you shouldn't have more MPI processes than you have physical cores available on your machine (whether it's your laptop or cluster that is available to you).

Usually the direction you'd go is to do something more like taking the entire range of data and dividing it up among the number of processes you have available. The more processes you have, the more the data is divided and (theoretically) the faster your program can execute.

An example might look like this (this will be in pseudo-C since I'm not familiar with mpi4py, but you can probably figure out what I mean):

datasize = 1000000;

MPI_Init(...);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

mydata_size = data size / size;
mystart = rank * mydata_size;

/* Do something for data[mystart...mydata_size] */

Instead of trying to create one process per element, you create as many processes as you have available and just divide up your data accordingly.

unwind loop in python for data parallelism, has to work in MPI environment

1 Answers