mpi4py: Spawn processes as Python threads for easier debugging

Question

To use mpi4py, the standard approach is to use mpiexec to start a program using multiple MPI processes. For example mpiexec -n 4 python3.5 myprog.py.

Now, that makes debugging difficult, because one can not straight-forwardly use the Python interpreter plus maybe an IDE debugger using the Python interpreter. However, it is no problem to debug a multi-threaded application.

So my idea would be: Instead of using mpiexec to spwan the processes, I have a Python script that will spwan several threads, each of them will act as an MPI process, all happening within the Python interpreter. So the use of mpiexec woud not be necessary and I could debug my application like any other multi-threaded Python program. Would that be possible, and how?

(In general, I'd be very happy to find some good example collections or tutorials for mpi4py, there's not very much available.)

Answer 1

You do not need to implement any hardcore idea. mpiexec is already spawning process for you. Assuming your using the pudb Python debugger you could do the following:

mpiexec -np 4 xterm -e python2.7 -m pudb.run helloword.py

The -e option of xterm is specifying which program xterm is going to execute.

PS: I have not test it with Python 3.5 but a similar solution will work