Different processes from different terminals communicating with MPI

Question

What I want to do is as follows:

I want to have a client-server type of relationship between one server process and several client processes. But my problem is, that these client processes will run in different terminal windows and they will require standard input. Therefore, running the MPI program with the same mpirun call will not do it for me, e.g.:

mpirun -np 2 --stdin 1 ./server : -np 3 ./client

In this example, standard input is also directed to only one process, which is of course another shortcoming for me.

One last point is, that I want to be able to create a new client process at any time, which can also communicate with a running server process.

So, what kind of an approach do I need to follow to accomplish these? I've been searching for a couple days, I am only getting more confused every time I read a different tutorial.

Background info, in case necessary: I am running on Ubuntu 12.04 and using Boost MPI. Still, a suggestion concerning any platform/MPI implementation is welcome.

What is the role of the server? How much communication do you need and will latency and/or bandwidth be important? — Jørgen Fogh
Why do they have to be started from different terminals? Also, it sounds like you want to be able to add and remove mpi tasks (in the form of client processes) on the fly. As far as I know, mpi isn't really suited for that. Perhaps 0MQ (zero MQ) would be something to look into? It sounds like it might do just what you want. — amaurea
for now there is nothing that requires a lot of resources. server will handle a database and clients will send queries / get results. one may say 'why MPI, then?'. It was a design choice to use MPI as these processes may get more complex tasks in the future. — incrediblehulk
we decided to look into other alternatives like 0MQ. it was possible to accomplish this with MPI,but it is indeed not meant for this. — incrediblehulk

amaurea amaurea · Accepted Answer · 2012-11-12T15:38:23

One thing you could do is to set up a fifo. It would look something like this: In one terminal, do:

mkfifo a.fifo
tee a.fifo | mpirun -np 2 ./server

And then in the other terminal, do:

mpirun -np 3 ./client < a.fifo

Not the most elegant solution, though. Also, this only takes care of the standard input part, not the part of making them part of the same communicator.

Dynamically adding and removing MPI tasks to a communicator is possible in MPI-2 through the dynamic process management commands, but I have never used them, and I am not sure how practical they are. Depending on what you are trying to do, zeromq might be a better choice. It makes it easy to set up broadcasting and gathering of data to a variable and dynamically changing number of tasks.

Different processes from different terminals communicating with MPI

1 Answers