MPI - Shared queue across processes

Question

I have a MPI program where the master node waits until certain number of tasks (say 1000) are completed by slave nodes. The slave nodes are in a while(True) loop and keep on generating output from their tasks. The runtime of these tasks can vary across tasks and nodes, so if there are 2 slave nodes and the master needs to wait for say 1000 tasks then slave-node-1 could have completed 450 tasks, and slave-node-2 the other 550.

What is the best way for the slave nodes to "tell" the master node that in total 1000 tasks have been completed ? It looks to me that I need some sort of a shared queue across processes where slaves can push data once their task is completed, and the master just polls on this queue's size until it hits 1000. Subsequently, the master can drain data from this queue to reset the queue size for slaves to fill in more data.

MPI stands for Message Passing Interface, so there is no "shared queue". Instead, processes send and receive messages to and from each other. A typical scenario is a slave ask some work to the master, the master send some work to the slave and the slave sends its result back. Have the master use MPI_ANY_SOURCE if there should be some imbalance between processes. — Gilles Gouaillardet

scaler scaler · Accepted Answer · 2020-01-24T15:24:12

There are two solutions I would recommend.

The first as Gilles points out is to use MPI_ANY_SOURCE to receive 1000 completion messages which can be sent from any of the workers.

The second is to use MPI_ACCUMULATE. In this case, the master node shares a window which is initialized to 0, then each worker uses MPI_ACCUMULATE to increment the value in the window after each task is completed. The master polls it's own local window until it reaches 1000.

In this case I'd stick to MPI_ANY_SOURCE rather than mess with creating and destroying windows. I don't think there is a compelling reason to add that complexity here.

MPI - Shared queue across processes

1 Answers