Suppose I am running an MPI program on 5 nodes, which will run a simulation 50 times. However, simulations may take notably different amounts of time. If I have a set of initial conditions, say ic1,ic2,...ic50, when one node/process completes a simulation, I would like it to run a new simulation using the next (not yet used) set of initial conditions. My initial thought was to use MPI_Bcast or MPI_Gather s.t. all nodes have an int holding the next index to run, and update this before starting a new simulation. Is this a reasonable idea/are there other and/or better solutions?
0
votes
1 Answers
2
votes
you need a shared work queue. There are lots of ways to do this. The simplest way is to designate one mpi rank the "master" and the others the "workers". Workers ask the master for a work unit (one of your initial conditions), go to work, then when done ask master for new one. Some additional detail here (it says OpenMPI, but there's nothing about it that's specific to OpenMPI): Non-blocking data sharing through OpenMPI
There is a project called ADLB (http://www.mcs.anl.gov/project/adlb-asynchronous-dynamic-load-balancer) that is a work queue on steroids. Possibly (definitely) overkill for a 5 node job, but might make sense for a 5,000 node job.
you could use RMA shared memory and use that to keep track of an index. If the process owning the RMA window is itself busy computing on a work unit, it may be a bit slow to respond to requests, but the benefit you would not need to burn one MPI process as a master process.
You could use the MPI shared file pointer routines to keep a work queue on disk. I'm generally quite down on the MPI shared file pointer routines, but in this case where the I/O is small (read an initial condition) relative to the amount of CPU work, it will be ok.
