Process MPI data as it's received (Pipe-lining)

2
votes

Say I have an MPI_Send from machine A with a huge message, and on another machine (say, B) it's being received with MPI_Recv (or MPI_isend and MPI_irecv respectively). Currently, B processes the data that's received ONLY after ALL the data has arrived; is there an MPI functionality that allows me to start processing some of the data as I receive it?

1
c++mpi
Anything wrong with sending it in smaller chunks? – eduffy
@eduffy Wouldn't that mean I have to send and receive many times? In that case, wouldn't the communication overhead be too costy? Thanks! – tj56

1 Answers

2
votes

Not really. You can have a buffer of a minimal size and start sending data through that and consume it as it is received.

You may want to have a look at persistent communication request support of MPI http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node65.htm#Node65 and create some higher level API that suits your application.

All the cases I've seen with pipelining where ad hoc solutions made with MPI_Send() / MPI_Recv() pairs or MPI_Sendrecv().