MPI blocking receive semantics

Question

I got confused when reading about blocking send and receives in MPI. As stated in the MPI Standard 2.2, section 3.4 Communication Modes:

... The receive operation described in the last section is blocking: it returns only after the receive buffer contains the newly received message. A receive can complete before the matching send has completed (of course, it can complete only after the matching send has started).

Can somebody explain to me how a blocking receive can complete before the corresponding send has completed? My understanding is that a blocking receive (MPI_recv) returns when data is ready to be used in the receive buffer (i.e. the data has been received fully). Is this not the case?

Answer 1

Your understanding is correct.

The described case (receive completes before send completes) in the standard is telling you not to rely on the ordering of the library calls. It does not necessary need to be the case on a specific machine / implementation.

If you think about a synchronous send and a blocking receive of a tiny message. The synchronous send may only complete after the matching receive has started, possibly indicated by 'confirmation message (ACK)'. The actual message transfer might completely transferred (receive may return) before the ACK arrives at the sender (send may return). Please keep in mind this is highly implementation / machine specific.

Answer 2

Can somebody explain to me how a blocking receive can complete before the corresponding send has completed?

The definition of "send is complete" is that the user code can safely alter the contents of the send buffer. There is no guarantee of the state of the corresponding recv.

In some cases, it is necessary for the recv to complete (and return) before the send buffer can be safely altered. For instance, an ack must be sent from the recv side to the send side to indicate that it is safe to alter the send buffer (common in RDMA rendezvous protocol message transfers).

My understanding is that a blocking receive (MPI_recv) returns when data is ready to be used in the receive buffer (i.e. the data has been received fully). Is this not the case?

That is correct.