MPI Send and receive questions

Question

I have questions about MPI send and receive operations.

Suppose, we have 2 MPI threads that try to send message to each other. Following are three code snippets doing that:

First (Blocking 'send' and 'receive'):

...
int data = ...;
...
MPI_Send( &data, sizeof( int ), MPI_INT, 
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD );

MPI_Status status; 
MPI_Recv( &data, sizeof( int ), MPI_INT,
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &status );
...

Second (Non-blocking 'send' but blocking 'receive'):

...
int data = ...;
...
MPI_Request request;
MPI_Isend( &data, sizeof( int ), MPI_INT, 
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &request);

MPI_Status status; 
MPI_Recv( &data, sizeof( int ), MPI_INT,
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &status );
// Synchronize sender & receiver
MPI_Wait( &request, &status);
...

Third (Non-blocking 'receive' with blocking 'send'):

...
int data = ...;
...
MPI_Request request;
MPI_Irecv( &data, sizeof( int ), MPI_INT,
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &request );

MPI_Send( &data, sizeof( int ), MPI_INT, 
        (my_id == 0)?1:0, 0, MPI_COMM_WORLD);

MPI_Status status; 

// Synchronize sender & receiver
MPI_Wait( &request, &status);
...

I guess there are potential problems with above three code but I want your opinion. So, I have the following questions:

What are the (potential) problems (if any) with 3 codes given above?
Which of the above three code are valid/correct considering MPI standard so that it can work with all MPI implementations?
What is the best way (if not one of above 3 please write it) to do that?
In the third code, what if we change the order of MPI_Irecv and MPI_Send call?

PS: By the way, I have tried executing them using Scali MPI and all of them worked!

François Févotte François Févotte · Accepted Answer · 2012-02-23T08:07:29

Your first implementation is likely to cause a deadlock, especially if the comminication is done in synchronized mode (maybe it worked in your tests, because the communication was buffered; it's not likely to be the case for large data).

The other two implementations should work without deadlocking. I believe it's considered better practice to initiate receive operations before sends, so I would personally favour the 3rd implementation. From the MPI standard, section 3.7:

Advice to users

[...]

The message-passing model implies that communication is initiated by the sender. The communication will generally have lower overhead if a receive is already posted when the sender initiates the communication (data can be moved directly to the receive buffer, and there is no need to queue a pending send request). However, a receive operation can complete only after the matching send has occurred. The use of nonblocking receives allows one to achieve lower communication overheads without blocking the receiver while it waits for the send.

The third implementation with order MPI_Send/MPI_Irecv can deadlock in the MPI_Send call for the same reasons as the first implementation.

MPI Send and receive questions

1 Answers