MPI_Send and MPI_Recv overhead

2
votes

So I am trying to figure out the communication overhead of sending and receiving information between processors using MPI in C code.

I need to pass a buffer in both the send and receive, but all I want to do is to time how long it takes to do n communications between two processors.

Here is the my entire code:

main(int argc, char** argv){

int n;
int rank;
int time;
int i;
MPI_Status status;

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

if(rank == 0){
    n = atoi(argv[1]);
    printf("Size of data set = %d\n", n);
}

MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);

for(i = 0; i < n; i++){
    if(rank == 0){
        MPI_Send(&n, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
    }

    else{
        MPI_Recv(&n, 1, MPI_INT, 0, 1, MPI_COMM_WORLD, &status);
    }
}

MPI_Barrier(MPI_COMM_WORLD);

time = clock();
printf("Time: %d\n", time);

MPI_Finalize();
}

I did some testing and I have found that it works the way I want it to when I take out the for loop. So what is wrong with the for loop that causes either an infinite loop or seg fault?

4
cmpi
I'm surprised that works even without the loop. Your send and receive calls are using different tags, so the receive should never return. How many processes are you running? Also, if you're going to use MPI, you should use its timer call, too: MPI_Wtime() – Phil Miller
All I am trying to do is send from node 0 to node 1 and receive at node 1. It is a simple gathering of overhead for communication between the processors. – user287979
There's several correct answers below; I'll just add an aside, that if you're looking for these sorts of timings, a very handy set of tools is the Intel MPI Benchmarks, software.intel.com/en-us/articles/intel-mpi-benchmarks which includes MPI "ping pong" timings. – Jonathan Dursi

4 Answers

1
votes

There are two problems:

1) There is no check in the code to make sure it received the "size of the data set". If you start the code without a valid command line option, it will segfault or depending on your system, proceed in an unpredictable fashion.

2) The tags in Send and receive do NOT match. The tags must match for a successful communication. Without a matching tag, the Recv will wait forever till it finds a matching send. And send waits till it finds a matching receive.

Change the 1 in the receive, next to the MPI_COMM_WORLD, to 0, and the code will run fine.

1
votes

Below is a complete program which does what you're asking. There were several problems which would prevent the original version from working:

  1. The tags did not match, this would cause the program to stall.
  2. There is no check that MPI_COMM_WORLD contains exactly 2 MPI processes, this would also cause a stall.
  3. If no argument is present on the command line, you may get a seg fault. I added a default value for n.
  4. The timing will not generate anything useful, you have to call clock() before the send/recv's start running.

Good luck!

#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <mpi.h>

#define TAG_NUMBER 777     // just something
#define DEFAULT_N 10000000 // takes ~3 seconds on my laptop


int main(int argc, char **argv)
{
  int i,n,rank,size,message=0;
  clock_t start = clock();
  MPI_Status status;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  // This test assumes two processes in MPI_COMM_WORLD
  // ---------------------------------------------------------------------------
  if (size != 2) {
    if (rank == 0) { // only rank 0 prints
      printf("Please run with exactly 2 processes.\n");
    }
    MPI_Finalize();
    return 1;
  }

  // Collect from the command line the number of messages to send, default to
  // DEFAULT_N.
  // ---------------------------------------------------------------------------
  if (rank == 0) {
    if (argc > 1) {
      n = atoi(argv[1]);
    }
    else {
      n = DEFAULT_N;
    }
    printf("Number of messages to send = %d\n", n);
  }

  // Make sure everyone has the same n.
  MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);

  // ---------------------------------------------------------------------------
  // Here we have ranks 1 and 2 exchange n messages via MPI_Send and MPI_Recv.
  // ---------------------------------------------------------------------------
  for (i=0; i<n; i++) {
    if (rank == 0) {
      MPI_Send(&message, 1, MPI_INT, 1, TAG_NUMBER, MPI_COMM_WORLD);
    }
    else{
      MPI_Recv(&message, 1, MPI_INT, 0, TAG_NUMBER, MPI_COMM_WORLD, &status);
    }
  }

  MPI_Barrier(MPI_COMM_WORLD); // not really necessary
  printf("rank %d: time = %f seconds\n", rank,
     (double)(clock() - start)/CLOCKS_PER_SEC);

  MPI_Finalize();
  return 0;
}
0
votes

I believe the code would hang unless you used exactly two MPI processes.