MPI status gets wrong sender

Question

I am solving a load balance problem using MPI: a Master process sends tasks to the slaves processes and collect results as they compute and send back their job. Since I want to improve performances more as possible I use non-blocking communications: Master sends several tasks and then wait until one process sends back its response so that the master can send additional work to it and so on.

I use MPI_Waitany() since I don't know in advance which slave process responses first, then I get the sender from the status and I can send the new job to it.

My problem is that sometimes the sender I get is wrong (a rank not in MPI_COMM_WORLD) and the program crashes; other times it works fine.

Here's the code. thanks!

//master
if (rank == 0) {

    int N_chunks = 10;
    MPI_Request request[N_chunks];
    MPI_Status status[N_chunks];
    int N_computed = 0;
    int dest,index_completed;

    //initialize array of my data structure
    vec send[N_chunks];
    vec recv[N_chunks];


    //send one job to each process in communicator
    for(int i=1;i<size;i++){

      MPI_Send( &send[N_computed], 1, mpi_vec_type, i, tag, MPI_COMM_WORLD);
      MPI_Irecv(&recv[N_computed], 1, mpi_vec_type, i, tag, MPI_COMM_WORLD,  
                &request[N_computed]);
      N_computed++;
    }

    // loop
    while (N_computed < N_chunks){

      //get processed messages
      MPI_Waitany(N_computed,request,&index_completed,status);

      //get sender ID dest
      dest = status[index_completed].MPI_SOURCE;

      //send a new job to that process
      MPI_Send( &send[N_computed], 1, mpi_vec_type, dest, tag, MPI_COMM_WORLD);
      MPI_Irecv(&recv[N_computed], 1, mpi_vec_type, dest, tag, MPI_COMM_WORLD,
                &request[N_computed]);

      N_computed++;
  }

  MPI_Waitall(N_computed,request,status);

  //close all process
  printf("End master\n");
}

Answer 1

You are not using MPI_Waitany() correctly.

It should be

MPI_Status status;
MPI_Waitany(N_computed,request,&index_completed,&status);
dest = status.MPI_SOURCE;

note :

• you need an extra loop to MPI_Wait() the last size - 1 requests
• you can revamp your algorithm and use MPI_Request request[size-1]; and hence save some memory

Answer 2

I forgot to add a line in the initial post in which I wait for all the pending requests: by the way if I initialize a new status everytime I do Waitany the master process crashes; I need to track which processes are still pending in order to wait the proper number of times... thanks by the way

EDIT: now it works even if I find it not very smart; would it be possible to initialize an array of MPI_Status at the beginning instead of doing it each time before a wait?

//master
if (rank == 0) {

    int N_chunks = 10;
    MPI_Request request[size-1];
    int N_computed = 0;
    int dest;
    int index_completed;

    //initialize array of vec
    vec send[N_chunks];
    vec recv[N_chunks];



    //initial case
    for(int i=1;i<size;i++){

      MPI_Send(&send[N_computed], 1, mpi_vec_type, i, tag, MPI_COMM_WORLD);
      MPI_Irecv(&recv[N_computed], 1, mpi_vec_type, i, tag, MPI_COMM_WORLD, &request[N_computed]);
      N_computed++;
    }



    // loop
    while (N_computed < N_chunks){

      MPI_Status status;
      //get processed messages
      MPI_Waitany(N_computed,request,&index_completed,&status);

      //get sender ID dest
      dest = status.MPI_SOURCE;

      MPI_Send(&send[N_computed], 1, mpi_vec_type, dest, tag, MPI_COMM_WORLD);
      MPI_Irecv(&recv[N_computed], 1, mpi_vec_type, dest, tag, MPI_COMM_WORLD, &request[N_computed]);

      N_computed++;
    }

    //wait other process to send back their load
    for(int i=0;i<size-1;i++){

      MPI_Status status;
      MPI_Waitany(N_computed, request, &index_completed,&status);

    }


    //end
    printf("Ms finish\n");
}