Open MPI Waitall() Segmentation Fault

Question

I'm new with MPI and I'm trying to develop a non-blocking programm (with Isend and Irecv). The functionality is very basic (it's educational):

• There is one process (rank 0) who is the master and receives messages from the slaves (rank 1-P). The master only receives results.
• The slaves generates an array of N random numbers between 0 and R and then they do some operations with those numbers (again, it's just for educational purpose, the operations don't make any sense)
• This whole process (operations + send data) is done M times (this is just for comparing different implementations; blocking and non-blocking)

I get a Segmentation Fault in the Master process when I'm calling the MPI_waitall() funcion

#include <stdio.h>
#include <stdlib.h>
#include "mpi.h"
#include <math.h>
#include <time.h>
#define M 1000      //Number of times
#define N 2000      //Quantity of random numbers
#define R 1000      //Max value of random numbers

double SumaDeRaices (double*);

int main(int argc, char* argv[]) {
    int         yo;            /* rank of process      */
    int         p;             /* number of processes  */
    int         dest;          /* rank of receiver     */

    /* Start up MPI */
    MPI_Init(&argc, &argv);

    /* Find out process rank  */
    MPI_Comm_rank(MPI_COMM_WORLD, &yo);

    /* Find out number of processes */
    MPI_Comm_size(MPI_COMM_WORLD, &p);

    MPI_Request  reqs[p-1];
    MPI_Status   stats[p-1];   

    if (yo == 0) {
        int i,j;
        double result;
        clock_t inicio,fin;

        inicio = clock();

        for(i = 0; i<M; i++){ //M times
            for(j = 1; j<p; j++){ //for every slave
                MPI_Irecv(&result, sizeof(double), MPI_DOUBLE, j, i, MPI_COMM_WORLD, &reqs[j-1]);
            }
            MPI_Waitall(p-1,reqs,stats); //wait all slaves (SEG_FAULT)
        }
        fin = clock()-inicio;

        printf("Tiempo total de ejecucion %f segundos \n", ((double)fin)/CLOCKS_PER_SEC);   
    }
    else {
        double* numAleatorios = (double*) malloc( sizeof(double) * ((double) N) ); //array with numbers
        int i,j;
        double resultado; 
        dest=0;

        for(i=0; i<M; i++){ //again, M times
            for(j=0; j<N; j++){
                numAleatorios[j] = rand() % R ;
            }
            resultado = SumaDeRaices(numAleatorios);
            MPI_Isend(&resultado,sizeof(double), MPI_DOUBLE, dest, i, MPI_COMM_WORLD,&reqs[p-1]); //send result to master
        }
    }

    /* Shut down MPI */
    MPI_Finalize();

    exit(0);
} /* main */



double SumaDeRaices (double* valores){
    int i;
    double sumaTotal = 0.0;

    //Raices cuadradas de los valores y suma de estos   
    for(i=0; i<N; i++){
        sumaTotal = sqrt(valores[i]) + sumaTotal;
    }

    return sumaTotal;
}

Answer 1

There are several issues with your code. First and foremost in your Isend you pass &resultado several times without waiting until previous non-blocking operation finishes. You are not allowed to reuse the buffer you pass to Isend before you make sure the operation finishes.

Instead I recommend you using normal Send, because in contrast to synchronous send (SSend) normal blocking send returns as soon as you can reuse the buffer.

Second, there is no need to use message tags. I recommend you to just set tag to 0. In terms of performance it is simply faster.

Third, the result shouldn't be a simple variable, but an array of size at least (p-1)

Fourth, I do not recommend you to allocate arrays on stack, like MPI_Request and MPI_Status if the size is not a known small number. In this case the size of array is (p-1), so you better use malloc for this data structure.

Fifth, if you do not check status, use MPI_STATUSES_IGNORE.

Also instead of sizeof(double) you should specify number of items (1).

But of course the absolutely best version is just to use MPI_Gather.

Moreover, generally there is no reason not to run computations on the root node.

Here is slightly rewritten example:

#include <stdio.h>
#include <stdlib.h>
#include "mpi.h"
#include <math.h>
#include <time.h>
#define M 1000      //Number of times
#define N 2000      //Quantity of random numbers
#define R 1000      //Max value of random numbers

double SumaDeRaices (double* valores)
{

  int i;
  double sumaTotal = 0.0;

  //Raices cuadradas de los valores y suma de estos
  for(i=0; i<N; i++) {
    sumaTotal = sqrt(valores[i]) + sumaTotal;
  }

  return sumaTotal;
}


int main(int argc, char* argv[]) {
  int         yo;            /* rank of process      */
  int         p;             /* number of processes  */

  /* Start up MPI */
  MPI_Init(&argc, &argv);

  /* Find out process rank  */
  MPI_Comm_rank(MPI_COMM_WORLD, &yo);

  /* Find out number of processes */
  MPI_Comm_size(MPI_COMM_WORLD, &p);

  double *result;
  clock_t inicio, fin;
  double *numAleatorios;
  if (yo == 0) {
    inicio = clock();
  }

  numAleatorios = (double*) malloc( sizeof(double) * ((double) N) ); //array with numbers
  result = (double *) malloc(sizeof(double) * p);

  for(int i = 0; i<M; i++){ //M times
    for(int j=0; j<N; j++) {
      numAleatorios[j] = rand() % R ;
    }
    double local_result = SumaDeRaices(numAleatorios);
    MPI_Gather(&local_result, 1, MPI_DOUBLE, result, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD); //send result to master
  }

  if (yo == 0) {
    fin = clock()-inicio;

    printf("Tiempo total de ejecucion %f segundos \n", ((double)fin)/CLOCKS_PER_SEC);
  }

  free(numAleatorios);

  /* Shut down MPI */
  MPI_Finalize();
} /* main */