Why my C++ parallel program gives MPI fatal error in MPI_Gather?

Question

My sorting program works fine with even number of elements in array, but gives error

" Fatal error in MPI_Gather: Message truncated, error stack: MPI_Gather(sbuf=0x00A2A700, scount=4, MPI_INT, rbuf=0x00A302C8, rcount=4, MPI_INT, root=0, MPI_COMM_WORLD) failed Message from rank 1 and tag -1342177184 truncated; 28 bytes received but buffer size is 16 "

for odd number of elements in array. The problem begins in the code if ((world_rank == 1) && (n % world_size != 0)). I tried everything and it didn't work. How can I fix this? Thanks in advance!

void merge(int*, int*, int, int, int);
void mergeSort(int*, int*, int, int);

int main(int argc, char** argv) {

    


    int world_rank;
    int world_size;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    int n = atoi(argv[1]);
    int* original_array{ new int[n] {} };
    //int original_array[]=new int[n];

    int c;
    srand(time(NULL));

    if (world_rank == 0) {
        printf("This is the unsorted array: ");
        for (c = 0; c < n; c++) {

        original_array[c] = rand() % n;
        printf("%d ", original_array[c]);

        }
        printf("\n");
        printf("\n");
    }
    
    
    int size = n / world_size;
    int* sub_array=NULL;
    int* tmp_array = NULL;
    int* sorted = NULL;

    if (world_rank == 0) {

        sorted = { new int[n] {} };

    }


    if ((world_rank == 1) && (n % world_size != 0)) {
        int r = n % world_size;
        int size2 = size + r;
        sub_array = { new int[size2] {} };
        MPI_Scatter(original_array, size2, MPI_INT, sub_array, size2, MPI_INT, 0, MPI_COMM_WORLD);
        tmp_array = { new int[size2] {} };
        mergeSort(sub_array, tmp_array, 0, (size2 - 1));
        MPI_Gather(sub_array, size2, MPI_INT, sorted, size2, MPI_INT, 0, MPI_COMM_WORLD);
    }
    else {
        sub_array = { new int[size] {} };
        MPI_Scatter(original_array, size, MPI_INT, sub_array, size, MPI_INT, 0, MPI_COMM_WORLD);
        tmp_array = { new int[size] {} };
        mergeSort(sub_array, tmp_array, 0, (size - 1));
        MPI_Gather(sub_array, size, MPI_INT, sorted, size, MPI_INT, 0, MPI_COMM_WORLD);
    }

    

    

    
    if (world_rank == 0) {

        printf("Array state before final mergeSort call: ");
        for (c = 0; c < n; c++) {

            printf("%d ", sorted[c]);

        }
        
        printf("\n");

        int* other_array{ new int[n] {} };
        mergeSort(sorted, other_array, 0, (n - 1));

        printf("This is the sorted array: ");
        for (c = 0; c < n; c++) {

            printf("%d ", sorted[c]);

        }

        printf("\n");
        printf("\n");

        delete[] sorted;
        delete[] other_array;

    }

    delete[] original_array;
    delete[] sub_array;
    delete[] tmp_array;

    /********** Finalize MPI **********/
    MPI_Finalize();

}

dreamcrash dreamcrash · Accepted Answer · 2021-02-17T17:23:11

TL;DR: With an even number of elements the processes call MPI_Scatter and MPI_Gather with the same count, with an odd number they don't.

My sorting program works fine with even number of elements in array

When the array size is even all processes execute the else part of:

   if ((world_rank == 1) && (n % world_size != 0)) {
        int r = n % world_size;
        int size2 = size + r;
        sub_array = { new int[size2] {} };
        MPI_Scatter(original_array, size2, MPI_INT, sub_array, size2, MPI_INT, 0, MPI_COMM_WORLD);
        tmp_array = { new int[size2] {} };
        mergeSort(sub_array, tmp_array, 0, (size2 - 1));
        MPI_Gather(sub_array, size2, MPI_INT, sorted, size2, MPI_INT, 0, MPI_COMM_WORLD);
    }
    else {
        sub_array = { new int[size] {} };
        MPI_Scatter(original_array, size, MPI_INT, sub_array, size, MPI_INT, 0, MPI_COMM_WORLD);
        tmp_array = { new int[size] {} };
        mergeSort(sub_array, tmp_array, 0, (size - 1));
        MPI_Gather(sub_array, size, MPI_INT, sorted, size, MPI_INT, 0, MPI_COMM_WORLD);
    }

but gives error " Fatal error in MPI_Gather: Message truncated, error stack: MPI_Gather(sbuf=0x00A2A700, scount=4, MPI_INT, rbuf=0x00A302C8, rcount=4, MPI_INT, root=0, MPI_COMM_WORLD) failed Message from rank 1 and tag -1342177184 truncated; 28 bytes received but buffer size is 16 " for odd number of elements in array.

However, when the size of the array is odd process 1 executes the if part of the aforementioned if and else, whereas the other processes executed the else part. Therefore some processes will call the routines MPI_Gather and MPI_Scatter with a different count. Those routines are supposed to be call the same way by all processes.

To fix your code you can changed it so that all processes call with the same count the MPI_Scatter and MPI_Gather routines.

In general when the size of the input is not evenly divided by the number of processes one will face the same issue that you did. To solve it, one can add dummy values into the array so that the size is evenly divided by the number of processes. Or one can use the MPI_Gatherv:

Gathers into specified locations from all processes in a group

and MPI_Scatterv

Scatters a buffer in parts to all processes in a communicator

From source one can read:

MPI_Gatherv and MPI_Scatterv are the variable-message-size versions of MPI_Gather and MPI_Scatter. MPI_Gatherv extends the functionality of MPI_Gather to permit a varying count of data from each process, and to allow some flexibility in where the gathered data is placed on the root process. It does this by changing the count argument from a single integer to an integer array and providing a new argument displs (an array) . MPI_Scatterv extends MPI_Scatter in a similar manner. More information on the use of these routines will be presented in an Application Example later in this module.

Why my C++ parallel program gives MPI fatal error in MPI_Gather?

1 Answers