MPI_Gather with unequal amounts?

Question

I am implementing matrix multiplication using MPI_Scatter and MPI_Gather. My code works fine if the number of processes divides evenly into the number of matrix rows,columns. However, it crashes on MPI_Gather when they do not divide evenly. This makes sense because MPI_Gather is expecting a certain amount from each process and will not receive that much from the last process. Is there a way to make this work? I looked into MPI_Gatherv, but I don't want spaces between the data - I just want it to receive less than the amount for the last process.

Currently, I am using MPI_Scatter to send each process the amount of:

ceil(N/size)

where size is the number of processes I am using and N is the number of rows and columns in the matrices. The last process will not receive as much data as the others. I am making sure to stop doing the multiplication when I reach the end of the data in the last process. I could change this scatter+multiplication part if there is no way to change my MPI_Gather call. Any help is appreciated.

firegurafiku firegurafiku · Accepted Answer · 2015-08-20T21:26:36

I think that wrapping MPI_Scatterv and MPI_Gatherv in helper functions is the way to go. Today I have written a pair of such wrappers as part of my summer school, feel free to use them if you still need a solution.

void fillCounts(int rank, int size, int totalCount, int *counts, int *offsets) {
    int i;
    int n;
    int sum;

    n = totalCount / size;
    for (i = 0; i < size; i++)
    counts[i] = n;

    for (i = 0; i < totalCount % size; i++)
        counts[i] += 1;

    sum = 0;
    for (i = 0; i < size; i++) {
    offsets[i] = sum;
    sum += counts[i];
    }
}

int scatter01(const void *sendBuf, int totalCount, MPI_Datatype dataType,
                    void *recvBuf, int *recvCount,
                    int root, MPI_Comm comm) {
    int rank;
    int size;
    int *counts;
    int *offsets;
    MPI_Comm_rank(comm, &rank);
    MPI_Comm_size(comm, &size);

    counts  = alloca(size * sizeof(int));
    offsets = alloca(size * sizeof(int));
    fillCounts(rank, size, totalCount, counts, offsets);

    *recvCount = counts[rank]; 
    return MPI_Scatterv(sendBuf, counts, offsets, dataType,
                        recvBuf, counts[rank, dataType, root, comm);
}

int gather01(void *recvBuf, int totalCount, MPI_Datatype dataType,
            const void *sendBuf, int root, MPI_Comm comm) {
    int rank;
    int size;
    int *counts;
    int *offsets;
    MPI_Comm_rank(comm, &rank);
    MPI_Comm_size(comm, &size);

    counts  = alloca(size * sizeof(int));
    offsets = alloca(size * sizeof(int));
    fillCounts(rank, size, totalCount, counts, offsets);

    return MPI_Gatherv(sendBuf, counts[rank], dataType,
               recvBuf, counts, offsets, dataType, root, comm);
}

Usage example:

double globData[N];
double partData[N/size+1];
int n;

scatter01(globData, N, MPI_DOUBLE, partData, &n, 0, MPI_COMM_WORLD);
gather01 (globData, N, MPI_DOUBLE, partData,     0, MPI_COMM_WORLD);

MPI_Gather with unequal amounts?

3 Answers