Trouble with MPI Allgatherv function

Question

I have been struggling to try to parallel this function which calculates interactions between particles. I had an idea to use Allgatherv which should distribute my original buffer to all other processes. Then using "rank" make a loop in which each process will calculate its part.In this program, MPI is overwritten to show stats that's why I am calling it mpi->function. Unfortunately, when I run it I receive following error. Can somebody advice what is wrong?

Error

void calcInteraction() {
    totalInteraction = 0.0;
    int procs_number;
    int rank;
    int particle_amount = particles;

    mpi->MPI_Comm_size(MPI_COMM_WORLD,&procs_number);
    mpi->MPI_Comm_rank(MPI_COMM_WORLD,&rank);
    //long sendBuffer[particle_amount];
    //int *displs[procs_number];
    //long send_counts[procs_number+mod];
    int mod = (particle_amount%procs_number);
    int elements_per_process = particle_amount/procs_number;
    int *send_buffer = new int[particle_amount]; //data to send
    int *displs = new int[procs_number]; //displacement
    int *send_counts = new int[procs_number];

    if(rank == 0)
    {
        for(int i = 0; i < particle_amount; i++)
        {
            send_buffer[i] = i; // filling buffer with particles
        }
    }
    for (int i = 0; i < procs_number;i++)
    {
        send_counts[i] = elements_per_process; // filling buffer since it can't be empty
    }

    // calculating displacement
    displs[ 0 ] = 0;
    for ( int i = 1; i < procs_number; i++ )
        displs[ i ] = displs[ i - 1 ] + send_counts[ i - 1 ];
    int allData = displs[ procs_number - 1 ] + send_counts[ procs_number - 1 ];

    int * endBuffer = new int[allData];

    int start,end; // initializing indices

    cout<<"entering allgather"<<endl;
    mpi->MPI_Allgatherv(send_buffer,particle_amount,MPI_INT,endBuffer,send_counts,displs,MPI_INT,MPI_COMM_WORLD);
    //                   ^from      ^how many        ^send   ^receive  ^how many  ^displ ^send    ^communicator
    //                               to  send         type    buffer    receive           type
    start = rank*elements_per_process;
    cout<<"start = "<< start <<endl;
    if(rank == procs_number) //in case that particle_amount is not even
    {
        end = (rank+1)*elements_per_process + mod;
    }
    else
    {
        end = (rank+1)*elements_per_process;
    }

    cout<<"end = "<< end <<endl;
    cout << "calcInteraction" << endl;

        for (long idx = start; idx < end; idx++) {
            for (long idxB = start; idxB < end; idxB++) {
                if (idx != idxB) {
                    totalInteraction += physics->interaction(x[idx], y[idx], z[idx], age[idx], x[idxB], y[idxB],
                                                             z[idxB], age[idxB]);
                }
            }
        }



    cout << "calcInteraction - done" << endl;
}

Gilles Gouaillardet Gilles Gouaillardet · Accepted Answer · 2017-11-09T01:40:52

You are not using MPI_Allgatherv() correctly.

I had an idea to use Allgatherv which should distribute my original buffer to all other processes.

The description suggests you need MPI_Scatter[v]() in order to slice your array from a given rank, and distributes the chunks to all the MPI tasks. If all tasks should receive the full array, then MPI_Bcast() is what you need.

Anyway, let's assume you need an all gather.

First, you must ensure all tasks have the same particles value.

Second, since you gather the same amout of data from every MPI tasks, and store them in a contiguous location, you can simplify your code with MPI_Allgather(). If only the last task might have a bit less data, then you can use MPI_Allgatherv() (but this is not what your code is currently doing) or transmit some ghost data so you can use the simple (and probably more optimized) MPI_Allgather().

Last but not least, you should send elements_per_process elements (and not particle_amount). That should be enough to get rid of the crash (e.g. MPI_ERR_TRUNCATE). But that being said, i am not sure that will achieve the result you need or expect.

Trouble with MPI Allgatherv function

1 Answers