Debugging distributed OR operation

Question

I am trying to perform a bitwise operation on integer arrays stored in different nodes of a distributed system. After computing the results I want to distribute the result to every node. For accomplishing this I am trying to use MPI_Allreduce operation. But I am getting runtime errors with the following code.

#include <bits/stdc++.h>
#include <mpi.h>
#include <unistd.h>
using namespace std;

int main (int argc, char* argv[]) {
    int numtasks, taskid, n=200, i;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &taskid);

    int *arr;
    arr = new int[n];
    for(i=0; i<n; i++) arr[i] = 0;
    for(i=taskid; i<n; i+=numtasks) arr[i] = 1;

    MPI_Allreduce(arr, arr, n, MPI_INT, MPI_BOR, MPI_COMM_WORLD);

    if(taskid == 0){
        for(i=0; i<n; i++) printf("%d ", arr[i]);
        printf("\n");
    }

    MPI_Finalize();
    return 0;
}

The program is working correctly when n is 1 but when n>1 it is giving the following error during runtime.

[user:17026] An error occurred in MPI_Allreduce
[user:17026] on communicator MPI_COMM_WORLD
[user:17026] MPI_ERR_BUFFER: invalid buffer pointer
[user:17026] MPI_ERRORS_ARE_FATAL: your MPI job will now abort

---

mpirun has exited due to process rank 2 with PID 17028 on node user exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here).

---

[user:17025] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[user:17025] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

I want to know does MPI_Allreduce works with n>1 or not because most of the examples available on the internet have taken n as 1 only. If my approach is completely wrong, please suggest me a better solution for my problem.

Answer 1

If you want to use the same buffer for sending and receiving, you can specify MPI_IN_PLACE as send buffer.

MPI_Allreduce(MPI_IN_PLACE, arr, n, MPI_INT, MPI_BOR, MPI_COMM_WORLD);

Note: This works only for intra-communicators. MPI_COMM_WORLD is an intra-communicator. If you don't know what an inter-communicator is, your communicator is probably an intra-communicator.

Answer 2

I don't know the MPI library, but from the MPI_Allreduce documentation:

int MPI_Allreduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)

I would guess that the "const void *sendbuf" and "void *recvbuf" HAVE to be different arrays, even more so in an heavily parallelized computing I suppose.

It would not surprise me that the library check if the two pointers are at the same address or overlap, hence the message:

[user:17026] MPI_ERR_BUFFER: invalid buffer pointer

So the code would be something like this:

#include <bits/stdc++.h>
#include <mpi.h>
#include <unistd.h>
using namespace std;

int main (int argc, char* argv[]) {
    int numtasks, taskid, n=200, i;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &taskid);

    int *arr_send = new int[n];
    int *arr_recv = new int[n];
    for(i=0; i<n; i++) arr_send[i] = 0;
    for(i=taskid; i<n; i+=numtasks) arr_send[i] = 1;

    MPI_Allreduce(arr_send, arr_recv, n, MPI_INT, MPI_BOR, MPI_COMM_WORLD);

    if(taskid == 0){
        for(i=0; i<n; i++) printf("%d ", arr_recv[i]);
        printf("\n");
    }

    MPI_Finalize();
    return 0;
}

Not a lot of difference from the original. Let me know if that works for you.