C, MPI threads comunication issue

Question

Given this struct:

typedef struct 
{
    double rx, ry, rz;
    double vx, vy, vz;
    double fx, fy, fz;
    double mass;
} Body;

I am trying to pass it via MPI multithreding interface. It is a custom struct so I've created a an MPI Type:

int bodyParamas=10;
int blocklengths[10] = {1,1,1,1,1,1,1,1,1,1};
MPI_Datatype types[10] = {MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE};
MPI_Datatype mpi_body_type;
MPI_Aint     offsets[10];
offsets[0] = offsetof(Body, rx);
offsets[1] = offsetof(Body, ry);
offsets[2] = offsetof(Body, rz);
offsets[3] = offsetof(Body, vx);
offsets[4] = offsetof(Body, vy);
offsets[5] = offsetof(Body, vz);
offsets[6] = offsetof(Body, fx);
offsets[7] = offsetof(Body, fy);
offsets[8] = offsetof(Body, fz);
offsets[9] = offsetof(Body, mass);
MPI_Type_create_struct(bodyParamas, blocklengths, offsets, types, &mpi_body_type);
MPI_Type_commit(&mpi_body_type);

Then in my for loop i am sending data, and reciving it in other threads (diffrent than the root one):

        if(my_id == root_process) {
            int starting_bodies_array_index = -1;
            for(an_id = 1; an_id < num_procs; an_id++) {
                start_body_index = an_id*num_of_bodies_per_process + 1;
                end_body_index = (an_id + 1)*num_of_bodies_per_process;

                num_of_bodies_to_send = end_body_index - start_body_index + 1;
                starting_bodies_array_index += num_of_bodies_to_send;

                ierr = MPI_Send( &starting_bodies_array_index, 1 , MPI_INT,
                      an_id, send_data_tag, MPI_COMM_WORLD);

                ierr = MPI_Send( &bodies[starting_bodies_array_index], num_of_bodies_to_send, mpi_body_type,
                      an_id, send_data_tag, MPI_COMM_WORLD);
            }
        }
        else {

            ierr = MPI_Recv(&num_of_bodies_to_recive, 1, MPI_INT, 
                   root_process, send_data_tag, MPI_COMM_WORLD, &status);

            ierr = MPI_Recv(&bodiesRecived, num_of_bodies_to_recive, mpi_body_type, 
                   root_process, send_data_tag, MPI_COMM_WORLD, &status);

            num_of_bodies_recived = num_of_bodies_to_recive;

        }

I do not know what is the problem with my code. I am pretty sure, that my custom MPI type is correct, it is not mentioned in the error. Here is the error I am seeing:

*** An error occurred in MPI_Recv
*** reported by process [1580531713,1]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_TRUNCATE: message truncated
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)

Anyone see anything wrong?

Gilles Gouaillardet Gilles Gouaillardet · Accepted Answer · 2018-01-26T01:56:53

The root cause is you MPI_Send() num_of_bodies_to_send elements, but you MPI_Recv() starting_bodies_array_index elements instead.

you should replace the first MPI_Send() with

ierr = MPI_Send( &num_of_bodies_to_send, 1 , MPI_INT,
                 an_id, send_data_tag, MPI_COMM_WORLD);

C, MPI threads comunication issue

1 Answers