How to intervleave MPI_Gather

Question

I have two MPI processes. Each process has one array with same size. I would like to merge two array into one with double size. Which mpi interface should I use?

For example, I have two array:

I want to have them go to rank 0:

I have tried MPI_Gather

MPI_Gather(
    arr, // void* send_data,
    4, // int send_count,
    MPI_INT, // MPI_Datatype send_datatype,
    arr_merged, // void* recv_data,
    1, // int recv_count,
    MPI_INT, // MPI_Datatype recv_datatype,
    0, // int root,
    MPI_COMM_WORLD); // MPI_Comm communicator

but I got error

David Henty David Henty · Accepted Answer · 2016-06-20T16:12:40

Your code above is incorrect in terms of the counts. Unless you are playing around with derived types, sendcount and recvcount should be the same (in this case 4). However, this would lead to the order: 0 2 4 6 1 3 5 7, i.e. the two messages received one after the other.

You can receive into strided pattens by defining a vector type. However, in order to receive into the correct place you need to specify different displacements for each sender using MPI_Gatherv - you also need to resize the vector type as the default extent is not useful.

The appended code seems to work OK where the first line of output uses the usual gather but the second uses gatherv and a vector datatype on the receive end. Unfortunately, can't see a simpler way to do it ...

user$ mpicc -o mpigather mpigather.c
user$ mpiexec -n 2 ./mpigather  
0 2 4 6 1 3 5 7 
0 1 2 3 4 5 6 7 
user$ mpiexec -n 3 ./mpigather
0 3 6 9 1 4 7 10 2 5 8 11 
0 1 2 3 4 5 6 7 8 9 10 11

Here's the code:

#include <stdio.h>
#include <mpi.h>

#define NBUFF 4

void main(void)
{
  int i;
  MPI_Comm comm = MPI_COMM_WORLD;
  MPI_Datatype vector, resizevector;
  int rank, size;

  MPI_Init(NULL, NULL);
  MPI_Comm_rank(comm, &rank);
  MPI_Comm_size(comm, &size);

  int sendbuff[NBUFF];
  int recvbuff[size*NBUFF];

  for (i=0; i < NBUFF; i++)
    {
      sendbuff[i] = rank+i*size;
    }

  for (i=0; i < NBUFF*size; i++)
    {
      recvbuff[i] = -1;
    }

  MPI_Gather(sendbuff, NBUFF, MPI_INT, recvbuff, NBUFF, MPI_INT, 0, comm);

  if (rank == 0)
    {
      for(i=0; i < NBUFF*size; i++)
      {
        printf("%d ", recvbuff[i]);
      }
      printf("\n");
    }

  MPI_Type_vector(NBUFF, 1, size, MPI_INT, &vector);
  MPI_Type_create_resized(vector, 0, sizeof(int), &resizevector);
  MPI_Type_commit(&resizevector);

  int recvcounts[size];
  int displs[size];

  for (i=0; i < size; i++)
    {
      recvcounts[i] = 1;
      displs[i] = i;
    }

  MPI_Gatherv(sendbuff, NBUFF, MPI_INT, recvbuff, recvcounts, displs, resizevector, 0, comm);

  if (rank == 0)
    {
      for(i=0; i < NBUFF*size; i++)
      {
        printf("%d ", recvbuff[i]);
      }
      printf("\n");
    }

  MPI_Finalize();
}

How to intervleave MPI_Gather

1 Answers