MPI Cartesian Topology: MPI_Neighbor_alltoall wrong data received

Question

I have an MPI Cartesian Topology and want to send every nodes rank to their neighbors with MPI_Neighbor_alltoall. I can't figure out, where the error is and i also implemented my own MPI_Neighbor_alltoall which doesn't work. I minimalised my code to an (hopefully) easy to understand code snippet.

alltoall.c

#include <mpi.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>


int main(int argc, char** argv) {


// MPI_Cart variables
MPI_Comm cart_comm;
MPI_Comm mycomm;
int ndims=2;
int periods[2]={0,0};
int coord[2];


    int dims[2]={3,3};
    int xdim = dims[0];
    int ydim = dims[1];
    int comm_size = xdim*ydim;

    // Initialize the MPI environment and pass the arguments to all the processes.
    MPI_Init(&argc, &argv);

    // Get the rank and number of the process
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    // output: dimensions
    if(rank==0){
        printf("dims: [%i] [%i]\n", xdim, ydim);
    }

    // enough nodes
    if(comm_size<=size){

        // Communicator count has to match nodecount in dims
        // so we create a new Communicator with matching nodecount
        int color;
        int graphnode;
        if(rank<comm_size){
            //printf("%d<%d\n",rank,comm_size);
            color=0;
            graphnode=1;
        } else {
            //printf("%d>=%d\n",rank,comm_size);
            // not used nodes
            color=1;
            graphnode=0;
        }

        MPI_Comm_split(MPI_COMM_WORLD, color, rank, &mycomm);
        MPI_Comm_rank(mycomm, &rank);
        MPI_Comm_size(mycomm, &size);


        // ***GRAPHNODE-SECTION***
        if(graphnode){

            // Create Dimensions
            MPI_Dims_create(size, ndims, dims);

            // Create Cartesian
            MPI_Cart_create(mycomm, ndims, dims, periods, 1, &cart_comm);

            // Get the name of the processor
            char processor_name[MPI_MAX_PROCESSOR_NAME];
            int len;
            MPI_Get_processor_name(processor_name, &len);

            // Get coordinates
            MPI_Cart_coords(cart_comm, rank, ndims, coord);

            // sending
            int *sendrank = &rank;
            int recvrank[4];
            MPI_Neighbor_alltoall(sendrank , 1, MPI_INT, recvrank, 1, MPI_INT, cart_comm);


            printf("my rank: %i, received ranks: %i %i %i %i\n", rank, recvrank[0], recvrank[1], recvrank[2], recvrank[3]);



        } else {
            // *** SPARE NODES SECTION ***

        }
    } else {
        // not enough nodes reserved
        if(rank==0)
            printf("not enough nodes\n");
    }

    // Finalize the MPI environment.
    MPI_Finalize();

}

So this code creates a 3x3 cartesian topology. It finalizes, if there are not enough nodes and let the spare nodes do nothing, when there are too many nodes. Even though this should be easy, i am still doing something wrong, because the output lacks some data.

output

$ mpicc alltoall.c
$ mpirun -np 9 a.out
dims: [3] [3]
my rank: 2, received ranks: -813779952 5 0 32621
my rank: 1, received ranks: 1415889936 4 0 21
my rank: 5, received ranks: 9 8 0 32590
my rank: 3, received ranks: 9 6 -266534912 21
my rank: 7, received ranks: 9 32652 0 21
my rank: 8, received ranks: 9 32635 0 32635
my rank: 6, received ranks: 9 32520 1372057600 21
my rank: 0, received ranks: -1815116784 3 -1803923456 21
my rank: 4, received ranks: 9 7 0 21

As you can see in the output, nobody has node 1,2 as neighbor and where does the 21 come from? Rank 4 should be the only node, that has 4 neighbors, but that should be {1,3,5,7} right? I really have no idea where my mistake here is.

coordinates should look like this:

[0,0] [1,0] [2,0]
[0,1] [1,1] [2,1]
[0,2] [1,2] [2,2]

and ranks like this:

0   3   6
1   4   7
2   5   8

When calling MPI_Cart_create it is not strictly necessary that the number of ranks in the parent communicator match the number of ranks in the topology. The count only has to be greater or equal. Excess ranks receive MPI_COMM_NULL in the output argument. The intermediate call to MPI_Comm_split is therefore redundant. — Hristo Iliev

Gilles Gouaillardet Gilles Gouaillardet · Accepted Answer · 2017-07-14T01:28:29

you are accessing a lot of uninitialized data (both in sendrank and recvrank)

here is a rewritten version of your test program that works for me

#include <mpi.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>


int main(int argc, char** argv) {


// MPI_Cart variables
MPI_Comm cart_comm;
MPI_Comm mycomm;
int ndims=2;
int periods[2]={0,0};
int coord[2];


    int dims[2]={3,3};
    int xdim = dims[0];
    int ydim = dims[1];
    int comm_size = xdim*ydim;

    // Initialize the MPI environment and pass the arguments to all the processes.
    MPI_Init(&argc, &argv);

    // Get the rank and number of the process
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    // output: dimensions
    if(rank==0){
        printf("dims: [%i] [%i]\n", xdim, ydim);
    }

    // enough nodes
    if(comm_size<=size){

        // Communicator count has to match nodecount in dims
        // so we create a new Communicator with matching nodecount
        int color;
        int graphnode;
        if(rank<comm_size){
            //printf("%d<%d\n",rank,comm_size);
            color=0;
            graphnode=1;
        } else {
            //printf("%d>=%d\n",rank,comm_size);
            // not used nodes
            color=1;
            graphnode=0;
        }

        MPI_Comm_split(MPI_COMM_WORLD, color, rank, &mycomm);
        MPI_Comm_rank(mycomm, &rank);
        MPI_Comm_size(mycomm, &size);


        // ***GRAPHNODE-SECTION***
        if(graphnode){

            // Create Dimensions
            MPI_Dims_create(size, ndims, dims);

            // Create Cartesian
            MPI_Cart_create(mycomm, ndims, dims, periods, 1, &cart_comm);

            // Get the name of the processor
            char processor_name[MPI_MAX_PROCESSOR_NAME];
            int len;
            MPI_Get_processor_name(processor_name, &len);

            // Get coordinates
            MPI_Cart_coords(cart_comm, rank, ndims, coord);

            // sending
            int sendrank[4];
            int recvrank[4];
            int i;
            char * neighbors[4];
            for (i=0; i<4; i++) {
                 sendrank[i] = rank;
                 recvrank[i] = -1;
            }
            MPI_Neighbor_alltoall(sendrank , 1, MPI_INT, recvrank, 1, MPI_INT, cart_comm);
            for (i=0; i<4; i++) {
                if (-1 != recvrank[i]) {
                    asprintf(&neighbors[i], "%d ", recvrank[i]);
                } else {
                    neighbors[i] = "";
                }
            }

            printf("my rank: %i, received ranks: %s%s%s%s\n", rank, neighbors[0], neighbors[1], neighbors[2], neighbors[3]);


        } else {
            // *** SPARE NODES SECTION ***

        }
    } else {
        // not enough nodes reserved
        if(rank==0)
            printf("not enough nodes\n");
    }

    // Finalize the MPI environment.
    MPI_Finalize();

}

MPI Cartesian Topology: MPI_Neighbor_alltoall wrong data received

1 Answers