Sum of the element of an array in parallel?

Question

I'm trying to write a MPI program that calculates the sum of an array of integers.

For this purpose I used MPI_Scatter to send chunks of the array to the other processes then MPI_Gather to get the sum of each chunk by the root process(process 0).

The problem is one of the processes receives two elements but the other one receives random numbers. I'm running my code with 3 processes.

Here is what I have:

#include <stdio.h>
#include <mpi.h>

int main(int argc,char *argv[]){

    MPI_Init(NULL,NULL); // Initialize the MPI environment

    int world_rank; 
    int world_size;

    MPI_Comm_rank(MPI_COMM_WORLD,&world_rank);
    MPI_Comm_size(MPI_COMM_WORLD,&world_size);

    int number1[2]; //buffer for processes
    int sub_sum = 0; 
    int sub_sums[2]; 
    int sum;

    int number[4];

    if(world_rank == 0){
       number[0]=1;
       number[1]=3;
       number[2]=5;
       number[3]=9;
    }
    //All processes
    MPI_Scatter(number, 2, MPI_INT, &number1, 2, MPI_INT, 0, MPI_COMM_WORLD);

    if(world_rank!=0){
       printf("I'm process %d , I received the array : ",world_rank);
       for(int i=0 ; i<2 ; i++){
           printf("%d ",number1[i]);
           sub_sum = sub_sum + number1[i];
       }
       printf("\n");
   }

   MPI_Gather(&sub_sum, 1, MPI_INT, &sub_sums, 1, MPI_INT, 0,MPI_COMM_WORLD);

   if(world_rank == 0){
      sum=0;
      for(int i=0; i<2;i++){
          sum+= sub_sums[i];
        }
        printf("\nthe sum of array is: %d\n",sum);
  }

   MPI_Finalize();
   return 0;
}

The result:

I'm process 1 , I received the array : 5 9 
I'm process 2 , I received the array : 1494772352 32767 

the sum of array is: 14

dreamcrash dreamcrash · Accepted Answer · 2021-01-18T18:04:21

It seems that you misunderstood how MPI works; Your code is hardcoded to work (correctly) with only two processes. However, you are trying to run the code with 3 processes, with the wrong assumption that the during the MPI_Scatter call the root rank will only send the data to the other processes. If you look at the following image (taken from source):

you notice that the root rank (i.e., rank = 0) also receives part of the data.

The problem is one of the processes receives two elements but the other one receives random numbers.

MPI_Scatter(number, 2, MPI_INT, &number1, 2, MPI_INT, 0, MPI_COMM_WORLD);

So you have hardcoded an input as follows number{1,3,5,9} (with only 4 elements); and what is happen during the MPI_Scatter call is that process 0 will get the first and second elements from array number (i.e., {1, 3}), whereas process 1 gets the other two elements (i.e., {5, 9}), and the process 2 will get some random values, consequently:

I'm process 2 , I received the array : 1494772352 32767

You get

the sum of array is: 14

because the array sub_sums will have the sums performed by process 0, which is zero since you excluded, and process 1 which is 3 + 9. Hence, 0 + 14 = 14.

To fix this you need to remove if(world_rank!=0) from:

  if(world_rank!=0){
    printf("I'm process %d , I received the array : ",world_rank);
    for(int i=0 ; i<2 ; i++){
        printf("%d ",number1[i]);
        sub_sum = sub_sum + number1[i];
    }
     printf("\n");
 }

and run your code with only 2 processes.

For the last step instead of the MPI_Gather you can used MPI_Reduce to perform the sum in parallel and collect the value directly on the root rank. Consequently, you would not need to performed the sum manually on the root rank.

A running example:

int main(int argc,char *argv[]){
    MPI_Init(NULL,NULL); // Initialize the MPI environment
    int world_rank; 
    int world_size;
    MPI_Comm_rank(MPI_COMM_WORLD,&world_rank);
    MPI_Comm_size(MPI_COMM_WORLD,&world_size);
    
    int number1[2];           
    int number[4];
    if(world_rank == 0){
      number[0]=1;
      number[1]=3;
      number[2]=5;
      number[3]=9;               
    }

    //All processes
    MPI_Scatter(number, 2, MPI_INT, &number1, 2, MPI_INT, 0, MPI_COMM_WORLD);
    printf("I'm process %d , I received the array : ",world_rank);
       
    int sub_sum = 0;
    for(int i=0 ; i<2 ; i++){
        printf("%d ",number1[i]);
        sub_sum = sub_sum + number1[i];
    }
    printf("\n");        
    int sum = 0;
    MPI_Reduce(&sub_sum, &sum, 1, MPI_INT, MPI_SUM,0,MPI_COMM_WORLD);            
    if(world_rank == 0)
      printf("\nthe sum of array is: %d\n",sum);
            

    MPI_Finalize();
    return 0;
 }

Input : {1,3,5,9} running with 2 processes

Output

I'm process 0 , I received the array : 1 3 
I'm process 1 , I received the array : 5 9 

the sum of array is: 18

If you really want to only have the process 1 and 2 receive the data and performed the sum, I would suggest to look into the routines MPI_Send and MPI_Recv.

Sum of the element of an array in parallel?

1 Answers