Using MPI_Send inside a pthread in C

Question

i am trying to create a ring of mpi processes where each MPI process will launch a pthread and the threads will perform the ring, i used pthread so i can use the MPI processes to do another task. It seems that i can't use MPI_send or MPI_Recv inside a pthread, i have no compilation error but i do have a run time error.

i compile using this command mpicc -lpthread threaded_ring.c

this is the runtime error

a.out:28372 terminated with signal 11 at PC=2aaaaaae312d SP=2aaab0771860.  Backtrace:
/usr/lib64/libpsm_infinipath.so.1(psmi_mpool_get+0xd)[0x2aaaaaae312d]

a.out:28366 terminated with signal 11 at PC=333c00c110 SP=2aaab02d9698.  Backtrace:
/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x333c00c110]
/usr/lib64/libpsm_infinipath.so.1(psmi_amsh_short_request+0x180)[0x2aaaaaad31b0]
/usr/lib64/libpsm_infinipath.so.1(+0xd9f6)[0x2aaaaaad49f6]
/usr/lib64/libpsm_infinipath.so.1(psm_mq_send+0x41)[0x2aaaaaaf5d51]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(psm_send_pkt+0xb1)[0x2aaaaae0af21]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(psm_istartmsgv+0x130)[0x2aaaaae0a010]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(MPIDI_CH3_iStartMsgv+0x6)[0x2aaaaaddf1e6]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(MPIDI_CH3_EagerContigSend+0x89)[0x2aaaaada6e39]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(MPID_Send+0x116)[0x2aaaaade3136]
/usr/local/mpi/mvapich2/intel12/1.8.1/lib/libmpich.so.3(MPI_Send+0xf8)[0x2aaaaae2a408]
./a.out[0x4022ba]
/lib64/libpthread.so.0[0x333c0077f1]
/lib64/libc.so.6(clone+0x6d)[0x333bce570d]

a.out:28373 terminated with signal 11 at PC=333bf9d428 SP=2aaab0771838.  Backtrace:

a.out:28370 terminated with signal 11 at PC=2aaaaaae312d SP=2aaab0771860.  Backtrace:

here is my code

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <time.h>

void *ring_func(void *p)
{
  int token=1;
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  if (world_rank==0){
   MPI_Send(&token, 1, MPI_INT, (world_rank + 1) % world_size, 0,
         MPI_COMM_WORLD);
   }

  if (world_rank != 0) {
    MPI_Recv(&token, 1, MPI_INT, world_rank - 1, 0, MPI_COMM_WORLD,
             MPI_STATUS_IGNORE);
    printf("Process %d received token %d from process %d\n", world_rank, token,
           world_rank - 1);
}
pthread_exit(NULL);
}

    int main(int argc, char** argv) {
  // Initialize the MPI threaded environment

  int provided;
  MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE , &provided);
 if (provided < MPI_THREAD_MULTIPLE)
 {
    printf("Error: the MPI library doesn't provide the required thread level\n");
       MPI_Abort(MPI_COMM_WORLD, 0);
       }
  pthread_t ring ;
  pthread_create (&ring, NULL, ring_func, NULL) ;

  MPI_Barrier(MPI_COMM_WORLD);
  MPI_Finalize();
}

possible duplicate of MPI+pthreads. Program stuck on MPI_Ssend and MPI_Recv — Hristo Iliev
i already added MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); but i still have the same error !! — Omar Al Oliyan
Check that provided is equal to MPI_THREAD_MULTIPLE after the call to MPI_Init_thread. Also, you never join the thread and there is no guarantee whatsoever that the main thread won't succeed in going through the barrier and calling MPI_Finalize() before the second thread has even started. — Hristo Iliev
Hi lliev thank you for helping me, i have actully added the provided and made sure that it is equal to MPI_THREAD_MULTIPLE but it does not seem to be solving the problem, i think you are right on the second part that some times the main thread will just call MPI_Finlize but i have no idea on how to solve that. can you think of something to stop the main thread from terminating before my pthread finish ? — Omar Al Oliyan

Omar Al Oliyan Omar Al Oliyan · Accepted Answer · 2014-10-15T06:51:37

Thanks to Hristo lliev i was able to solve the problem, the problem was that the main thread was finsishing before my pthread but when i added pthread_join the main thread waited for the pthread to join before calling MPI_Finalize(). Here is the new code

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <time.h>

void *ring_func(void *p)
{
  int token;
  // Receive from the lower process and send to the higher process. Take care
  // of the special case when you are the first process to prevent deadlock.

  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  if (world_rank != 0) {
    MPI_Recv(&token, 1, MPI_INT, world_rank - 1, 0, MPI_COMM_WORLD,
             MPI_STATUS_IGNORE);
    printf("Process %d received token %d from process %d\n", world_rank, token,
           world_rank - 1);
  } else {
// Set the token's value if you are process
 token = -1;
  }
MPI_Send(&token, 1, MPI_INT, (world_rank + 1) % world_size, 0,
           MPI_COMM_WORLD);

 if (world_rank == 0) {
//    sleep(20);
    MPI_Recv(&token, 1, MPI_INT, world_size - 1, 0, MPI_COMM_WORLD,
             MPI_STATUS_IGNORE);
    printf("Process %d received token %d from process %d\n", world_rank, token,
           world_size - 1);
  }

pthread_exit(NULL);
}

int main(int argc, char** argv) {
  // Initialize the MPI threaded environment

  int provided;
  MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE , &provided);
 if (provided != MPI_THREAD_MULTIPLE)
 {
    printf("Error: the MPI library doesn't provide the required thread level\n");
       MPI_Abort(MPI_COMM_WORLD, 0);
       }
  pthread_t ring ;
  pthread_create (&ring, NULL, ring_func, NULL) ;
  pthread_join(ring,NULL);
  MPI_Barrier(MPI_COMM_WORLD);
  MPI_Finalize();
}

Using MPI_Send inside a pthread in C

1 Answers