I'm trying to run a simple helloworld test of my OpenMPI installation. I have set up a two node cluster on Amazon AWS and I'm using SUSE SLES11 SP3, OpenMPI 1.4.4 (a bit old but there is no new binaries available for my Linux distribution). I am down to the last step and I am having some problems setting up the btl flags properly.
He is what I CAN do:
I can scp between nodes, both directions, so passwordless SSH is up and running correctly
If I run iptables -L, it indicates that no firewall is up, so I think communication between nodes should work.
I can compile my helloworld.c program using mpicc and I have confirmed that the script runs correctly on another working cluster, so the local paths are set up correctly I think and the script definitely works.
If I execute mpirun from my master node, and using only the master node, helloworld executes correctly:
ip-xxx-xxx-xxx-133: # mpirun -n 1 -host master --mca btl sm,openib,self ./helloworldmpi ip-xxx-xxx-xxx-133: hello world from process 0 of 1If I execute mpirun from my master node, using only the worker node, helloworld executes correctly:
ip-xxx-xxx-xxx-133: # mpirun -n 1 -host node001 --mca btl sm,openib,self./helloworldmpi ip-xxx-xxx-xxx-210: hello world from process 0 of 1
Now, my problem is that if I try to run helloworld on both nodes, I get an error:
ip-xxx-xxx-xxx-133: # mpirun -n 2 -host master,node001 --mca btl openib,self ./helloworldmpi
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[5228,1],0]) is on host: ip-xxx-xxx-xxx-133
Process 2 ([[5228,1],1]) is on host: node001
BTLs attempted: self
Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
PML add procs failed
--> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
*** The MPI_Init() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[ip-xxx-xxx-xxx-133:7037] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 7037 on
node ip-xxx-xxx-xxx-133 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
*** The MPI_Init() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[ip-xxx-xxx-xxx-210:5838] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
[ip-xxx-xxx-xxx-133:07032] 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc
[ip-xxx-xxx-xxx-133:07032] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[ip-xxx-xxx-xxx-133:07032] 1 more process has sent help message help-mpi-runtime / mpi_init:startup:internal-failure
Finally, if I leave out the -mca btl sm,openib,self flag then nothing works at all. I will admit that my understanding of these flags is almost zero. There is very little info on the web about their use however. I have looked in my data.conf file and I'm not sure that all of the devices listed are actually present, but the -mca flag appears to be taking care of most of the problem, since I can at least execute on each node in the cluster individually. Any pointers on what I may be doing wrong, or where I may look would be greatly appreciated.
