OpenMPI: have each process write to stdout

2
votes

Child processes started by mpirun redirect their output to the mpirun process, so all output ends up on one node.

Instead, I'd like each of the processes spawned by MPI to write to STDOUT on their own nodes, or to a file or named pipe.

I read the faq and tried out some things:

mpirun -host host1,host2 my_script >&1

Just redirects stdout from all hosts to stdout on the invoking node (like default). Doing

mpirun -host host1,host2 my_script

Where my_script redirects output to >&1 just captures output from processes on the invoking node.

Is there a way I can get each node to write to their local filesystems (for example) without redirecting to the invoking node's mpirun process?

Thanks.

1
openmpi

1 Answers

3
votes

Open MPI has the --output-file option, it is pretty close but not exactly what you are asking for.

I do not think there is a native way to achieve what you expect.

That being said, that can be easily achieved via a wrapper

For example, via the command line

mpirun --host host1,host2 sh -c 'my_script > /tmp/log.$OMPI_COMM_WORLD_RANK'

Each MPI task will redirect its stdout to /tmp/log.<id>.

An other method is to use the fork_agent

mpirun --host host1,host2 --mca orte_fork_agent /.../wrapper my_script

basically, instead of exec'ing my_script, Open MPI will exec /.../wrapper my_script and with a bit of creativity, the wrapper you have to write can do whatever you need. Within this wrapper, you will likely want to check the following environment variables

  • OMPI_COMM_WORLD_SIZE
  • OMPI_COMM_WORLD_RANK
  • OMPI_COMM_WORLD_LOCAL_SIZE
  • OMPI_COMM_WORLD_LOCAL_RANK