Standard method for file distribution prior to MPI execution

Question

I am new to the world of MPI and HPC in general. I have designed a distributed algorithm which I have implemented using MPI. Each MPI process accesses a set of files which is different for each MPI process. On my computer this already works because the files are already there.

The problem comes now that I want to run the program on more than one computer because I need to transfer files to each workstation manually and then run my program. This is okay for one or two computers because I can do it by hand but what happens if I have 1000 computers and I want to distribute files to each computer before running my program.

Is there a generally acceptable/standard way of transferring files from a host node in a cluster to all the destination nodes prior to running the actual MPI program?

UPDATE As an example Matlab has a batch distritbution function argument called FileDependencies which zips relevant file on the host machine and distributes them to the nodes and then unzips them into a file structure on those machines.

Answer 1

Usually with a shared file system, like NFS.

Edit: MPI is generally used for parallel programs that require tight coupling (high performance computing). It sounds like there is no coupling among your tasks, you just want a lot running at once. This type of computing is called high-throughput computing, or embarrassingly parallel computing. The most widely used platform for HTC is Condor, it even has a file transfer mechanism built in (just list the files you want to transfer, and they'll end up on your execute hosts).