How to use python modules in parfor matlab loops and optimize computational time?

Question

I'm currently working with a project where I use python functions wrapped in a module within a matlab code. The matlab part of the code is a MCMC (monte carlo multi chains) computation, therefore for speeding up the code I'm using a parfor loop on a cluster.

To be more specific the algorithm can be thought as follow:

1. for i=1:number of chain steps(==number of iterations)
2. parfor j=1:number of chains(==number of workers)
3. load the module and use python functions defined there
4. end
5. do something related to the evolution of the chains
6. end

My problem is that the only way I have for matlab to use the python's defined function is to re-load the python module per each parfor iteration, but as the code work this means also per each chain step (the parfor is nested inside) and there I spend some times.

My question is: is there a smarter-faster way to use python libraries within matlab? (something equivalent to MEX-?-) otherwise, is there a way to "store" the python module infos in each worker at the beginning without the need to reload the module every time I step forward in the outer loop as well?

Any hint will be really really appreciated!! Thanks a lot

Giulia

Answer 1

I believe you're looking for pctRunOnAll. From the Matlab documentation:

This is useful if there are setup changes that need to be performed on all the workers and the client.

You should be able to modify your algorithm with

1. pctRunOnAll load the module
2. for i=1:number of chain steps(==number of iterations)
3. parfor j=1:number of chains(==number of workers)
4. use python functions from pre-loaded modeule
5. end
6. do something related to the evolution of the chains
7. end

Answer 2

You might be able to take advantage of parallel.pool.Constant here. This allows you to set up some "constant" data to be used by multiple iterations of a parfor loop, even multiple parfor loops. The linked reference page shows you how to build a parallel.pool.Constant using a function handle - you probably want that to be a function handle that loads your module.