Julia parallel programming - Making existing function available to all workers

20
votes

I am faced with the following problem:

I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing, it returns a function as an output argument, let's call it f. When I enquire the type of this f, Julia returns:

(generic function with 1 method)

(I am not sure of this last piece of information is useful to anyone reading this)

Now in a second step, I need to apply function f on a very large array of values. This is a step that I would like to parallelise. Having had started Julia with multiple processes, e.g.

julia -p 4

ideally, I would use:

pmap(f, my_values)

or perhaps:

aux = @parallel (hcat) for ii=1:100000000
        f(my_values[ii])
      end

Unfortunately, this doesn't work. Julia complains that the workers are not aware of the function f, i.e. I get a messsage:

ERROR: function f not defined on process 2

How can I make function f available to all workers? Obviously a "dirty" solution would be to run the time-consuming function TrainModel on all workers, like this perhaps: 

@everywhere f = TrainModel( ... )

but this would be a waste of cpu when all I want is that just the result f is available to all workers.

Though I searched for posts with similar problems, so far I could not find an answer...

Thanks in advance! best,

N.

2
for-loopparallel-processingjulia
is this the issue here? docs.julialang.org/en/release-0.3/manual/parallel-computing/… - Claies
I think this is exactly the issue. I do understand how to make a function available to all workers when the function is declared: for instance, one can use the macro @everywhere when declaring the function. Also the same for variables. But how about doing the same for functions that are not located in a module or declared, but are instead created during runtime? - user1438310
This is a good question and I don't think directing OP to the Julia docs helps. In fact, it is very clear from the question that OP has read and understood the relevant section in the docs and they do not address the issue of functions generated and passed between different scopes at run-time. Also, there is an additional consideration: since the function is generated at run-time, the compiler may not be able to determine the output type (ie it won't be type-stable) and this may cause additional problems (possibly only performance related). - Colin T Bowers
Thanks for all the comments, thanks Colin for noting that the issue I am raising is not exactly addressed by that part of the Julia docs. The problem I am raising is very general in statistical/machine learning applications: having trained a model (function TrainModel) that returns a prediction function (function f), how can we parallelise the prediction task on multiple workers when we would like to predict on a large dataset? Maybe I should rework my original post to this problem situation... - user1438310

2 Answers

3
votes

The approach to return the function seems elegant but unfortunately, unlike JavaScript, Julia does not resolve all the variables when creating the functions. Technically, your training function could produce the source code of the function with literal values for all the trained parameters. Then pass it to each of the worker processes, which can parse it in their environment to a callable function.

I suggest to return a data structure that contains all the information to produce the trained function: weights of an ANN, support vectors, decision rules ... Define a the "trained" function on the worker processes, such that it will utilized the trained parameters. You might want to have the ability of saving the results of the training to disk anyway, so that you can easily re-produce your computations.

0
votes

There is a Unix-only solution based on the PTools.jl package (https://github.com/amitmurthy/PTools.jl).

It relies on parallelism via forking instead of the Julia in-built mechanism. Forked processes are spawned with the same workspace as the main process, so all functions and variables are directly available to the workers.

This is a similar to the Fork clusters in R parallel package, so it can be used as the mclapply function.

The function of interest is pfork(n::Integer, f::Function, args...) and one noticeable difference with mclapply in R is that the function f must take as first argument the index of the worker.

An example:

Pkg.add("PTools")
Pkg.checkout("PTools") #to get the last version, else the package does not build at the time of writing

using PTools
f(workid,x) = x[workid] + 1
pfork(3, f, [1,2,3,4,5]) #Only the three first elements of the array will be computed

3-element Array{Any,1}:  
 2  
 3  
 4

I expect that an interface to pfork will be built so that the first argument of the function will not need to be the index of the worker, but for the time being it can be used to solve the problem