Using addprocs() and pmap() inside a function in Julia

Question

In Julia, I want to use addprocs and pmap inside a function that is defined inside a module. Here's a silly example:

module test

using Distributions

export g, f

function g(a, b)
  a + rand(Normal(0, b))
end

function f(A, b)

  close = false
  if length(procs()) == 1    #  If there are already extra workers,
    addprocs()               #  use them, otherwise, create your own.
    close = true
  end

  W  = pmap(x -> g(x, b), A)

  if close == true
    rmprocs(workers())       #  Remove the workers you created.
  end

  return W

end

end

test.f(randn(5), 1)

This returns a very long error

WARNING: Module test not defined on process 4
WARNING: Module test not defined on process 3
fatal error on fatal error on WARNING: Module test not defined on process 2
43: : WARNING: Module test not defined on process 5
fatal error on fatal error on 5: 2: ERROR: UndefVarError: test not defined
 in deserialize at serialize.jl:504
 in handle_deserialize at serialize.jl:477
 in deserialize at serialize.jl:696

...

 in message_handler_loop at multi.jl:878
 in process_tcp_streams at multi.jl:867
 in anonymous at task.jl:63
Worker 3 terminated.
Worker 2 terminated.ERROR (unhandled task failure): EOFError: read end of file
WARNING: rmprocs: process 1 not removed

Worker 5 terminated.ERROR (unhandled task failure): EOFError: read end of file

4-element Array{Any,1}:Worker 4 terminated.ERROR (unhandled task failure): EOFError: read end of file


 ERROR (unhandled task failure): EOFError: read end of file
ProcessExitedException()
 ProcessExitedException()
 ProcessExitedException()
 ProcessExitedException()

What I'm trying to do is write a package that contains functions that perform operations that can be optionally parallelized at the user's discretion. So a function like f might take an argument par::Bool that does something like I've shown above if the user calls f with par = true and loops otherwise. So from within the definition of f (and within the definition of the module test), I want to create workers and broadcast the Distributions package and the function g to them.

Michael Ohlrogge Michael Ohlrogge · Accepted Answer · 2016-07-06T17:13:44

What's wrong with using @everywhere in your function? The following, for example, works fine on my computer.

function f(A, b)

  close = false
  if length(procs()) == 1    #  If there are already extra workers,
    addprocs()               #  use them, otherwise, create your own.
    @everywhere begin
      using Distributions
      function g(a, b)
        a + rand(Normal(0, b))
      end
    end
    close = true
  end

  W  = pmap(x -> g(x, b), A)

  if close == true
    rmprocs(workers())       #  Remove the workers you created.
  end

  return W

end

f(randn(5), 1)

Note: when I first ran this, I needed to recompile the Distributions package since it had been updated since I had last used it. When I first tried the above script right after the recompiling, it failed. But, I then quit Julia and reopened it and it worked fine. Perhaps that was what is causing your error?

Using addprocs() and pmap() inside a function in Julia

1 Answers