Module loading with multiple processes

2
votes

I'm puzzled by Julia behavior around loading modules when worker processes are used.

I need to use a rather heavy PyPlot module which takes a considerable amount of time to load. This program:

using PyPlot
pygui(true)
println("Loaded")

takes around 11 seconds to load on my laptop:

% time julia test.jl 
INFO: Loading help data...
Loaded
julia test.jl  11,10s user 0,18s system 99% cpu 11,323 total

Note the INFO: Loading help data... line. It seems to be emitted by the PyPlot module as it does not appear if I omit using PyPlot line.

However, when I run this program:

using PyPlot
pygui(true)
@everywhere println("Loaded")

I get these results:

% time julia -p 4 test.jl 
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
Loaded
    From worker 2:  Loaded
    From worker 5:  Loaded
    From worker 3:  Loaded
    From worker 4:  Loaded
julia -p 4 test.jl  88,94s user 1,19s system 266% cpu 33,865 total

Not only it runs for whopping 33 seconds (three times longer!), but it also seems to load PyPlot module on every worker!

But I was sure that in order for module to be available on each worker, it has to be @everywhered! Indeed, this simple program crashes:

module Example
    export x
    x = 10
end

using Example

@everywhere println("x: $x")

Invocation:

% julia -p 4 test2.jl
x: 10
exception on 2: exception on exception on exception on 4: 5: 3: ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so

So why is PyPlot module loaded on all workers even if I didn't request it?

What's even more interesting, there is a workaround:

using PyPlot
pygui(true)
addprocs(4)
@everywhere println("Loaded")

When I run this program with julia test.jl, I get 15 seconds:

% time julia test.jl     
INFO: Loading help data...
Loaded
    From worker 2:  Loaded
    From worker 4:  Loaded
    From worker 5:  Loaded
    From worker 3:  Loaded
julia test.jl  21,98s user 0,46s system 143% cpu 15,678 total

which is exactly what I'd have expected for the original version ran with julia -p 4 test.jl. But I don't like this workaround because it forces my program to use addprocs().

Is there a way to restrict module loading to the master process when Julia is started with -p X argument?

1
parallel-processingmultiprocessingjulia

1 Answers

2
votes

Let's test using DummyModule.jl:

module DummyModule

export MyType, f

type MyType
    a::Int
end

f(x) = x^2+1

println("loaded")

end

With this, we can see that there are actually 3 possibilities. All of these experiments are run from a fresh julia session invoked as julia -p 2.

Use DummyModule on all processes

julia> @everywhere using DummyModule
loaded
        From worker 3:  loaded
        From worker 2:  loaded

julia> @everywhere println(f(4))
17
        From worker 2:  17
        From worker 3:  17

julia> rr = RemoteRef(2)
RemoteRef(2,1,24)

julia> put!(rr, MyType(7))
RemoteRef(2,1,24)

julia> fetch(rr)
MyType(7)

Use DummyModule only on the driver process

julia> include("DummyModule.jl")
loaded

julia> using DummyModule

julia> @everywhere println(f(4))
exception on 2: 17
exception on 3: ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6
ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6

julia> rr = RemoteRef(2)
RemoteRef(2,1,14)

julia> put!(rr, MyType(7))
WARNING: Module DummyModule not defined on process 2
fatal error on 2: ERROR: DummyModule not defined
 in deserialize at serialize.jl:376
 in handle_deserialize at serialize.jl:351
 in deserialize at serialize.jl:505
 in handle_deserialize at serialize.jl:351
 in deserialize at serialize.jl:334
 in anonymous at serialize.jl:354
 in ntuple at tuple.jl:30
 in deserialize_tuple at serialize.jl:354
 in handle_deserialize at serialize.jl:346
 in anonymous at task.jl:824
Worker 2 terminated.ERROR: ProcessExitedException()
 in wait at ./task.jl:284
 in wait at ./task.jl:194
 in wait_full at ./multi.jl:574
 in remotecall_fetch at multi.jl:675
 in remotecall_fetch at multi.jl:680
 in call_on_owner at multi.jl:722
 in put! at multi.jl:743

(Note the order in which different messages are printed is not deterministic; in exception on 2: 17, the exception ... part is due to an error being issued from process 2, and the 17 comes from printing f(4) on process 1.)

Using only on process 1, but passing data among workers

julia> using DummyModule
loaded
        From worker 3:  loaded
        From worker 2:  loaded

julia> @everywhere println(f(4))
exception on 2: 17
exception on 3: ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6
ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6

julia> rr = RemoteRef(2)
RemoteRef(2,1,19)

julia> put!(rr, MyType(7))
RemoteRef(2,1,19)

julia> using DummyModule

julia> fetch(rr)
MyType(7)

The distinction in the last case is that the serializer actually knows how to handle MyType on all processes, making it safe to pass data among workers.