Julia-Lang UndefVarError: loop_inner is not defined

Question

im trying to use Multi-Threadding (or better Multi-Processor) in Julia-Lang. Just using Base.Threads just made my Application Slower so i wanted to try Distributed.

module Parallel
# ... includes ..
using Distributed
@Distributed.everywhere include("...jl") 
#... Includes Needed in Proccesses

export loop_inner
@Distributed.everywhere function loop_inner(parentValue, value, i, depth)
...
end

function langfordSequence(parentValue, depth)
    ...

    if depth < 4 && depth > 1
        futures = [@spawnat :any loop_inner(parentValue, value, i, depth)   for i = 0:possibilites]       
        return sum(fetch.(futures))
       % 2) PROBLEMATIC LINE ^^ 
    else
        return sum([loop_inner(parentValue, value, i, depth) for i = 0:possibilites]) 
       % 1) PROBLEMATIC LINE ^^ 
    end
end
end

But i run into $julia -L ...jl -L Parallel.jl main.jl -p 8

ERROR: LoadError: UndefVarError: loop_inner not defined (At Line [see Code Above at % 1) PROBLEMATIC LINE ^^ )

I hope someone can tell me what im doing wrong. If im chaning if depth < 4 && depth > 1 to if depth < 4 im getting UndefVarError: Parallel not defined (At Line [see Code Above at % 2) PROBLEMATIC LINE ^^ )

Ty in advance

Przemyslaw Szufel Przemyslaw Szufel · Accepted Answer · 2020-07-07T18:46:04

It does not work because each worker process needs to separately load your module.

Your module should look like this:

module MyParallel

include("somefile.jl") 
export loop_inner
function loop_inner(parentValue, value, i, depth)
end

end

Now you use it in the following way (this assumes that the module is inside your private package):

using Distributed
using Pkg
Pkg.activate(".") # or wherever is the package
using MyParallel # enforces module compilation which should not occur in parallel 
addprocs(4) # should always happen before any `@everywhere` macro or use `-p` command line option instead. 
@everywhere using Pkg, Distributed
@everywhere Pkg.activate(".")
@everywhere using MyParallel

Now you are ready to work with the MyParallel module.

EDIT

To make it more clear. The important goal of a module is to provide a common namespace for a set of functions, types and global (module) variables. If you put any distributed code inside a module you obviously break this design because each Julia's worker is a totally separate system process, having it's own memory and namespace. Hence the good design, in my opinion, is to keep all the do-the-work code inside the module and distributed computation management outside the module. Perhaps in some scenarios one might want the distributed orchestration code to be in a second module - but normally it is just more convenient to keep such code outside of the module.

Julia-Lang UndefVarError: loop_inner is not defined

1 Answers

EDIT