How to add processors (addprocs) for Pyjulia 0.5.2 and return the value of Julia function to Python?

Question

Does anyone know how to add processors for Pyjulia when calling Julia function in Python? I need to use Python to call a Julia function using multiple cores and return values to Python.

The Python code, testJulia_addproc.py:

import julia
j = julia.Julia()
print("Core number = {}".format(j.nprocs()))
j.include("test_addproc.jl")
a = j.eval("test_addproc()")
print a

The Julia code test_addproc.jl:

function test_addproc()
    println("line 1")
    np = nprocs()
    println("line 2")
    if np < 12
        println("line 3")
        addprocs(12-np)
        println("line 4")
    end
    println("line 5")
    return(1+1)
end

==
$ python testJulia_addproc.py
I got this error.

WARNING: redefining constant JULIA_HOME
Core number = 1
line 1
line 2
line 3
ERROR (unhandled task failure): could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): no such file or directory (ENOENT)
 in _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.DevNullStream, ::Base.PipeEndpoint, ::Base.TTY) at ./process.jl:321
 in #424 at ./process.jl:478 [inlined]
 in setup_stdio(::Base.##424#425{Cmd,Ptr{Void},Base.Process}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:466
 in #spawn#423(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:477
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#420(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:359
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in open(::Base.CmdRedirect, ::String, ::Base.DevNullStream) at ./process.jl:539
 in launch(::Base.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./managers.jl:318
 in (::Base.##666#670{Base.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:68
could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): no such file or directory (ENOENT)Traceback (most recent call last):
  File "testJulia_addproc.py", line 5, in <module>
    j.eval("test_addproc()")
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 436, in eval
    ans = self._call(src)
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 399, in _call
    self.check_exception(src)
  File "//anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/core.py", line 419, in check_exception
    .format(exception_type, src))
julia.core.JuliaError: Exception 'UVError' occurred while calling julia code:
test_addproc()

Based on this post, I found the number of processors in pyjulia needs to match the number in Julia (@Isaiah: nprocs() in pure Julia must match j.nprocs()).

I tried to add procs in pyjulia.

import julia
j = julia.Julia()
j.addprocs(2)

Got errors:

ERROR (unhandled task failure): could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker 8WpA7O0EArLzXh25`; dir="/Applications/Julia-0.5.app/Contents/Resources/julia/bin"): no such file or directory (ENOENT)
 in _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.DevNullStream, ::Base.PipeEndpoint, ::Base.TTY) at ./process.jl:321
 in #424 at ./process.jl:478 [inlined]
 in setup_stdio(::Base.##424#425{Cmd,Ptr{Void},Base.Process}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:466
 in #spawn#423(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:477
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#420(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./process.jl:359
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}, ::Bool, ::Bool) at ./<missing>:0
 in open(::Base.CmdRedirect, ::String, ::Base.DevNullStream) at ./process.jl:539
 in launch(::Base.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./managers.jl:318
 in (::Base.##666#670{Base.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:68
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-bfa063d91dec> in <module>()
----> 1 j.addprocs(2)

RuntimeError: Julia exception: could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker 8WpA7O0EArLzXh25`; dir="/Applications/Julia-0.5.app/Contents/Resources/julia/bin"): no such file or directory (ENOENT)

Based on this post, I found there is a way around to add processors in pyjulia, but didn't return values of the Julia function.

# One way to add procs in pyjulia
import ctypes
JLPATH=b"/Applications/Julia-0.5.app/Contents/Resources/julia"
jl = ctypes.PyDLL(JLPATH+b"/lib/libjulia.dylib", ctypes.RTLD_GLOBAL)
jl.jl_init(JLPATH+b"/bin/")
jl.jl_eval_string(" include(test_addproc.jl) ")
a = jl.jl_eval_string(" test_addproc()) ")
print a
# 0
# It should be 2

My MacPro version is OS X Yosemite 10.10.5
My Python version is Python 2.7.12 :: Anaconda 4.2.0 (x86_64).
My Julia version is

Julia Version 0.5.2
Commit f4c6c9d4bb (2017-05-06 16:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, ivybridge)

Any suggestion or thought are welcome. Thank you.

Chih-Hsu Jack Lin Chih-Hsu Jack Lin · Accepted Answer · 2017-05-23T21:37:14

The first error message says,

ERROR (unhandled task failure): could not spawn setenv(`/anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/julia/../fake-julia/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker pjVTa5WBhFbDB7vs`; dir="/Users/chlin/Dropbox/Projects/StackOverflow"): no such file or directory (ENOENT)

I found there is no fake-julia directory in the python package folder.
So I copied fake-julia directory to /anaconda/lib/python2.7/site-packages/julia-0.1.1-py2.7.egg/ from PyJulia.
Now I am able to add processor by j.addprocs(1) and get result 2 from testJulia_addproc.py without error.

So far that's a way around it.
I am also posting issues on PyJulia GitHub and hoping to get it fixed in future version.

How to add processors (addprocs) for Pyjulia 0.5.2 and return the value of Julia function to Python?

1 Answers