Julia parallel computing in IPython Jupyter

8
votes

I'm preparing a small presentation in Ipython where I want to show how easy it is to do parallel operation in Julia.

It's basically a Monte Carlo Pi calculation described here

The problem is that I can't make it work in parallel inside an IPython (Jupyter) Notebook, it only uses one.

I started Julia as: julia -p 4

If I define the functions inside the REPL and run it there it works ok.

@everywhere function compute_pi(N::Int)
    """
    Compute pi with a Monte Carlo simulation of N darts thrown in [-1,1]^2
    Returns estimate of pi
    """
    n_landed_in_circle = 0  
    for i = 1:N
        x = rand() * 2 - 1  # uniformly distributed number on x-axis
        y = rand() * 2 - 1  # uniformly distributed number on y-axis

        r2 = x*x + y*y  # radius squared, in radial coordinates
        if r2 < 1.0
            n_landed_in_circle += 1
        end
    end
    return n_landed_in_circle / N * 4.0    
end

 

function parallel_pi_computation(N::Int; ncores::Int=4)
    """
    Compute pi in parallel, over ncores cores, with a Monte Carlo simulation throwing N total darts
    """
    # compute sum of pi's estimated among all cores in parallel
    sum_of_pis = @parallel (+) for i=1:ncores
        compute_pi(int(N/ncores))
    end

    return sum_of_pis / ncores  # average value
end

 

julia> @time parallel_pi_computation(int(1e9))
elapsed time: 2.702617652 seconds (93400 bytes allocated)
3.1416044160000003

But when I do:

 using IJulia
 notebook()

And try to do the same thing inside the Notebook it only uses 1 core:

In [5]:  @time parallel_pi_computation(int(10e8))
elapsed time: 10.277870808 seconds (219188 bytes allocated)

Out[5]:  3.141679988

So, why isnt Jupyter using all the cores? What can I do to make it work?

Thanks.

2
parallel-processingipython-notebookjuliajupyter
Have you tried modifying the corresponding kernel.json file and add the -p switch there? – cel
What happens if addprocs(4) is issued first within the notebook? – rickhg12hs
@rickhg12hs, I think this should work and if so this is a much nicer solution than my ugly kernel file hack. – cel
@rickhg12hs Thank you. It worked perfectly. – Esteban
I'll convert my comment to an answer to make it easier for others to find it. – rickhg12hs

2 Answers

12
votes

Using addprocs(4) as the first command in your notebook should provide four workers for doing parallel operations from within your notebook.

3
votes

One way to solve this is to create a kernel that always uses 4 cores. For that some manual work is required. I assume that you are on a unix machine.

In the folder ~/.ipython/kernels/julia-0.x, you will find following kernel.json file:

{
  "display_name": "Julia 0.3.9",
  "argv": [
    "/usr/local/Cellar/julia/0.3.9_1/bin/julia",
    "-i",
    "-F",
    "/Users/ch/.julia/v0.3/IJulia/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia"
}

If you copy the whole folder cp -r julia-0.x julia-0.x-p4, and modify the newly copied kernel.json file:

{
  "display_name": "Julia 0.3.9 p4",
  "argv": [
    "/usr/local/Cellar/julia/0.3.9_1/bin/julia",
    "-p",
    "4",
    "-i",
    "-F",
    "/Users/ch/.julia/v0.3/IJulia/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia"
}

The paths will probably be different for you. Note that I only gave the kernel a new name and added the command line argument `-p 4.

You should see a new kernel named Julia 0.3.9 p4 which should always use 4 cores.

Also note that this kernel file will not get updated when you update IJulia, so you have to update it manually whenever you update julia or IJulia.