Julia: Unexpected Result when Using @parallel for Loop with SharedArray

Question

I am trying to learn parallel for loop in Julia, while I was confused by the result of the example code below:

addprocs(4)
@everywhere begin
N = 10
V = SharedArray{Int64,1}(N)
@sync @parallel for i = 1:N
    V[i] = i
    println(V[i])
end 
end

By using println, I tried to identify which worker went through which step. Surprisingly, what I got from the above code was that each worker kept going through the entire iteration until the last worker (worker 3 in my case) finished the for loop:

From worker 4:  1
From worker 4:  2
From worker 4:  3
From worker 5:  1
From worker 5:  2
From worker 5:  3
From worker 5:  4
From worker 5:  5
From worker 5:  6
From worker 5:  4
From worker 5:  5
From worker 5:  6
From worker 2:  7
From worker 2:  8
From worker 2:  4
From worker 2:  5
From worker 2:  6
From worker 2:  9
From worker 2:  10
From worker 4:  1
From worker 4:  2
From worker 2:  7
From worker 2:  8
From worker 3:  9
From worker 5:  4
From worker 5:  5
From worker 5:  6
From worker 3:  10
From worker 3:  7
From worker 3:  8
From worker 3:  7
From worker 3:  8
From worker 3:  9
From worker 3:  10
From worker 2:  4
From worker 2:  5
From worker 2:  6
From worker 3:  7
From worker 3:  8
From worker 4:  3
From worker 4:  9
From worker 4:  10
From worker 4:  1
From worker 4:  2
From worker 4:  3
From worker 4:  1
From worker 4:  2
From worker 4:  3
From worker 5:  9
From worker 5:  10

This is not what I would expect from parallel for loop, as I thought the work should be distributed among the workers and then combined together, rather than being done separately for each of the worker. I mean, should the concept of parallel for loop be that each worker only goes through 2-3 steps?

Is something wrong with my code or I misunderstand the concept of parallel for loop?

Thanks!

Edit: I just realized that there is something to do with @everywhere. After I eliminated @everywhere, things work as expected.

Your example is not precisely reproducible. May I suggest that you fix a random seed (e.g. srand(2017)) and specify how many processes you initialized, e.g. via addprocs(X) or julia -p X? — Kevin L Keys

carstenbauer carstenbauer · Accepted Answer · 2018-01-08T11:56:46

You told julia (through @everywhere) to run the parallel loop on every worker and on the host process, hence you count from 1 to 10 in a parallel fashion 5 times. (Check that in the output you posted every number from 1 to 10 occurs precisely 5 times)

Slightly more detailed: First we note that the total number of processes is nprocs() == 5 (one "host" and 4 workers, check workers()). @everywhere tells Julia to run the content of the begin end block on every process, hence 5 times in our example. The content of the begin end block is "do a parallel loop, counting from 1 to 10". That is exactly what happened.

When you remove the @everywhere you do just a single parallel loop. Thus you will get exactly what you want:

julia> N = 10
       V = SharedArray{Int64,1}(N)
       @sync @parallel for i = 1:N
           V[i] = i
           println(V[i])
       end
        From worker 5:  9
        From worker 5:  10
        From worker 3:  4
        From worker 3:  5
        From worker 3:  6
        From worker 2:  1
        From worker 2:  2
        From worker 2:  3

Suggested reading: https://docs.julialang.org/en/stable/manual/parallel-computing

Julia: Unexpected Result when Using @parallel for Loop with SharedArray

1 Answers