Parallel processing using DataFrames in Julia

Question

I am referring to the example in the documentation for processing Parallel loops and trying to adapt it for my use case. In each independent iteration in my case, I get a DataFrame as a result which I need to finally combine across all iterations using vcat(). This is a simplified version of my attempt so far:

using DataFrames, Distributed

function test()
    if length(workers()) < length(Sys.cpu_info())
        addprocs(length(Sys.cpu_info()); exeflags="--project=" * Base.active_project())
    end

    nheads = @distributed (vcat) for i = 1:20
        DataFrame(a=[Int(rand(Bool))])
    end
end

But on running test(), I get the error:

ERROR: On worker 2: UndefVarError: DataFrame not defined

What do I need to do to correct this?

Another approach to load the processes with your environment is to launch julia in this way: julia --project="path_to_environment" -e "using Distributed;addprocs(length(Sys.cpu_info()); exeflags=\"--project=\" * Base.active_project());include(\"path_to_file_to_run.jl\");" to run the file with the code in the answer, without the line having addprocs() — WebDev

Art Art · Accepted Answer · 2019-05-30T08:44:45

Your using DataFrames ... statement on the first line only applies to the main "thread". So your worker threads didn't import the required libraries.

To fix this, you should add the keyword @everywhere on the first line. That would ask all the processes to import those libraries.

Edit

Just noticed that you did addprocs in the function. Then my suggestion wouldn't work. Here's a working version:

using Distributed

addprocs(length(Sys.cpu_info()))

@everywhere using DataFrames

function test()
    nheads = @distributed (vcat) for i = 1:20
        DataFrame(a=[Int(rand(Bool))])
    end
end

test()

Parallel processing using DataFrames in Julia

1 Answers