I am trying to send data from the master process to worker processes. I am able to do so just fine with relatively small pieces of data. But, as soon as they get above a certain size, I encounter a serialize error.
Is there a way to resolve this, or would I just need to break my objects down into smaller pieces and then reassemble them on the workers? If so, is there a good way to determine ahead of time the max size that I can send (which I suppose may be dependent upon system variables)? Below is code showing a transfer that works and one that fails. It's possible the sizes might need to be tinkered with to reproduce on other systems.
function sendto(p::Int; args...)
for (nm, val) in args
@spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
end
X1 = rand(10^5, 10^3);
X2 = rand(10^6, 10^3);
sendto(2, X1 = X1) ## works fine
sendto(2, X2 = X2)
ERROR: write: invalid argument (EINVAL)
in yieldto at /Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib
in wait at /Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib
in stream_wait at /Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib
in uv_write at stream.jl:962
in buffer_or_write at stream.jl:982
in write at stream.jl:1011
in serialize_array_data at serialize.jl:164
in serialize at serialize.jl:181
in serialize at serialize.jl:127
in serialize at serialize.jl:310
in serialize_any at serialize.jl:422
in send_msg_ at multi.jl:222
in remotecall at multi.jl:726
in sendto at none:3
Note: I have plenty of system memory, even for two copies of the larger object, so the problem isn't in that.
