Issues with parallel Julia code on windows cluster

2
votes

I'm setting up a small windows cluster for parallel speedup of my Julia code (2x32 cores).

I have following questions:

  1. Is there a way to suppress loading of a module (e.g. "using PyPlot") on the remote machines? In my code, I use my workstation for initialization and data presentation, whereas the cluster is used for heavy calculation without any need for PyPlot, Dataframes etc.

  2. This code loading on the remote machines is even more annoying as the PyPlot (and any other package) fails to populate help database by giving following error message: (actually a lot of errors from every worker)

    exception on : 1: 1ERROR: opening file C:\Users\phlavenk\AppData\Local\Julia-0.3.6\bin/../share/julia\helpdb.jl: No such file or directory

Running on Julia 3.6/ x64 / Windows7, identical directory structure and versions everywhere.

My addprocs command is following:

addprocs(machines, 
  sshflags=`-i c:\\cygwin64\\home\\phlavenk\\.ssh\\id_rsa`, 
  dir=`/cygdrive/c/Users/phlavenk/AppData/Local/Julia-0.3.6/bin`, 
  tunnel=true)

Thank you very much for your advice

2
windowsparallel-processingjulia
AFAIK, there is no automatic package loading. Is there something in your code or juliarc.jl file that loads packages on the remotes? - rickhg12hs

2 Answers

1
votes

"using" causes a module to be loaded on all the processes. To load a module on a specific machine you use "include". e.g.

if myid()==1 include("/home/user/.julia/PyPlot/src/PyPlot.jl") end

You can then do your plotting by PyPlot.plot(...) on your local machine.

0
votes

You could sequence the statements in this order:

using PyPlot
using ModuleNeededOnMasterProcessOnly
addprocs(...)
using ModuleNeededOnAllProcesses