Using Julia package manager offline

5
votes

We recently decided to make Julia Language available on our cluster systems. The cluster system is not able to connect to the internet. Is there any way to download all Julia packages and make them available for our different users to install and use them offline?

Another option that we have is a system that can connect to the internet temporarily, but it is always connected to the main cluster system. Is there any way to use this system as a mirror for the Julia packages or not?

We want to use "Julia 1.0.1".

our cluster operation system is: "CentOS 5.5

notes: I have seen the question asked before here, but it is for Julia 0.6 and a single package that will be copied by hand. I want that user uses the Pkg.add <pkgName> command but instead of the internet, the package manager gets the packages from our offline system.

Thank you for your help and time.

1
julia
xref: discourse.julialang.org/t/using-julia-package-manager-offline/… – carstenbauer
in Julia it is easy to reattach all packages via the JULIA_DEPOT_PATH. If you have Internet on one node and all nodes are identical you could use it as "installation & compilation node" and then replicate the contents to other servers in your cluster. – Przemyslaw Szufel
I would try This: 1. Attach machine A to the internet. 2. export JULIA_DEPOT_PATH="/my/package/folder". 3. julia. 4. Install whatever you need. 5. In the Julia package manager run (v1.0) pkg> precompile to precompile everything. 6. Copy "/my/package/folder" to machine B (without internet). 7. Run on machine B: export JULIA_DEPOT_PATH="/my/package/folder" before running julia. If machine A and B have the same hardware configuration this should work. – Przemyslaw Szufel
As far as I know Julia uses JULIA_DEPOT_PATH to store all compiled module files. As long as they are not compiled (that no new modules are added to Julia installation) it should be possible to share them in read-only mode across many users (I have not tested in though). – Przemyslaw Szufel
Now regarding the size: the main reason for huge size of JULIA_DEPOT_PATH is the Python installation. Bu here it is easy. Simply do not use the inbuilt Python!. This can be achieved by setting ENV["PYTHON"] Julia variable before running Pkg.add("PyCall"). This will allow you to use an external Python Anaconda and I am pretty sure there are already many recipes how to share that across large user bases. – Przemyslaw Szufel

1 Answers

1
votes

Caution:
Side effects are not known!
May please be tested properly before put into production!

a) Collect the required packages along with their dependent packages in compiled form, put them in folder, stdlib (for example: /opt/julia/julia-1.1.0/shared/julia/stdlib/v1.1/)

b) add stdlib path to environment variables, JULIA_DEPOT_PATH and JULIA_LOAD_PATH