How to make Julia PkgServer.jl work offline

Question

I work mostly on offline machines and really want to begin to migrate from Python to Julia. Currently the biggest problem I face is how can I setup a package server on my own network that does not have access to the internet. I can copy files to the offline computer/network and want to be able to just cache a good percentage of the Julia Package Ecosystem and copy it to my network, so that I and others can install packages as needed.

I have experimented with PkgSever.jl by using the deployment docker-compose script they have, then just installing a long list of packages so that the PkgServer instance would cache everything. Next took the PkgServer machine offline and attempted to install packages from it. This worked well, however when I restarted the docker container the server was running in, everything fell apart quickly.

It seems that maybe the PkgServer needs to be able to talk to the Storage Server at least once before being able to serve packages. I tried setting: JULIA_PKG_SERVER_STORAGE_SERVERS from: "https://us-east.storage.juliahub.com,https://kr.storage.juliahub.com" to: "" but that failed miserably.

Can someone please point me in the right direction.

TIA

It looks like the PkgServer is actually trying to contact the Registry before it starts. I don't know enough about the registry stuff enough to know if there is a way to hack this to look locally or just ignore this..

pkgserver_1   | ERROR: LoadError: DNSError: kr.storage.juliahub.com, temporary failure (EAI_AGAIN)
pkgserver_1   | Stacktrace:
pkgserver_1   |  [1] getalladdrinfo(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/addrinfo.jl:112
pkgserver_1   |  [2] getalladdrinfo at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/addrinfo.jl:121 [inlined]
pkgserver_1   |  [3] getconnection(::Type{TCPSocket}, ::SubString{String}, ::String; keepalive::Bool, connect_timeout::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},NTuple{4,Symbol},NamedTuple{(:require_ssl_verification, :iofunction, :reached_redirect_limit, :status_exception),Tuple{Bool,Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:630
pkgserver_1   |  [4] #getconnection#29 at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:682 [inlined]
pkgserver_1   |  [5] newconnection(::HTTP.ConnectionPool.Pod, ::Type{T} where T, ::SubString{String}, ::SubString{String}, ::Int64, ::Bool, ::Int64; kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:597
pkgserver_1   |  [6] getconnection(::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}; connection_limit::Int64, pipeline_limit::Int64, idle_timeout::Int64, reuse_limit::Int64, require_ssl_verification::Bool, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:541
pkgserver_1   |  [7] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}; proxy::Nothing, socket_type::Type{T} where T, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionRequest.jl:73
pkgserver_1   |  [8] (::Base.var"#56#58"{Base.var"#56#57#59"{ExponentialBackOff,HTTP.RetryRequest.var"#2#3"{Bool,HTTP.Messages.Request},typeof(HTTP.request)}})(::Type{T} where T, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at ./error.jl:301
pkgserver_1   |  [9] #request#1 at /depot/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
pkgserver_1   |  [10] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :status_exception),Tuple{Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/MessageRequest.jl:51
pkgserver_1   |  [11] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :status_exception),Tuple{Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
pkgserver_1   |  [12] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:status_exception,),Tuple{Bool}}}) at /depot/packages/HTTP/IAI92/src/RedirectRequest.jl:24
pkgserver_1   |  [13] request(::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; headers::Array{Pair{SubString{String},SubString{String}},1}, body::Array{UInt8,1}, query::Nothing, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:status_exception,),Tuple{Bool}}}) at /depot/packages/HTTP/IAI92/src/HTTP.jl:314
pkgserver_1   |  [14] #get#12 at /depot/packages/HTTP/IAI92/src/HTTP.jl:391 [inlined]
pkgserver_1   |  [15] get_registries(::String) at /app/src/resource.jl:21
pkgserver_1   |  [16] update_registries() at /app/src/resource.jl:130
pkgserver_1   |  [17] start(; kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:listen_addr, :storage_root, :storage_servers),Tuple{Sockets.InetAddr{IPv4},String,Array{SubString{String},1}}}}) at /app/src/PkgServer.jl:88
pkgserver_1   |  [18] top-level scope at /app/bin/run_server.jl:43
pkgserver_1   |  [19] include(::Function, ::Module, ::String) at ./Base.jl:380
pkgserver_1   |  [20] include(::Module, ::String) at ./Base.jl:368
pkgserver_1   |  [21] exec_options(::Base.JLOptions) at ./client.jl:296
pkgserver_1   |  [22] _start() at ./client.jl:506
pkgserver_1   | in expression starting at /app/bin/run_server.jl:43

This might be helpful but I'm not sure, yet, how to get it started LocalRegistry.jl

Answer 1

Here is a solution that seems to work, based on LocalPackageServer.

Preliminary steps

Install all required packages. You can either put them in your default environment (e.g. @v1.5) or in a dedicated project.

• LocalRegistry
• LocalPackageServer

In order to use LocalPackageServer, we'll need to set up a local registry, even though we won't really use it (but it can still be handy if you also have to serve local packages).

Something like this should create an empty local registry as local-registry.gitt in the current folder:

# Create an empty (bare) git repository to host the registry
run(`git init --bare local-registry.git`)

# Populate the repository with a new, empty local registry
using LocalRegistry
Base.Filesystem.mktempdir() do dir
    create_registry(joinpath(dir, "local-registry"),
                    abspath("local-registry.git"),
                    description = "(unused) local registry",
                    push=true)
end

Step 1a - run the local package server (online)

A script like the following should run a local package server listening on http://localhost:8000.

#################
# run_server.jl #
#################

using LocalPackageServer

config = LocalPackageServer.Config(Dict(
    # Server parameters
    "host"           => "127.0.0.1",
    "port"           => 8000,
    "pkg_server"     => "https://pkg.julialang.org",

    # This is where persistent data will be stored
    # (I use the current directory here; adjust to your constraints)
    "local_registry" => abspath("local-registry.git"), # In accordance with the preliminary step above
    "cache_dir"      => abspath("cache"),
    "git_clones_dir" => abspath("data"),
))

# The tricky part: arrange for the server to never update its registries
# when it is offline
if get(ENV, "LOCAL_PKG_SERVER_OFFLINE", "0") == "1"
    @info "Running offline => no registry updates"
    config.min_time_between_registry_updates = typemax(Int)
end

# Start the server
LocalPackageServer.start(config)

Use this script to run the server online first:

shell$ julia --project run_server.jl

Step 1b - cache some packages (online)

Configure a Julia process to use your local server, and install the packages you want to cache:

# Take care to specify the "http://" protocol
# (otherwise https might be used by the client, and the server won't answer)
shell$ JULIA_PKG_SERVER=http://localhost:8000 julia
julia> using Pkg
julia> pkg"add Example"
[...]

At this point, the server should be caching things.

Step 2 - run the package server (offline)

When you're offline, simply restart the server, ensuring it won't try to update the registries:

shell$ LOCAL_PKG_SERVER_OFFLINE=1 julia --project run_server.jl

As before, set JULIA_PKG_SERVER as needed for Julia clients to use the local server; they should now be able to install packages, provided that the project environments resolve to the exact same dependency versions that were cached.

(You might want to resolve and instantiate your project environments online, and then transfer the relevant manifests to offline systems: this might help guarantee the consistency between the package server cache and what clients ask for.)