limit size JLD2 Julia

Question

Is there a size limit to load data using JLD2 in Julia?

The same code:

using JLD2
using FileIO
exp=load("myfile.jld2");

seems to work fine until a certain size of data. For some experiments I ran, the file is 5GB and I obtain the following :

Error encountered while loading "/pathtofile/myfile.jld2". Fatal error:

ERROR: EOFError: read end of file

Stacktrace:

[1] handle_error(::EOFError, ::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/error_handling.jl:80

[2] handle_exceptions(::Array{Any,1}, ::String) at /path/.julia/packages/FileIO/Y0Lre/src/error_handling.jl:75

[3] #load#27(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:193

[4] load(::File{DataFormat{:JLD2}}) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:172

[5] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:118

[6] load(::String) at /path/.julia/packages/FileIO/Y0Lre/src/loadsave.jl:118

[7] top-level scope at none:0

I tried using jldopen which I have no experience in using at all but I also get similar error messages.

What could I do ?

And, btw, what is the difference between load and jldopen("my file.jld2","r")?

I am using Julia Version 1.0.2 on Mac OSHighSierra 10.13.6.

Note that there are good (and sometimes more stable) alternatives to JLD2. For short term serialization have a look at the standard library docs.julialang.org/en/v1.0/stdlib/Serialization and for long term serialization github.com/MikeInnes/BSON.jl might be more appropriate. — Przemyslaw Szufel
Glad to help. Note that on Stackoverflow you click the "up" arrow when you find answers of other people useful rather than writing it. — Przemyslaw Szufel

carstenbauer carstenbauer · Accepted Answer · 2018-12-14T15:03:27

I regularily read >5GB files without problems. So I assume that, for some reason, your file wasn't written correctly. This is also indicated by the error message.

To answer your other question, the differene between load("myfile.jld2") and jldopen("myfile.jld2", "r") is that the former loads all the contents of the file into a dictionary whereas the latter opens the file and returns a JLDFile object which you can use to get fine grained access to your data file. For example it allows you to write multiple datafields to the same file (Note that multiple save commands overwrite the same file).

Demonstration:

julia> using FileIO, JLD2

julia> jldopen("myfile.jld2", "w") do f
           write(f, "x", rand(10))
           write(f, "y", "test")
       end

julia> load("myfile.jld2")
Dict{String,Any} with 2 entries:
  "x" => [0.918336, 0.608631, 0.757459, 0.935133, 0.548579, 0.909, 0.913573, 0.0278975, 0.…

  "y" => "test"

julia> load("myfile.jld2", "y")
"test"

julia> f = jldopen("myfile.jld2", "r")
JLDFile C:\Users\carsten\myfile.jld2 (read-only)
 ├─� x
 └─� y

julia> typeof(f)
JLD2.JLDFile{JLD2.MmapIO}

julia> f["x"]
10-element Array{Float64,1}:
 0.9183355611466055
 0.6086314948624771
 0.757458522055442
 0.9351333595616453
 0.5485794420648191
 0.9089998574850378
 0.9135729509843764
 0.027897482037234633
 0.5827560900500541
 0.9831034815173016

julia> close(f)

limit size JLD2 Julia

1 Answers