Julia 1.1 with JLD HDF5 package and memory release in Windows

Question

I'm using Julia 1.1 with JLD and HDF5 to save a file onto the disk, where I met a couple of question about the memory usage.

Issue 1:

First, I defined a 4 GB matrix A.

A = zeros(ComplexF64,(243,243,4000));

When I type the command and look at windows task manager:

A=nothing

It took several minutes for Julia to release those memory back to me. Most of the time, (In Task manager) Julia just doesn't release the memory usage at all, even though the command returned results saying that A occupied 0 bytes instantly.

varinfo()

    name                    size summary
–––––––––––––––– ––––––––––– –––––––
A                    0 bytes Nothing
Base                         Module
Core                         Module
InteractiveUtils 162.930 KiB Module
Main                         Module
ans                  0 bytes Nothing

Issue 2:

Further, when I tried to use JLD and HDF5 to save file onto the disk. This time, the task manager told me that, when using the save("test.jld", "A", A) command, an extra 4GB memory was used.

using JLD,HDF5
A = zeros(ComplexF64,(243,243,4000));
save("test.jld", "A", A)

Further, after I typed

A=nothing

Julia won't release the 8 GB memory back to me.

Finding 3:

An interesting thing I found was that, if I retype the command

A = zeros(ComplexF64,(243,243,4000));

The task manager would told me the cashed memory was released, and the total memory usage was again only 4GB.

Question 1:

What's going on with memory management in Julia? Was it just a mistake by Windows, or some command in Julia? How to check the Julia memory usage instantly?

Question 2:

How to tell the Julia to instantly release the memory usage?

Question 3:

Is there a way to tell JLD package not use those extra 4GB meomory?

(Better, could someone tell me how to create A directly on the disk without even creating it in the memory? I knew there's memory mapped I/O in JLD package. I have tried it, but it seemed to require me to create matrix A in the memory and save A onto the disk first, before I could recall the memory mapped A again. )

This is a long question, so thanks ahead!

This has nothing to do with dates, so I removed your last tag. — phipsgabler
Sometimes the garbage collector does not need to run immedietaly. What happens when you call GC.gc()? — Przemyslaw Szufel
@PrzemyslawSzufel Thanks you! It seemed to be working(Memory reduced instantly) after I called GC.gc() . Do you know what's going on? and what did GC.gc() do? — ShoutOutAndCalculate
I added an answer (the comment field was to short for good explanation) — Przemyslaw Szufel

Przemyslaw Szufel Przemyslaw Szufel · Accepted Answer · 2019-03-23T21:32:15

Julia uses garbage collector to de-alocate the memory. Usually a garbage collector does not run after every line of code but only when needed.

Try to force garbage collection by running the command:

GC.gc()

This releases memory space for unreferenced Julia objects. In this way you can check whether the memory actually has been released.

Side note: JLD used to be somewhat not-always-working (I do not know the current status). Hence you first consideration for non-cross-platform object persistence always should be the serialize function from the in-built Serialization package - check the documentation at https://docs.julialang.org/en/v1/stdlib/Serialization/index.html#Serialization.serialize

Julia 1.1 with JLD HDF5 package and memory release in Windows

1 Answers