Erlang garbage collection

3
votes

I need your help in investigation of issue with Erlang memory consumption. How typical, isn't it?

We have two different deployment schemes.

  • In first scheme we running many identical nodes on small virtual machines (in Amazon AWS), one node per machine. Each machine has 4Gb of RAM.
  • In another deployment scheme we running this nodes on big baremetal machines (with 64 Gb of RAM), with many nodes per machine. In this deployment nodes are isolated in docker containers (with memory limit set to 4 Gb).

I've noticed, that heap of processes in dockerized nodes are hogging up to 3 times much more RAM, than heaps in non-dockerized nodes with identical load. I suspect that garbage collection in non-dockerized nodes is more aggressive. Unfortunately, I don't have any garbage collection statistics, but I would like to obtain it ASAP.

To give more information, I should say that we are using HiPE R17.1 on Ubuntu 14.04 with stock kernel. In both schemes we are running 8 schedulers per node, and using default fullsweep_after flag.

My blind suggestion is that Erlang default garbage collection relies (somehow) on /proc/meminfo (which is not actual in dockerized environment). I am not C-guy and not familiar with emulator internals, so could someone point me to places in Erlang sources that are responsible for garbage collection and some emulator options which I can use to tweak this behavior?

2
garbage-collectionerlangdocker
The Erlang community is pretty thin on this site (the ones here who know the internals know the internals, though). You might want to put this on the mailing list also; a lot of folks over there know a lot about the internals. - zxq9
Also, please tell how you are getting that memory consumption figure. If it is from an OS utility like top or /proc/meminfo the reason you see this is probably very different than if you are getting the memory figure from erlang:memory/0. - zxq9
I am collecting statistics directly on erlang node using some different ways. Particulary, I use erlang:memory/0 and see that erlang:memory(processes) is abnormal. Also I have statistics on heaps of processes under some certain supervisors (I obtain it via erlang:process_info(Pid, total_heal_size) on each process under supervisor. - Viacheslav Kovalev

2 Answers

3
votes

Unfortunately VMs often try to be smarter with memory management than necessary and that not always plays nicely with the Erlang memory management model. Erlang tends to allocate and release a large number of small chunks of memory, which is very different to normal applications, which usually allocate and release a small number of big chunks of memory.

One of those technologies is Transparent Huge Pages (THP), which some OSes enable by default and which causes Erlang nodes running in such VMs to grow (until they crash).

https://access.redhat.com/solutions/46111

https://www.digitalocean.com/company/blog/transparent-huge-pages-and-alternative-memory-allocators/

https://docs.mongodb.org/manual/tutorial/transparent-huge-pages/

So, ensuring THP is switched off is first thing you can check.

The other is trying to tweak the memory options used when starting the Erlang VM itself, for example see this post:

Erlang: discrepancy of memory usage figures

Resulting options that worked for us:

-MBas aobf -MBlmbcs 512 -MEas aobf -MElmbcs 512

Some more theory about memory allocators:

http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf

And more detailed description of memory allocator flags:

http://erlang.org/doc/man/erts_alloc.html

1
votes

First thing to know, is that garbage collection i Erlang is process based. Each process is GC in their own time, and independently from each other. So garbage collection in your system is only dependent on data in your processes, not operating system itself.

That said, there could be some differencess between memory consumption from Eralang point of view, and System point of view. That why comparing erlang:memory to what your system is saying is always a good idea (it could show you some binary leaks, or other memory problems).

If you would like to understand little more about Erlang internals I would recommend those two talks:

https://www.youtube.com/watch?v=QbzH0L_0pxI

https://www.youtube.com/watch?v=YuPaX11vZyI

And from little better debugging of your memory management I could reccomend starting with http://ferd.github.io/recon/