Memory Leak Observed in Erlang Application

Question

Let me put my question as simple as below. Mine is a network router software built in erlang, but at a particular scenario I am observing very high memory growth as shown by VM.

I have one process which receives binary packet from some other process from socket.

This process, parses the binary packet and passes the binary packet to a gen_server (handle_cast is called)

The gen_server again stores some information in ETS table and send the packet to the peer server.

When the peer server responds back the entry from the ETS is deleted and the gen_server responds back to the first process

Also if the first process (which sent packet to gen_server) gets timedout after 5 seconds waiting for response from gen_server , it also deletes the ETS entry in the gen_server and exits.

Now I am observing high memory growth when lots of events gets timed out (due to unavailability of peer server) and from what i have researched its the "**binary**" and "**processes_used**" given by erlang:memory command thats using most of the memory.

but the same is not true when events are processed successfully.

Answer 1

The memory lost can be basically only in three places:

1. The state of your gen_server

   • look at you state, find out if there is some big or growing stuff there

2. Your processes mailboxes

   • see to it that there is some way to always drain unmatched messages (for gen_server handle_info callback) in normal receives a Any ->clause.

   • if the mailbox only fills up temporarily its probably because of the receiving process being too slow for the rate of messages produced. This is usually a problem for asynchronous communication. If its only temporary bursts that don't break anything this could be intended.

     • In this case you can either optimize the receiving process

     • or fix your protocol to use fewer messages

     • if you have multiple functions that receive some messages, make sure all receiving parts are being called regularly. Dont forget the Any -> clauses.

   • Be aware that while you are processing in a gen_servers callback no messages will be received, so if you need more time in a callback that would be necessary asyncronous messages might pile up (e.g. random message arrival + fixed processing time builds a unbounded growing queue, for details see Queueing theory

3. In your ETS table

   • maybe the information in the ETS is not be completely removed? Forgot to remove something in certain cases?

Answer 2

trigger GC by hand and see what happens with memory.

[garbage_collect(Pid) || Pid <- processes()] 

Answer 3

Most likely you're leaving processes running that has references to binaries. If a process dies, all memory related to that process will be cleaned up (including any binaries that only belonged to that process).

If you still have leaking binaries, it means you have some long running process (server, singleton etc) that keeps references to binaries, either in its process state or by non-tail recursive functions. Make sure you clean up your state once process communication times out or they die. Also, check that you don't leave references to binaries on the heap by using non-tail recursive calls.