tcmalloc's fragmentation

Question

Our software implement a actor model system, and we allocate/deallocate the small object very often.I am very sure the each object be destroyed without memory leak. ( I have used valgrind and tcmalloc tool to check the memory leak in my software. No leak be found.)

When we change to use tcmalloc for replacing the malloc in glibc, we found the memory continually increasing until the process be killed by OOM(Out of Memory). Then we found glibc also have same issue, but increase rate is less than tcmalloc.

I used the malloc_stats() to show the memory information

after first execution(top show 0.96G)'

---

• MALLOC: 960110592 ( 915.6 MB) Heap size
• MALLOC: 15886016 ( 15.2 MB) Bytes in use by application
• MALLOC: 907419648 ( 865.4 MB) Bytes free in page heap
• MALLOC: 0 ( 0.0 MB) Bytes unmapped in page heap
• MALLOC: 27121208 ( 25.9 MB) Bytes free in central cache
• MALLOC: 151040 ( 0.1 MB) Bytes free in transfer cache
• MALLOC: 9532680 ( 9.1 MB) Bytes free in thread caches
• MALLOC: 14275 Spans in use
• MALLOC: 27 Thread heaps in use
• MALLOC: 7602176 ( 7.2 MB) Metadata allocated

---

After 5th same execution(top show 1.2G)

• MALLOC: 1173131264 ( 1118.8 MB) Heap size
• MALLOC: 18001048 ( 17.2 MB) Bytes in use by application
• MALLOC: 1082458112 ( 1032.3 MB) Bytes free in page heap
• MALLOC: 21168128 ( 20.2 MB) Bytes unmapped in page heap
• MALLOC: 37992328 ( 36.2 MB) Bytes free in central cache
• MALLOC: 252928 ( 0.2 MB) Bytes free in transfer cache
• MALLOC: 13258720 ( 12.6 MB) Bytes free in thread caches
• MALLOC: 17651 Spans in use
• MALLOC: 27 Thread heaps in use
• MALLOC: 8126464 ( 7.8 MB) Metadata allocated

---

We can seen from thus data. after 5th same behavior, only 17.2 be used in our software. But the tcmalloc hold 1.1G memory without return to system. Of course, It does not matter tcmalloc hold those memory. But it keep increasing when our program be killed by OOM(actual used memory is less than 1G).

We have doubt that it is related to heap fragmentation. Anybody have some experience could share with us? I think I have same situation as https://bugzilla.redhat.com/show_bug.cgi?id=843478

Thanks very much.

Answer 1

I would suggest using Boehm's conservative GC and use GC_MALLOC and GC_MALLOC_ATOMIC instead of malloc in your application (and GC_FREE instead of free, but you could then even avoid any explicit free-ing, the GC would do them). Or maybe use valgrind (with the system Glibc malloc) to find memory leaks. If using Boehm's GC, don't forget to explicitly clear the allocated memory zones.

It would be preferable to make sure that your many small objects have coarse grained sizes. E.g. allocate objects of 8 or 12 or 16 words, not objects of 8, 9, 10, 11, 12, 13, 14, 15 or 16 words... For instance, you might allocate only zones of size some power of 2 or 3 times a power of 2.

Also, don't forget you can limit the memory space with setrlimit(2), e.g. with the ulimit builtin of the bash running in your terminal. This should ease testing. Also, perhaps using pmap or /proc/$(pidof yourapp)/maps could help you to understand the address space used.

PS. Neither Boehm GC, nore any kind of malloc (including tcmalloc or Glibc malloc) can tackle with memory fragmentation. If you suspect fragmentation you have to move memory zones in the address space (i.e. you may want to code your own precise, copying, generational GC or re-use some existing one).

Answer 2

tcmalloc tries to do some smart things to anticipate your memory use, but it isn't very good about releasing memory back to the system even after you have freed it. in fact, it might be resident in memory and leading to OOM.

do this :

MallocExtension::instance()->ReleaseFreeMemory();

when your application has expectedly freed memory that it expects to not use soon.

look at the section "Releasing memory back to the system" for more information http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html

there are other methods, such as setting the Release rate, but as a start you can sprinkle this in your code and inspect if resident memory goes down as you expect.