I have a fairly complicated program (Python with SWIG'ed C++ code, long running server) that shows a constantly growing resident memory usage. I've been digging with the usual tools for the leak (valgrind, Pythons gc module, etc.) but to no avail so far. I'm a bit afraid that the actual problem is memory fragmentation within Python and/or libc managed memory.
Anyway, my question is more specific right now: Is there a tool to visualize resident memory usage and ideally show how it develops over time? I think the raw data is in /proc/$PID/smaps but I was hoping there's some tool that shows me a nice graph of the amounts used by mmap'ed files vs. anonymous mmap'ed memory vs. heap over time so that it's easier to see (literally) what's changing. I couldn't find anything though.
Does anybody know of a tool that graphs the memory map (type of memory and amount) of a specific process over space and time in an intuitive way?
Update: I found the "pmap" tool but the version on my system doesn't seem to handle RSS and provides no option to merge sizes for all mapped files resp. mapped "anon" areas. I ended up hacking a little script that parses /proc/$PID/smaps every two minutes while the original program runs and prints lines like this:
12:00:28 {'_TOTAL': 729.20703125, 'file': 53.609375, 'heap': 22.08984375,
'anon': 653.5, 'stack': 0.0078125}
...
15:42:47 {'_TOTAL': 940.16015625, 'file': 53.484375, 'heap': 22.2109375,
'anon': 864.45703125, 'stack': 0.0078125}
No nice graph but after a few hours of runtime I think it's a safe bet now that I have to have a closer look at the 'anon' memory segments :-)
Update: The latest release of valgrind made its memory profiler ("massif") support page level profiling by using --pages-as-heap=yes. Yay! Running my program for a few hours through massif and then feeding the resulting file into Massif Visualizer resulted in a nice graph of memory consumption per page type over time, including stack traces to see where all the mmap calls were coming from. \o/
