Measuring memory usage of a process on Linux

Question

I am trying to measure the memory usage of a process (a java program) on linux and have two questions related to that:

1. I tried using the script ps_mem.py(sums values from /proc/$PID/smaps) and the peak of total memory usage was about 135MB (private and shared memory). The amount of shared memory is less than 1MB. Trying to use Valgrind with the massif tool valgrind --tool=massif --trace-children=yes --stacks=yes java myProgram yields about 10MB at the peak of memory usage.
   Now as I understand it, heap is where the variables of my program are stored, does it mean that the difference between the two methods is the space taken by the code itself (including the jvm)?

2. Does the same program use different amount of memory on different machines if they have different amount of RAM or/and use different processors (ARM or x86)?

Answer 1

1. Depends.
   • Many of the shared memory mappings in smaps are directly backed by libraries/binaries on disk. While the footprint of these does matter, it's less important as the system can drop these pages at any time and reload them from disk when needed again.
   • Anything that is dirty or private belongs exclusively to the current process (well, process tree if your program forks without execs). This is more important because the system must save them to swap if it needs to push these pages out of memory.
   • What massif is measuring is probably correlated with the latter. However, the memory taken by the JVM itself (without your program) is in both.
2. Yes. Java or a library it uses might adjust its memory model depending on the size of available RAM. On a different architecture, you are using completely different binaries, which may be larger or smaller or arranged differently or using different strategies for JIT and memory management.

Answer 2

There's a similar question other than this and answering the same here to let people know about how linux proc stat vm info currently is not accurate.
Valgrind can show detailed information but it slows down the target application significantly, and most of the time it changes the behavior of the app.

I assume what everyone wants to know WRT "memory usage" is the following...
In linux, the amount of physical memory a single process might use can be roughly divided into following categories.

• M.a anonymous mapped memory
  • .p private
    • .d dirty == malloc/mmapped heap and stack allocated and written memory
    • .c clean == malloc/mmapped heap and stack memory once allocated, written, then freed, but not reclaimed yet
  • .s shared
    • .d dirty == there should be none
    • .c clean == there should be none
• M.n named mapped memory
  • .p private
    • .d dirty == file mmapped written memory private
    • .c clean == mapped program/library text private mapped
  • .s shared
    • .d dirty == file mmapped written memory shared
    • .c clean == mapped library text shared mapped

I would prefer to get the numbers as follows to get the real numbers in least overhead.
You have to sum these up in order to divide what ps shows as RSS and get more accurate numbers not to confuse.
/proc/(pid)/status tries to show these numbers, but they are failing.
So instead of trying to label [anon], [stack], correctly to each mapping, my wish is that linux kernel people will mainline the proc entry code to sum and show these M.a.p.d, M.a.p.c, M.n.p.d, .... numbers.
Embedded linux people will get really happy IMHO.

M.a.p.d:

 awk '/^[0-9a-f]/{if ($6=="") {anon=1}else{anon=0}} /Private_Dirty/{if(anon) {asum+=$2}else{nasum+=$2}} END{printf "sum=%d\n",asum}' /proc/<pid>/smaps

M.a.p.c:

 awk '/^[0-9a-f]/{if ($6=="") {anon=1}else{anon=0}} /Private_Clean/{if(anon) {asum+=$2}else{nasum+=$2}} END{printf "sum=%d\n",asum}' /proc/<pid>/smaps

M.n.p.d:... and so on

Answer 3

For #1, Shared memory is memory (potentially) used by more than one process. This is basically if you run the same binary file in multiple processes or different processes are using a shared library. The heap is where allocated memory is stored (when you use new in Java). Since Java has its VM, it is allocating a lot of memory on the process level that you don't see in your java code. I think that yes, the majority of that 135 MB is from the JVM code/data itself. However, there is also the memory taken up by the stack (when you make a function call and have local variables) as well.

For #2, different amount of RAM would not affect how much "memory" is used when we let memory equal RAM + swap space. However, different processors (especially if we're talking about 32-bit vs. 64-bit) may use different amount of memory. Also, the way a process is compiled may change the amount of memory used because you can instruct a compiler to optimize for memory footprint over speed, as well as disabling some or all optimization altogether.

Answer 4

You might want to take a look at JConsole. Things can be tricky depending on the purpose of your measurement. If you want to know the memory usage of your Java program then tools which measure the memory usage of a process will be inaccurate because they will show memory used by the JVM as well as your program.

As for the massif tool you should know that parts of the JVM will be stored on the stack, and the java code itself might be on the heap (since it's a variable of the JVM), I don't know enough about the JVM to say.