Memory leak in tomcat 7 org.apache.juli.ClassLoaderLogManager$RootLogger

Question

I'm trying to diagnose a production memory leak in my web application. The application is a GWT app running on Tomcat 7 and mySQL, and OpenJDK 1.6.0_18 on Debian.

I booted tomcat and let the app leak for several days -- no reloads of the app -- and then took a heap dump and opened it in Eclipse MAT plugin. I'm trying to make sense of what I'm seeing. It appears that a massive amount of memory is being held from a tomcat loggin class (see screenshot below).

I do basic logging in my GWT servlet using the java logging framework. Just basic calls like

Logger.getAnonymousLogger().log(Level.INFO, "User" + userId + " did something interesting");
Logger.getAnonymousLogger().log(Level.SEVERE, exception.getMessage(), exception);

The only similar issue I could find on the web to this was this blog post.

Can anyone explain what might be going on here? Why would tomcat logging classes be holding onto so much memory?

Screenshot of MAT analysis

To confound me even further, it seems to be holding onto a massive array of weak references -- but I can't seem to figure out in MAT how to find out what those weak references are pointing at, and I also would expect weak references to get collected when the heap limits are reached, but tomcat throws out of memory exceptions instead.

Giant Array of Weak References

I don't have a crisp answer, unfortunately, but one potential explanation for the OOMs without reaping WeakReferences could be that the referents of the WeakReferences also have strong references in the heap. If you can prove that the WeakReferences survived a GC that resulted in an OOM, that's a likely answer. Hope that helps! — sigpwned
Here is the heap dump: dl.dropbox.com/u/9617020/dump.hprof.gz — Aaron Davidson

Andrew Alcock Andrew Alcock · Accepted Answer · 2013-04-09T04:28:32

Interesting problem :)

Looks like 10's of millions of classes have registered loggers with your RootLogger. Now, unless you have millions classes in your project, either:

A class regularly calls Logger.getAnonymousLogger(), or
Loggers or classes are being generated dynamically:
- These could be coming from:
  - java.lang.reflect.Proxy
  - A scripted language (like Groovy) in which you can create classes dynamically
  - A bytecode manipulation library
  - Direct calls to new Logger(name)
- Each created class registers a logger - LogManager.addLogger()
- After a while the Logger (and any class that was created) is GCed

Either way, each additional logger adds another row to the ArrayList containing a weak reference. The weak reference starts by pointing to the logger, but the loggers are being GC'ed leaving only the empty weak reference. It is the vast number of weak references that is causing the OOM rather than the objects they (used to) point to.

A fix?

If you are using anonymous logging, don't. This is for use by Applets.
Take a look to see if you have LogManager.addLogger in a dynamic class. If so, force the logger name to be a constant
Check if you have anything like new Logger(variable). If you do, then you need to change the variable to a constant.

Memory leak in tomcat 7 org.apache.juli.ClassLoaderLogManager$RootLogger

1 Answers