WebSphere 7 - Can excessive Garbage Collection lead to out of memory?

3
votes

Issue: Getting out of native memory exception and was wondering if excessive garbage collection can lead to this? Also any advice on GC policy or tuning would be helpful. I'm not sure if what I have warrants a change yet.

Good Reference StackOverflow Question: Which GC Policy to Use

Specs:

  • Server Environment: Websphere Version 7
  • GC Policy : Default ( optthruput )
  • Java 1.5
  • Heap: 8 GB
  • Running in a VM
  • Analysis Tool: App Dynamics

Preliminary Analysis:

  1. I assumed memory leak however the garbage collection looks ok as it reclaims memory
  2. Native memory exception makes me think that memory outside of the VM is being exhausted, however i am not sure how.

Attached Screenshot:

1.1 - 4 hour period of heap utilization . Each little green garbage can represents a major garbage collection point. 1.2 GC time spend in the above graph. 1.3 What the heap utilization looked like during a out of memory exception.

enter image description here

Exception:

EJB threw an unexpected (non-declared) ejb Exception data: java.lang.OutOfMemoryError: native memory exhausted at garbagecollection.mycode.test at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1658) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1598) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:149) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:935) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:503) at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:875) at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83) at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

1
javamemory-leaksgarbage-collectionwebsphere
Please post the stacktrace of the out of memory error – K Erlandsson
Done. Not much to it. Native out of memory exception. – haju
Is that all you get? You don't see a complete stack trace? – K Erlandsson
Copy and paste mistake. Updated – haju
Garbage collection is a symptom, not a cause, of memory leaks. – user207421

1 Answers

2
votes

The error message you get indicate a problem with native memory, that is, memory outside the heap. The garbage collector is not responsible for off-the-heap memory, why you can not affect this error with garbage collector settings. Excessive garbage collecting should not lead to native memory issues (unless there is a bug in the GC, of course).

What is causing this particular error is hard to say without more information. I assume you have at least 12 GB of available RAM on the machine? Otherwise you are simply running a heap that is too big for the available memory.

Some examples of things in Java that can exhaust native memory:

  • Leaks in native (JNI) code

  • Direct allocation of many ByteBuffers

Or maybe there is no leak, this behavior is expected for your application, and you only need to allow larger processes or buy some more memory.

First step to troublehsoot would be to check the process size against possible process size limits. You could simply run into such a limit without there being an actual leak.

If that is not the problem, track the process size over time, especially in relation to available memory to see if you have a "leaky" behavior.

If you actually think you have a native memory leak, identify which libraries that use JNI you are using (e.g. JDBC drivers) and try to replace them with Java-only versions. Also, check for known memory leak bugs in such libraries and your JVM version and WebSphere version.

If that fails you will have to resort to native memory leak troubleshooting tools. That is another question.