Help and guidance is highly appreciated.
The issue reported is of High CPU utilization (>95%) by JBOSS.
- JBOSS EAP 6.3 alpha is configured as a service through NSSM (https://nssm.cc) on windows server
- I have captured the thread dump and identified High CPU Utilizing threads (5 in all) which are as under : nativeId(in base 10 Decimal): 6428, 2768, 5904, 6100 and 6840. These threads are consuming around 15% of CPU each.
- No deadlocks observed through thread dump
- I have not written the deployed code nor do i have the source code for these threads so i'm a bit stuck as part of the delivery/implementation team.
High CPU threads identified through psexplorer.exe
For these 5 threads, 2 types/variations of stack trace are there as under:
http-/0.0.0.0:8080-187 priority:6 - threadId:0x0000000022d29800 - nativeId:0x191c - nativeId (decimal):6428 - state:RUNNABLE stackTrace: java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007b394a958> (a java.util.concurrent.ExecutionException) at java.lang.Throwable.(Throwable.java:310) at java.lang.Exception.(Exception.java:102) at java.util.concurrent.ExecutionException.(ExecutionException.java:92) at sun.nio.ch.CompletedFuture.get(CompletedFuture.java:78) at org.apache.tomcat.util.net.NioChannel.readBytes(NioChannel.java:597) at org.apache.coyote.http11.InternalNioInputBuffer.blockingRead(InternalNioInputBuffer.java:504) at org.apache.coyote.http11.InternalNioInputBuffer.fill0(InternalNioInputBuffer.java:449) at org.apache.coyote.http11.InternalNioInputBuffer.access$100(InternalNioInputBuffer.java:50) at org.apache.coyote.http11.InternalNioInputBuffer$InputBufferImpl.doRead(InternalNioInputBuffer.java:528) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:116) at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:407) at org.apache.coyote.Request.doRead(Request.java:438) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:357) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:422) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:389) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:943) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:922) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:280) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:223) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:123) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:347) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:115) at com.newgen.wfdesktop.servlet.filehandler.processRequest(filehandler.java:70) at com.newgen.wfdesktop.servlet.filehandler.doPost(filehandler.java:122) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.ibps.security.xss.SECRequestValidator.doFilter(SECRequestValidator.java:231) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.wfdesktop.filter.WDSessionCheckFilter.doFilter(WDSessionCheckFilter.java:378) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.wfdesktop.filter.WDClickJackFilter.doFilter(WDClickJackFilter.java:179) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911) at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers:
- <0x000000069bcbde88> (a java.util.concurrent.ThreadPoolExecutor$Worker)
http-/0.0.0.0:8080-146 priority:6 - threadId:0x0000000024579000 - nativeId:0xad0 - nativeId (decimal):2768 - state:RUNNABLE stackTrace: java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007b4558488> (a java.nio.channels.ClosedChannelException) at java.lang.Throwable.(Throwable.java:250) at java.lang.Exception.(Exception.java:54) at java.io.IOException.(IOException.java:47) at java.nio.channels.ClosedChannelException.(ClosedChannelException.java:52) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:233) at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:283) at org.apache.tomcat.util.net.NioChannel.readBytes(NioChannel.java:597) at org.apache.coyote.http11.InternalNioInputBuffer.blockingRead(InternalNioInputBuffer.java:504) at org.apache.coyote.http11.InternalNioInputBuffer.fill0(InternalNioInputBuffer.java:449) at org.apache.coyote.http11.InternalNioInputBuffer.access$100(InternalNioInputBuffer.java:50) at org.apache.coyote.http11.InternalNioInputBuffer$InputBufferImpl.doRead(InternalNioInputBuffer.java:528) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:116) at org.apache.coyote.http11.InternalNioInputBuffer.doRead(InternalNioInputBuffer.java:407) at org.apache.coyote.Request.doRead(Request.java:438) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:357) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:422) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:389) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:943) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:922) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:280) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:223) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:123) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:70) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:347) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:115) at com.newgen.wfdesktop.servlet.filehandler.processRequest(filehandler.java:70) at com.newgen.wfdesktop.servlet.filehandler.doPost(filehandler.java:122) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.ibps.security.xss.SECRequestValidator.doFilter(SECRequestValidator.java:231) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.wfdesktop.filter.WDSessionCheckFilter.doFilter(WDSessionCheckFilter.java:378) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at com.newgen.wfdesktop.filter.WDClickJackFilter.doFilter(WDClickJackFilter.java:179) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911) at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers:
- <0x000000069a402d48> (a java.util.concurrent.ThreadPoolExecutor$Worker)
