We have J2EE based web application. Intermittently we are facing high CPU usage (80-90%) on our production environment. We are unable to replicate it on our QA environment.
Production Environment: Windows 2012 Server (64 bit), JDK 1.8 (64 bit)
For troubleshooting we have taken thread dump. It shows total 215 threads.
111 threads are in WAITING status
34 threads are in RUNNABLE status
67 threads are in TIMED_WAITING status
3 threads are in BLOCKED status
How can we find which threads are causing high CPU usage?
Thread Dump
2016-03-01 11:07:52 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.5-b02 mixed mode):
"Thread-739969" - Thread t@807668 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <229fc226> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"Thread-739968" - Thread t@807667 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <5a578345> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"Thread-739963" - Thread t@807662 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <20851ef7> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"Thread-739962" - Thread t@807661 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <35061b89> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"Thread-739960" - Thread t@807659 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <3491a931> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"Thread-739959" - Thread t@807658 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <6395c27d> (a org.apache.solr.util.ConcurrentLRUCache$CleanupThread) at java.lang.Object.wait(Object.java:502) at org.apache.solr.util.ConcurrentLRUCache$CleanupThread.run(ConcurrentLRUCache.java:604)
Locked ownable synchronizers: - None
"httpShardExecutor-3-thread-57429-processing-{core=actionscomments, replica=core_node1, ShardRequest.shards=s1.asite.com:8983//solr//actionscomments|s1r1.asite.com:8983//solr//actionscomments, node_name=10.120.112.31:8983_solr, shard=shard1, collection=actionscomments, ShardRequest.urlList=[http:////s1r1.asite.com:8983//solr//actionscomments, http:////s1.asite.com:8983//solr//actionscomments]}" - Thread t@807648 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.SocketInputBuffer.isDataAvailable(SocketInputBuffer.java:95) at org.apache.http.impl.AbstractHttpClientConnection.isStale(AbstractHttpClientConnection.java:310) at org.apache.http.impl.conn.ManagedClientConnectionImpl.isStale(ManagedClientConnectionImpl.java:158) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:433) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328) at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:246) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:221) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers: - locked (a java.util.concurrent.ThreadPoolExecutor$Worker)
