Orphaned Threads In Java Thread Dump

2
votes

I'm seeing a frequent occurrence of "Orphaned" threads in thread dumps a java application. My application itself is appearing to be frozen with very low cpu activity and my thread dumps show many threads waiting for a lock on an object but no thread has a lock on that object.

Has anybody seem a similar situation and have any suggestions on how to diagnose this further?

Here's the thread dump:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_12-b04 mixed mode):

"RMI TCP Connection(17)-10.162.107.103" daemon prio=1 tid=0x0000002c094b6600 nid=0x3856 runnable [0x000000004499a000..0x000000004499aeb0]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked  (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(16)-10.162.107.103" daemon prio=1 tid=0x0000002c08067990 nid=0x3855 runnable [0x0000000044899000..0x0000000044899d30]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked  (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(13)-10.162.107.103" daemon prio=1 tid=0x0000002c080680d0 nid=0x3594 runnable [0x0000000044596000..0x0000000044596b30]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked  (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"JMX server connection timeout 86" daemon prio=1 tid=0x0000002c08a9f000 nid=0x34c8 in Object.wait() [0x0000000044394000..0x0000000044394e30]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a [I)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
    - locked  (a [I)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(11)-10.162.107.103" daemon prio=1 tid=0x0000002c096a6d80 nid=0x34c6 in Object.wait() [0x0000000044191000..0x0000000044192eb0]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
    at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:344)
    - locked  (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
    at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:122)
    at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:169)
    at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1215)
    at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
    at sun.rmi.transport.Transport$1.run(Transport.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(10)-10.162.107.103" daemon prio=1 tid=0x0000002c084513b0 nid=0x3067 runnable [0x0000000043c8d000..0x0000000043c8dd30]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked  (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(8)-10.162.107.103" daemon prio=1 tid=0x0000002c07f7f980 nid=0x2acd runnable [0x0000000044293000..0x0000000044293c30]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
    - locked  (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
    at java.lang.Thread.run(Thread.java:595)

"RMI LeaseChecker" daemon prio=1 tid=0x0000002c080f6990 nid=0x264c waiting on condition [0x0000000043d8e000..0x0000000043d8ecb0]
    at java.lang.Thread.sleep(Native Method)
    at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
    at java.lang.Thread.run(Thread.java:595)

"OracleTimeoutPollingThread" daemon prio=1 tid=0x0000002c08b70320 nid=0x9ae waiting on condition [0x0000000043b8c000..0x0000000043b8cbb0]
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.driver.OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:158)

"ProcessActivities_29 Thread" prio=1 tid=0x0000002c06961d50 nid=0x8e4 waiting for monitor entry [0x0000000043a8b000..0x0000000043a8be30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_28 Thread" prio=1 tid=0x0000002c06960930 nid=0x8e3 waiting for monitor entry [0x000000004398a000..0x000000004398aeb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_27 Thread" prio=1 tid=0x0000002c0695f4e0 nid=0x8e2 waiting for monitor entry [0x0000000043889000..0x0000000043889d30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_26 Thread" prio=1 tid=0x0000002c0695e4a0 nid=0x8e1 waiting for monitor entry [0x0000000043788000..0x0000000043788db0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_25 Thread" prio=1 tid=0x0000002c05e745c0 nid=0x8e0 waiting for monitor entry [0x0000000043687000..0x0000000043687c30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_24 Thread" prio=1 tid=0x0000002c05e73490 nid=0x8df waiting for monitor entry [0x0000000043586000..0x0000000043586cb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_23 Thread" prio=1 tid=0x0000002c05e71db0 nid=0x8de waiting for monitor entry [0x0000000043485000..0x0000000043485b30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_22 Thread" prio=1 tid=0x0000002c05e70990 nid=0x8dd waiting for monitor entry [0x0000000043384000..0x0000000043384bb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_21 Thread" prio=1 tid=0x0000002c05e6f540 nid=0x8dc waiting for monitor entry [0x0000000043283000..0x0000000043283e30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_20 Thread" prio=1 tid=0x0000002c05e6e410 nid=0x8db waiting for monitor entry [0x0000000043182000..0x0000000043182eb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_19 Thread" prio=1 tid=0x0000002c05ee7a90 nid=0x8da waiting for monitor entry [0x0000000043081000..0x0000000043081d30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_18 Thread" prio=1 tid=0x0000002c05ee6960 nid=0x8d9 waiting for monitor entry [0x0000000042f80000..0x0000000042f80db0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_17 Thread" prio=1 tid=0x0000002c05ee5280 nid=0x8d8 waiting for monitor entry [0x0000000042e7f000..0x0000000042e7fc30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_16 Thread" prio=1 tid=0x0000002c05ee3e60 nid=0x8d7 waiting for monitor entry [0x0000000042d7e000..0x0000000042d7ecb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_15 Thread" prio=1 tid=0x0000002c05ee2a10 nid=0x8d6 waiting for monitor entry [0x0000000042c7d000..0x0000000042c7db30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_14 Thread" prio=1 tid=0x0000002c05ee19f0 nid=0x8d5 waiting for monitor entry [0x0000000042b7c000..0x0000000042b7cbb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_13 Thread" prio=1 tid=0x0000002c0a55c820 nid=0x8d4 waiting for monitor entry [0x0000000042a7b000..0x0000000042a7be30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_12 Thread" prio=1 tid=0x0000002c0a55b6f0 nid=0x8d3 waiting for monitor entry [0x000000004297a000..0x000000004297aeb0]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpProcessManager.putInActivityQueue(CwpProcessManager.java:1343)
    at com.conceptwave.process.CwpPEContext.commitMemoryTransaction(CwpPEContext.java:99)
    at com.conceptwave.process.CwpThread.commitPETransaction(CwpThread.java:90)
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:466)

"ProcessActivities_11 Thread" prio=1 tid=0x0000002c0a55a010 nid=0x8d2 waiting for monitor entry [0x0000000042879000..0x0000000042879d30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_10 Thread" prio=1 tid=0x0000002c0a558bf0 nid=0x8d1 waiting for monitor entry [0x0000000042778000..0x0000000042778db0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_09 Thread" prio=1 tid=0x0000002c0a5577a0 nid=0x8d0 waiting for monitor entry [0x0000000042677000..0x0000000042677c30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_08 Thread" prio=1 tid=0x0000002c083ffb80 nid=0x8cf waiting for monitor entry [0x0000000042576000..0x0000000042576cb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_07 Thread" prio=1 tid=0x0000002c083fe470 nid=0x8ce waiting for monitor entry [0x0000000042475000..0x0000000042475b30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_06 Thread" prio=1 tid=0x0000002c083fd340 nid=0x8cd runnable [0x0000000042374000..0x0000000042374bb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:231)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_05 Thread" prio=1 tid=0x0000002c083fbc60 nid=0x8cc waiting for monitor entry [0x0000000042273000..0x0000000042273e30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_04 Thread" prio=1 tid=0x0000002c083fa840 nid=0x8cb waiting for monitor entry [0x0000000042172000..0x0000000042172eb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_03 Thread" prio=1 tid=0x0000002c087bcb90 nid=0x8ca runnable [0x0000000042071000..0x0000000042071d30]
    at com.conceptwave.process.CwpActivityInfoCache.checkForActivity(CwpActivityInfoCache.java:159)
    at com.conceptwave.process.CwpActivityInfoCache.getExistingActivity(CwpActivityInfoCache.java:190)
    at com.conceptwave.process.CwpProcess.getActivityIfExists(CwpProcess.java:2200)
    at com.conceptwave.process.CwpProcess.isRunning(CwpProcess.java:1984)
    at com.conceptwave.process.CwpActivityQueueThread.startPendingException(CwpActivityQueueThread.java:787)
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:405)

"ProcessActivities_02 Thread" prio=1 tid=0x0000002c087bb770 nid=0x8c9 waiting for monitor entry [0x0000000041f70000..0x0000000041f70db0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_01 Thread" prio=1 tid=0x0000002c08a6dc30 nid=0x8c8 waiting for monitor entry [0x0000000041e6f000..0x0000000041e6fc30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"ProcessActivities_00 Thread" prio=1 tid=0x0000002c08d15a50 nid=0x8c7 waiting for monitor entry [0x0000000041d6e000..0x0000000041d6ecb0]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

"Participants Queue" prio=1 tid=0x0000002c0a9b5330 nid=0x8c6 waiting for monitor entry [0x0000000041c6d000..0x0000000041c6db30]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpProcessManager.putInActivityQueue(CwpProcessManager.java:1343)
    at com.conceptwave.process.CwpParticipantsThread.processMatchedActivity(CwpParticipantsThread.java:436)
    at com.conceptwave.process.CwpParticipantsThread.processWaitingQueue(CwpParticipantsThread.java:367)
    at com.conceptwave.process.CwpParticipantsThread.run(CwpParticipantsThread.java:248)
    - locked  (a com.conceptwave.process.CwpParticipantsThread)

"GlobalRequests Queue" prio=1 tid=0x0000002c067cecf0 nid=0x8c5 waiting for monitor entry [0x0000000041b6c000..0x0000000041b6cbb0]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpProcessManager.putInActivityQueue(CwpProcessManager.java:1343)
    at com.conceptwave.process.CwpGlobalRequestsThread.processRequest(CwpGlobalRequestsThread.java:323)
    at com.conceptwave.process.CwpGlobalRequestsThread.run(CwpGlobalRequestsThread.java:175)

"TaskDistribution Queue" prio=1 tid=0x0000002c096a7e40 nid=0x8c4 waiting on condition [0x0000000041a6b000..0x0000000041a6be30]
    at java.lang.Thread.sleep(Native Method)
    at com.conceptwave.system.CwfBaseThread.waitForInterval(CwfBaseThread.java:144)
    at com.conceptwave.process.CwpIncrementalWaitThread.waitCurrently(CwpIncrementalWaitThread.java:173)
    at com.conceptwave.process.CwpTaskDistributionThread.run(CwpTaskDistributionThread.java:22)

"StartRestart Queue" prio=1 tid=0x0000002c09696c10 nid=0x8c3 waiting for monitor entry [0x000000004196a000..0x000000004196aeb0]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpStartProcessesThread.restartTheProcessList(CwpStartProcessesThread.java:181)
    at com.conceptwave.process.CwpStartProcessesThread.run(CwpStartProcessesThread.java:87)

"GlobalProcesses Queue" prio=1 tid=0x0000002c08c87990 nid=0x8c2 waiting for monitor entry [0x0000000041869000..0x0000000041869d30]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpGlobalProcessThread.enqueueCheckCommand(CwpGlobalProcessThread.java:199)
    at com.conceptwave.process.CwpGlobalProcessThread.run(CwpGlobalProcessThread.java:126)

"SocketListener" prio=1 tid=0x0000002c061cfc50 nid=0x8a8 waiting for monitor entry [0x0000000041767000..0x0000000041768d30]
    at com.conceptwave.process.CwpParticipantsThread.getDBCacheData(CwpParticipantsThread.java:450)
    - waiting to lock  (a com.conceptwave.process.CwpParticipantsThread)
    at com.conceptwave.process.CwpProcess.getParticipantsQueueData(CwpProcess.java:1700)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at com.conceptwave.javascript.CwfScriptFunctionObject.call(CwfScriptFunctionObject.java:368)
    at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:76)
    at org.mozilla.javascript.gen.c1422._c2( (a com.conceptwave.process.CwpActivityQueue)
    at com.conceptwave.process.CwpActivityQueue.enqueue(CwpActivityQueue.java:71)
    at com.conceptwave.process.CwpProcessManager.putInActivityQueue(CwpProcessManager.java:1343)
    at com.conceptwave.process.CwpDelayedActivitiesThread.run(CwpDelayedActivitiesThread.java:271)

"HeartBeat" prio=1 tid=0x0000002c0c84ff90 nid=0x8a6 waiting on condition [0x0000000041465000..0x0000000041465c30]
    at java.lang.Thread.sleep(Native Method)
    at com.conceptwave.system.CwfBaseThread.waitForInterval(CwfBaseThread.java:144)
    at com.conceptwave.system.CwfAVMHeartBeat.run(CwfAVMHeartBeat.java:288)

"ClockThread" prio=1 tid=0x0000002c09dc2a20 nid=0x8a5 waiting on condition [0x0000000041566000..0x0000000041566cb0]
    at java.lang.Thread.sleep(Native Method)
    at com.conceptwave.system.CwfBaseThread.waitForInterval(CwfBaseThread.java:144)
    at com.conceptwave.system.CwfClockThread.run(CwfClockThread.java:32)

"Thread-5" daemon prio=1 tid=0x0000002c07eec270 nid=0x706 waiting on condition [0x0000000041364000..0x0000000041364cb0]
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:91)

"Thread-4" daemon prio=1 tid=0x0000002c088ecde0 nid=0x705 waiting on condition [0x0000000041263000..0x0000000041263b30]
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:91)

"RMI TCP Accept-10001" daemon prio=1 tid=0x0000002c07e7c730 nid=0x6c6 runnable [0x0000000041061000..0x0000000041061eb0]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked  (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Accept-0" daemon prio=1 tid=0x0000002c07e78c70 nid=0x6c5 runnable [0x0000000040f60000..0x0000000040f60d30]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked  (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
    at java.lang.Thread.run(Thread.java:595)

"Timer-0" daemon prio=1 tid=0x0000002c061fb720 nid=0x6c4 in Object.wait() [0x0000000040e5f000..0x0000000040e5fdb0]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:474)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked  (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=1 tid=0x0000002c02f2ecd0 nid=0x6c3 runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x0000002c02f2d0a0 nid=0x6c2 waiting on condition [0x0000000000000000..0x0000000040c5c610]

"CompilerThread0" daemon prio=1 tid=0x0000002c02f2baf0 nid=0x6c1 waiting on condition [0x0000000000000000..0x0000000040b5b460]

"AdapterThread" daemon prio=1 tid=0x0000002c02f2a390 nid=0x6c0 waiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x0000002c02f29030 nid=0x6bf waiting on condition [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x0000002c02f14d60 nid=0x6be in Object.wait() [0x0000000040859000..0x0000000040859eb0]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked  (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x0000002c02f14530 nid=0x6bd in Object.wait() [0x0000000040758000..0x0000000040758d30]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:474)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked  (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x00000000401164d0 nid=0x6b6 in Object.wait() [0x0000007fbfffc000..0x0000007fbfffd080]
    at java.lang.Object.wait(Native Method)
    - waiting on  (a java.lang.Class)
    at java.lang.Object.wait(Object.java:474)
    at com.conceptwave.process.CwpProcessManager.main(CwpProcessManager.java:397)
    - locked  (a java.lang.Class)

"VM Thread" prio=1 tid=0x0000002c02f10060 nid=0x6bc runnable 

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040133d50 nid=0x6b7 runnable 

"GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040134bd0 nid=0x6b8 runnable 

"GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040135a50 nid=0x6b9 runnable 

"GC task thread#3 (ParallelGC)" prio=1 tid=0x00000000401368d0 nid=0x6ba runnable 

"VM Periodic Task Thread" prio=1 tid=0x0000002c07e87830 nid=0x6c7 waiting on condition
1
javathread-dump
It could be a JVM bug. Have you tried a recent version of Java? – Peter Lawrey
Even a recent version of Java 5.0? – Peter Lawrey
Can you show us code around CwpActivityQueueThread:490 and CwpActivityPriorityQueue:55? Also what is happening in CwpActivityQueueThread:466? For the time being I think you have a custom lock/object that you are not releasing in finally properly. – Tomasz Nurkiewicz

1 Answers

2
votes

First off, in the future please whittle down your thread dump. A large number of the threads are duplicates and could have been marked as such.

Looks to me like you have some sort of lock contention. I'm surprised that the JVM didn't flag it as a deadlock but maybe it isn't a nested lock issue.

You have a couple of these:

"Participants Queue" prio=1 tid=0x0000002c0a9b5330 nid=0x8c6 waiting for monitor entry [0x0000000041c6d000..0x0000000041c6db30]
    at com.conceptwave.process.CwpActivityPriorityQueue.putInQueue(CwpActivityPriorityQueue.java:55)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

and then you have one thread that seems to be working:

"ProcessActivities_03 Thread" prio=1 tid=0x0000002c087bcb90 nid=0x8ca runnable [0x0000000042071000..0x0000000042071d30]
    at com.conceptwave.process.CwpActivityInfoCache.checkForActivity(CwpActivityInfoCache.java:159)
    at com.conceptwave.process.CwpActivityInfoCache.getExistingActivity(CwpActivityInfoCache.java:190)
    at com.conceptwave.process.CwpProcess.getActivityIfExists(CwpProcess.java:2200)

Maybe it is stuck in a loop? What locks is it holding?

You also have a large number of these:

"ProcessActivities_29 Thread" prio=1 tid=0x0000002c06961d50 nid=0x8e4 waiting for monitor entry [0x0000000043a8b000..0x0000000043a8be30]
    at com.conceptwave.process.CwpActivityQueueThread.run(CwpActivityQueueThread.java:490)
    - waiting to lock  (a com.conceptwave.process.CwpActivityQueue)

Can't tell what is holding them up. None of these threads are marked as daemon so they all will have to finish by themselves or otherwise detect a shutdown signal (like a volatile boolean shutdown field) before they will quit. If they don't quit then your JVM will never shutdown on its own.