Tomcat does not respond

Question

I'm in charge of deploying a Java application. My problem is that after an hour tomcat stop process request.

curl localhost:8080 does not answer.

The Tomcat process is still alive and the RAM and the load average of the server are OK. I got no error in tomcat's log neither in my application.

I did a Thread Dump using kill -3 tomcat_pid. Almost all Thread are in state WAITING or TIMED_WAITING.

I use Tomcat v7.0.56. The application use JackRabbit JCR as data source.

How can I debug my server?

Edit:

Here is some thread stack exemple: Most of the "http-bio-8080-exec-N" Threads looks like this:

    "http-bio-8080-exec-16" daemon prio=10 tid=0x00007f0cacfc1000 nid=0x56b6 waiting on condition [0x00007f0c62abc000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007f574af30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

There is also reference to JCR and Derby Database:

    "derby.antiGC" daemon prio=10 tid=0x00007f0ca0953000 nid=0x5610 in Object.wait() [0x00007f0ca8d11000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007f7543aa8> (a org.apache.derby.impl.services.monitor.AntiGC)
    at java.lang.Object.wait(Object.java:503)
    at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
    - locked <0x00000007f7543aa8> (a org.apache.derby.impl.services.monitor.AntiGC)
    at java.lang.Thread.run(Thread.java:745)

And here:

    "jackrabbit-pool-1" daemon prio=10 tid=0x00007f0ca08c7800 nid=0x560f waiting on condition [0x00007f0ca8d92000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007fa29ccb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

WAITING=A thread that is waiting indefinitely for another thread to perform a particular action is in this state. \n\n TIMED_WAITING=A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state. What are your threads waiting for ? — Svetlin Zarev
Seems like derby database connection related issue which is halting the whole application. Check if you have any of these derby versions issues.apache.org/jira/browse/DERBY-2905 — Amit Vyas

krishna garikapati krishna garikapati · Accepted Answer · 2020-07-17T17:30:35

This will be due to one of the thread got blocked(This could be due to network or DB issue) and on top of it multiple requests are queued from your web server to app(tomcat).

You will see lot of errors like below when you face this error and restarting tomcat service won't help in this case as it could be cached.

  "Thread-14" #83 daemon prio=5 os_prio=0 tid=0x00007f702c017000 nid=0x3830 waiting on condition [0x00007f707125e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.tridion.licensing.events.LicenseTracker.run(LicenseTracker.java:91)
at java.lang.Thread.run(Thread.java:748)

And

 "Thread-14" #83 daemon prio=5 os_prio=0 tid=0x00007f702c017000 nid=0x3830 waiting on condition [0x00007f707125e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.tridion.licensing.events.LicenseTracker.run(LicenseTracker.java:91)
at java.lang.Thread.run(Thread.java:748)

This can be fixed by rebooting the server as it clears all the queue.

Tomcat does not respond

1 Answers