Behavior of Tomcat Thread Pool in Apache + Tomcat Deployement

Question

I have deployed a popular web application benchmark called TPC-W in a 3 tier setting. I have two physical nodes, one running Apache Web Server and another running Tomcat + MySQL. I'm using a shared executor in Tomcat and I'm trying to understand it's behavior. I'm using JMX to monitor the Tomcat JVM.

Here's my question. The completedTaskCount attribute of the Tomcat shared executor MBean does not update according to the number of served requests (actually it keeps unchanged at 0). I can see that there's a very large number of requests served under AJP-connector in GlobalRequestProcessor MBean. Why is that? Shouldn't completedTaskCount increment as requests are processed.

The only possible explanation I could come with is that this is because of some behavior in the AJP protocol. I assumed that for whatever the reason, each Apache process gets connected to a thread in the Tomcat executor thread pool and sends all incoming requests as a single request to the Apache. Even any new incoming request is sent as a part of that request. in that case, the Tomcat thread would never see as that request from Apache is fully served. Does this make any sense?

Any pointers or help regarding this is highly appreciated.

Edit 1

Tomcat Version: 7.0.94

Apache Version: 2.4.38

This is the connector definition:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" executor="tomcatThreadPool"/>

Here is the shared executor definition:

<Executor name="tomcatThreadPool" namePrefix="tpcw-exec-" maxThreads="200" minSpareThreads="25"/>

Answer 1

AJP uses keep-alive connections by default. BIO connections have a thread allocated to them when the are opened and that thread stays allocated until the connection closes.

The use of keep-alive connections means the connection never closes so the thread never returns to the pool so the Executor never sees a completed task. If you bounce one of the httpd instances that should close the open connections and you should see the completed task count on the executor rise - but only by the number of connections, not the number of processed requests since each connection processes many requests.