Tomcat and acceptCount not working

Question

I am using tomcat 8 as part of Spring Boot project and my acceptCount setting seems to be not working. Instead of only accepting 300 connections, my server accepted almost 1000 connection I threw at it, though of course processing no more than 200 of them simultaneously.

Tomcat documentation seems to be clear on acceptCount: "The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused." But clearly that does not happen.

There is of course another setting, maxConnections, and its documentation says: "Note that once the limit has been reached, the operating system may still accept connections based on the acceptCount setting" - but that note alone does not yet mean acceptCount is "count on top of maxConnections, not maxThreads" (like the topicstarter at https://coderanch.com/t/647733/application-servers/Tomcat-BIO-connector-configurations thinks). It just means that it will have its effect in both cases: when all processing threads are busy, and when all available connections are exhausted. (And even if that guy was correct, that would mean Tomcat documentation is outright wrong when defining acceptCount...)

Then why is it ignored? I found some mentions of discussions where people supposedly claimed that acceptCount does not work for them, but did not find actual discussions :( Even on the contrary, I was able to find some complains about how Tomcat chokes after 300 connections (which is exactly default maxThreads + acceptCount). So I can see that for some people it works, and I am asked to believe that "maybe" for some people it does not. Not for me either. Should I trust that the Tomcat manual is wrong in a sense that this option is not always respected?

Answer 1

There are complicated relationships between all of these settings, and they are further complicated by your choice of connector (e.g. BIO, NIO, APR, etc.).

The BIO connector is essentially dead... it does not exist past Tomcat 8.0.x. It cannot accept any more connections than the thread pool can process at a time. Therefore, the BIO connector essentially makes maxConnections == maxThreads. Therefore, for a BIO connector, the number of connections the server is willing to accept should be maxConnections + acceptCount, but maxConnections is limited to some relatively small number.

The other connectors, which are more complicated, allow more than one connection per thread to be accepted in the TCP/IP sense.The defaults for maxConnections are closer to 10k (varies by exact type), and the thread pool size is not relevant, so the number of connections the server is willing to accept is maxConnections + acceptCount.

The reason the non-BIO connectors can accept so many more connections is because the two states where an active request-processing thread are not necessary (waiting for the next HTTP-keepalive-request - using read() and waiting for the next connection - using accept()) are done by a separate thread, allowing the request-processing thread to go back into the pool as quickly as possible in order to service other requests.

If you are using BIO, I'd expect connections to fail after e.g. 300 connections (you didn't post your configuration, so it's impossible to say what the actual numbers would be) but for NIO, I'd expect it to be north of 10k connections.

For testing, it's important that you don't actually do anything with your connection in order to count them properly. Basically, you need to do this:

foreach(i from 0..10301)
    conn[i] = connect('host:port')

You should find that there is some i for which the server will not accept the connection. If you connect and issue GET / then the server will respond and the connection and the request-processing thread will go back into their respective pools.

** UPDATE after reading some comments**

I'd expect to be able to simultaneously-process 200 requests, but Tomcat will happily queue another 800 requests internally. The additional 200 in the acceptQueue are being queued by the OS's TCP/IP stack, not by Tomcat/Java

Assuming infinite read-timeouts on the clients, I'd expect you to be able to launch 1201 instances of curl where the first 200 in the door immediately get a thread (and sleep), the next 800 get queued in Tomcat/Java, the next 200 get queued in the TCP/IP stack, and instance #1201 gets a "connection refused" response.

Once the first set of 200 requests are completed, Tomcat will process another batch of 200 requests, the 200 connections in the TCP/IP queue will move from that queue into the Tomcat/Java queue, and after 20 seconds the second set of 200 requests will complete. That will repeat until all 1200 of the initial requests return a response to their clients. Only 1 of the 1200 requests will be refused.

Answer 2

acceptCount for NIO is passed in to ServerSocket.bind(SocketAddress,int) as the 2nd parameter (backlog). The Javadoc for that method says:

The backlog argument is the requested maximum number of pending connections on the socket. Its exact semantics are implementation specific. In particular, an implementation may impose a maximum length or may choose to ignore the parameter altogether.

It looks like it is being ignored in your JVM.