Multiple requests go into same thread of Java gRPC Server

Question

I have some micro services written by java, communicating with each other by gRPC, and are put into AWS ECS for running. When I run the service and use a client program to call these services, I have noticed a very strange thing.

Normally, Tomcat's thread pool setting will make sure multiple requests will be handled by same amount of threads. But when I run my program sometimes it will happen a thread start handling a new request when it is already handling a previous request. It makes the process thread-unsafe and thus errors were happened. It happens very rare on normal services, and must happens when I send 10 requests into same service at the same time.

May I ask is there any setting that I have missed? What I can search on the internet is I can only increase thread pool size.

----------2020/12/22 Edited----------

By entering debug mode, I have seen that one RPC request will be split to two tasks. One for message itself and another for listener returned from the first task. 90% of my previous errors were happened between the time gap of these two tasks. Moreover, when running listener, ServerImpl class will use context to handle each RPC message, and each step in unit of methods that covers all written codes within over-rided listeners. Therefore it will happen thread-switching during listener's run.

What does it mean that a thread starts a new request while it is processing another request; how can a thread be executing two pieces of code simultaneously? Can you explain more about the thread-unsafety you were noticing? — Eric Anderson
@EricAnderson From the log, I see that the thread started processing request A for a few steps then it started to process request B. Some cases can handle both request A and B at the end, but other cases just like request B take over request A. — Chung

Eric Anderson Eric Anderson · Accepted Answer · 2020-12-16T19:58:28

In StreamObserver's documentation:

Implementations are not required to be thread-safe (but should be thread-compatible).

The "thread-compatible" is the important part for your question. It means you can't assume that every call to your instance will happen on the same Thread. This is true for virtually all of the gRPC API that is not-thread-safe. That doesn't matter to most code, but would impact things like ThreadLocals.

Basically, a thread is not "owned" by an RPC. Instead, the RPCs share all the threads and when a callback needs to be delivered it just finds a thread to run the callback on. This an async model that allows greater scaling compared to the mostly-blocking model used with servlets, as you can service more RPCs with fewer threads. But it means as soon as you return from a callback that same thread may be used for some other RPC callback.

Multiple requests go into same thread of Java gRPC Server

1 Answers