Netty threads being blocked

Question

I have 3 ThreadPoolExecutors in my system.

One for Netty's Master process, another for netty's worker process and last one for processing ad-hoc processing (sending request to mail server).

ExecutorService bossExecutors = Executors.newFixedThreadPool(1, 
                 new ServerThreadFactory("netty-boss")); 
ExecutorService workerExecutors = Executors.newFixedThreadPool(10, 
                 new ServerThreadFactory("netty-worker"));
ChannelFactory factory = new NioServerSocketChannelFactory(
                 bossExecutors, 
                 workerExecutors,
                 Runtime.getRuntime().availableProcessors());

ExecutorService mailExecutor = Executors.newFixedThreadPool(40);

This works perfectly fine until mailExecutor starts making request to mail server. Until, that batch requests using mailExecutor, generally making 5000+ requests to mail server is completed, netty threads get blocked.

I don't understand why netty threads seem to be getting blocked that time since, I have allocated definite thread pools. During that time, Netty can't even process single request.

Any idea why it's happening or what I'm doing wrong?

Answer 1

Can you provide a thread-dump ?

jstack <pid>

Also you should never use a fixed threadpool for the worker / poss threadpool. Use a cached one, this way you can be sure you never get into any starvation. You should specify the worker count with the 3 argument in the constructor.

Answer 2

It sounds like a scheduling issue. You have 40 threads under heavy load, vs the availableProcessors number of threads for handling Netty work (what is your availableProcessors() count at the time you create your factory?).

So it could just be that the Netty threads are too few and are being starved since they never happen to be picked for execution compared to the 40 threads handling the mail work.

It may also be that for some reason, your worker threads are blocked on the mail threads finishing, perhaps due to some shared object that is being synchronized on (is there some queue or list of mail to be sent that the netty threads need to write to, and which the mail threads have locked while they send?).