Increasing worker threads has no effect on Netty IO performance

Question

I am running Http server using Netty I/O library on a quad-core Linux machine. Running with default worker thread pool size (which is set to 2 x number of cores internally in Netty), performance analysis show throughput caps at 1k requests/second and further increases in request rate causes increase in latency almost linearly.

As max CPU utilization shows 60%, I increased number of worker threads as per code below. However, there is hardly any change in performance and CPU is still capped at 60-70%. The process is not bounded by memory, I/O or network bandwidth. Why isn't there change in performance by increasing worker threads? What other things I can do to improve my server performance to increase it's capacity.

EventLoopGroup group = new NIOEventLoopGroup(100);
ServerBootStrap serverBootStrap = new ServerBootStrap();
serverBootStrap.group(group)
               .channel(NioServerSocketChannel.class)
               .localAddress(..)
               ...

What makes you certain that the process is not bounded by memory, I/O, or network bandwidth? — Jeremy Friesner
i capture various stats on the machine and export them to metric server for analysis (prometheus). these include memory/heap usage, swap usage, network I/O, etc. Everything is well under 40%. — user236215

Leo Gomes Leo Gomes · Accepted Answer · 2020-05-16T22:16:47

If your code is using purely non-blocking I/O, you should be reaching more than 1k TPS with a quad-core. You should analyse what the Netty threads are doing, i.e. if they are getting blocked by any call done within the event loop. VisualVM should already give you a good idea of what's happening, e.g. here you can see Vert.x threads, which use Netty behind the scenes sleeping:

Another thing you could try is to disable hyperthreading and checking how CPU utilisation behaves: https://serverfault.com/questions/235825/disable-hyperthreading-from-within-linux-no-access-to-bios

Increasing worker threads has no effect on Netty IO performance

1 Answers