boost asio multithreaded tcp server with thread pool

Question

I have a single threaded asynchronous tcp server written using boost asio. Each incoming request will go through several processing steps (synchronous and asynchronous) and finally send back the response using async write.

For small loads with 10 concurrent requests, it works decently. However, when I test using a parallelism of 100, things start worsening. Response latency starts increasing as time progresses. So, I want to try with some multi-threaded processing for handling requests.

I am looking for a decent example / help on creating and running multiple threads for asynchronous reading/writing to clients. I have the following doubts:

1. Should I use a single IOS object and call its run method in all of the threads of the thread pool, or should I use a separate IOS per thread?
2. If I use a single IOS, is there a possibility that part of the tcp data goes to one thread, while another part going to another thread and so on.. Is this understanding correct?
3. Is there any other better way?

Thanks for any help and pointers here.

Answer 1

Without seeing your code I can only guess what goes wrong. Most probably you're running long actions inside async completion handlers. The completion handlers should be fast - get the data, hand it off for further processing, done.

As a first priority, I would go full-asynchronous and run all processing in a thread pool. You can find an example here, where a new thread is started for every new client, which you can replace with a thread pool.

Use a single io_service. A single io_service can handle a lot of parallelism, provided you don't delay it inside completion handlers. This simplifies the implementation because you don't have to worry about completion handlers running in parallel, which will happen if you run multiple IOS in multiple threads.

Answer 2

Q1: Should I use a single IOS object and call its run method in all of the threads of the thread pool, or should I use a separate IOS per thread?

Either you can

• HTTP Server 2 - IOS per thread
• HTTP Server 3 - single IOS with thread pool

Q2: If I use a single IOS, is there a possibility that part of the tcp data goes to one thread, while another part going to another thread and so on.. Is this understanding correct?

Yes, there is a race condition, but boost.asio support strand to avoid it.

Q3: Is there any other better way?

To me, not find a better way, if you find, tell me or past here, thank you.

BTW, as @rustyx said, your program is blocked at sync calls, turn to full-asynchronous calls will help.