Handle multiple TCP clients

Question

I am writing a simple TCP server, and despite a lot of Googling I can't quite figure out the answer to a question of mine.

I want my server to handle multiple clients.

For performance and simplicity reasons, I want my server to be single-threaded, and I use select() to handle all the sockets.

Also, my protocol is message-based. Message framing is done using a length prefix. This means the server needs a buffer to reconstruct messages fragment after fragment.

The maximum message length is 64K bytes (although I could reduce it to 256 bytes).

Note: this will run on a tiny embedded device, so using a messaging layer like ZMQ is not an option (not enough memory).

I can either:

have a single buffer, but it means once I started to receive/reconstruct a message from a client socket, I ignore other client sockets until the current message has been completely received. This is prone to DOS attacks: one client sending a huge message, byte per byte, very slowly, would block the server.
have one buffer per client socket, and then my server would be really parallel. But it does not scale well with the number of clients.

Is there another way which has the benefits of both techniques and none of the drawbacks?

One idea I had was to use the socket buffer to store the entire message. I would set the buffer size to 64K using setsockopt() and SO_RCVBUFSIZ.

But I would need to do a recv() with both MSG_WAITALL and MSG_DONTWAIT, so that either the message is entirely available in the socket buffer and I get it, or it is not entirely received yet and then recv() does not block. However, these 2 options do not work together.

Maybe I can do a recv() with MSG_PEEK to read the size, then another recv() with MSG_PEEK to test if all bytes are available, and if yes then re-do a recv() without MSG_PEEK to actually read the bytes from the socket?

Anyway, I have the impression it is a trivial question that must have been solved a long time ago.

How many clients are you anticipating being connected to your device simultaneously? How much memory does your device have available? Those kind of answers will help determine what kind of buffering strategies you can use. Using a per-client 256B buffer won’t take up very much memory even with thousands of clients. Even a per-client 64K buffer is likely to be doable on modern hardware. — Remy Lebeau
You don't have anhy choice but to do (2), but the buffer doesn't have to be big enough to hold a complete message if your message parsing is smart enough. — user207421
My idea with MSG_PEEK does not work. First, once a fragment of message is available in a socket, but the message is not received entirely yet, select() will return immediately and the server will use 100% of the CPU. Second, I cannot detect a connection closed with recv() returning 0. — Julien REINAULD

Erdal Küçük Erdal Küçük · Accepted Answer · 2021-01-04T02:32:11

If you can have both, why not use both?

Have a large single buffer for a 'few' clients.
Keep time between the reads for each client, if you think there is a DOS attack or it takes a long time for a read (due to a slow connection), kick the client(s) away. The idea here is to implement a time window.
If you are at 'prime time', increase your buffer (for more clients) or allocate another buffer (bank based).
Shrink the buffer otherwise.

With a little management, you are able to serve some clients simultaneously and don't waste precious memory.

Handle multiple TCP clients

2 Answers