Separating messages in a simple TCP echo server using Winsock DLL

2
votes

Please consider a simple echo server using TCP and the Winsock DLL. The client application sends messages from multiple threads. The recv call on the server sometimes returns with multiple messages stored in the passed buffer. At this point, there's no chance for the server to know, whether this is one huge message or multiple small messages.

I've read that one could use setsockopt in combination with the TCP_NODELAY option. Besides that MSDN states, that this option is implemented for backward compatibility only, it doesn't even change the behavior described above.

Of course, I could introduce some kind of delimiter at the end of each message and split the message on server-side. But I don't think that's way one should do it. So, what is the right way to do it?

1
c++socketstcpwinsock

1 Answers

3
votes

Firstly, TCP_NODELAY was not the right way to do this... TCP is a byte stream protocol and any given connection only maintains the byte ordering - not necessarily the boundaries of any given send/write. It's inherently broken to rely on multiple threads that don't use any synchronisation being able to even keep the messages they want to send together on the stream. For example, say thread 1 wants to send the two-byte message "AB" and thread 2 wants to send "XY"... say thread 1 starts first and the output buffer only has room for one byte, send will enqueue "A" and let thread 1 know it's only sent one byte (so it should loop and retry - preferable after waiting for notification that the output queue has more space). Then, thread 2 might get some or all of "XY" into the queue before thread 1 can get "Y". These sorts of problems become more severe on slower connections, for slow and loaded machines (e.g. perhaps a low-powered phone that's playing video and multitasking while your app runs over 3G).

The ways to ensure the logical messages stay together over TCP include:

  • have a single sending thread that picks up messages sequentially from a shared queue (a mutex might be used to let the threads enqueue messages)

  • contest a lock (mutex) so the threads' sends have an uninterrupted ability to loop to send until a complete message is sent (this wouldn't suit some apps because any of the threads could be held up for quite a while doing comms work)

  • use a separate TCP connection per thread