Are repeated recv() calls expensive?

6
votes

I have a question about a situation that I face quite often. From time to time I have to implement various TCP-based protocols. Most of them define variable-length data packets that begin with a common header ([packet ID, length, payload] or something really similar). Obviously, there can be two approaches to reading these packets:

  1. Read header (since header length is usually fixed), extract the payload length, read the payload
  2. Read all available data and store it in a buffer; parse the buffer afterwards

Obviously, the first approach is simple, but requires two calls to read() (or probably more). The second one is slightly more complicated, but requires less calls.

The question is: does the first approach affect the performance badly enough to worry about it?

5
csocketstcp
Personally I wouldn't worry about the difference between one call per connection and two, because accepting a connection is likely much more expensive than the overhead of a system call. But I'm not a network guru so just a comment. Also perhaps consider that if the header indicates some kind of error or impossibility then in the first case, you don't have to read all the data and in the second you do. Consider for example an HTTP POST trying to upload 100MB to a non-existent URL. Ultimately receive() is called in a loop so it's not really up to you how many calls you make... – Steve Jessop
@Steve: Aside from a few extremely expensive syscalls (and of course all the ones that can be interrupted during filesystem access, etc.), most of the cost of a syscall is the fixed overhead from entering and exiting kernelspace. Making 2 recv calls where one would do will probably double the time spent there. If each is already accompanied by accept/close then it's only 33% more time (or less) you'll spend on the extra recv, but if this is a persistent connection, the increased cost could be high. – R.. GitHub STOP HELPING ICE
@R..: possibly "expensive" was the wrong word, I suppose I actually meant "time-consuming". Proper analysis really requires finding out what Roman D means by "performance". I had unthinkingly assumed a smallish number of simultaneous connections on the machine, so that the cost of syscalls is pretty much irrelevant provided you can keep up with data arriving, and 1 vs. 2 likely won't affect that. Accepting a connection therefore is "expensive" in the sense that a TCP handshake takes essentially forever. But you're right, that might not be relevant in any given case. – Steve Jessop
... if each header and each message-payload is 1 byte long, all on a single persistent connection operating at high bandwidth, and the server is CPU-bound, then I guess 1 vs 2 syscalls per message could double the throughput. – Steve Jessop
@Steve: Even if not, it could affect your electric bill. I don't know if many hosting services bill that way, but I'd love to find a colo that charges based on your electricity usage... I suspect if more hosting services took that into consideration for billing, people might start writing more efficient code in a hurry.... – R.. GitHub STOP HELPING ICE

5 Answers

9
votes

yes, system calls are generally expensive, compared to memory copies. IMHO it is particularly true on x86 architecture, and arguable on RISC machine (arm, mips, ...).

To be honest, unless you must handle hundreds or thousands of request per second, you will hardly notice the difference.

Depending on what is exactly the protocol, an hybrid approach could be the best. When the protocol uses a lot of small packets and less big ones, you can read the header and a partial amount of data. When it is a small packet, you win by avoiding a large memcpy, when the packet is big, you win by issuing a second syscall only for that case.

4
votes

If your application is a server capable of handling multiple clients simultaneously and non-blocking sockets are used to handle multiple clients in one thread, you have little choice but to only ever issue one recv() syscall when a socket becomes ready for read.

The reason for that is if you keep calling recv() in a loop and the client sends a large volume of data, what can happen is that your recv() loop may block the thread for long time from doing anything else. E.g., recv() reads some amount of data from the socket, determines that there is now a complete message in the buffer and forwards that message to the callback. The callback processes the message somehow and returns. If you call recv() once more there can be more messages that have arrived while the callback was processing the previous message. This leads to a busy recv() loop on one socket preventing the thread from processing any other pending events.

This issue is exacerbated if the socket read buffer in your application is smaller than the kernel socket receive buffer. In other words, the whole contents of the kernel receive buffer can not be read in one recv() call. Anecdotal evidence is that I hit this issue on a busy production system when there was a 16Kb user-space buffer for a 2Mb kernel socket receive buffer. A client sending many messages in succession would block the thread in that recv() loop for minutes because more messages would arrive when the just read messages were being processed, leading to disruption of the service.

In such event-driven architectures it is best to have the user-space read buffer equal to the size of the kernel socket receive buffer (or the maximum message size, whichever is bigger), so that all the data available in the kernel buffer can be read in one recv() call. This works by doing one recv() call, processing all complete messages in the user-space read buffer and then returning control to the event loop. This way a connections with a lot of incoming data is not blocking the thread from processing other events and connections, rather it round-robin's processing of all connections with incoming data available.

3
votes

The best way to get your answer is to measure. The strace program is decent for the purpose of measuring system call times. Using it adds a lot of overhead in itself, but if you merely compare the cost of one recv for this purpose versus the cost of two, it should be reasonably meaningful. Use the -tt option to get times. Or you can use the -c option to get an overview of time spent separated by which syscall it was spent on.

A better way to measure, albeit with more of a learning curve, is oprofile.

Also note that if you do decide buffering is worthwhile, you may be able to use fdopen and the stdio functions to take care of it for you. This is extremely easy and will work well if you're only dealing with a single connection or if you have a thread/process per connection, but won't work at all if you want to use a select/poll-based model.

1
votes

Note that you generally have to "read all the available data into a buffer and process it afterwards" anyway, to account for the (unlikely, but possible) scenario where a recv() call returns only part of your header - so you might as well go the whole hog and use option 2.

-2
votes

Yes, depending upon the scenario the read/recv calls may be expensive. For example, if you are issuing huge number of recv() calls to read very small amount of data every small interval, it would be a performance hit. In such scenario you could issue a recv() with reasonably large buffer, let say 4k, and then parse that 4k buffer. It may contain multiple header+data combo. By reading header first you can find the data and its length. And to avoid the mem copy of data into a new buffer, you can just use the offset from where the actual data start, and store that pointer.