How to programmatically increase the per-socket buffer for UDP sockets on LInux?

3
votes

I'm trying to understand the correct way to increase the socket buffer size on Linux for our streaming network application. The application receives variable bitrate data streamed to it on a number of UDP sockets. The volume of data is substantially higher at the start of the stream and I've used:

# sar -n UDP 1 200

to show that the UDP stack is discarding packets and

# ss -un -pa

to show that each socket Recv-Q length grows to the nearly the limit (124928. from sysctl net.core.rmem_default) before packets are discarded. This implies that the application simply can't keep up with the start of the stream. After discarding enough initial packets the data rate slows down and the application catches up. Recv-Q trends towards 0 and remains there for the duration.

I'm able to address the packet loss by substantially increasing the rmem_default value which increases the socket buffer size and gives the application time to recover from the large initial bursts. My understanding is that this changes the default allocation for all sockets on the system. I'd rather just increase the allocation for the specific UDP sockets and not modify the global default.

My initial strategy was to modify rmem_max and to use setsockopt(SO_RCVBUF) on each individual socket. However, this question makes me concerned about disabling Linux autotuning for all sockets and not just UDP.

udp(7) describes the udp_mem setting but I'm confused how these values interact with the rmem_default and rmem_max values. The language it uses is "all sockets", so my suspicion is that these settings apply to the complete UDP stack and not individual UDP sockets.

Is udp_rmem_min the setting I'm looking for? It seems to apply to individual sockets but global to all UDP sockets on the system.

Is there a way to safely increase the socket buffer length for the specific UDP ports used in my application without modifying any global settings?

Thanks.

1
linuxsocketsnetwork-programmingudp
possible duplicate of Specifying UDP receive buffer size at runtime in Linux – Adam Liss
If this was Windows, I would be saying 'move now to overlapped IO, preferably IOCP, so that the kernel threads running the stack can receive data directly into user buffers that you provide. Create a pool of 100 64K buffers and keep a few overlapped reads queued up'. Surely, something similar is possible on Linux? NIO/asio? – Martin James
You might consider moving your UDP-recv() loop into a separate, high-priority thread that does nothing except receive packets, then hand then off to the main thread (or some other thread) for the actual processing. In my experience, when an application can't keep up with incoming UDP, it's usually not because the CPU isn't fast enough in general, but rather because the receiving thread is occasionally busy doing something else and doesn't call recv() soon enough. By dedicating a thread to only doing recv() and little else, you minimize the chances of a hold off like that occurring. – Jeremy Friesner
Jeremy - I'm not sure how that will help. Even if the recv() is the highest priority (and only) thread running on the system, the data could still fill the queues if the time required to process a packet is larger than the arrival rate. Obviously the app can't sustain that data rate but, since it only occurs at the beginning, I'd like to increase the buffer depth to accommodate it. – bfallik-bamboom

1 Answers

0
votes

Jim Gettys is armed and coming for you. Don't go to sleep.

The solution to network packet floods is almost never to increase buffering. Why is your protocol's queueing strategy not backing off? Why can't you just use TCP if you're trying to send so much data in a stream (which is what TCP was designed for).