Receive an high rate of UDP packets with python

Question

I'm working with python in order to receive a stream of UDP packets from an FPGA, trying to lose as few packets as possible. The packet rate goes from around 5kHz up to some MHz and we want to take data in a specific time window (acq_time in the code). We have this code now:

BUFSIZE=4096
dataSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
dataSock.settimeout(0.1)
dataSock.bind((self.HOST_IP, self.HOST_PORT))
time0=time.time()
data_list = []
while time.time()-time0<acq_time:
     fast_acquisition(data_list)

def fast_acquisition(data_list_tmp):
    data, addr = dataSock.recvfrom(self.BUFSIZE)
    data_list_tmp.append(data) 
    return len(data)

And after the acquisition we save our data_list on disk.

This code is meant to be as simple and fast as possible, but it's still too slow and we lose too many packets even at 5kHz, and we think that this happens because while we read, and store in the list one packet and check the time, the next one (ore ones) arrives and is lost. Is there any way to keep the socket open? Can we open multiple sockets "in series" with parallel processing, so that when we are saving the file from the first the second can receive another packet? We can even think to use another language only to receive and store the packets on disk.

game0ver game0ver · Accepted Answer · 2018-07-20T20:00:44

You could use tcpdump (which is implemented in C) to capture the UDP traffic, since it's faster than python:

#!/bin/bash

iface=$1 # interface, 1st arg
port=$2  # port, 2nd arg

tcpdump -i $iface -G acq_time_in_seconds -n udp port $port -w traffic.pcap

And then you can use e.g. scapy to process that traffic

#!/usr/bin/env python

from scapy.all import *

scapy_cap = rdpcap('traffic.pcap')
for packet in scapy_cap:
    # process packets...

Receive an high rate of UDP packets with python

2 Answers