Receive an high rate of UDP packets with python

Question

I'm working with python in order to receive a stream of UDP packets from an FPGA, trying to lose as few packets as possible. The packet rate goes from around 5kHz up to some MHz and we want to take data in a specific time window (acq_time in the code). We have this code now:

BUFSIZE=4096
dataSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
dataSock.settimeout(0.1)
dataSock.bind((self.HOST_IP, self.HOST_PORT))
time0=time.time()
data_list = []
while time.time()-time0<acq_time:
     fast_acquisition(data_list)

def fast_acquisition(data_list_tmp):
    data, addr = dataSock.recvfrom(self.BUFSIZE)
    data_list_tmp.append(data) 
    return len(data)

And after the acquisition we save our data_list on disk.

This code is meant to be as simple and fast as possible, but it's still too slow and we lose too many packets even at 5kHz, and we think that this happens because while we read, and store in the list one packet and check the time, the next one (ore ones) arrives and is lost. Is there any way to keep the socket open? Can we open multiple sockets "in series" with parallel processing, so that when we are saving the file from the first the second can receive another packet? We can even think to use another language only to receive and store the packets on disk.

Answer 1

You could use tcpdump (which is implemented in C) to capture the UDP traffic, since it's faster than python:

#!/bin/bash

iface=$1 # interface, 1st arg
port=$2  # port, 2nd arg

tcpdump -i $iface -G acq_time_in_seconds -n udp port $port -w traffic.pcap

And then you can use e.g. scapy to process that traffic

#!/usr/bin/env python

from scapy.all import *

scapy_cap = rdpcap('traffic.pcap')
for packet in scapy_cap:
    # process packets...

Answer 2

There are several reasons why UDP packets can be lost, and certainly the speed of being able to take them off the socket queue and store them can be a factor, at least eventually. However, even if you had a dedicated C language program handling them, it's unlikely you'll be able to receive all the UDP packets if you expect to receive more than a million a second.

The first thing I'd do is to determine if python performance is actually your bottleneck. It is more likely in my experience that, first and foremost, you're simply running out of receive buffer space. The kernel will store UDP datagrams on your socket's receive queue until the space is exhausted. You might be able to extend that capacity a little with a C program, but you will still exhaust the space faster than you can drain the socket if packets are coming in at high enough speed.

Assuming you're running on linux, take a look at this answer for how to configure the socket's receive buffer space -- and examine the system-wide maximum value, which is also configurable and might need to be increased. https://stackoverflow.com/a/30992928/1076479

(If you're not on linux, I can't really give any specific guidance, although the same factors are likely to apply.)

It is possible that you will be unable to receive packets fast enough, even with more buffer space and even in a C program. In that case, @game0ver's idea of using tcpdump might work better if you only need to withstand a short intense burst of packets as it uses a much lower-level interface to obtain packets (and is highly optimized). But then of course you won't just have the UDP payload, you'll have entire raw packets and will need to strip the IP and Ethernet layer headers as well before you can process them.