libpcap: what is the efficiency of pcap_dispatch or pcap_next

Question

I use libpcap to capture a lot packets, and then process/modify these packets and send them to another host.

First, I create a libpcap handler handle and set it NON-BLOCKING, and use pcap_get_selecable_fd(handle) to get a corresponding file descriptor pcap_fd.

Then I add an event for this pcap_fd to a libevent loop(it is like select() or epoll()).

In order to avoid frequently polling this file descriptor, each time there are packet arrival event, I use pcap_dispatch to collect a bufferful of packets and put them into a queue packet_queue, and then call process_packet to process/modify/send each packet in the queue packet_queue.

  pcap_dispatch(handle, -1, collect_pkt, (u_char *)packet_queue);
  process_packet(packet_queue);

I use tcpdump to capture the packets that are sent by process_packet(packet_queue), and notice:

1. at the very beginning, the interval between sent packets is small
2. after that several packets are sent, the interval becomes around 0.055 second
3. after 20 packets are sent, the interval becomes 0.031 second and keeps on being 0.031 second

I carefully checked my source code and find no suspicious blocks or logic which leads to so big intervals. So I wonder whether it is due to the problem of the function pcap_dispatch.

are there any efficiency problem on pcap_dispatch or pcap_next or even the libpcap file descriptor? thanks!

Answer 1

On many platforms libpcap uses platform-specific implementations for faster packet capture, so YMMV. Generally they involve a shared buffer between the kernel and the application.

1. At the very beginning you have a time window between the moment packets start piling up on the RX buffer and the moment you start processing. The accumulation of these packets may cause the higher frequency here. This part is true regardless of implementation.
2. I haven't found a satisfying explanation to this. Maybe you got behind and missed a few packets, so you the time between packets resent becomes higher.
3. This is what you'd expect in normal operation, I think.

pcap_dispatch is pretty much as good as it gets, at least in libpcap. pcap_next, on the other hand, incurs in two penalties (at least on Linux, but I think it does in other mainstream platforms too): a syscall per packet (libpcap calls poll for error checking, even in non-blocking mode) and a copy (libpcap releases the "slot" in the shared buffer ASAP, so it can't just return that pointer). An implementation detail is that, on Linux, pcap_next just calls pcap_dispatch for one packet and with a copy callback.