python - Scapy fails to sniff packets when using multiple threads

Question

I'll try to demonstrate my problem with a simplified example.

Following is a very simple (single threaded) packet sniffer (ICMP):

from scapy.all import *

m_iface = "wlan0"
m_dst = "192.168.0.1"

def print_summary(pkt):
  print pkt.summary()

def plain_sniff():
  sniff(iface = m_iface, count = 10, filter = "icmp and src {0}".format(m_dst), prn = print_summary)

This sniffer works just fine and I get the output:

WARNING: No route found for IPv6 destination :: (no default route?)
Ether / IP / ICMP 192.168.0.1 > 192.168.0.9 echo-reply 0 / Raw
Ether / IP / ICMP 192.168.0.1 > 192.168.0.9 echo-reply 0 / Raw
Ether / IP / ICMP 192.168.0.1 > 192.168.0.9 echo-reply 0 / Raw
...

Next, I create a separate thread for sniffing packets and use a Queue to communicate the captured packets between the sniffer thread and the main thread:

from threading import Thread
from Queue import Queue, Empty
from scapy.all import *

m_iface = "wlan0"
m_finished = False
m_dst = "192.168.0.1"

def print_summary(pkt):
  print pkt.summary()

def threaded_sniff_target(q):
  global m_finished
  sniff(iface = m_iface, count = 10, filter = "icmp and src {0}".format(m_dst), prn = lambda x : q.put(x))
  m_finished = True

def threaded_sniff():
  q = Queue()
  sniffer = Thread(target = threaded_sniff_target, args = (q,))
  sniffer.daemon = True
  sniffer.start()
  while (not m_finished):
    try:
      pkt = q.get(timeout = 1)
      print_summary(pkt)
    except Empty:
      pass

This sniffer also works fine and I get the same output as above. However, when I modify the main thread just a little bit so that it uses the send() function between reads from the packet queue as below:

def threaded_sniff_with_send():
  q = Queue()
  sniffer = Thread(target = threaded_sniff_target, args = (q,))
  sniffer.daemon = True
  sniffer.start()
  while (not m_finished):
    send(IP(dst = m_dst) / ICMP()) # Here
    try:
      pkt = q.get(timeout = 1)
      print_summary(pkt)
    except Empty:
      pass

Then I get the following bizarre output (the filter doesn't seem to work):

WARNING: No route found for IPv6 destination :: (no default route?)
Sent 1 packets.
Ether / ARP who has 192.168.0.1 says 192.168.0.9
Sent 1 packets.
Ether / ARP is at a0:21:b7:1a:7a:db says 192.168.0.1
Sent 1 packets.
Ether / IP / ICMP 192.168.0.9 > 192.168.0.1 echo-request 0
Sent 1 packets.
Ether / IP / ICMP 192.168.0.1 > 192.168.0.9 echo-reply 0
...

The script for the three sniffers can be downloaded from here.

My current system configuration is as below:

Python: 2.7.3
Scapy: 2.2.0
OS: Fedora 18

Interestingly, all the three sniffers work fine on my older computer:

Python: 2.6.4
Scapy: 2.0.0.10 (beta)
OS: Fedora 13

First I thought it might be the Scapy / Python versions. But even when I installed the exact same versions on my new computer, the behaviour persisted.

I'm not entirely sure if this is a question suitable for SO (may be a bug report to Scapy?). Please excuse me in that case.

@ChathurangaChandrasekara: The issue I'm facing is related to multi-threading. The code works fine without threading. — Asiri Rathnayake
@AsiriRathnayake This doesn't happen to have something to do with the fact that the network interface ends up in "Premiscious mode" and that only one source can bind to that interface? If not, have you tried using the python threading library instead of queue? i'm not entire sure how the queue system works but with threading you'll get more control of what does when at least, and you can tinker around for a bit. — Torxed
@AsiriRathnayake and i guess if you put a sleep(30) before executing the second thread when using threading it works? — Torxed
@Torxed: I put a time.sleep(1) just after starting the sniffer thread (so, delayed the main thread a little), and it works!!! My guess is that there is a resource overlap between Scapy's sniff() and send() functions which is not thread-safe, and when we delay the main thread, this problem goes away. Please articulate all this into an answer and I'll award you the points. Cheers!!! — Asiri Rathnayake

Torxed Torxed · Accepted Answer · 2013-05-07T08:09:24

The main reason for this is most likely because of resource locks. Scapy's sniff() function probably have to lock on to low-level network resources in order to be able to sniff packets.

Delaying the two threads (after starting the sniffer thread) you'll ensure that Scapy will get the time it needs to do so.

To see how we came to this conclusion see the discussion above in the comment section. Gl Asiri Rathnayake :)

python - Scapy fails to sniff packets when using multiple threads

1 Answers