How to make setsockopt IP_ADD_MEMBERSHIP honor the local interface address to receive multicast packets on only one specific interface?

Question

Question

How to make setsockopt IP_ADD_MEMBERSHIP honor the local interface address to receive multicast packets on only one specific interface?

Summary

In a single Python process, I create three receive sockets (sockets S1, S2, and S3).

Each socket joins the same multicast group (same multicast address, same port) using setsockopt IP_ADD_MEMBERSHIP.

However, each socket joins that multicast group on a different local interface by setting imr_address in struct ip_mreq (see http://man7.org/linux/man-pages/man7/ip.7.html) to the address of particular local interface: socket S1 joins on interface I1, socket S2 joins on interface I2, and socket S3 joins on interface I3.

Here is the code for creating a receive socket:

def create_rx_socket(interface_name):
    local_address = interface_ipv4_address(interface_name)
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    try:
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    except AttributeError:
        pass
    try:
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    except AttributeError:
        pass
    sock.bind((MULTICAST_ADDR, MULTICAST_PORT))
    report("join group {} on {} for local address {}".format(MULTICAST_ADDR, interface_name,
                                                             local_address))
    req = struct.pack("=4s4s", socket.inet_aton(MULTICAST_ADDR), socket.inet_aton(local_address))
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, req)
    return sock

where the IP address of the local interface is determined as follows (I used debug prints to verify it obtains the correct address):

def interface_ipv4_address(interface_name):
    interface_addresses = netifaces.interfaces()
    if not interface_name in netifaces.interfaces():
        fatal_error("Interface " + interface_name + " not present.")
    interface_addresses = netifaces.ifaddresses(interface_name)
    if not netifaces.AF_INET in interface_addresses:
        fatal_error("Interface " + interface_name + " has no IPv4 address.")
    return interface_addresses[netifaces.AF_INET][0]['addr']

On the sending socket side, I use setsockopt IP_MULTICAST_IF to choose the outgoing local interface on which outgoing multicast packets are to be sent. This works fine.

def create_tx_socket(interface_name):
    local_address = interface_ipv4_address(interface_name)
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    try:
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    except AttributeError:
        pass
    try:
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    except AttributeError:
        pass
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF, socket.inet_aton(local_address))
    # Disable the loopback of sent multicast packets to listening sockets on the same host. We don't
    # want this to happen because each beacon is listening on the same port and the same multicast
    # address on potentially multiple interfaces. Each receive socket should only receive packets
    # that were sent by the host on the other side of the interface, and not packet that were sent
    # from the same host on a different interface. IP_MULTICAST_IF is enabled by default, so we have
    # to explicitly disable it)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_LOOP, 0)
    sock.bind((local_address, MULTICAST_PORT))
    sock.connect((MULTICAST_ADDR, MULTICAST_PORT))
    return sock

When the Python process receives a single individual multicast packet on (say) interface I1, all three socket (S1, S2, and S3) report a received UDP packet.

The expected behavior is that only socket S1 reports a received packet.

Here is some output from my script, along with some output from tcpdump, to illustrate the problem (<<< are manually added annotations):

beacon3: send beacon3-message-1-to-veth-3-1a on veth-3-1a from ('99.1.3.3', 911) to ('224.0.0.120', 911)     <<< [1]
TX veth-3-1a: 15:49:13.355519 IP 99.1.3.3.911 > 224.0.0.120.911: UDP, length 30   <<< [2]
RX veth-1-3a: 15:49:13.355558 IP 99.1.3.3.911 > 224.0.0.120.911: UDP, length 30   <<< [3]
beacon1: received beacon3-message-1-to-veth-3-1a on veth-1-2 from     99.1.3.3:911   <<< [4a]
beacon1: received beacon3-message-1-to-veth-3-1a on veth-1-3a from 99.1.3.3:911   <<< [4b]
beacon1: received beacon3-message-1-to-veth-3-1a on veth-1-3b from 99.1.3.3:911   <<< [4c]

A single call to socket send [1] causes a single packet to be sent on the wire [2] which is received on the wire once [3] but which causes three sockets to wake up and report the received packet [4abc].

The expected behavior is that only one of the receive sockets would wake up and report the received packet, namely the one socket that joined the multicast group on the interface

Topology details

I have three Python scripts (1, 2, and 3) that are connected in the following network topology:

              veth-1-2      veth-2-1
          (1)------------------------(2)
          | |                         |
veth-1-3a | | veth-1-3b               | veth-2-3
          | |                         |
          | |                         | 
veth-3-1a | | veth-3-1b               |
          | |                         |
          (3)-------------------------+
              veth-3-2

Each Python script is running in a network namespace (netns-1, netns-2, netns-3).

The network namespaces are connected to each other using veth interface pairs (veth-x-y).

Each Python process periodically sends a UDP multicast packet on each of its directly connected interfaces.

All UDP packets sent by any Python process on any interface always uses the same destination multicast address (224.0.0.120) and the same destination port (911).

Update:

I noticed the following comment in function ospf_rx_hook in file packet.c in BIRD (https://github.com/BIRD/bird):

int
ospf_rx_hook(sock *sk, uint len)
{
  /* We want just packets from sk->iface. Unfortunately, on BSD we cannot filter
     out other packets at kernel level and we receive all packets on all sockets */
  if (sk->lifindex != sk->iface->index)
    return 1;

  DBG("OSPF: RX hook called (iface %s, src %I, dst %I)\n",
      sk->iface->name, sk->faddr, sk->laddr);

Apparently BIRD has the same issue (at least on BSD) and solves it by checking which interface a received packet arrived on, and ignoring it when it is not the expected interface.

Update:

If the BIRD comment is true, then the question becomes: how can I determine which interface a packet was received on for a received UDP packet?

Some options:

• Use IP_PKTINFO (http://man7.org/linux/man-pages/man7/ip.7.html and comments in Python socket bind 0.0.0.0, inspect incoming message). It appears that this may not be supported on all operating systems and/or in Python.

• Check whether the source address is on the same subnet as the interface. This only works if the packets are sent over one hop which is indeed the case in my scenario. But does it work for IPv6 (yes, I think so) and on unnumbered interfaces (clearly not, but I wont support those)?

Some interesting observations on BIRD:

• BIRD uses raw sockets for OSPF (because OSPF runs directly over IP, not over UDP) and in function sk_setup (file io.c) it calls sk_request_cmsg4_pktinfo which on Linux set the IP_PKTINFO socket option and on BSD sets the IP_RCVIF socket option.

• BIRD has a macro INIT_MREQ4 to initialize the request for IP_ADD_MEMBERSHIP. Interestingly, for Linux it sets the multicast address (imr_multiaddr) and the local interface index (imr_ifindex) whereas for BSD it sets the multicast address (imr_multiaddr) and the local interface address (imr_interface)

Answer 1

I am "answering" my own question by documenting the work-around that I ended up using.

Note that this is only a work-around. In this work-around we detect packets that are received on the "wrong" socket and ignore them. If someone has a "real" solution (i.e. one that avoids the packets being received on the wrong socket in the first place), I would love to hear it.

Let's say that we join a multicast group when we create a receive socket as follows:

req = struct.pack("=4s4s", socket.inet_aton(MULTICAST_ADDR), socket.inet_aton(local_address))
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, req)

Here, MULTICAST_ADDR is the multicast address and local_address is the IP address of the local interface on which the group is joined.

Apparently, what this really means is that IP multicast packets received on that interface will be accepted and passed on to the higher layer, which is UDP in this case.

Note, however, that if there are multiple UDP sockets S1, S2, S3, ... that have joined the same multicast group (same multicast address and same port) on different interfaces I1, I2, I3, ... then if we receive a multicast packet on ANY of the interfaces I1, I2, I3... then ALL the sockets S1, S2, S3, ... will be notified that a packet has been received.

My work-around works as follows:

(1) For each socket on which we join the mulitcast group, keep track of the if_index for the interface on which we did the join, and

interface_index = socket.if_nametoindex(interface_name)

(2) Use the IP_PKTINFO socket option to determine on which interface index the packet was actually received,

To set the option:

sock.setsockopt(socket.SOL_IP, socket.IP_PKTINFO, 1)

To retrieve the if_index from the PKTINFO ancillary data when the packet is received:

rx_interface_index = None
for anc in ancillary_messages:
    if anc[0] == socket.SOL_IP and anc[1] == socket.IP_PKTINFO:
        packet_info = in_pktinfo.from_buffer_copy(anc[2])
        rx_interface_index = packet_info.ipi_ifindex

(3) Ignore the received packet if the if_index of the socket does not match the if_index on which the packet was received.

A complicating factor is that neither IP_PKTINFO nor in_pktinfo are part of the standard Python socket module, so both must be manually defined:

SYMBOLS = {
    'IP_PKTINFO': 8,
    'IP_TRANSPARENT': 19,
    'SOL_IPV6': 41,
    'IPV6_RECVPKTINFO': 49,
    'IPV6_PKTINFO': 50
}

uint32_t = ctypes.c_uint32

in_addr_t = uint32_t

class in_addr(ctypes.Structure):
    _fields_ = [('s_addr', in_addr_t)]

class in6_addr_U(ctypes.Union):
    _fields_ = [
        ('__u6_addr8', ctypes.c_uint8 * 16),
        ('__u6_addr16', ctypes.c_uint16 * 8),
        ('__u6_addr32', ctypes.c_uint32 * 4),
    ]

class in6_addr(ctypes.Structure):
    _fields_ = [
        ('__in6_u', in6_addr_U),
    ]

class in_pktinfo(ctypes.Structure):
    _fields_ = [
        ('ipi_ifindex', ctypes.c_int),
        ('ipi_spec_dst', in_addr),
        ('ipi_addr', in_addr),
    ]

for symbol in SYMBOLS:
    if not hasattr(socket, symbol):
        setattr(socket, symbol, SYMBOLS[symbol])

For a complete working example and for important information about the copyright and 2-clause BSD-license for the source of this code see file https://github.com/brunorijsman/python-multicast-experiments/blob/master/beacon.py