Can't perform TCP-handshake through a NAT between two NICs with SO_BINDTODEVICE

Question

I'm trying to connect my computer to both sides of a NAT (run by OpenWRT) and to establish a TCP connection through the NAT:

I run a DHCP server on my first NIC (eth0, ip address 129.104.0.1) and connect it to the WAN port of the router (ip address 129.104.0.198)
I connect my wifi (wlan0, ip address 192.168.1.119) to the router's SSID behind the NAT

I'm using python and the SO_BINDTODEVICE option to send packet between a server (on eth0) and a client (on wlan0) through the NAT:

For the server:

self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind((str(self.local_ip_addr),self.handler.port))
self.server.setsockopt(socket.SOL_SOCKET,25,self.iface.name+"\0")    
self.server.listen(10)

while self.stopped() is False:
    connect = self.server.accept()[0]
    connect.settimeout(1)
    connect.close()
self.server.close()

For the client:

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, 25, self.iface.name + "\0")
sock.settimeout(1)
try:
    sock.connect((self.dest,self.handler.port))
    sock.close()
expect socket.timeout, socket.error as e:
    return -1

My problem is that the connection times out before. I wiresharked both my interfaces and it seems the problem resides on the client's side:

wlan0 sends a TCP SYN packet to 129.104.0.1
The packet is correctly NATed by the router and is received from 129.104.0.198 by eth0
eth0 replies with a SYN,ACK packet, which is correctly NATed back to wlan0
wlan0 does not understand this SYN,ACK and tries to retransmit the first SYN packet

I'm thinking it might have something to do with the linux-kernel refusing to receive a packet from an address that belongs to the machine but if anyone has a clue it would be of great help!

EDIT: I narrowed it down: it is indeed a kernel issue, the packets sent from eth0 are perceived as "martians" by the kernel because they have a local ip address as source. Setting net.ipv4.conf.all.accept_local=1 did not help, neither did deactivating net.ipv4.conf.all.rp_filter=0.

CaptainCap CaptainCap · Accepted Answer · 2014-06-03T15:52:15

After browsing the kernel sources and adding a lot of KERNEL_WARNING we found where it came from: the linux kernel is configured on certain mainstream distributions (Ubuntu...) to act as a router and drop packets where the source address is suspect in order to prevent spoofing (search "rp_filter" on https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt and RFC3704).

To allow such traffic you have to set some variables on your machine (as root):

sysctl -w net.ipv4.conf.all.accept_local=1
sysctl -w net.ipv4.conf.all.rp_filter=0
sysctl -w net.ipv4.conf.your_nic.rp_filter=0

where your_nic is the network interface receiving the packet. Beware to change both net.ipv4.conf.all.rp_filter and net.ipv4.conf.your_nic.rp_filter, it will not work otherwise (the kernel defaults to the most restrictive setting).

Can't perform TCP-handshake through a NAT between two NICs with SO_BINDTODEVICE

1 Answers