I think that my issue is similar to this question which never got answered. In my case, I have a large, multi-threaded server application that communicates using UDP. Every client that connects to the server gets its own UDP socket and thread. In one particular use-case, it's very advantageous to have all of these sockets bound to the same local address and port, but connected to different destination addresses. On macOS and Linux, this works great, but it doesn't work on Windows.
The easiest way to illustrate the problem is to compare it to TCP. Let's assume that every socket is a 5-tuple (protocol, src_addr, src_port, dst_addr, dst_port). If I call listen() on port 5000 and then do an accept(), I wind up with something like the following situation:
- listen socket:
(TCP, 0.0.0.0, 5000, 0.0.0.0, 0) - accept socket:
(TCP, 10.1.1.1, 5000, 10.2.2.2, 12345)
which can be confirmed with netstat (this one's on a mac, but Windows looks similar):
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.1.220.5000 192.168.1.220.65282 ESTABLISHED
tcp4 0 0 *.5000 *.* LISTEN
When incoming packets are received, they are given to the "best" matching socket and fall back to the listen socket if there's no better match. This happens on all platforms.
To do the same thing with UDP, I create the "listen" socket by setting SO_REUSEPORT or SO_REUSEADDR (depending on the platform), then calling bind(). The "accept" socket can be made using SO_REUSEPORT, bind(), then calling connect() to set its destination address. Which gives a similar situation:
- "listen" socket:
(UDP, 0.0.0.0, 5000, 0.0.0.0, 0) - "accept" socket:
(UDP, 10.1.1.1, 5000, 10.2.2.2, 12345)
again confirmed by netstat (on a mac):
Proto Recv-Q Send-Q Local Address Foreign Address (state)
udp4 0 0 10.211.55.2.5000 10.211.55.2.6666
udp4 0 0 *.5000 *.*
On macOS and Linux, incoming packets go to the "best" socket just like with TCP. But on Windows, packets are always dispatched to the first socket, completely ignoring the connect(). The call to connect() really did bind to both a local and foreign address (verified with getsockname() and getpeername()), but even netstat thinks the 2 sockets are the same:
Proto Local Address Foreign Address State
UDP 0.0.0.0:5000 *:*
UDP 0.0.0.0:5000 *:*
I find this strange since it works on other platforms, it works with TCP, and netstat doesn't show the real bound addresses. It's almost as if connect() is doing filtering at a higher level and it's never making its way down to the network drivers.
I realize that I could just open a single socket and handle the dispatching myself, but it's a pain since there are lots of threads that can currently act independently and that would add a single sync point, as well as adding an extra layer of buffering. Ideally there would be an easy way to convince Windows to just treat UDP dispatching the same way as everything else.
Here is a minimal working example that compiles and runs on macOS (clang/llvm), Linux (gcc), and Windows (Visual Studio). Error handling has been omitted for brevity. Note that I'm explicitly using an external address since localhost is special and isn't relevant to my use-case. The sample code creates 2 sockets on local port 9999, one of which is also connected to a hardcoded IP address on port 6666. I then send a packet to each socket using something like:
echo 1 | nc -w 1 -u 10.211.55.9 9999 && echo 2 | nc -w 1 -p 6666 -u 10.211.55.9 9999
On Linux and macOS this gives:
Socket listen got packet [49] from [0ad33702:56651].
Socket accept got packet [50] from [0ad33702:6666].
but on Windows I get:
Socket listen got packet [49] from [0ad33702:58940].
Socket listen got packet [50] from [0ad33702:6666].
Here's the C program. There a bit of boiler-plate, but main() should hopefully be self-explanatory:
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#ifdef _WIN32
#include <ws2tcpip.h>
typedef SOCKET SocketHandle;
static int poll(struct pollfd *fds, unsigned long nfds, int timeout) { return WSAPoll(fds, nfds, timeout); } // rename WSAPoll to poll
#pragma comment(lib, "Ws2_32.lib")
#else
#include <netinet/in.h>
#include <poll.h>
#include <sys/socket.h>
typedef int SocketHandle;
#endif
// Explicitly send data to an external IP address, not localhost. In this case 10.211.55.2. Change as needed.
static uint32_t sIpAddr = 10u<<24 | 211u<<16 | 55u<<8 | 2u;
static void setReusePort(SocketHandle s)
{
int trueval = 1;
#ifdef SO_REUSEPORT
setsockopt(s, SOL_SOCKET, SO_REUSEPORT, (const char*) &trueval, sizeof(trueval));
#else
setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (const char*) &trueval, sizeof(trueval));
#endif
}
static void bindSocket(SocketHandle s, uint32_t addr, uint16_t port)
{
struct sockaddr_in sourceAddr;
memset(&sourceAddr, 0, sizeof(sourceAddr));
sourceAddr.sin_family = AF_INET;
sourceAddr.sin_addr.s_addr = htonl(addr);
sourceAddr.sin_port = htons(port);
#ifdef __APPLE__
sourceAddr.sin_len = sizeof(sourceAddr);
#endif
bind(s, (const struct sockaddr*) &sourceAddr, sizeof(sourceAddr));
}
static void connectSocket(SocketHandle s, uint32_t addr, uint16_t port)
{
struct sockaddr_in peerAddr;
memset(&peerAddr, 0, sizeof(peerAddr));
peerAddr.sin_family = AF_INET;
peerAddr.sin_addr.s_addr = htonl(addr);
peerAddr.sin_port = htons(port);
#ifdef __APPLE__
peerAddr.sin_len = sizeof(addr);
#endif
connect(s, (const struct sockaddr*) &peerAddr, sizeof(peerAddr));
}
static void receive(SocketHandle s, const char* name)
{
struct sockaddr_in retAddr;
memset(&retAddr, 0, sizeof(retAddr));
retAddr.sin_family = AF_INET;
#ifdef __APPLE__
retAddr.sin_len = sizeof(retAddr);
#endif
socklen_t retAddrLen = sizeof(retAddr);
// Recv up to 10 packets over 10 seconds.
for (int i = 0; i < 10; ++i) {
struct pollfd pollSet = {s, POLLRDNORM, 0};
int r = poll(&pollSet, 1, 1000);
if (1 == r && POLLRDNORM == (pollSet.revents & POLLRDNORM)) {
char data[256];
if (recvfrom(s, data, (int) sizeof(data), 0, (struct sockaddr*) &retAddr, &retAddrLen) > 0) {
printf("Socket %s got packet [%u] from [%8.8x:%u].\n", name, (unsigned int)data[0], ntohl(retAddr.sin_addr.s_addr), ntohs(retAddr.sin_port));
}
}
}
}
int main()
{
#ifdef _WIN32
WSADATA data;
WSAStartup(MAKEWORD(2, 2), &data);
#endif
// Create a UDP socket on port 9999 (essentially the "listen" socket).
SocketHandle s = socket(AF_INET, SOCK_DGRAM, 0);
setReusePort(s);
bindSocket(s, INADDR_ANY, 9999);
// At this point, all incoming packets on port 9999 are delivered to `s`.
// Create another UDP socket on port 9999 (will be the "accept" socket).
SocketHandle s2 = socket(AF_INET, SOCK_DGRAM, 0);
setReusePort(s2);
bindSocket(s2, INADDR_ANY, 9999);
// At this point, all incoming packets on port 9999 are still delivered to `s`.
// Restrict `s2` to only communicate with a particular address ("accept").
connectSocket(s2, sIpAddr, 6666);
// At this point, things are different between platforms:
// Linux/BSD: packets from 10.211.55.2:6666 are delivered to `s2`, everything else to `s`
// Windows: all packets still delivered to `s`
receive(s, "listen");
receive(s2, "accept");
return 0;
}
Does anyone know how Windows handles this dispatching and if there a setsockopt or WSAIoctl or similar to work around it?
