Linux UDP Socket/Port Reuse

Question

I'm in the middle of experimenting with Linux UDP sockets. I have a server and client that are talking to one another in the following way.

Server: Sends broadcast announcement on port X to all IPs on it's subnet.

Client: Responds to the announcement by sending a unicast message back to the server. This message gets sent back directly to port X.

On the server, I have one thread that periodically sends the announcement. I now want to add more code to receive the response from the client as well as send unicast messages to a particular client. I have my proposed scheme for the server outlined below.

Thread 1

• Broadcast on socket_a, which is set up using setsockopt() to set SO_BROADCAST. Sends broadcast announcements on port X.

Thread 2

• Reads in data from port X, using socket_b. Due to limitations with the code's current structure, this thread does not have access to socket_a. Note that the port is shared with socket_a. bind() is called to read in data on this socket.

Thread 3

• Sends unicast data to particular clients using socket_b on port X. Note that both the socket and port are shared with thread 2, and the port is common across all three threads and the two sockets.

In no particular order, here are my questions (bear with me, I'm fairly new to unix socket programming):

1. Do I need to do any locking between the two sockets or is that handled in the OS? That is, can I send data concurrently to both sockets and not get some weird collisions since both use the same port?

2. Do I need to worry about Thread 2 receiving data that is sent out by either of the "sender" threads? How do I prevent this?

3. Are there any options that I need to enable to make this work? Does it matter if the broadcast socket (socket_a) gets set up prior to the unicast one (socket_b)?

4. Should I create a dedicated "receive" socket on port X? That is, would it make sense to have a socket that is solely used to receive data? This is the only socket that would have bind() called on it.

5. Are there any other issues I should be worried about given that all three threads are using the same port? Unfortunately, this is a constraint that I am working with. If the above scheme is impossible, I may have to revisit my one port design.

Thank you in advance for your help.

Answer 1

Do I need to do any locking between the two sockets or is that handled in the OS? That is, can I send data concurrently to both sockets and not get some weird collisions since both use the same port?

For UDP, locking isn't necessary. That said, you may still find it easier to simply create a separate socket for each thread, if only because safely co-ordinating the shutdown and destruction of a socket that is in use by multiple threads can be a bit tricky.

Do I need to worry about Thread 2 receiving data that is sent out by either of the "sender" threads? How do I prevent this?

Yes, Thread 2 is likely to see the broadcast packets at least, and possibly the unicast packets as well (if they ever get sent to the computer that thread 2 is running on). The easiest way to avoid that would be to listen on a different port instead (e.g. send your broadcast packets to port X, but have the clients send their unicast reply packets back to port X+1; that way any packet you receive on port X+1 is almost certainly a unicast reply from a client)

Are there any options that I need to enable to make this work?

Just the SO_BROADCAST one, AFAIK. (if you wanted several sockets on the same machine to listen to the same port, you'd also need to do SO_REUSEADDR and [under MacOS/X] SO_REUSEPORT, but hopefully you don't need that)

Does it matter if the broadcast socket (socket_a) gets set up prior to the unicast one (socket_b)?

No, the ordering should not matter.

Should I create a dedicated "receive" socket on port X? That is, would it make sense to have a socket that is solely used to receive data? This is the only socket that would have bind() called on it.

You could do that, but having the same socket send and receive will work as well. (Note that if you use blocking I/O you have to watch out for things like recv() not returning for a long time, which can prevent your thread from doing anything else like sending in the meantime -- which is why I almost always use non-blocking I/O and select() in my programs. That way it's possible to do everything I want using a single thread, which allows me to avoid all of the non-determinism and potential race conditions and deadlocks that come with multiple threads)

Are there any other issues I should be worried about given that all three threads are using the same port? Unfortunately, this is a constraint that I am working with. If the above scheme is impossible, I may have to revisit my one port design.

Using a single port will work, just be prepared to receive your own broadcast packets back on it (i.e. don't go into a feedback loop when that happens).

Answer 2

I don't think this will work. When you bind UDP sockets multiple times to the same ip+port, packets destined for this ip+port will be delivered to ANY of the sockets. So the unicast replies from the peer to your ip+port might be delivered to any of the three sockets. When locking at such a packet this behavior is quite logical: from the view of the OS the udp packet contains a source ip+port and a destination ip+port. And because all your sockets just bind to the destination ip+port and nothing more, any of the sockets might get the reply at the end.

So you probably need to restructure your application around a single socket, which will be used for unicast and broadcast and which will receive all replies and handle them. If you insist on using multiple threads you can use the same socket to send from all threads without locking, but you should only receive in one thread. In this thread you need to determine the kind of packet and then you can "give" it to the right thread.