Speeding up UDP-based file transfer with loss protection?

Question

I'm trying to learn UDP, and make a simple file transferring server and client. I know TCP would potentially be better, because it has some reliability built in. However I would like to implement some basic reliability code myself.

I've decided to try and identify when packets are lost, and resend them.

What I've implemented is a system where the server will send the client a certain file in 10 byte chunks. After it sends each chunk, it waits for an acknowledgement. If it doesn't receive one in a few seconds time, it sends the chunk again.

My question is how can a file transfer like this be done quickly? If you send a file, and lets say theirs 25% chance a packet could be lost, then there will be a lot of time built up waiting for the ACK.

Is there some way around this? Or is it accepted that with high packet loss, it will take a very long time? Whats an accepted time-out value for the acknowledgement?

Thanks!

Answer 1

There are many questions in your post, I will try to address some. The main thing is to benchmark and find the bottleneck. What is the slowest operation?

I can tell you now that the bottleneck in your approach is waiting for an ACK after each chunk. Instead of acknowledging chunks, you want to acknowledge sequences. The second biggest problem is the ridiculously small chunk. At that size there's more overhead than actual data (look up the header sizes for IP and UDP).

In conclusion:

What I've implemented is a system where the server will send the client a certain file in 10 byte chunks.

You might want to try a few hundred bytes chunks.

After it sends each chunk, it waits for an acknowledgement.

Send more chunks before requiring an acknowledgement, and label them. There is more than one way:

• Instead of acknowledging chunks, acknowledge data: "I've received 5000 bytes" (TCP, traditional)
• Acknowledge multiple chunks in one message. "I've received chunks 1, 5, 7, 9" (TCP with SACK)

Answer 2

What you've implemented is Stop-and-wait ARQ. In a high-latency network, it will inevitably be slower than some other more complex options, because it waits for a full cycle on each transmission.

For other possibilities, see Sliding Window and follow links to other variants. What you've got is basically a degenerate form of sliding window with window-size 1.

As other answers have noted, this will involve adding sequence numbers to your packets, sending additional packets while waiting for acknowledgement, and retransmitting on a more complex pattern.

If you do this, you are essentially reinventing TCP, which uses these tactics to supply a reliable connection.

Answer 3

You want some kind of packet numbering, so that the client can detect a lost packet by the missing number in the sequence of received packets. Then the client can request a resend of the packets it knows it is missing.

Example:

Server sends packet 1,2,3,4,5 to client. Client receives 1,4,5, so it knows 2 and 3 were lost. So client acks 1,4 and 5 and requests resend of 2 and 3.

Then you still need to work out how to handle acks / requests for resends, etc. In any case, assigning a sequence of consecutive numbers to the packets so that packet loss can be detected by "gaps" in the sequence is a decent approach to this problem.

Answer 4

Your question exactly describes one of the problems that TCP tries to answer. TCP's answer is particularly elegant and parsimonious, imo, so reading an English-language description of TCP might reward you.

Just to give you a ballpark idea of UDP in the real world: SNMP is a network-management protocol that is meant to operate over UDP. SNMP requests (around 1500 payload bytes) sent by a manager to a managed node are never explicitly acknowledged and it works pretty well. Twenty-five percent packet loss is a huge number -- real-life packet loss is an order of magnitude somaller, at worst -- and, in that broken environment, SNMP would hardly work at all. Certainly a human being operating the network management system -- the NMS -- would be on the phone to network hardware support very quickly.

When we use SNMP, we generally understand that a good value for timeout is three or four seconds, meaning that the SNMP agent in the managed network node will probably have completed its work in that time.

HTH

Answer 5

Have a look at the TFTP protocol. It is a UDP-based file transfer protocol with built-in ack/resend provisions.