Python: Sending large object over UDP

4
votes

I am new to socket programming and recently picked up Python for it. I have a few questions in mind which I can't seems to find a definite answer for.

I am looking into sending data over UDP and have written a simple python script to do just that. Works fine sending small objects (Small pickled objects to be exact) across but how should I handle objects that are too large to be fitted in one UDP packet?

I've thought of first sizing up the object in bytes. Nothing will be done if the object is small enough to be fitted in a UDP packet, but if the object is too huge, the object will then be split up evenly (if possible) into many smaller chunks so that it can be fitted into multiple UDP packets and be sent across to the client. Once the client receive the chunks, the client will reassemble the multiple UDP packets into the original state.

I immediately hit my first brick wall when trying to implement the mentioned above.

From my research done, it doesn't seems like there is any 'effective' way in getting the byte size of an object. This means I am unable to determine if an object is too large to fit in a UDP packet.

What happen if I insist on sending an large object across to the client? Will it get fragmented automatically and be reassembled on the client side or will the packet be dropped by the client?

What is the right way to handle large object over UDP? Keeping in mind that the large object could be a file that is 1GB in size or a byte object that is 25MB in size.

Thanks in advance.

Side Notes:

  • I do understand that UDP packets may not always come in order and therefore I have already implemented countermeasure for it which is to tag a sequence number to the UDP packets sent out to the client.

  • I do understand that there is no assurance that the client will receive all of the UDP packets. I am not concerned about packet loss for now.

  • I do understand that TCP is the right candidate for what I am trying to do but I am focusing on understanding UDP and on how to handle situations where acknowledgement of packets from client is not possible for now.

  • I do understand the usage of pickle is insecure. Will look into it at later stage.

3
pythonsocketsnetworkingudp

3 Answers

3
votes

A UDP packet can be as large as approximately 64k. So if you want to send a file that is larger than that you can fragment yourself into packets of 64k. That is the theoretical maximum. My advice is to use fragments of smaller chunks of 500 bytes.

IP is responsible for fragmentation and reassembly of the packets if you do use 64k packets. Smaller packets of 500 bytes are not likely to be fragmented because the mtu is usually around 1500 bytes. If you use larger packets that are fragmented, IP is going to drop them if one of those fragments is lost.

You are right that using TCP is probably better to use for something like this or even an existing protocol like TFTP. It implements a per packet acking mechanism and sequence numbers just like you did.

1
votes

Most applications dealing with sockets might store data in memory until it is all sent. Bad idea! I have a product application that has to send very large files over the web and have used chunking methods in the past. I just rewrote some of my code in Python ( binfileio ). In my applications, I have sent chuck files to a reserved folder and, once all chunked files were tucked in bed, I reassembled them. I've never trusted sending large files across a wire that could get cut at any time. Hope this helps.

0
votes

Pytho 3.0

Code for Sending Data Through UDP Communication

import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
MESSAGE = "Hello, World!"
print("UDP target IP:",UDP_IP)
print("UDP target port:",UDP_PORT)
print("message:",MESSAGE)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
sock.sendto(bytes(MESSAGE, "utf-8"), (UDP_IP, UDP_PORT))