client does not receive all messages if server sends messages too quickly with pickle python

Question

My client side cannot recv the two messages if the sender sends too quickly.

sender.py

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


sock.bind(('', int(port)))
sock.listen(1)

conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))

Where both message 1 and message 2 are pickled objects.

client.py

sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))

message1 = pickle.loads(sock.recv(1024))

print(message1)

message2 = pickle.loads(sock.recv(1024))

When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.

Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages

Steffen Ullrich Steffen Ullrich · Accepted Answer · 2018-11-10T05:03:21

Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.

The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.

client does not receive all messages if server sends messages too quickly with pickle python

1 Answers