Python Sockets Not Sending Full Data

0
votes

I have a client-server application where the client is on a different box then my server. When sending small packets all the data from the client makes it to the server and vise versa. The client needs to send the server a status report that's 43175 bytes in length and the server needs to send instructions of a similar size. Of the 43175 byte packet only 11711 bytes are making it to the server.

All packets are setup so the first line is the length of the packet in bytes followed by the packet.

import socket, sys

# recv expects the first line to the the integer length of the following packet data
# this would look like
#
# 12
# Hello World!
#

def recv(socket):
    char = ''
    num = ''
    while char != '\n':
        num += char
        char = socket.recv(1)

        if char:
            dummy=1
        else:
            return False
    text = socket.recv(int(num))
    if sys.getsizeof(text) < int(num):
        print("The received packet was not the length the heading specified: " + str(sys.getsizeof(text)) + " ~ " + num)

    return text

"The received packet was not the length the heading specified: 11711 ~ 43175"

The sending process looks like this-

def emit(socket, packet):
    send = str(packet.length()) + "\n" + str(packet)
    socket.send(send)

# packet class in another file

import json, sys

class Packet:
    def __init__(self, name, data):
        self.name = name
        self.data = data

    def length(self):
        return sys.getsizeof(self.__str__())

    def __str__(self):
        obj = {"name":self.name,"data":self.data}
        return json.dumps(obj)

I've also set a linger sockopt on the client and server sockets but it didn't help-

self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((self.host, self.port))

linger_struct = struct.pack("ii", 1, 10)
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, linger_struct)

Why is this happening and what can I do about it? Anyone else had a similar problem?

2
pythonsocketsnetwork-programming
Sockets are byte streams, not message streams. This is documented in the socket documentation, and explained in the Socket Programming HOWTO, and fundamental to TCP. - abarnert
Anyway, you've already done the hard part, designing a protocol; you just need to implement it properly by recv-ing and buffering in a loop—as your edited code does. However, instead of posting your solution as an edit to the question, post it as an answer, and accept your answer (and upvote Joachim's answer if it helped you write your final code but isn't enough on its own). - abarnert
Will do. I try to read those documents but I'm not a smart man. It's not the easiest to follow. - micah

2 Answers

1
votes

When using TCP sockets, you are not guaranteed to receive all that you ask for, you might receive less. You might have to receive in a loop to receive everything.

0
votes

From Joachim's recommendation and other posts found on stack overflow the recv length it only an upper bound to the amount that could be received. recv doesn't wait for that amount of data to be pulled before it continues so it continues short of what I needed. To fix this I keep track of the amount I pulled and while the amount I pulled doesn't equal the expected amount of loop the recv until I get all the data I'm expecting.

Works flawlessly.

Thanks for everyones help.

Edit:

Also changed all sys.getsizeof to len in the code (in recv and packet). sys.getsizeof didn't work properly.

def recv(socket):
    char = ''
    num = ''
    while char != '\n':
        num += char
        char = socket.recv(1)

        if char:
            dummy=1
        else:
            return False

    expectedlength = int(num)
    recvlength = 0

    text = ""
    while expectedlength-recvlength > 0:
        text += socket.recv(expectedlength-recvlength)
        recvlength = len(text)

    return text