Python pySerial read data from arduino breaks when sending "(char)0"

Question

I send some data from an arduino using pySerial.

My Data looks like

bytearray(DST, SRC, STATUS, TYPE, CHANNEL, DATA..., SIZEOFDATA)

where sizeofData is a test that all bytes are received.

The problem is, every time when a byte is zero, my python program just stops reading there:

serial_port = serial.Serial("/dev/ttyUSB0")
while serial_port.isOpen():
  response_header_str = serial_port.readline()
  format = '>';
  format += ('B'*len(response_header_str));
  response_header = struct.unpack(format, response_header_str)
  pprint(response_header)
serial_port.close()

For example, when I send bytearray(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15) everything is fine. But when I send something like bytearray(1,2,3,4,0,1,2,3,4) I don't see everything beginning with the zero.

The problem is that I cannot avoid sending zeros as I am just sending the "memory dump" e.g. when I send a float value, there might be zero bytes.

how can I tell pyserial not to ignore zero bytes.

Brett Lempereur Brett Lempereur · Accepted Answer · 2014-09-19T12:55:51

I've looked through the source of PySerial and the problem is in PySerial's implementation of FileLike.readline (in http://svn.code.sf.net/p/pyserial/code/trunk/pyserial/serial/serialutil.py). The offending function is:

def readline(self, size=None, eol=LF):
    """\
    Read a line which is terminated with end-of-line (eol) character
    ('\n' by default) or until timeout.
    """
    leneol = len(eol)
    line = bytearray()
    while True:
        c = self.read(1)
        if c:
            line += c
            if line[-leneol:] == eol:
                break
            if size is not None and len(line) >= size:
                break
        else:
            break
    return bytes(line)

With the obvious problem being the if c: line. When c == b'\x00' this evaluates to false, and the routine breaks out of the read loop. The easiest thing to do would be to reimplement this yourself as something like:

def readline(port, size=None, eol="\n"):
    """\
    Read a line which is terminated with end-of-line (eol) character
    ('\n' by default) or until timeout.
    """
    leneol = len(eol)
    line = bytearray()
    while True:
        line += port.read(1)
        if line[-leneol:] == eol:
            break
        if size is not None and len(line) >= size:
            break
    return bytes(line)

To clarify from your comments, this is a replacement for the Serial.readline method that will consume null-bytes and add them to the returned string until it hits the eol character, which we define here as "\n".

An example of using the new method, with a file-object substituted for the socket:

>>> # Create some example data terminated by a newline containing nulls.
>>> handle = open("test.dat", "wb")
>>> handle.write(b"hell\x00o, w\x00rld\n")
>>> handle.close()
>>>
>>> # Use our readline method to read it back in.
>>> handle = open("test.dat", "rb") 
>>> readline(handle)
'hell\x00o, w\x00rld\n'

Hopefully this makes a little more sense.

Python pySerial read data from arduino breaks when sending "(char)0"

1 Answers