Reading serial data in realtime in Python

39
votes

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.

The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)

Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)

On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)

The problem:

  • When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
  • When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.

Here is the code (I omitted most part of the code, but the loop is the same):

ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
  data_raw = ser.readline()
  print(data_raw)

Anyone knows why pySerial takes so much time to read from the serial port till the end of the line? Any help?

I want to have this in real time.

Thank you

4
pythonpython-2.7serial-portpyserial
Can you try using ser.read() instead of ser.readline()? – Tim
I used ser.read(), to read while a \n was not received, adding character by character to the string as they were received, still the same speed. – Vasco Baptista
Don't wait until a \n is received - just print out each character as it arrives. Do you get a sudden flood of characters at once or does each individual character arrive on its own? – Tim
since i was doing ser.read() and then print() i was getting one character per line – Vasco Baptista
@VascoBaptista With the accepted solution (to remove timeout and use inWaiting()), it could still be possible that the Python had to wait for all bytes to be received before printing out the entire message (multi-bytes). This would logically cause a delay. Was your system still real-time Can you confirm it ? – Pe Dro

4 Answers

39
votes

You can use inWaiting() to get the amount of bytes available at the input queue.

Then you can use read() to read the bytes, something like that:

While True:
    bytesToRead = ser.inWaiting()
    ser.read(bytesToRead)

Why not to use readline() at this case from Docs:

Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.

You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

5
votes

You need to set the timeout to "None" when you open the serial port:

ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False)

This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end: line = ser.readline()

Once you have the data, it will return ASAP.

3
votes

From the manual:

Possible values for the parameter timeout: … x set timeout to x seconds

and

readlines(sizehint=None, eol='\n') Read a list of lines, until timeout. sizehint is ignored and only present for API compatibility with built-in File objects.

Note that this function only returns on a timeout.

So your readlines will return at most every 2 seconds. Use read() as Tim suggested.

3
votes

A very good solution to this can be found here:

Here's a class that serves as a wrapper to a pyserial object. It allows you to read lines without 100% CPU. It does not contain any timeout logic. If a timeout occurs, self.s.read(i) returns an empty string and you might want to throw an exception to indicate the timeout.

It is also supposed to be fast according to the author:

The code below gives me 790 kB/sec while replacing the code with pyserial's readline method gives me just 170kB/sec.

class ReadLine:
    def __init__(self, s):
        self.buf = bytearray()
        self.s = s

    def readline(self):
        i = self.buf.find(b"\n")
        if i >= 0:
            r = self.buf[:i+1]
            self.buf = self.buf[i+1:]
            return r
        while True:
            i = max(1, min(2048, self.s.in_waiting))
            data = self.s.read(i)
            i = data.find(b"\n")
            if i >= 0:
                r = self.buf + data[:i+1]
                self.buf[0:] = data[i+1:]
                return r
            else:
                self.buf.extend(data)

ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)

while True:

    print(rl.readline())