How to parse indeterminate length packet from MCU UART?

Question

I am attempting to attach a new microcontroller to an existing OSC (Open Sound Control) network via an ethernet-to-serial converter.

The OSC messages are sent with the following formatting:

• OSC Address Pattern: /string/optional substring --> always 32-bit packed/DWORD aligned with extra NULL bytes at the end
• Type Tags: int32 (i), float32 (f), string (s)
• Data: for certain patterns, this is a known length; for others, it is indeterminate in length

I'm a bit confused as to how best to divide up the code and how to buffer this. In my UART receive ISR, I was hoping to just dump the incoming bytes to a SRAM buffer and then decode them in the main loop, but I'm not certain how to do this with indeterminate length packets. I've never worked with a variable length packet that didn't indicate the packet size in the header somewhere. There also doesn't appear to be any EOP/footer to indicate the completion of the packet and you are left with decoding x number of bytes based on the type tag.

Some concerns: 1. How to size the buffer? Yes, I could do this dynamically, but I'd prefer to avoid that overhead, if possible. That leads me towards a circular buffer, but how do I pull out what I need to in the main loop while continuing to decode the variable length packets? 2. How much of the decode should be done in the ISR? Do I set up some type of state machine in the main loop that advances based on the ISR decode of the incoming bytes and then flush those bytes as I advance the FSM?

Looking for any advice/guidance on how best to tackle this problem/architect the code. Thank you in advance.

Answer 1

The UART receive ISR should simply read the incoming byte from the UART and copy it to a circular buffer. I wouldn't do any decoding/parsing/interpretation of the bytes in the ISR. The circular buffer should be large enough to contain the largest message you expect to receive (or larger if bursts of messages can arrive faster than your main loop can handle them).

The main loop should use a state machine to parse the byte stream into a message. The states could be named READ_ADDRESS, READ_TYPE, and READ_DATA. Or you may want to have a unique READ_DATA state for each message type.

From a high level view, each state should read the appropriate number of bytes from the circular buffer and then advance to the next state. It sounds like the data state handler function will have to be smart enough to calculate the appropriate number of data bytes. If you have to interpret some of the data in order to figure the data length then you might want to consider breaking the READ_DATA state into two states, one for reading enough to figure the length and the other for reading the rest of the data.

When you're designing your state handler functions, consider that the appropriate number of bytes may not have been received into the circular buffer yet. From a low level view, the state handlers should return to the main loop when not enough bytes are available. This allows the main loop to service other parts of the application while the ISR receives more bytes. Then when the your main loop calls the state machine again, the state handler should pick up where it left off.

Answer 2

So let's say you have a protocol where packets are of arbitrary size, but they have some patterns by which they are indentifiable. For some identification techniques, see below, first the reading process.

• Interrupt: It just loads bytes into a circular buffer, incrementing the write pointer (and pushing forth the read pointer when the buffer is full, so it drops bytes in a somewhat clean manner).

• Main program: Polls the buffer. It starts at the current read pointer and seeks whether from that a packet can be identified. If nothing is found, it tries to seek from read pointer + 1, and so on until covering the buffer. The read pointer is left unchanged (this is important). If a packet could be identified, it processes it, and (only then) sets the read pointer at the packet's end, thus discarding its contents from the circular buffer.

Why leaving the read pointer unchanged is important when no packet could be identified? Since as further bytes trickle in, a larger packet might complete later.

When the buffer is full with junk, thus the interrupt is pushing forth the read pointer, you might experience hazards depending on how cautiously you designed the interaction between the IT and the main thread. Usually with any reasonably sane implementation, there shouldn't be too many problems here. The push forth behavior in the IT is more consistent than simply passing the read pointer by the write pointer (thus indirectly discarding the entire buffer).

Usually this approach works fine for just about anything. Once I had a situation where I had packet types which contained a smaller otherwise correct packet for the protocol. Now that was nasty to fix, to get the outer packet these cases! (The above algorithm will fetch the inner packet since it likely arrives sooner)

So what about identification?

• If you have any identifiable header, a synchronization byte or whatever, it is easier to get the beginning (you can bail off fast if from the given start offset in the buffer it doesn't look like a proper header).

• You may need to do a partial processing in the identification (in the main program of course) to figure out the length of the packet. The simplest if there is any direct length indication, but without it, if there is a packet type identification (which implies a length, this seems to be your case) it is nothing much harder.

• If there is any checksum or CRC, it is also a good thing to detect the packet. Use it as part of the identification even if a complex header seems to be right: if it's there correct, then you definitely have a valid packet.

• If there is no checksum or CRC at all, use whatever fixed bits the packet has, or whatever fixed ranges the protocol imposes on certain values within the packet to see whether they match a valid packet.

Using these principles I was able to get around quite wrecked protocols proper, I mean things like which assumed delays between packets with hardly any framing, the delays acting as separators, which ceased to exist after the data travelled through the Internet and / or was fetched by a PC running an operating system (in an operating system environment the interrupt and circular buffer is replaced by the OS' own buffering mechanisms, the principles to detect the packets are the same).