Serial port basestream gives partial / scrambled / duplicated data

Question

I am currently writing a simple app that will communicate with arduino based device by a ftdi232 chips (serial port)

I've hit a wall with basestream read - my app works great when testing on virtual ports (com0com), but when i have swithed to ftdi devices received data became scrambled and / or duplicated.

Both sending and receiving ports are configured the same:

19200 baud rate
8 data bits
parity = none
stop bits = 1
dtr and rts are disabled
DiscardNull is enabled.

For now (testing) both ftdi boards are connected to the same machine. Boards are connected by 3 wires (rx, tx [crossed] and ground).

Problematic method:

public async Task StartReceivingAsync()
    {
        _isLissening = true;
        string errorData = string.Empty;
        byte[] mainBuffer = new byte[_completeCommandSizeWithSep];

        while(_port.IsOpen && !_receiveToken.IsCancellationRequested)
        {                
            int bytesRead = await _port.BaseStream.ReadAsync(mainBuffer, 0, _completeCommandSizeWithSep, _receiveToken);                

            string rawData = Encoding.GetEncoding(_port.Encoding.CodePage).GetString(mainBuffer);

            if(_port.Encoding.CodePage == Encoding.ASCII.CodePage)
                _receivedBuffer.Append(RemoveNonAsciiChars(rawData));
            else
                _receivedBuffer.Append(rawData);


            if(_receivedBuffer.Length >= _completeCommandSizeWithSep)
            {
                ICommandModel command = _commandModelFac();

                string workPiece = _receivedBuffer.ToString(0, _completeCommandSizeWithSep);
                int whereToCut = CheckRawData(workPiece);

                if(whereToCut == -1)
                {
                    command.Data = workPiece;
                    _receivedBuffer.Remove(0, _completeCommandSizeWithSep);
                }

                else if(whereToCut > 0)
                {
                    command.Data = _receivedBuffer.ToString(0, whereToCut);
                    _receivedBuffer.Remove(0, whereToCut);
                }                   

                if (whereToCut != 0)                    
                    command.CommandType = CommandType.Error;

                else
                {                        
                    command = CommandTranslator(workPiece);
                    _receivedBuffer.Remove(0, _completeCommandSizeWithSep);                        
                }   
                DataReceived?.Invoke(this, command);
            }
            else
            {
                continue;
            }                
        } 
        _isLissening = false;
    }

Sending method:

while (true)
            {
                serial.Write($@"*{"0015"}*" +  i.ToString().PadLeft(4, '0') + '*' + i.ToString().PadLeft(30, '.'));
                Console.WriteLine(serial.ReadExisting());

                i++;
                Console.WriteLine("...");
                Console.ReadLine();
            }

_completeCommandSizeWithSep is a length of command that should be received.

Command format is *0001*AAAA*123456789123asdfgh12345678584a

Example of data from workPiece variable (almost vanilla, just non - ascii chars removed) when sending data:

Sended data:

*0015*0EF9*.............................0

*0015*2K3R*.............................1

*0015*C80S*.............................2

Received data:

*0015*0EF9*.0EF9*.0EF9*.0EF9*.0EF9*.0EF9*
...F9*...F9*...F9*...F9*...F9*...F9*...F9
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*...F
*...F9*...F9*...F9*...F9*...F9*...F9*0..F
*...F9*...F9*...F9*...F9*...F9*0..F9**..F
*...F9*...F9*...F9*...F9*0..F9**..F9*00.F
*...F9*...F9*...F9*0..F9**..F9*00.F9*10.F
*...F9*...F9*0..F9**..F9*00.F9*10.F9*50.F
*...F9*0..F9**..F9*00.F9*10.F9*50.F9**0.F
*0..F9**..F9*00.F9*10.F9*50.F9**0.F9*00.F
**..F9*00.F9*10.F9*50.F9**0.F9*00.F9*E0.F
*..F9*00.F9*10.F9*50.F9**0.F9*00.F9*E0.F9
*00.F9*10.F9*50.F9**0.F9*00.F9*E0.F9*F0.F
*10.F9*50.F9**0.F9*00.F9*E0.F9*F0.F9*90.F
*50.F9**0.F9*00.F9*E0.F9*F0.F9*90.F9**0.F
**0.F9*00.F9*E0.F9*F0.F9*90.F9**0.F9*.0.F
*0.F9*00.F9*E0.F9*F0.F9*90.F9**0.F9*.0.F9
*00.F9*E0.F9*F0.F9*90.F9**0.F9*.0.F9*...F
*E0.F9*F0.F9*90.F9**0.F9*.0.F9*...F9*....
*F0.F9*90.F9**0.F9*.0.F9*...F9*....9*....
*90.F9**0.F9*.0.F9*...F9*....9*....9*....
**0.F9*.0.F9*...F9*....9*....9*....9*....
*0.F9*.0.F9*...F9*....9*....9*....9*....9
*.0.F9*...F9*....9*....9*....9*....9*....
*...F9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*0...
*....9*....9*....9*....9*....9*0...9**...
*....9*....9*....9*....9*0...9**...9*0...
*....9*....9*....9*0...9**...9*0...9*01..
*....9*....9*0...9**...9*0...9*01..9*51..
*....9*0...9**...9*0...9*01..9*51..9**1..
*0...9**...9*0...9*01..9*51..9**1..9*21..
**...9*0...9*01..9*51..9**1..9*21..9*K1..
*...9*0...9*01..9*51..9**1..9*21..9*K1..9
*0...9*01..9*51..9**1..9*21..9*K1..9*31..
*01..9*51..9**1..9*21..9*K1..9*31..9*R1..
*51..9**1..9*21..9*K1..9*31..9*R1..9**1..
**1..9*21..9*K1..9*31..9*R1..9**1..9*.1..
*1..9*21..9*K1..9*31..9*R1..9**1..9*.1..9
*21..9*K1..9*31..9*R1..9**1..9*.1..9*.1..
*K1..9*31..9*R1..9**1..9*.1..9*.1..9*.1..
*31..9*R1..9**1..9*.1..9*.1..9*.1..9*.1..
*R1..9**1..9*.1..9*.1..9*.1..9*.1..9*....
**1..9*.1..9*.1..9*.1..9*.1..9*....9*....
*1..9*.1..9*.1..9*.1..9*.1..9*....9*....9
*.1..9*.1..9*.1..9*.1..9*....9*....9*....
*.1..9*.1..9*.1..9*....9*....9*....9*....
*.1..9*.1..9*....9*....9*....9*....9*....
*.1..9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*1...
*....9*....9*....9*....9*....9*1...9**...
*....9*....9*....9*....9*1...9**...9*0...
*....9*....9*....9*1...9**...9*0...9*0...
*....9*....9*1...9**...9*0...9*0...9*1...
*....9*1...9**...9*0...9*0...9*1...9*5...
*1...9**...9*0...9*0...9*1...9*5...9**...
**...9*0...9*0...9*1...9*5...9**...9*2...
*...9*0...9*0...9*1...9*5...9**...9*2...9
*0...9*0...9*1...9*5...9**...9*2...9*K...
*0...9*1...9*5...9**...9*2...9*K...9*3...
*1...9*5...9**...9*2...9*K...9*3...9*R*..
*5...9**...9*2...9*K...9*3...9*R*..9*.*..
**...9*2...9*K...9*3...9*R*..9*.*..9*.*..
*...9*2...9*K...9*3...9*R*..9*.*..9*.*..9
*2...9*K...9*3...9*R*..9*.*..9*.*..9*.*..
*K...9*3...9*R*..9*.*..9*.*..9*.*..9*.*..
*3...9*R*..9*.*..9*.*..9*.*..9*.*..9*.*..
*R*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..
*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*
*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..
*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*
*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..
*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*
*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..
*..9*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*
*.*..9*.*..9*.*..9*.*..9*.*..9*.*..9*....
*..9*.*..9*.*..9*.*..9*.*..9*.*..9*....9*
*.*..9*.*..9*.*..9*.*..9*.*..9*....9*....
*..9*.*..9*.*..9*.*..9*.*..9*....9*....9*
*.*..9*.*..9*.*..9*.*..9*....9*....9*....
*..9*.*..9*.*..9*.*..9*....9*....9*....9*
*.*..9*.*..9*.*..9*....9*....9*....9*....
*..9*.*..9*.*..9*....9*....9*....9*....9*
*.*..9*.*..9*....9*....9*....9*....9*....
*..9*.*..9*....9*....9*....9*....9*....9*
*.*..9*....9*....9*....9*....9*....9*....
*..9*....9*....9*....9*....9*....9*....9*
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*....
*....9*....9*....9*....9*....9*....9*1...
*....9*....9*....9*....9*....9*1...9**...
*....9*....9*....9*....9*1...9**...9*0...
*....9*....9*....9*1...9**...9*0...9*01..
*....9*....9*1...9**...9*0...9*01..9*5*..
*....9*1...9**...9*0...9*01..9*5*..9*C8..
*1...9**...9*0...9*01..9*5*..9*C8..9*08..
**...9*0...9*01..9*5*..9*C8..9*08..9*S8..
*...9*0...9*01..9*5*..9*C8..9*08..9*S8..9
*0...9*01..9*5*..9*C8..9*08..9*S8..9**...
*01..9*5*..9*C8..9*08..9*S8..9**...9*....
*5*..9*C8..9*08..9*S8..9**...9*....9*....
*..9*C8..9*08..9*S8..9**...9*....9*....9*
*C8..9*08..9*S8..9**...9*....9*....9*....
*08..9*S8..9**...9*....9*....9*....9*....
*S8..9**...9*....9*....9*....9*....9*....
**...9*....9*....9*....9*....9*..........
*...9*....9*....9*....9*....9*...........
*....9*....9*....9*....9*................
*....9*....9*....9*......................
*....9*....9*............................
*....9*..................................
*........................................
....................2*....0*....015...*C5

Additionally - data from port listener:

1619781: 2019-09-21 23:37:52,7484713 +0,2099872

 2A 30 30 31 35 2A 30 45 46 39 2A 2E 2E 2E 2E 2E   *0015*0EF9*.....
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2E 2E 2E 2E 2E 2E 2E 2E 30 2A 30 30 31 35 2A 30   ........0*0015*0
 45 46 39 2A 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   EF9*............
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2E 30                                             .0
1620349: 2019-09-21 23:37:54,4023089 +0,3609614

 2A 30 30 31 35 2A 32 4B 33 52 2A 2E 2E 2E 2E 2E   *0015*2K3R*.....
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2E 2E 2E 2E 2E 2E 2E 2E 31 2A 30 30 31 35 2A 32   ........1*0015*2
 4B 33 52 2A 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   K3R*............
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2E 31                                             .1
1620937: 2019-09-21 23:37:54,8706713 +0,3255464

 2A 30 30 31 35 2A 43 38 30 53 2A 2E 2E 2E 2E 2E   *0015*C80S*.....
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2E 2E 2E 2E 2E 2E 2E 2E 32 2A 30 30 31 35 2A 43   ........2*0015*C
 38 30 53 2A 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   80S*............
 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E 2E   ................
 2

The _receivedBuffer variable is currently defined outside the StartReceivingAsync method. Could it be getting shared between two or more concurrent tasks? Have you tried pulling it inside the method? — AlwaysLearning
Yes, there was no difference unfortunately. The StringBuilder _receivedBuffer is accessed in a single task only - writing data is handled separately with different set of variables. There is also no other helper method that could access _receivedBuffer concurrently. — quain

quain quain · Accepted Answer · 2019-09-22T11:26:29

I have found a solution!

In StartReceivingAsync method byte[] mainBuffer is initialized once.
After the initialization comes a 'while...' loop that does actual stream reading of data.

I do not know the exact specification for FTDI chips, but it does send data in a rather random parts when sent constantly (no timeouts in senders and receivers)

So, when executing
int bytesRead = await _port.BaseStream.ReadAsync(mainBuffer, 0, _completeCommandSizeWithSep, _receiveToken);
I would get a random portion of data sent. That data would be, as intended, written to mainBuffer byte array, which in turn, would be added to _receivedBuffer.

And here comes the thing: mainBuffer is not beeing cleaned when stream received portion of data that is smaller than _completeCommandSizeWithSep.
BaseStream reader just replaces as many chars in mainBuffer from 0 index as were received, ignoring rest.

The fix is to just add a substring of length equal to bytesRead value instead of whole mainBuffer to received buffer:

int bytesRead = await _port.BaseStream.ReadAsync(mainBuffer, 0, _completeCommandSizeWithSep, _receiveToken);

Bad code:
string rawData = Encoding.GetEncoding(_port.Encoding.CodePage).GetString(mainBuffer);

Working code:
string rawData = Encoding.GetEncoding(_port.Encoding.CodePage).GetString(mainBuffer).Substring(0, bytesRead);

Alternate solution would be to reinitialize mainBuffer array inside While loop.

Serial port basestream gives partial / scrambled / duplicated data

1 Answers