Windows application wont read all data from serial port

Question

I've created a Windows Form that I intend on using to pull data from a data logging Arduino. Currently, the Arduino has an SD card mounted on it that has a log file containing the data I want to receive. Using the Arduino IDE serial monitor, I was able to test sending and receiving serial data. The Ardunio sketch waits for an 'r' character and once it receives it, it sends all of the data in the log file to the serial monitor. This has been tested and works 100% of the time within the Arduino IDE. I now have a Windows form GUI to allow for reading the data directly to a log file on the PC without having to remove the SD card (there are reasons why this is necessary). I have no issue connecting to the serial port, and it seems like there is no issue reading data, until I get a little ways into the file.

When outputting the read data to a list box, I noticed it would stop way before the end of the file. Assuming this might be a multi-thread issue, I decided to try directly writing to a file instead of updating the GUI. This also failed and upon inspecting the created log file, I noticed that sometimes, it would have data in it and sometimes it wouldn't. Then instead of trying to read the whole file, I began attempting to read only certain amounts of data. I can read up to about 110 lines of data before the program stops being able to do so consistently. When the program fails and I close it, checking the log file shows that I have 2048 characters in the file, which is the write buffer size of the serial port. Changing this number causes the number of characters to match the buffer size. Attempting to flush data after each write to the file causes the number of characters left in the file to be seemingly random instead of matching the buffer size.

VB Code:

Public Class mainForm
    Public Shared logFile As System.IO.StreamWriter
    Public Shadows num As Int16 = 0

    'this is the sub that checks for serial data and is writing the read data to the previously created log file
    Private Sub SerialPort_DataReceived(sender As Object, e As IO.Ports.SerialDataReceivedEventArgs) Handles serialPort.DataReceived
        Dim readData As String = serialPort.ReadLine()
        num += 1
        If readData = "EOF" Then
            logFile.Close()
            MsgBox("Done")
        End If
        logFile.Write(readData)
        If num = 110 Then
            MsgBox("Done")
            serialPort.Close()
            logFile.Close()
        End If
    End Sub

    'this sub allows the user to connect to the correct serial port
    'I'm almost positive the problem does not lie here
    Private Sub BtnConnect_Click(sender As Object, e As EventArgs) Handles btnConnect.Click
        serialPort.Close()
        Dim comPortName As String = "COM" + txtComPort.Text
        serialPort.PortName = comPortName
        Try
            serialPort.Open()
            lstLog.Items.Add("Connected to serial port " + comPortName)
        Catch
            lstLog.Items.Add("Could not connect to serial port " + comPortName)
        End Try
    End Sub

    'this sub creates the log file and sends the 'r' to the Arduino to begin reading the sd card
    Private Sub BtnReadData_Click(sender As Object, e As EventArgs) Handles btnReadData.Click
        logFile = My.Computer.FileSystem.OpenTextFileWriter("C:\Users\AHNEHR\Documents\Docs\test1.txt", True)
        serialPort.WriteLine("r")
    End Sub
End Class

Arduino Sketch:

#include <SdFat.h>
#include <SPI.h>

const int relayPin = 9;
const int sdChip = 10;
SdFat sd;
SdFile dataLog;


void setup() {
  Serial.begin(115200);
  pinMode(relayPin,OUTPUT);
}

void loop() {
  if (Serial.available()) {
    char data = Serial.read();
    if (data == 'r') {
      if (!sd.begin(sdChip)) {
        Serial.println("Failed to Initialize SD Card");
      }
      else Serial.println("SD Card Initialized");
      if (!dataLog.open("log.txt", O_READ)) {         
        Serial.println("Failed to open file.");
      }
      else Serial.println("Opened Log");
      while (dataLog.available()) {
        Serial.write(dataLog.read());
      }
      Serial.println("EOF");
      dataLog.close();
    }
  }
}

The "if readData = EOF" is supposed to check for the end of the file, since the Arduino sketch writes this once the entire log file has been output to serial, but I never made it there, so the other if statement checks the number of times a read has been done and stops if it is 110. For numbers less than 100 I have no problem writing all of the data to the log file, but once I get around 100-110, the program begins to malfunction.

The data in the log file on the SD card is formatted like:
04 Aug 2019 06:57 T:ERRC P:ERRpsi
04 Aug 2019 06:58 T:ERRC P:ERRpsi
And I expect the created log file to match the log file on the sd card directly, but instead I get around 80/90 lines that match, and the program stops working and won't continue to write to the log file. I believe the log file on the sd card has about 900 lines total. My vb skills aren't great, so I'm hoping there's something obvious that I'm missing, but I've tried everything I know and don't know why I can only read some of the serial data. Any help figuring out why this happens and how to fix it is greatly appreciated.

*Edit: I have also made sure the baud rates of the Arduino sketch and the serial port match at 115200. Upon some more testing, it definitely seems as though this is a problem with the serial port and not the file.

Answer 1

Several things, I know you are new to SerialPorts and you made a good first attempt. I would recommend the following:

1. The DataReceived Event fires somewhat randomly, example it could fire mid string so you might only get the EO in one event and the F in another event. To solve this, you need to build a string and rely on a character that lets you know when you have the whole string. This is generally done with a Carriage Return (\r) or a Line Feed (\n).
2. As @gunnerone suggested, you should use the ReadExisting() function as it returns ALL the data in the Data Received buffer, ReadLine may cause you to miss data.
3. Make sure your baud rates match. serialPort.BaudRate = 115200;

Here is some example code to get you started, sorry it's C# but you should be able to translate to VB.

char LF = (char)10;
StringBuilder sb = new StringBuilder();
string currentLine = string.Empty;

private void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
    string Data = serialPort1.ReadExisting();

    foreach (char c in Data)
    {
        if (c == LF)
        {
            sb.Append(c);

            CurrentLine = sb.ToString();
            sb.Clear();

            //parse CurrentLine here or print it to textbox
        }
        else
        {
            sb.Append(c);
        }
    }
}

Couple of things to note:

1. LF is the character I'm looking for in this example to let me know I have the whole string. I'm basically building a string until I see this character then I parse the string or do whatever you need to do.
2. I clear my string I'm building right away, because the DataReceived event is multi-threaded so its possible to fire while in it. So we want to clear it asap so it can start building the next line.