Constructing a line based ByteBuffer from an uint8List on-line

Question

I guess my problem is simple but I'm new to dart...

Situation

On the client side I have an open socket, listening.
The server sends data chunks of arbitrary length (In reality between 2 and 1024 bytes per chunk)
Everytime new data arrives, it gets added to a list (uint8List)
The connection may stay open for a longer period of time and there may be "lots" of data chunks coming in

This is only basic code and working.

main() async {
  await SecureSocket.connect('SERVER', 123).then(handleResponse);
}

void handleResponse(SecureSocket ss) {

  print('Connected to: '
      '${ss.remoteAddress.address}:${ss.remotePort}');

  ss.listen(
      (data) {
          print ('New chunk available:' + data.lengthInBytes.toString() + ' Bytes');
          String msg = String.fromCharCodes(data).trim();
          print(data.toString());
      },
      onDone: () {
          ss.destroy();
      },
      onError: (error) {
        print('An Error occured while communicating with the server.');
      },
      cancelOnError: true,
  );

  ss.write('FGG1223' + '\r\n');
}

Now, I want to incrementally update my front end with the incoming data. That's why I want to start parsing and processing the data as soon as it arrives.

From a semantic point of view the delimiter in the data is CRLF [13,10].
It separates the command sequences from each other eg.:
CMD1CMD2CRLFCMD3CRLF

So, my idea was to store the arriving data line-based, separated by the CRLF in another buffer (ByteBuffer List).
The data would be much easier to process (feeding the parser) and I could update the front-end much earlier.

Problem

I want to process the data on-line (on arrival). That's why I can't wait for the whole data to come in
One chunk may have more then 1 delimiter sequences (See example response 0)
The delimiter sequence (CRLF) may be split between 2 or more responses (See example response 1 and 2)

In this case I have to wait until the next LF arrives with one of the next chunks or throw an exception if the LF doesn't arrive after n chunks.

Is there an easy and elegant way of accomplishing this in dart?

Thank you.

Example server response at time 0,1,2:
0 Chunk: [32, 79, 101, 97, 100, 13, 10, 121, 46, 13, 10]
1 Chunk: [70, 76, 65, 71, 13]
2 Chunk: [10, 66, 89, 69, 32, 13, 10]

julemand101 julemand101 · Accepted Answer · 2020-01-03T20:14:07

I know my suggested solution does not match 100% with you requirements but I hope it will still give some inspiration to a solution. Instead of writing all the logic of handling the splitting of data chunks I have made this example using methods from the Dart SDK:

import 'dart:convert';
import 'dart:io';

Future<void> main() async {
  await SecureSocket.connect('SERVER', 123).then(handleResponse);
}

void handleResponse(SecureSocket ss) {
  print('Connected to: ${ss.remoteAddress.address}:${ss.remotePort}');

  ss
      .map((uint8list) => uint8list.toList())
      .transform(const Utf8Decoder())
      .transform(const LineSplitter())
      .listen(
    (line) {
      print('New line available:' + line);
    },
    onDone: () {
      ss.destroy();
    },
    onError: (Exception error) {
      print('An Error occured while communicating with the server.');
    },
    cancelOnError: true,
  );

  ss.write('FGG1223\r\n');
}

First, I want to note that the use of the map method are used to convert from Uint8List (which are used by the SecureSocket class) to a normal List which are used by Utf8Decoder. I hope this would be unnecessary in future Dart releases since it seem a bit silly.

Then I convert the received bytes to UTF8 and sending it to a LineSplitter which returns lines from the chunk.

All of this works as data is received. So you listen method are triggered each time a full line has been parsed.

Again, this is not completely like you wanted. But I hope it gives some inspiration to a working solution. And if you want to make something more specific to your requirements you can take a look into how to make you own StreamTransformer.

Constructing a line based ByteBuffer from an uint8List on-line

1 Answers