Delphi: Indy TIdTCPClient Reading XML

Question

I'm opening a socket connection with TidTCPClient to a server. Server starts sending me tones of different XML Data. However ReadLn does not help me because I don't know where did the xml start & stops. So I decided to use Capture. This works correct. But as I read from the documentation capture also calling ReadLn, means it reads line by line. A 30~40 lines of XML takes 1~2 secs to read sometimes. So I need to read it faster. Can I use ReadStream or any other thing else which makes difference?

The code I'm using now:

procedure TfrmMain.ThreadRun(Sender: TIdThreadComponent);
var
  FData: TStringList;
  FMemory: TStringStream;
begin
  if Client.Connected = False then Exit;
  FData := TStringList.Create;
  Client.IOHandler.Capture(FData, '', False);
  FMemory := TStringStream.Create('',TEncoding.UTF8);
  FMemory.WriteString(FData.Text);
  CoInitialize(Nil);
  QInsert.ParamByName('XMLData').LoadFromStream(FMemory,ftBlob);
  QInsert.Execute;
  CoUninitialize;
  FMemory.Free;
  FData.Free;
end;

As you can see I'm inserting XML data to a local table and this is done really quick, 15ms ~ 60ms depending on size of the XML, but getting data is really painful.

Can you provide an example of the TCP data you are trying to read? Please show how multiple XML documents are being delimited (if at all) within the TCP stream. — Remy Lebeau
BTW, on a side note, your CoInitialize() and CoUninitialize() calls belong in the OnBeforeRun and OnAfterRun events of TIdThreadComponent, not in the OnRun event. — Remy Lebeau
Thanks for the CoInitialize() and CoUninitialize() warnings. I'd fixed that. — wshmstr

Remy Lebeau Remy Lebeau · Accepted Answer · 2015-06-25T23:26:12

Capture() calls ReadLn() in a loop until the specified terminator line is read (in your example, a blank line).

You need to know where each XML document ends and the next begins. If there is no distinct marker in between each document (a blank line is fine, as long as every XML document ends with a line break before the blank line, and does not contain any blank lines inside of the XML), then your only remaining option is to read the first document's opening tag, keep reading until you find the corresponding closing tag, then read the next document's opening tag and wait for its closing tag, and so on.

In that case, you are best off using an XML parser that supports a push model to handle that. You would simply read raw bytes from the socket and push them as-is into the parser, and let it tell you whenever it encounters the start and end of each top-level document element. You can buffer any data the parser gives you in between those two events, using the start event to clear/prepare the buffer and the end event to finalize/flush the buffer to your database.

If you are not willing/able to use a pre-exising XML library, you will have to write your own parser.

Update: The example Capture() you showed has its ADelim parameter set to '', which will only work if the XML documents are delimited by either a LF LF or CR LF CR LF sequence. In that case, you can use that delimiter as the ATerminator parameter of ReadLn() and let it read 1 whole document at a time, instead of using Capture() at all:

procedure TfrmMain.ThreadBeforeRun(Sender: TIdThreadComponent);
begin
  CoInitialize(Nil);
end;

procedure TfrmMain.ThreadAfterRun(Sender: TIdThreadComponent);
begin
  CoUninitialize;
end;

procedure TfrmMain.ThreadRun(Sender: TIdThreadComponent);
var
  Xml: String;
  FMemory: TStringStream;
begin
  // read until a 'CR LF CR LF' sequence is reached...
  Xml := TrimRight(Client.IOHandler.ReadLn(CR+LF+CR+LF, IndyTextEncoding_UTF8));
  // alternatively:
  // Xml := Client.IOHandler.WaitFor(CR+LF+CR+LF, True, False, IndyTextEncoding_UTF8);

  FMemory := TStringStream.Create(Xml, TEncoding.UTF8);
  try
    QInsert.ParamByName('XMLData').LoadFromStream(FMemory, ftBlob);
  finally
    FMemory.Free;
  end;

  QInsert.Execute;
end;

Delphi: Indy TIdTCPClient Reading XML

1 Answers