Receiving Unicode strings with Indy 10

Question

I am using the latest Delphi 10.4.2 with Indy 10.

In a REST server, JSON commands are received and handled. It works fine except for Unicode.

A simple JSON like this:

{"driverNote": "Test"}

is shown correctly

If I now change to Unicode Russian characters:

{"driverNote": "Статья"}

Not sure where I should begin to track this. I expect ARequestInfo.FormParams to have the same value in debugger as s variable.

If I debug Indy itself, FormParams are set in this code:

if LRequestInfo.PostStream <> nil then
begin
  // decoding percent-encoded octets and applying the CharSet is handled by
  // DecodeAndSetParams() further below...
  EnsureEncoding(LEncoding, enc8Bit);
  LRequestInfo.FormParams := 
    ReadStringFromStream( LRequestInfo.PostStream, 
                          -1, 
                          LEncoding
                          {$IFDEF STRING_IS_ANSI}, LEncoding{$ENDIF});
  DoneWithPostStream(AContext, LRequestInfo); // don't need the PostStream anymore
end;

It use enc8Bit. But my string has 16-bits characters.

Is this handled incorrect in Indy?

Remy Lebeau Remy Lebeau · Accepted Answer · 2021-03-10T10:22:24

The code snippet you quoted from IdCustomHTTPServer.pas is not what is in Indy's GitHub repo.

In the official code, TIdHTTPServer does not decode the PostStream to FormParams unless the ContentType is 'application/x-www-form-urlencoded':

if LRequestInfo.PostStream <> nil then begin
  if TextIsSame(LContentType, ContentTypeFormUrlencoded) then
  begin
    // decoding percent-encoded octets and applying the CharSet is handled by DecodeAndSetParams() further below...
    EnsureEncoding(LEncoding, enc8Bit);
    LRequestInfo.FormParams := ReadStringFromStream(LRequestInfo.PostStream, -1, LEncoding{$IFDEF STRING_IS_ANSI}, LEncoding{$ENDIF});
    DoneWithPostStream(AContext, LRequestInfo); // don't need the PostStream anymore
  end;
end;

That ContentType check was added way back in 2010, so I don't know why it is not present in your version.

In your example, the ContentType is 'application/json', so the raw JSON should be in the PostStream and the FormParams should be blank.

That being said, in your version of Indy, TIdHTTPServer is simply reading the raw bytes from the PostStream and zero-extending each byte to a 16-bit character in the FormParams. To recover the original bytes, simply truncate each Char to an 8-bit Byte. For instance, you can use Indy's ToBytes() function in the IdGlobal unit, specifying enc8Bit/IndyTextEncoding_8Bit as the byte encoding.

JSON is most commonly transmitted as UTF-8 (and that is the case in your example), so when you have access to the raw bytes, in any version, make sure you parse the JSON bytes as UTF-8.

Receiving Unicode strings with Indy 10

1 Answers