No mapping for the Unicode character exists in the target multi-byte code page : how to convert MemoryStream to String

Question

My goal is to be able to convert TMemoryStream to string. I have this code to get the data into the TMemoryStream:

var
  idHttp : TIdHTTPEx;
  url : string;
  slTemp : TStringList;
  memoryStream : TMemoryStream;
begin
  try

    idHttp := TIdHTTPEx.Create(nil);
    slTemp := TStringList.Create;
    memoryStream := TMemoryStream.Create;
    try
      url := GetURL;
      SetParams(slTemp);

  idHttp.Request.Accept := 'application/json, text/javascript, */*; q=0.01';
  idHttp.Request.AcceptEncoding := 'gzip, deflate, br';
  idHttp.Request.AcceptLanguage := 'en-US,en;q=0.9';
  idHttp.Request.CacheControl := 'no-cache';
  idHttp.Request.Connection := 'keep-alive';
  idHttp.Request.ContentLength := 16;
  idHttp.Request.ContentType := 'application/x-www-form-urlencoded; charset=UTF-8';

      idHttp.Post(url, slTemp, memoryStream);
      Result := MemoryStreamToString(memoryStream);
    finally
      memoryStream.Free;
      slTemp.Free;
      idHttp.Free;
    end;

  except on E : Exception do
    begin
      Result := 'e:' + E.Message;
    end;
  end;
end;

And this is my code to convert it to a string:

function MemoryStreamToString(MemoryStream : TMemoryStream): string;
var
  StringStream: TStringStream;
begin
  Result:='';

    StringStream:= TStringStream.Create('', TEncoding.UTF8);
  try
    MemoryStream.Position := 0;
    StringStream.CopyFrom(MemoryStream, MemoryStream.Size);

    Result:= StringStream.DataString;
    Result := Result;
  finally
    FreeAndNil(StringStream);
  end;

end;

My function works fine in most of the conversion but not this. I checked these links: link1, link2 but they are different than my situation. I tried link3 too but still fail.

Any idea how to solve the problem?

Are you sure that the returned data is encoded in UTF-8? You don't seem to check this in your code... — HeartWare
The error message would suggest that the stream does not contain UTF8 encoded text. We don't know what's in your stream. — David Heffernan
one of the request is idHttp.Request.AcceptEncoding := 'gzip, deflate, br'; — Ago
It makes me sad to see so much delphi code like this. A memory stream is just a point to a contiguous block of memory. No need to bring yet another stream object into play, create yet another copy of the data. Decode the payload directly. — David Heffernan
i have updated my code in the question.. i set the charset in my idhttp.request to utf8 but still no avail — Ago

Remy Lebeau Remy Lebeau · Accepted Answer · 2019-06-14T14:46:39

You don't need to decode the raw data manually. Just let TIdHTTP do it for you. The Post() method has an overload that returns a decoded string:

Result := idHttp.Post(url, slTemp);

Also, you need to get rid of this line completely:

idHttp.Request.AcceptEncoding := 'gzip, deflate, br';

Otherwise TIdHTTP will not be able to decode the response correctly if the server decides to send a compressed response. You are manually giving the server permission to do so, but you are not setting up the TIdHTTP.Compressor property so TIdHTTP can handle decompression. Do not set the AcceptEncoding manually unless you are willing and able to manually detect and decode a response that has been actually been encoded in one of the formats you specify. Otherwise, just let TIdHTTP manage the AcceptEncoding property internally based on its actual capabilites.

No mapping for the Unicode character exists in the target multi-byte code page : how to convert MemoryStream to String

1 Answers