Encoding Issues After Indy Upgrade

Question

The following code requests an OAuth2 token. It works with Indy 10.0.52 but with Indy 10 SVN 5412 it generates a 400 Bad Request error due to invalid request credentials.

WXYZUserID   :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZUserID, ''));
WXYZPassword :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZPassword, ''))
WXYZSecret   :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZSecret, ''));
OAuthURL     :=      Trim( GetSettingsValue( mySQLQuery,  wcsOAuthURL, ''));
WxyzHttp := TIdHttp.Create(nil);
Serial := TStringList.Create;
if (WXYZUserID <> '') and (WXYZPassword <> '') and (WXYZSecret <> '') then
  Serial.Add('grant_type=password&username=' + WXYZUserID + '&password=' +    WXYZPassword )
Else
  Serial.Add('grant_type=password&username=****-*****&password=***************');

Output := TMemoryStream.Create;
Serial.SaveToStream(Output);

WxyzHttp.ConnectTimeout := 60000;

IdLogFile := TIdLogFile.Create;
IdLogFile.Filename := 'Logs/WxyzHTTP' + FOrmatDateTime('yyyymmdd_hhnnsszzz',now) + '.log';
IdLogFile.Active := True;

IdSSLIOHandlerSocketOpenSSL1 := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
IdSSLIOHandlerSocketOpenSSL1.SSLOptions.Method := sslvSSLv23;
IdSSLIOHandlerSocketOpenSSL1.Intercept := IdLogFile;
WxyzHttp.IOHandler := IdSSLIOHandlerSocketOpenSSL1;

if (WXYZUserID <> '') and (WXYZPassword <> '') and (WXYZSecret <> '') then
  WxyzHttp.Request.CustomHeaders.Values['Authorization'] := 'Basic ' + WXYZSecret
Else
  WxyzHttp.Request.CustomHeaders.Values['Authorization'] := 'Basic ****************************************************************';

WxyzHttp.Request.ContentType := 'application/x-www-form-urlencoded; charset="utf-8"';

Try
Token := WxyzHttp.Post( OAuthURL, Output );
Except
On E: Exception do
Begin
  DbgWebCatLog( wcmtDbg, 'GetSerialToken', E.ClassName + ' error raised, with message: ' + E.Message, '' );
  Token := '';
  end;
end;

I added code to capture IdLogFile date. The log from the Indy 10.0.52 version contains the following line containing the credentials.

Sent 05/01/2017 18:34:35: grant_type=password&username=*************_*****-*****&password=***}%25**(**(***

For security purposes I’ve replaced all the alphanumeric characters with ‘*’. The “%25” characters were originally a “%” in the actual password. Something translated the original “%” to “%25” yet did not alter any of the other special characters.

Initial inspection of the log from the Indy 10 SVN 5412 detected a message from the server indicating there were invalid request credentials. Furthermore all the special characters in the line corresponding to the one above showed all the special characters had been encoded. The grant_type, username and password were rendered using a tStringList which I found resulted in urlencoding. I changed the code to render the same data using a stream and now nothing is being encoded; not even the “%” is being translated to “%25”, and I continue to get the invalid request credentials message.

So my questions are why would Indy 10.0.52 translate only the “%” character to “%25” and how can I replicate that behavior in Indy 10 SVN 5412?

Following is the full log from the Indy 10.0.52 version which works.

Stat Connected.
Sent 05/01/2017 18:34:35: POST /auth/realms/hvac/tokens/grants/access HTTP/1.0<EOL>Content-Type: application/x-www-form-urlencoded; charset="utf-8"<EOL>Content-Length: 82<EOL>Authorization: Basic<EOL> **********<EOL>Host: services.ccs.utc.com:443<EOL>Accept: text/html, */*<EOL>Accept-Encoding: identity<EOL>User-Agent: Mozilla/4.0 (compatible; MSIE 8.0)<EOL><EOL>
Sent 05/01/2017 18:34:35: grant_type=password&username=*************_*****-*****&password=***}**%25**(**(***
Recv 05/01/2017 18:34:35: HTTP/1.1 200 OK<EOL>Server: Apache-Coyote/1.1<EOL>Pragma: no-cache<EOL>Cache-Control: no-store<EOL>Content-Type: application/json;charset=UTF-8<EOL>Content-Length: 569<EOL>Date: Mon, 01 May 2017 22:34:35 GMT<EOL>Connection: close<EOL><EOL>{<EOL>  "access_token":"**********",<EOL>  "token_type":"Bearer",<EOL>  "expires_in":1800,<EOL>  "refresh_token":"**********",<EOL>  "id_token":"**********"<EOL>}
Stat Disconnected.
Stat Disconnected

Following is the full log from the Indy 10 SVN 5412 version which fails with invalid request credentials.

Stat Connected.
Sent 05/02/2017 15:11:08: POST /auth/realms/hvac/tokens/grants/access HTTP/1.0<EOL>Connection: keep-alive<EOL>Content-Type: application/x-www-form-urlencoded; charset=utf-8<EOL>Content-Length: 82<EOL>Authorization: Basic **********<EOL>Host: services.ccs.utc.com<EOL>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<EOL>User-Agent: Mozilla/3.0 (compatible; Indy Library)<EOL><EOL>
Sent 05/02/2017 15:11:08: grant_type=password&username=*************_*****-*****&password=***}**%**(**(***<EOL>
Recv 05/02/2017 15:11:08: HTTP/1.1 400 Bad Request<EOL>Server: Apache-Coyote/1.1<EOL>Content-Type: application/json;charset=UTF-8<EOL>Content-Length: 84<EOL>Date: Tue, 02 May 2017 19:11:08 GMT<EOL>Connection: close<EOL><EOL>{<LF>  "error": "invalid_grant",<LF>  "error_description": "Invalid request credentials"<LF>}
Stat Disconnected.
Stat Disconnected.

Remy Lebeau Remy Lebeau · Accepted Answer · 2017-05-03T00:07:03

You are posting a TMemoryStream, which gets posted as-is, TIdHTTP does not encode it in any way, in any version of Indy. You are responsible for ensuring the content of your TMemoryStream is formatted correctly. That is on you, not Indy.

TIdHTTP.Post() has an overload that posts a TStrings instead of a TStream, formatting the strings in application/x-www-webform-urlencoded format for you. You do not need to use a TMemoryStream at all.

In Indy 10.0.52, TIdHTTP.Post(TStrings) encodes only the value of name=value pairs, and it encodes them using TIdURI.ParamsEncode(), which percent-encodes any character that is in the set '*#%<> []' or is not an ASCII character between #33..#128, inclusive. 10.0.52 is not Unicode-aware at all, so it won't encode the strings to UTF-8, and it does not support Delphi 2009+'s UnicodeString type (or even WideString, for that matter). So any strings you store in the TStrings must already be in UTF-8 format to begin with (and in the case of Delphi 2009+, they would still have to be in UTF-8, using 16-bit characters for each codeunit instead of 8-bit characters).

In the latest Indy version (10.6.2.5418 at the time of this writing), TIdHTTP.Post(TStrings) is fully Unicode-aware, and supports UnicodeString. It will encode both name and value of name=value pairs, and manually encodes them (not using TIdURI) in compliance with the HTML5 standard by encoding any Unicode character not in the set 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789*-._' to UTF-8 (by default, can be overridden using the AByteEncoding parameter of Post()) before percent-encoding the resulting byte octets.

With that said, try something more like this:

try
  WXYZUserID   :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZUserID, ''));
  WXYZPassword :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZPassword, ''))
  WXYZSecret   :=      Trim( GetSettingsValue( mySQLQuery,  wcsWXYZSecret, ''));
  OAuthURL     :=      Trim( GetSettingsValue( mySQLQuery,  wcsOAuthURL, ''));

  WxyzHttp := TIdHttp.Create(nil);
  try
    WxyzHttp.ConnectTimeout := 60000;

    IdLogFile := TIdLogFile.Create(WxyzHttp);
    IdLogFile.Filename := 'Logs/WxyzHTTP' + FormatDateTime('yyyymmdd_hhnnsszzz',now) + '.log';
    IdLogFile.Active := True;

    IdSSLIOHandlerSocketOpenSSL1 := TIdSSLIOHandlerSocketOpenSSL.Create(WxyzHttp);
    IdSSLIOHandlerSocketOpenSSL1.SSLOptions.Method := sslvSSLv23;
    IdSSLIOHandlerSocketOpenSSL1.Intercept := IdLogFile;
    WxyzHttp.IOHandler := IdSSLIOHandlerSocketOpenSSL1;

    if (WXYZSecret <> '') then
      WxyzHttp.Request.CustomHeaders.Values['Authorization'] := 'Basic ' + WXYZSecret
    else
      WxyzHttp.Request.CustomHeaders.Values['Authorization'] := 'Basic ****************************************************************';

    WxyzHttp.Request.ContentType := 'application/x-www-form-urlencoded';
    WxyzHttp.Request.CharSet := 'utf-8';

    Serial := TStringList.Create;
    try
      Serial.Add('grant_type=password');

      if (WXYZUserID <> '') and (WXYZPassword <> '') then
      begin
        Serial.Add('username=' + WXYZUserID);
        Serial.Add('password=' + WXYZPassword);
      end else
      begin
        Serial.Add('username=****-*****');
        Serial.Add('password=***************');
      end;

      Token := WxyzHttp.Post( OAuthURL, Serial );
    finally
      Serial.Free;
    end;
  finally
    WxyzHttp.Free;
  end;
except
  on E: Exception do
  begin
    DbgWebCatLog( wcmtDbg, 'GetSerialToken', E.ClassName + ' error raised, with message: ' + E.Message, '' );
    Token := '';
  end;
end;

Encoding Issues After Indy Upgrade

1 Answers