IdHTTP Post returns connection reset by peer for Streams over 32KB

Question

I have a problem posting to a web server using HTTPS. I am not sure if the problem is with me or with the server. So it appeared that if I try to post a Stream greater than 32KB, Delphi crashes with Socket Error 10054 - Connection reset by peer. I am using Delphi XE5 with the internal version of Indy and latest to date open ssl dlls. I also try this on XE with latest to date Indy and ssl dlls.

Here is part of my code

function TForm1.SendItemsList(aDataList: TStringList): Boolean;
var
  aHTTP: TIdHTTP;
  aRes: String;

  aURL: String;
  aErrMsg: String;

  aStrm: TMemoryStream;
  aResStrm: TMemoryStream;
  aXML: TNativeXML;
  aTmpNode: TXmlNode;
  aErrNode: TXmlNode;
  aList: TList;
  i: Integer;

begin
  Result := False;

  aStrm := TMemoryStream.Create;
  aResStrm := TMemoryStream.Create;
  aXML := TNativeXML.Create(nil);
  aHTTP := CreateHTTP('application/x-www-form-urlencoded');
  try
    aDataList.SaveToStream(aStrm);
    aStrm.Position := 0;
    aURL := Format(cIRPURL, ['1']);
    try
      aHTTP.Post(aURL, aStrm, aResStrm);
      aResStrm.Position := 0;

      aXML.LoadFromStream(aResStrm);

      aTmpNode := aXML.Root.FindNode('ResponseCode');
      if aTmpNode <> nil then
      begin
        if aTmpNode.Value <> '0' then
        begin
          aErrNode := aXML.Root.FindNode('ResponseText');
          aErrMsg := '';
          if aErrNode <> nil then
            aErrMsg := aErrNode.Value;
          aList := TList.Create;
          try
            aXML.Root.FindNodes('Detail', aList);
            for i := 0 to aList.Count-1 do
            begin
              aErrMsg := aErrMsg+#13#10+TXmlNode(aList[i]).Value;
            end;
          finally
            aList.Free;
          end;
        end;
      end;
    except
      on E:Exception do
      begin
        if E is EIdHTTPProtocolException then
          aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
        else
          aErrMsg := E.Message;

        Exit;
      end;
    end;
  finally
    aXML.Free;
    aStrm.Free;
    aHTTP.Free;
    aResStrm.Free;
  end;

  Result := True;
end;

Where CreateHTTP looks like

function TForm1.CreateHTTP(aContentType: String): TIdHTTP;
begin
  Result := TIdHTTP.Create(nil);
  Result.ConnectTimeout:=60000;
  Result.ReadTimeout:=90000;
  Result.ProtocolVersion:=pv1_1;
  Result.HTTPOptions := [hoForceEncodeParams];
  Result.HandleRedirects:=True;
  Result.IOHandler := SSLHandler;
  SSLHandler.ReadTimeout := 30000;
  Result.Request.Accept:='*/*';
  Result.Request.AcceptLanguage:='en-US';
  Result.Request.ContentType:=aContentType;
  Result.Request.CharSet:='utf-8';
  Result.Request.UserAgent := 'Mozilla/5.0';
end;

All these timeouts exist just because I was testing why I get that error. Then I realized that the problem is when the stream to be sent is larger than 32KB. I can't really say that there is something wrong with the code at all, because in the same way I send data to several other services like Amazon and Walmart for example where I send sometimes megabytes and I don't receive any errors.

The server is IIS but I don't know what version, the support doesn't seem to believe me that I am doing everything OK.

What I notice is that the SSL handler has some default buffer sizes - SendBufferSize and RecvBufferSize which default to 32KB. Well I tried setting that to 1MB but still I get the same error. If I send something which is less than 32KB then everything is OK. The error is returned immediately after line with POST is executed - there is no delay, just immediate error. Otherwise sending small streams results in having a delay of a second or two before it gets processed and then the debugger goes to the next line. I started believing it is a setting of the IIS and there is really such a setting, but the guys there say everything on their side is ok and they have 4MB of limit for the requests.

The service provider is IRPCommerce but unfortunately I can't give links for testing because of IP filtering which takes place at the moment there.

I spent several days discussing this with them, searching the web for problems any limitations etc. So is there something I am missing here, any limitations in Indy which may cause this problem, I doubt but just to be sure I am asking? Anything else I can do to make it clearer where the problem might be?

EDIT: Here is excerpt of the aDataList:

Stock_ExternalStockID|Brands_Active|Brands_Brand|Models_Active|Models_Model|Models_Description|Models_AdditionalInformation1|Models_AdditionalInformation2|Stock_DisplayOrder|Stock_Option|Stock_Price|Stock_RRP|Stock_SupplierCost|Categories_Active|Categories_Name|Stock_PostageWeight|Stock_PartCode|Stock_ISBNNumber|Stock_UPCAPartCode|Stock_EAN13PartCode|Models_ImageURLs|OptionSelector|OptionSelectorAttributes|OptionSelectorCount|Stock_OutOfStockStatus
17664-00001|TRUE|Polypads|TRUE|Polypads Plus One Outsider Pet Bed|<ul><li>The perfect pet bed for any animal around the house or for covering car seats or boots for travelling. </li><li>Convenient to use.  </li><li>6cm Plus One thickness.  </li><li>Fully machine washable and quick drying. </li><li>As there is such an extensive range of colours available for the Polypad collection many colour combinations will have to be ordered in specifically; this service could take up to two weeks. </li><li>If you do not have any specific colours in mind please select, Colour Not Important, from the drop down menu.</li></ul>| | |10|Royal Blue-Navy|43.95|48.99|21.69|TRUE|Dog Beds|1000|160||||https://saddlery.biz/media/catalog/product/o/u/outp1.jpg|1|21,44|2|10
17664-00002|TRUE|Polypads|TRUE|Polypads Plus One Outsider Pet Bed|<ul><li>The perfect pet bed for any animal around the house or for covering car seats or boots for travelling. </li><li>Convenient to use.  </li><li>6cm Plus One thickness.  </li><li>Fully machine washable and quick drying. </li><li>As there is such an extensive range of colours available for the Polypad collection many colour combinations will have to be ordered in specifically; this service could take up to two weeks. </li><li>If you do not have any specific colours in mind please select, Colour Not Important, from the drop down menu.</li></ul>| | |20|Soft Blue-Royal Blue|43.95|48.99|21.69|TRUE|Dog Beds|1000|160||||https://saddlery.biz/media/catalog/product/o/u/outp1.jpg|1|21,44|2|10
17664-00003|TRUE|Polypads|TRUE|Polypads Plus One Outsider Pet Bed|<ul><li>The perfect pet bed for any animal around the house or for covering car seats or boots for travelling. </li><li>Convenient to use.  </li><li>6cm Plus One thickness.  </li><li>Fully machine washable and quick drying. </li><li>As there is such an extensive range of colours available for the Polypad collection many colour combinations will have to be ordered in specifically; this service could take up to two weeks. </li><li>If you do not have any specific colours in mind please select, Colour Not Important, from the drop down menu.</li></ul>| | |30|Black-Purple|43.95|48.99|21.69|TRUE|Dog Beds|1000|160||||https://saddlery.biz/media/catalog/product/o/u/outp1.jpg|1|21,44|2|10

Here I have 165 rows. If I send about 40 of them they go, just because 40 are just about 32KB. I have confirmed that data is not the problem, because I have tried sending one by one each of the lines.

I tried multipart/form-data with no luck. Actually they haven't told me what to use, no matter how much times I have asked about that, so I used the same thing I am using with Walmart.

I think the server is IIS 8.5.

Answer 1

It seems that the "solution" is to change the Content-type to text/xml. None of the other mentioned content-types work with streams larger than 32KB. At the same time I got a confirmation from the developers of the site that the content-type is not considered at all on the server side. So I am really confused what is going on here and why only 'text/xml' works fine.