TidHTTPServer “Out of memory” on large file upload

Question

I'm using Delphi 10.3.1 and Indy TIdHTTP / TIdHTTPServer

I created a client / server application to archive files. The client uses a TIdHTTP component, the code is something like this:

procedure TForm1.SendFileClick (Sender: TObject);
var
    Stream: TIdMultipartFormDataStream;
begin
    Stream: = TIdMultipartFormDataStream.Create;
    try
       Stream.AddFormField ('field1', 'hello world');
       Stream.AddFile ('field2', 'c:\temp\gigafile.mp4');
       idHTTP.Post ('http://192.168.1.100:1717', Stream);
    finally
       Stream.Free;
    end;
end;

The server uses a TIdHTTPServer component. Everything seemed to work perfectly until I uploaded very large video files (>= 1GB), because I got the error "Out of memory".

By debugging, I saw that I get the error in function PreparePostStream (line 1229 of the IdCustomHTTPServer unit) when it calls LIOHandler.ReadStream, the event OnCommandGet is not fired yet.

The function LIOHandler.ReadStream goes wrong when it runs AdjustStreamSize (line 2013 of the IdIOHandler unit)

In my last test, with a large video file, in the AdjustStreamSize function, the value of ASize was 1091918544 and I got the error during the execution of

AStream.Size line: = ASize

I think that the origin point of the error is in the System.Classes unit in the following procedure when at the SetPointer ... line.

procedure TMemoryStream.SetCapacity (NewCapacity: NativeInt);
{$ IF SizeOf (LongInt) = SizeOf (NativeInt)}
begin
  SetPointer (Realloc (LongInt (NewCapacity)), FSize);
  FCapacity: = NewCapacity;
end;

I read many articles on the web but I didn't understand if there is something wrong in my code. How can I solve it or is there a limit to the size of the files I can upload with TIdHTTPServer?

@J: TIdMultipartFormDataStream is client side, his problem is that the http server uses a Memorystream to get the file... — whosrdaddy
The fastest solution to your problem is switch to 64bit application (and offcourse 64bit server- with enough RAM). The longterm solution will have to be found on the Indy side. Maybe @RemyLebeau can comment on this... — whosrdaddy
Possible workaround: use a plain binary HTTP payload instead of multipart, and direct the HTTP input stream of the TIdHTTPRequest info to a TFileStream — mjn
@mjn TIdFormDataField defaults to quoted-printable only for text fields. It defaults to binary for files and streams — Remy Lebeau

Remy Lebeau Remy Lebeau · Accepted Answer · 2019-06-06T20:22:23

By default, TIdHTTPServer receives posted data using a TMemoryStream, which will obviously not work well for such large files. You can use the server's OnCreatePostStream event to provide an alternative TStream object to receive into, such as a TFileStream.

TidHTTPServer “Out of memory” on large file upload

2 Answers