Why does my Delphi Indy idHTTpServer stop responding when CLOSE_WAIT?

Question

The Environment

I've created a web server in Delphi using Indy component TidHTTPServer. I'm using Delphi XE2 which came with Indy version 10.5.8. The server is running as a desktop app with a form that displays a log of the connections and their requests. It is running on Windows 7 Professional. Requests are for SQL data from a Firebird database. The response is JSON. All traffic is HTTP.

The Challenge

When I was testing it with a small number of users everything worked great. Now that I have rolled it out to about 400 users there are communication problems. The server stops responding to requests and the only way I can get it to respond again is to reboot the machine it is running on and then restart it. The need to reboot occurs more frequently during high volume times.

The Symptoms

Using Windows netstat I have noticed that whenever a TCP connection of type CLOSE_WAIT occurs the server stops responding to requests and I have to reboot again

The Test Procedure

I have been able to simulate this hanging even when there is no traffic on the server. I created a web page that sends multiple requests with a delay between each request.

The web page let's me specify the number of requests to make, how long to wait between each request, and how long to wait before timing out. Even at one millisecond between requests the server seems to respond without issue.

The Test Results

If I set the time out period of each request to a very small number, like 1 msec, I can make my Delphi HTTP Server hang. At a 1 msec timeout requests to my server fail every time, as I would expect. The time out is so short my server can't possibly respond quickly enough.

What I don't understand is that after I force this timeout at the client side, even a relatively small number of requests (fewer than 50), my Delphi web server no longer responds to any requests. When I run netstat on the server machine there are a number of CLOSE_WAIT socket connections. Even after an hour and after closing my server the CLOSE_WAIT socket connections persist.

The Questions

What is going on? Why does my Delphi Indy idHTTPServer stop responding when there are (even just one) CLOSE_WAIT socket connection? The CLOSE_WAITs don't go away and the server does not start responding again. I have to reboot.

What am I not doing?

Here is the results of netstat command showing CLOSE_WAITs:

C:\Windows\system32>netstat -abn | findstr 62000
TCP    0.0.0.0:62000          0.0.0.0:0             LISTENING
TCP    10.1.1.13:62000        9.49.1.3:57036        TIME_WAIT
TCP    10.1.1.13:62000        9.49.1.3:57162        CLOSE_WAIT
TCP    10.1.1.13:62000        9.49.1.3:57215        CLOSE_WAIT
TCP    10.1.1.13:62000        9.49.1.3:57244        CLOSE_WAIT
TCP    10.1.1.13:62000        9.49.1.3:57263        CLOSE_WAIT
TCP    10.1.1.13:62000        9.49.1.3:57279        ESTABLISHED
TCP    10.1.1.13:62000        104.236.216.73:59051  ESTABLISHED

Here is the essence of my web server:

unit MyWebServer;

interface

Uses
...

Type
  TfrmWebServer = class(TForm)
    ...
    IdHTTPServer: TIdHTTPServer;
    ...
    procedure IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
    procedure IdHTTPServerDisconnect(AContext: TIdContext);
    procedure btnStartClick(Sender: TObject);
    ...  
    dbFirebird : TIBDatabase;
    txFireird  : TIBTransaction;
    ...
  private
    function CreateSomeResponseStringData: string;
  end;


implementation

procedure TfrmWebServer.btnStartClick(Sender: TObject);
  begin
    {set the IP's and proit to listen on}
    IdHTTPServer.Bindings.Clear;
    IdHTTPServer.Bindings.Add.IP   := GetSetting(OPTION_TCPIP_ADDRESS);
    IdHTTPServer.Bindings.Add.Port := Str2Int(GetSetting(OPTION_TCPIP_PORT));
    {start the web server}
    IdHTTPServer.Active := TRUE;
    ...
    dbFirebird.Transactrion := txFirebird;
    ...
  end;

procedure TfrmWebServer.IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
  var
    qryFirebird : TIBSql;

  function CreateSomeResponseStringData: string;
    begin
      qryFirebird := NIL;
      qryFirebird := TIBSql.Create(IdHTTPServer);
      qryFirebird.Database := dbFirebird;
      dbFirebird.Connected := FALSE;
      dbFirebird.Connected := TRUE;
      qryFirebird.Active := TRUE;
      Result := {...whatever string will be returned}
    end;

  function CreateAnErrorResponse: string;
    begin
      Result := {...whatever string will be returned}
    end;

  begin
    try        
      AResponseInfo.ContentText := CreateSomeResponseStringData;
      {Clean up: What do I do here to make sure that the connection that was served is:
         - properly closed so that I don't run out of resourses?
         - anything that needs to be cleaned up is freed so no memory leaks
         - TIME_WAIT, CLOSE_WAIT, any other kind of _WAITs are not accumulating?}
    except;
      AResponseInfo.ContentText := CreateAnErrorResponse;
    end;
    qryFirebird.Free;
  end;

procedure TfrmWebServer.IdHTTPServerDisconnect(AContext: TIdContext);
  begin
    {Maybe I do the "Clean Up" here? I tried Disconnect as shown but still lots of 
    TIME_WAIT tcp/ip connections accumulate. even after the app is closed}    
    AContext.Connection.Disconnect;
  end;

end.  

Answer 1

There are at least two major issues with this code that could cause the crashing:

1. The database and transaction objects are global to all threads created by IdHTTPServer. When you disconnect the database it would disconnect for all threads.

2. If there is a run time error assigning content text this line AResponseInfo.ContentText := CreateAnErrorResponse; is not in an exception block.

Here is how I would fix this:

...
procedure TfrmWebServer.btnStartClick(Sender: TObject);
  begin
    {set the IP's and port to listen on}
    IdHTTPServer.Bindings.Clear;
    IdHTTPServer.Default.Port    := Str2Int(GetSetting(OPTION_TCPIP_PORT));
    IdHTTPServer.Bindings.Add.IP := GetSetting(OPTION_TCPIP_ADDRESS);
    {start the web server}
    IdHTTPServer.Active := TRUE;
    ...
  end;

procedure TfrmWebServer.IdHTTPServerCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
  var
    {make these local to each thread}
    qryFirebird : TIBSql;
    dbFirebird  : TIBDatabase;
    txFirebird  : TIBTransaction;

  function CreateSomeResponseStringData: string;
    begin
      dbFirebird  := TIBDatbase.Create(IdHTTPServer);
      txFirebird  := TIBTransaction.Create(IdHTTPServer);
      qryFirebird := TIBSql.Create(IdHTTPServer);
      dbFirebird.Transaction := txFirebird;
      qryFirebird.Database := dbFirebird;
      ...Add params that do the log in to database
      dbFirebird.Connected := TRUE;
      qryFirebird.Active := TRUE;
      Result := {...whatever string will be returned}
    end;

  function CreateAnErrorResponse: string;
    begin
      Result := {...whatever string will be returned}
    end;

  begin
    try
      try        
        ...
        AResponseInfo.ContentText := CreateSomeResponseStringData;
        ...
      except;
        try
          AResponseInfo.ContentText := CreateAnErrorResponse;
        except
          {give up}
        end;
      end;
    finaly
      qryFirebird.Free;
      dbFirebird.Free;
      txFirebird.Free;
    end;
  end;

end.