Indy error 10038 "Socket operation on non-socket" after 61 seconds of inactivity

Question

I want to download some large files (GB) from an FTP server. The download of the first file always works. Then when trying to get the second file I get:

"Socket Error # 10038. Socket operation on non-socket."

The error is on 'Get'. After 'Get' I see these messages (via FTP status event):

   Starting FTP transfer
   Disconnecting.
   Disconnected.

The code is like this:

{pseudo-code}
for 1 to AllFiles do 
 begin 
   if Connect2FTP then 
    begin
     FTP.Get(Name, GzFile, TRUE, FALSE);  <--- Socket operation on non-socket" error (I also get EIdConnClosedGracefully 'Connection Closed Gracefully' in IDE, F9 will resume execution without problems, but this is OK)
     Unpack(GzFile); <--- this takes more than 60 seconds
    end;
 end;
if FTP.Connected
then FTP.Disconnect;

--

function Connect2FTP(FTP: TIdFTP; RemoteFolder: string; Log: TRichLog): Boolean;      
begin
 Result:= FTP.Connected;
 if NOT Result then
  begin         { We are already connected }
   FTP.Host    := MyFTP;
   FTP.Username:= usr;
   FTP.Password:= psw;

   TRY
    FTP.Connect;
   EXCEPT
    on E: Exception DO

   Result:= FTP.Connected;
   if Result then FTP.ChangeDir(RemoteFolder); 
  end;
end;

Full code here: http://pastebin.com/RScj86R8 (PAS) or here https://ufile.io/26b54 (ZIP)

I think the problem appears after calling 'Unpack' which takes few minutes.

UPDATE: CONFIRMED: The problem appears after calling 'Unpack'. I removed the call and everything is fine. Pausing (sleep or break point) the program between downloads for a while (I think for more than 60 sec) will create the same problem.

That is debatable. The code does not compile, you have made up significant portions of it. I do suggest that you spend more time on the question. Clearly you have a problem, but you'll get better help if you put more work into your question. After having asked 290 questions here I find it hard to believe that you don't know how to do this, why you need to provide a minimal reproducible example. Ah, now I see that you are still adding more, even after the comment above. Anyway, 10038 means that you closed the socket and then tried to use it again. — David Heffernan
Since you're the only one with a repro, and actual code, I guess you need to debug it. — David Heffernan
OK, I guess everything is fine then, and the socket being used is fine, and the system is wrong when it throws 10038. — David Heffernan

Remy Lebeau Remy Lebeau · Accepted Answer · 2016-08-11T19:07:29

FTP uses multiple socket connections, one for commands/responses, and separate connections for each transfer.

The most likely cause of the socket error is an FTP-unaware proxy/router/firewall sitting in between TIdFTP and the FTP server is closing the command connection after a short period of inactivity. During the Unpack() (or manual pause), no commands/responses are being transmitted on the command connection, it is sitting idle, and thus is subject to being closed by a timeout on such a proxy/router/firewall.

During a transfer, the command connection is sitting idle, no FTP commands/responses are being transmitted on it (unless you abort the transfer), until the transfer is complete. An FTP-unaware proxy/router/firewall may close the command connection prematurely during this time.

To avoid that, TIdFTP has a NATKeepAlive property that can enable TCP keep-alives on the command connection while it is sitting idle. This usually prevents premature closes.

However, when there is no transfer in prgress, TIdFTP disables TCP keep-alives on the command connection if NATKeepAlive.UseKeepAlive is True. TIdFTP uses TCP keep-alives only during transfers, with the assumption that you are not going to perform long delays in between FTP commands. If you need to delay for awhile, either close the FTP connection, or send an FTP command at regular intervals (such as calling TIdFTP.Noop() in a timer/thread).

Alternatively, you can try manually enabling TCP keep-alives after connecting to the server, and set NATKeepAlive.UseKeepAlive to False so TIdFTP will not automatically disable the keep-alives after each transfer, eg:

function Connect2FTP(FTP: TIdFTP; RemoteFolder: string; Log: TRichLog): Boolean;      
begin
  Result := FTP.Connected;
  if not Result then
  begin         { We are not already connected }
    FTP.Host    := MyFTP;
    FTP.Username:= usr;
    FTP.Password:= psw;

    try
      FTP.Connect;
      try
        FTP.ChangeDir(RemoteFolder); 

        // send a TCP keep-alive every 5 seconds after being idle for 10 seconds
        FTP.NATKeepAlive.UseKeepAlive := False; // False by default, but just in case...
        FTP.Socket.Binding.SetKeepAliveValues(True, 10000, 5000);
      except
        FTP.Disconnect(False);
        raise;
      end;
    except
      Exit;
    end;

    Result := True;
  end;
end;

Note that while most platforms support enabling TCP keep-alives on a per-connection basis (keep-alives are part of the TCP spec), setting keep-alive intervals is platform-specific. At this time, Indy supports setting the intervals on:

Windows 2000+ and WinCE 4.x+, for Win32 and .NET
Linux, when Indy is used in Kylix
Unix/Linux and NetBSD, when Indy is used in FreePascal.

Otherwise, OS default intervals get used, which may or may not be too large in value for this situation, depending on OS configuration.

At this time, the intervals are not customizable in Delphi FireMonkey, except for Windows.

If a particular platform supports setting custom TCP keep-alive intervals, but Indy does not implement them in SetKeepAliveValues(), you can use TIdFTP.Socket.Binding.SetSockOpt() instead to set the values manually as needed. Many platforms do support TCP_KEEPIDLE/TCP_KEEPINTVL or equivalent socket options.

Indy error 10038 "Socket operation on non-socket" after 61 seconds of inactivity

1 Answers