FTPSClient retrievefile() hanging

Question

I'm creating an apache FTPS client (because the remote server won't allow plain FTP). I can connect and delete files without problem, but when using retrieveFile() or retrieveFileStream(), it hangs.

For some reason, very small files do transfer (up to 5792 bytes), but anything else gives the following PrintCommandListener output:

run:
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 50 allowed.
220-Local time is now 19:42. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
AUTH TLS
234 AUTH TLS OK.
USER
331 User OK. Password required
PASS
230 OK. Current restricted directory is /
TYPE A
200 TYPE is now ASCII
EPSV
229 Extended Passive mode OK (|||53360|)
RETR test.txt
150-Accepted data connection
150 7.3 kbytes to download

Here is the code:

try {

    FTPSClient ftpClient = new FTPSClient("tls",false);

    ftpClient.addProtocolCommandListener(new PrintCommandListener(new  PrintWriter(System.out)));

    ftpClient.connect(host, port);

    int reply = ftpClient.getReplyCode();

    if (FTPReply.isPositiveCompletion(reply)) {
        ftpClient.enterLocalPassiveMode();
        ftpClient.login(username, password);
        ftpClient.enterLocalPassiveMode();
        FileOutputStream outputStream = new FileOutputStream(tempfile);
        ftpClient.setFileType(FTPClient.ASCII_FILE_TYPE);
        ftpClient.retrieveFile("test.txt", outputStream);
        outputStream.close();
        ftpClient.logout();
        ftpClient.disconnect();
    }
} catch (IOException ioe) {
    System.out.println("FTP client received network error");
}

Any ideas are greatly appreciated.

Answer 1

Typically the FTP command sequence for FTPS connections goes (per RFC 4217) AUTH TLS, PBSZ 0, then USER, PASS, etc. Thus you might try:

FTPSClient ftpClient = new FTPSClient("tls",false);
ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftpClient.connect(host, port);
int reply = ftpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
    ftpClient.execPBSZ(0);
    reply = ftpClient.getReplyCode();
    // Check for PBSZ error responses...
    ftpClient.execPROT("P");
    reply = ftpClient.getReplyCode();
    // Check for PROT error responses...

    ftpClient.enterLocalPassiveMode();

This explictly tells the server to not buffer the data connection (PBSZ 0), and to use TLS for protecting the data transfer (PROT P).

The fact that you are able to transfer some bytes indicates that the issue is not the usual complication with firewalls/routers/NAT, which is another common FTPS issue.

Hope this helps!

Answer 2

Even if PBSZ 0 and PROT P are called in the correct sequence, sometimes the server does require SSL session reuse which is not the case by default for the client.

For example, the following reply comes when trying to list a directory. As a result no content listing is returned, this way the client seeing as if the directory is empty:

LIST /
150 Here comes the directory listing.
522 SSL connection failed; session reuse required: see require_ssl_reuse option in sftpd.conf man page

To overcome that, custom initialization of the FTPSClient is needed by overriding _prepareDataSocket_() method.

The solution is explained in details here: https://eng.wealthfront.com/2016/06/10/connecting-to-an-ftps-server-with-ssl-session-reuse-in-java-7-and-8/

Working code sample taken from the above link:

import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.Socket;
import java.util.Locale;

import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSessionContext;
import javax.net.ssl.SSLSocket;

import org.apache.commons.net.ftp.FTPSClient;

import com.google.common.base.Throwables;

public class SSLSessionReuseFTPSClient extends FTPSClient {

  // adapted from: https://trac.cyberduck.io/changeset/10760
  @Override
  protected void _prepareDataSocket_(final Socket socket) throws IOException {
    if(socket instanceof SSLSocket) {
      final SSLSession session = ((SSLSocket) _socket_).getSession();
      final SSLSessionContext context = session.getSessionContext();
      try {
        final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
        sessionHostPortCache.setAccessible(true);
        final Object cache = sessionHostPortCache.get(context);
        final Method putMethod = cache.getClass().getDeclaredMethod("put",Object.class, Object.class);
        putMethod.setAccessible(true);
        final Method getHostMethod = socket.getClass().getDeclaredMethod("getHost");
        getHostMethod.setAccessible(true);
        Object host = getHostMethod.invoke(socket);
        final String key = String.format("%s:%s", host, String.valueOf(socket.getPort())).toLowerCase(Locale.ROOT);
        putMethod.invoke(cache, key, session);
      } catch(Exception e) {
        throw Throwables.propagate(e);
      }
    }
  }

}

Answer 3

Hope someone finds my comment useful after several years. In my case, I replaced retrieveFile with retrieveFileStream. It requires more code, but at least it works.