Mule FTP polling stops without error or warning

Question

I encountered an issue with the FTP polling of Mule ESB standalone: The application was running for a few days without issues, and then the FTP polling stopped without giving warning or error.

Logs were showing signs of activity for the FTP polling until it stopped. Afterward nothing, but other connectors were still active (mainly SFTP polling). I enabled DEBUG log on runtime to see if there was still activity, and the corresponding connector threads were completely silent, as if stopped or blocked.

In the end, restarting the application solved the issue temporarily, but I am trying to understand why this occured to avoid facing it again. My suspicion is that the FTP connector threads either stopped or were blocked, preventing further polls.

It may be caused by an extended FtpMessageReceiver we used to prevent file deletion after poll (overriding postProcess() function). However looking in the source code of both this component and the base FTP receiver and connector, I cannot see how it could happen.

Any idea why the poll would suddenly stop without throwing an error?

Here is the current connector configuration:

    <ftp:connector name="nonDeletingFtpConnector" doc:name="FTP"
        pollingFrequency="${frequency}"
        validateConnections="true">
    <reconnect frequency="${frequency}" count="${count}"/>
    <service-overrides messageReceiver="my.comp.NonDeletingFtpMessageReceiver" />
</ftp:connector>

And the corresponding endpoint:

    <ftp:inbound-endpoint host="${ftp.source.host}" 
            port="${ftp.source.port}" 
            path="${ftp.source.path}" 
            user="${ftp.source.login}" 
            responseTimeout="10000"
            password="${ftp.source.password}" 
            connector-ref="archivingFtpConnector" 
            pollingFrequency="${ftp.default.polling.frequency}">
        <file:filename-wildcard-filter pattern="*.zip"/>
    </ftp:inbound-endpoint>

The messageReceiver code:

public class NonDeletingFtpMessageReceiver extends FtpMessageReceiver {
    public NonDeletingFtpMessageReceiver(Connector connector, FlowConstruct flowConstruct, InboundEndpoint endpoint, long frequency) throws CreateException {
        super(connector, flowConstruct, endpoint, frequency);
    }

    @Override
    protected void postProcess(FTPClient client, FTPFile file, MuleMessage message) throws Exception {
        //do nothing
    }
}

As you can see we defined a FtpMessageReceiver to avoid the file deletion on poll (this is done further on the flow), but looking in the code I can't see how skipping the super.postProcess() call (which is responsible for deleting the file) may cause issues.

FtpMessageReceiver source code I looked in: https://github.com/mulesoft/mule/blob/mule-3.5.0/transports/ftp/src/main/java/org/mule/transport/ftp/FtpMessageReceiver.java

Technical config:

Mule Standalone 3.5.0
Ubuntu 14.04.2 LTS
Java OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)

Any help would be appreciated. Thanks by advance !

we encountered this issue few times in our application where there were no errors in log file and no clue what happened.Finally resulting restart of application. I prefer check is there any issues in FTP server on threads blocking or waiting for the connection from FTP. — RamakrishnaN
Well done, I checked for hanging FTP connection with netstat and there was several connections opened for several days still open. Using jstack I was able to trace the bogus code and write a workaround with a custom FtpConnectionFactory; (I created another post as this was more related to Apache Commons Net FTP Client) — Pierre B.

Pierre B. Pierre B. · Accepted Answer · 2015-10-27T16:31:16

As discussed in the comments, the bug was more linked to the Apache FTP Client and I created a specific post here.

Here is the solution found: using a custom FtpConnectionFactory to configure the client correctly with a timeout value > 0. This way the hanging is interrupted with a timeout exception being thrown.

public class SafeFtpConnectionFactory extends FtpConnectionFactory{

        //define a default timeout
        public static int defaultTimeout = 60000;
        public static synchronized int getDefaultTimeout() {
            return defaultTimeout;
        }
        public static synchronized void setDefaultTimeout(int defaultTimeout) {
            SafeFtpConnectionFactory.defaultTimeout = defaultTimeout;
        }

        public SafeFtpConnectionFactory(EndpointURI uri) {
            super(uri);
        }

        @Override
        protected FTPClient createFtpClient() {
            FTPClient client = super.createFtpClient();

            //Define the default timeout here, which will be used by the socket by default,
            //instead of the 0 timeout hanging indefinitely
            client.setDefaultTimeout(getDefaultTimeout());

            return client;
        }
    }

And then attaching it to the connector:

<ftp:connector name="archivingFtpConnector" doc:name="FTP"
        pollingFrequency="${frequency}"
        validateConnections="true"
        connectionFactoryClass="my.comp.SafeFtpConnectionFactory">
    <reconnect frequency="${reconnection.frequency}" count="${reconnection.attempt}"/>
</ftp:connector>

I'll try to update this answer if there is any notable change on the other.

Mule FTP polling stops without error or warning

1 Answers