FTPS with PHP Curl getting partial download

Question

I'm having an issue retrieving a file using php curl over ftps with implicit ssl (as discussed here: ftp_ssl_connect with implicit ftp over tls). The problem is that sometimes - probably 5% of the time, I end up with a partial download.

My class is more or less adapted from adapted from Nico Westerdale's answer and here are the relevant methods:

class ftps {

    private $server;
    private $username;
    private $password;
    private $curlhandle;
    public $dir = '/';

    public function __construct($server, $username, $password) {
        $this->server = $server;
        $this->username = $username;
        $this->password = $password;
        $this->curlhandle = curl_init();
    }

    private function common($remote) {
        curl_reset($this->curlhandle);
        curl_setopt($this->curlhandle, CURLOPT_URL, 'ftps://' . $this->server . '/' . $remote);
        curl_setopt($this->curlhandle, CURLOPT_USERPWD, $this->username . ':' . $this->password);
        curl_setopt($this->curlhandle, CURLOPT_SSL_VERIFYPEER, FALSE);
        curl_setopt($this->curlhandle, CURLOPT_SSL_VERIFYHOST, FALSE);
        curl_setopt($this->curlhandle, CURLOPT_FTP_SSL, CURLFTPSSL_TRY);
        curl_setopt($this->curlhandle, CURLOPT_FTPSSLAUTH, CURLFTPAUTH_TLS);
        return $this->curlhandle;
    }

    public function download($filepath, $local = false) {
        $filename = basename($filepath);
        $remote   = dirname($filepath);
        if ($remote == '.') {
            $remote = $this->dir;
        }
        if ($local === false) {
            $local = $filename;
        }

        if ($fp = fopen($local, 'w')) {
            $this->curlhandle = self::common($remote . $filename);
            curl_setopt($this->curlhandle, CURLOPT_UPLOAD, 0);
            curl_setopt($this->curlhandle, CURLOPT_FILE, $fp);
            curl_exec($this->curlhandle);
            if (curl_error($this->curlhandle)) {
                return false;
            } else {
                return $local;
            }
        }
        return false;
    }
}

I'm using it like this:

$ftps = new ftps('example.com','john_doe','123456');
$ftps->download('remote_filename','local_filename');

As I mentioned above, this works almost flawlessly except about 5% of the time the result is a partially downloaded file. I then check the remote server and am able to verify that the file is indeed there in it's entirety - try the script again and it invariably gets the whole file on a second attempt.

What would cause an intermittent issue using curl like this? My next move would be to implement some kind of checksum and continue download attempts until everything hashes but this feels more like a sloppy workaround than a true solution and it would be nice to know the actual root of the problem.

Try checking the results of curl_exec($this->curlhandle); in case of failures. Also check the content size and also content length header on response, see if there is a mismatch. Next I will add timeout values also. CURLOPT_CONNECTIMEOUT, CURLOPT_FTP_RESPONSE_TIMEOUT, CURLOPT_ACCEPTTIMEOUT_MS, CURLOPT_TIMEOUT. The last one is for complete request. Rest are for phases of the request. So try playing around with these timeouts and see if the error rate drops. Also make sure your script gets enough time to execute and is not killed by webserver on execution time limit — Tarun Lalwani

hanshenrik hanshenrik · Accepted Answer · 2017-11-06T11:28:26

curl probably notices, and curl_error() probably reports it (as a CURLE_PARTIAL_FILE error), but your code is completely ignoring that error. instead of

        if (curl_error($this->curlhandle)) {
            return false;
        } else {

try

        if (curl_errno($this->curlhandle)) {
            throw new \RuntimeException('curl error: '.curl_errno($this->curlhandle).': '.curl_error($this->curlhandle));
    } else {

now you should get a proper error if the curl download failed. however, to give you something to debug, i also suggest adding a protected $curldebugfileh; to class ftps and in __construct do: curl_setopt_array($this->curlhandle,array(CURLOPT_VERBOSE=>true,CURLOPT_STDERR=>($this->curldebugfileh=tmpfile())));

and then change the exception to:

            throw new \RuntimeException('curl error: '.curl_errno($this->curlhandle).': '.curl_error($this->curlhandle).' curl verbose log: '.file_get_contents(stream_get_meta_data($this->curldebugfileh)['uri']));

and add to __destruct: fclose($this->curldebugfileh);

now you should get a verbose log in the exception about what happened up to the corrupted download, which would probably reveal why the download was corrupted.

edit: after reading closer, i see that you have no __destruct, and are never closing the curl handle, and is thus leaking memory. you should probably fix that too. adding function __destruct(){curl_close($this->curlhandle);fclose($this->curldebugfileh);} would prevent that memory/resource leak.

edit 2: i see you're doing $fp = fopen($local, 'w') - don't use w, this will corrupt just about everything you download, on certain OSs, like microsoft windows (it wont do any harm on linux, though..), use wb, and your code is safe to run on windows (and other OSs, including pre-OSX Mac, DOS, CP/M, OS/2, Symbian, and probably others)

edit 3: you're also leaking resources because you never fclose($fp); , you should probably fix that too.

FTPS with PHP Curl getting partial download

1 Answers