Downloading files from SFTP server listed by remote command with Paramiko in Python

Question

I am using Paramiko to connect to the SFTP server from my local machine and download txt files from remote path. I am able to make successful connection and can also print the remote path and the files but i cannot get the files locally. I can print the file_path and file_name but not able to download all the files. Below is the code I am using:

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect(hostname=hostname, username=username, password=password, port=port)

remotepath = '/home/blahblah'
pattern = '"*.txt"'
stdin,stdout,stderr = ssh.exec_command("find {remotepath} -name {pattern}".format(remotepath=remotepath, pattern=pattern))
ftp = ssh.open_sftp()
for file_path in stdout.readlines():
   file_name = file_path.split('/')[-1]
   print(file_path)
   print(file_name)
   ftp.get(file_path, "/home/mylocalpath/{file_name}".format(file_name=file_name))

I can see the file_path and file_name like below from print statement but get error while using ftp.get for multiple files. I can copy a single file by hardcoding the name on source and destination.

file_path = '/home/blahblah/abc.txt'
file_name = 'abc.txt'
file_path = '/home/blahblah/def.txt'
file_name = 'def.txt'

I see one file is downloaded and then i get the following error:

FileNotFoundErrorTraceback (most recent call last)

Error trace:

Traceback (most recent call last):  
File "<stdin>", line 1, in <module>
File "...anaconda3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 769, in get
  with open(localpath, 'wb') as fl:
FileNotFoundError: [Errno 2] No such file or directory: 'localpath/abc.txt\n'

Unless you provide the text of the stack trace (not just its name or type), it's not obvious if it's the local file or the remote file that isn't found. If it's a local file that gives the error, then you probably need to create a directory first if that directory doesn't exist. — Charles Duffy
That said, additional to the immediate issue this question is about, you have at least two additional potentially security-impacting bugs. The first of those is shell injection: Use shlex.quote() in Python 3 or pipes.quote() in Python 2 to get strings that are safe to substitute into ssh.exec_command(); otherwise, someone asking this program to retrieve files from /tmp/$(rm -rf ~) could cause you to have a really bad day. — Charles Duffy
The second security-impacting bug is caused by the reliance on a newline-delimited stream to list filenames. If someone runs d=/home/blahblah/$'\n'/etc/passwd$'\n'; mkdir "$d" && touch "$d/foo", then your find would return /etc/passwd as a result, and your code would copy it over despite not being under /home/blahblah. — Charles Duffy
To fix that latter one, use -print0 on the find command, and iterate over stdout.read().split('\0')[:-1] instead of using stdout.readlines(). There's probably a more efficient way to fix that too. — Charles Duffy
You might also want to look at what the remote file is -- if it's a broken symlink, well, there's your error. (And really, you should be putting a type filter on the find to only look for types of things that sftp.get() will work with). — Charles Duffy

Martin Prikryl Martin Prikryl · Accepted Answer · 2018-06-29T05:54:21

readlines does not remove newline from the line. So as you can see in the traceback, you are trying to create a file named abc.txt\n, what is not possible on many file systems, and mainly, it's not what you want.

Trim the trailing new lines from file_path:

for file_path in stdout.readlines():
    file_path = file_path.rstrip()
    file_name = file_path.split('/')[-1]
    # ...

Though you would have saved yourself lot of troubles, had you used a pure SFTP solution, instead of hacking it by executing a remote find command (what is a very fragile solution, as hinted in comments by @CharlesDuffy).

See List files on SFTP server matching wildcard in Python using Paramiko.

Side note: Do not use AutoAddPolicy. You lose security by doing so. See Paramiko "Unknown Server".

Downloading files from SFTP server listed by remote command with Paramiko in Python

1 Answers