Proper usage of Python 3 ftp class

Question

As a python newbie still learning the language, I struggled for a couple of days trying to do a simple ASCII file transfer (STOR/PUT) using the ftp class in ftplib (running Python 3.3).

After using the storbinary() method and consistently getting a TypeError: "Type str doesn't support the buffer API", I discovered the discussion on this thread, which implies that there is a bug in the port of ftplib to Python 3:

http://bugs.python.org/issue6822

I then tried using storbinary() instead of storlines(), using a file object opened using the 'rb' switch and it seems to work perfectly. I'm working on a Windows system, and for testing/learning purposes I'm uploading to my own site which is on a Linux host. After uploading both .zip and .txt files and copying them back down to my workstation using FileZilla, both files are intact.

In my day-to-day work I need to upload gzipped and ASCII files to a mainframe, and am concerned that I may be leaving myself open to file transfer errors using this counter-intuitive work-around. I've screwed up so many manual FTP transfers when forgetting to switch to the appropriate transfer mode, that it feels creepy to be able to transfer both binary and ASCII files using exactly the same code!

Can anyone comment on how I'm implementing this library class?

Thanks.

fileName = 'F:\\Data_Folder\\Test_File.txt'
fileParts = os.path.split(fileName)
putFile = fileParts[1]
cmd = 'STOR {}'.format(putFile)
fileObject = open(fileName, 'rb')
ftp.storbinary(cmd, fileObject)

fileName = 'F:\\Data_Folder\\Test_File.zip'
fileParts = os.path.split(fileName)
putFile = fileParts[1]
cmd = 'STOR {}'.format(putFile)
fileObject = open(fileName, 'rb')
ftp.storbinary(cmd, fileObject)

6/28/2013 - Coming back here to kinda "close the loop" on this issue. While I can use open(fileName, 'rb') together with ftp.storbinary() successfully for both binary and ASCII text files, with both Windows and Linux hosts as the target, when I do so with the mainframe as the target, the text file is getting garbled, appearing as a binary file.

By adding a switch to my wrapper class to continue to open the file with the 'rb' argument, but using storlines() instead to do the transfer, the file arrives at the destination intact. I'm willing to bet that there are likely some configuration options on the mainframe side that could make this behavior vary from one host to another, but I'm hoping that mentioning this will alert anyone encountering this thread to the possibility that the apparently "safe" combination of open(fileName, 'rb') and storbinary() may not succeed with all FTP hosts, most notably mainframe systems. It may be only determined through trial-and-error, but there are cases in which the correct approach for transferring ASCII data will require open(fileName, 'rb') together with storlines().

Keep in mind that FTP text mode is explicitly ASCII, while most "text files" are UTF-8 or Latin-1 or CP-850 or something else. Plus, FTP text mode is allowed to munge line endings. So, are you sure you want to use text mode anyway? — abarnert
Meanwhile, it's always safe to transfer ASCII in binary mode, unless you want the other side to convert line endings (or transcode ASCII to EBCDIC or something), so… what are you worried about? — abarnert

abarnert abarnert · Accepted Answer · 2013-05-08T18:26:36

The 3.3 documentation for storlines explicitly says:

Lines are read until EOF from the file object file (opened in binary mode)…

So, passing it a file opened in text mode isn't supposed to work.

FTP's text (ASCII) mode isn't the same think as Python's text mode. In particular, FTP text is required to be ASCII (and real 7-bit ASCII, not an extended codepage with values > 127). But Python text has an explicitly-specified character set, and is treated as Unicode. If your files are actually UTF-8, Latin-1, CP-850, etc., you can't use text mode.

On top of that, both Python and FTP are allowed to munge line endings for text files. Sometimes you want that, so you can upload a Windows text file to a linux box and have it show up with Unix line-endings instead of Windows (although that may not actually happen, depending on a variety of things…). But otherwise, you don't want to use text mode.

In short, you're probably doing the right thing by opening your text files in binary mode and uploading them in binary (image) mode.

Meanwhile, your code is fine as-is, but if you're looking for ways to improve it, there's always room for minor changes.

First, if you've got the same 7 lines copied and pasted twice, and the only difference is the string in 1 line, factor it out into a function.

Also, close your files. Either add an explicit fileObject.close(), or, better, use a with statement. If you've only got 2 files in a short-lived script, it won't make much difference, but it's still a good idea—and you might later expand this into something that opens more than 2 files or lives longer.

If you just want the basename of a file, it's clearer to call basename than to call split and then access [1].

Getting into the nitpicky, unless you've got lots of "legacy" or wrapper code using a different style, it's better to stick with PEP 8 than to invent your own style.

Finally, if you want to leave open the possibility of sending text and binary files differently, even though at present they're implemented the same, just write upload_binary_file and upload_text_file, and make the latter call the second, or be a reference to the same function. However, you probably don't want this. For the reasons explained above, and in J.F. Sebastian's comments, a upload_text_file function is more likely to be a misleading attractive nuisance than a useful hook for future expansion.

So:

def upload_file(filename):
    put_file = os.path.basename(filename)
    cmd = 'STOR {}'.format(put_file)
    with open(filename, 'rb') as file_object:
        ftp.storbinary(cmd, file_object)

upload_file('F:\\Data_Folder\\Test_File.txt')
upload_file('F:\\Data_Folder\\Test_File.zip')

Proper usage of Python 3 ftp class

2 Answers