Python 2 and 3 csv module text-binary mode backwards compatibility

Question

I would like to create a code which is Python 2.7-3.6 compatible I am trying to fix a problem with the csv module where initially I used outfile=open('./test.csv','wb') in Python 2.7 now I have to use outfile=open('./test.csv','w') like in this question otherwise I will incur in a TypeError: a bytes-like object is required, not 'str'.

A the moment I am fixing it using this code:

import sys
w = 'w'
if sys.version_info[0] < 3:
   w = 'wb'
# Where needed
outfile=open('./test.csv',w)

Not very nice, is there any better solution for opening the file in 'wb' if I am using Python 2.7 and in w if I am using Python 3.x? To clarify I have to use wb in Python 2.7 because otherwise, I'll have a blank line every time I add a new line to a file.

Patrick Artner Patrick Artner · Accepted Answer · 2019-05-19T09:19:37

When opening files to be used with module csv on python 3 you always should add newline="" the the open statement:

import sys
mode = 'w'
if sys.version_info[0] < 3:
   mode  = 'wb'

# python 3 write 
with open("somefile.txt", mode, newline="") as f:
    pass  # do something with f

The newline parameter does not exist in python 2 - but if you skip it in python 3 you get misshaped csv output on windows with additional empty lines in it.

See csv.writer (python 3):

If csvfile is a file object, it should be opened with newline=''. If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use \r\n linendings on write an extra \r will be added. It should always be safe to specify newline='', since the csv module does its own (universal) newline handling.

You should use a contextmanaging with as well:

with open("somefile.txt", mode) as f:  # works in 2 and 3
    pass  # do something with f

to get your filehandles closed even if you run into some kind of exception. This is python 2 safe - see Methods of file objects:

It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes, even if an exception is raised on the way. It is also much shorter than writing equivalent try-finally blocks.

Your solution - ugly but works:

import sys
python3 = sys.version_info[0] >= 3 

if python3:
    with open("somefile.txt","w",newline="") as f:
        pass
else:
    with open("somefile.txt","wb") as f:
        pass

The problem is the parameter newline does not exist in python 2. To fix that you would have to wrap/monkypath open(..) including the contextmanaging.

Python 2 and 3 csv module text-binary mode backwards compatibility

1 Answers