Code works in Python 2 but not Python3 TypeError: a bytes-like object is required, not 'str'

1
votes

The following code works in Python 2.7:

import os
import pickle

modelpath = "models/"

gmm_files = [os.path.join(modelpath,fname) for fname in 
          os.listdir(modelpath) if fname.endswith('.gmm')]

models    = [pickle.load(open(fname,'r')) for fname in gmm_files]

However, when I run the code in Python3, I get the following error from the last line:

TypeError: a bytes-like object is required, not 'str'

In order to get a better idea, I tried printing print([type(open(fname,'r')) for fname in gmm_files]) in both versions and found out that in python 2 the type is <type 'file'> and in Python 3 the type is <class '_io.TextIOWrapper'>.

I've checked out these stackoverflow questions but neither of them have helpful answers for this:

python 3.5: TypeError: a bytes-like object is required, not 'str' when writing to a file

Python sockets error TypeError: a bytes-like object is required, not 'str' with send function

UPDATE

A bunch of the answers here said to change open(fname, 'r')to open(fname, 'rb') but that just leads to another error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 0: ordinal not in range(128)

2
pythonpython-3.xpicklepython-2.x
You need to open the file in binary mode, rather than (the default) text mode: open(fname,'rb') - Tom Dalton
I also suggest using the the pathlib library rather than os. In my opinion it is a lot nicer to work with. - Bram Vanroy
Thanks everyone. @TomDalton I updated the question to show what happens when I implement changing r to rb - Philip7899
Are you trying to unpickle data in python 3.x that you pickled with 2.7? Can you post the full traceback of the error? - Tom Dalton
(It's probably worth asking this as a completely new question) - Tom Dalton

2 Answers

2
votes

Ref https://docs.python.org/3.6/library/pickle.html#pickle.load, the file-like object you pass to pickle.load needs to return binary data. Files are opened in text-mode by default, which is why you're seeing this error. If you open the file in binary mode (by adding 'b' to the mode), everything should be work.

E.g.

models = [pickle.load(open(fname, 'rb')) for fname in gmm_files]
1
votes

As the documentation for the pickle.load method says (emphasis mine):

The argument file must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both methods should return bytes.

open(stuff, 'r') will open the file for reading text, not raw bytes. Thus, open(stuff, 'r').read will return str, not bytes. To fix that, open the file in binary mode: open(stuff, 'rb').