CSV to bytes to DF to bypass UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte?

Question

I have a csv which I have previously read to a dataframe without issue, but now is giving me the following error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

df = pd.read_csv(r'\\blah\blah2\csv.csv')

I tried this:

df = pd.read_csv(r'\\blah\blah2\csv.csv', encoding = 'utf-8-sig')

but that gave me this error: UnicodeDecodeError: 'utf-8-sig' codec can't decode byte 0xff in position 10423: invalid start byte

So then I tried 'utf-16', but that gave me this error: UnicodeError: UTF-16 stream does not start with BOM

Then I tried this:

with open(r'\\blah\blah2\csv.csv', 'rb') as f:
contents = f.read()

and that worked, but I need that csv as a dataframe, so then I tried:

new_df = pd.DataFrame.to_string(contents)

but I got this error: AttributeError: 'bytes' object has no attribute 'columns'

Could someone please help me get my dataframe?

Thank you.

UPDATE:

This fixed it. It read the csv into a dataframe without the unicode errors.

df = pd.read_csv(r'\\blah\blah2\csv.csv', encoding='latin1')

Without seeing the raw data, difficult to guess the encoding. From your 'rb' opening, update your question to report what print(contents[:100]) displays. — Mark Tolonen
Hello. The results of print(contents[:100]) are roughly this:b'longitude,produce,open,last_entitle,one,more_strings,etc,' — One2three
That doesn’t match your error since it doesn’t have an 0xff byte in position 0. Also note latin1 will decode anything because it maps every byte to a Unicode code point, but not necessarily the correct code point. — Mark Tolonen
That is the output, starting with b', and that is the error. Do you mean that latin1 will have random str characters throughout the output? That's not good. Do you have any other suggestions? — One2three
Other than use the correct encoding, no. You haven’t shown a part of the string that is giving the errors reported. That would help to guess the encoding. — Mark Tolonen

Ehsan Ehsan · Accepted Answer · 2020-05-15T09:20:20

Try to find correct encoding with the code below:

# import the chardet library
import chardet 

# use the detect method to find the encoding
# 'rb' means read in the file as binary
with open(your_file, 'rb') as file:
    print(chardet.detect(file.read()))

However it is not guaranteed to find the encoding since the context may contain different encodings or different languages but, if it is encoded by only 1 code then you can see that.

pip(3) install chardet

if you dont have it installed

EDIT1: Following is another way to find the right encoding. May this help if the above didn't:

from encodings.aliases import aliases
alias_values = set(aliases.values())

for value in alias_values:
    try:
        df = pd.read_csv(your_file, encoding=value) # or pd.read_excel
        print(value)
    except:
        continue

CSV to bytes to DF to bypass UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte?

2 Answers