python - UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte

Question

I am new to Python, I am trying to read csv file using below script.

Past=pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv",encoding='utf-8')

But, getting error "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte", Please help me to know issue here, I used encoding in script thought it will resolve error.

Answer 1

This happens because you chose the wrong encoding.

Since you are working on a Windows machine, just replacing

Past=pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv",encoding='utf-8') 

with

Past=pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv",encoding='cp1252')

should solve the problem.

Answer 2

Use this solution it will strip out (ignore) the characters and return the string without them. Only use this if your need is to strip them not convert them.

with open(path, encoding="utf8", errors='ignore') as f:

Using errors='ignore' You'll just lose some characters. but if your don't care about them as they seem to be extra characters originating from a the bad formatting and programming of the clients connecting to my socket server. Then its a easy direct solution. reference

Answer 3

Try using :

pd.read_csv("Your filename", encoding="ISO-8859-1")

The code that I parsed from some website was converted in this encoding instead of default UTF-8 encoding which is standard.

Answer 4

The following works very well for me:

encoding = 'latin1'

Answer 5

Using the code bellow works for me:

with open(keeniz_dir + '/world_cities.csv',  'r', encoding='latin1') as input:

Answer 6

Don't pass encoding option unless you are sure about file encoding. Default value encoding=None passes errors="replace" to open() function called. Characters with encoding errors will be substituted with replacements, you can then figure out correct encoding or just use the resulting Dataframe. If wrong encoding is provided pd will pass errors="strict" to open() and get ValueError if encoding is incorrect.