Python - Reading CSV UnicodeError

Question

I have exported a CSV from Kaggle - https://www.kaggle.com/ngyptr/python-nltk-sentiment-analysis. However, when I attempt to iterate through the file, I receive unicode errors concerning certain characters that cannot be encoded.

File "C:\Program Files\Python35\lib\encodings\cp850.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]

UnicodeEncodeError: 'charmap' codec can't encode character '\u2026' in position 264: character maps to

I have enabled utf-8 encoding while opening the file, which I assumed would have decoded the ASCII characters. Evidently not.

My Code:

with open("sentimentDataSet.csv", "r", encoding="utf-8" ,errors='ignore', newline='') as file:
    reader = csv.reader(file)-
    for row in reader:
        if row:
            print(row)
            if row[sentimentCsvColumn] == sentimentScores(row[textCsvColumn]):
                accuracyCount += 1
    print(accuracyCount)

AKX AKX · Accepted Answer · 2019-11-11T17:35:12

That's an encode error as you're printing the row, and has little to do with reading the actual CSV.

Your Windows terminal is in CP850 encoding, which can't represent everything.

There are some things you can do here.

A simple way is to set the PYTHONIOENCODING environment variable to a combination that will trash things it can't represent. set PYTHONIOENCODING=cp850:replace before running Python will have Python replace characters unrepresentable in CP850 with question marks.
Change your terminal encoding to UTF-8: chcp 65001 before running Python.
Encode the thing by hand before printing: print(str(data).encode('ascii', 'replace'))
Don't print the thing.

Python - Reading CSV UnicodeError

1 Answers