Dictionary keys with unicode characters raise Error

1
votes

I write CSV parser.

CSV file have strings with unidentified characters and JSON file have map with the correct strings.

file.csv

0,�urawska A.
1,Polnar J�zef

dict.json

{
  "\ufffdurawska A.": "\u017burawska A.",
  "Polnar J\ufffdzef": "Polnar J\u00f3zef"
}

parse.py

import csv
import json

proper_names = json.load(open('dict.json'))

with open('file.csv') as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    for row in reader:
        print proper_names[row[1].decode('utf-8')]

Traceback (most recent call last): File "parse.py", line 9, in print proper_names[row[1].decode('utf-8')] UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in position 0: ordinal not in range(128)

How can I use that dict with decoded strings ?

2
pythonpython-2.7dictionaryunicode
To me it looks like you console cannot handle utf-8 . What do you get if you directly try to print the values to console , like print proper_names.values()[0] ? - Anand S Kumar
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 8: ordinal not in range(128) - CodeNinja

2 Answers

2
votes

If I look at the error message, I think that the issue is the value, not the key. (\u017b is in the value)

So you also have to encode the result:

print proper_names[row[1].decode('utf-8')].encode('utf-8')

(edit: fixes to address comments for future reference)

2
votes

I could reproduce the error, and identify where it occurs. In fact, a dictionnary with unicode keys causes no problem, the error occurs when you try to print a unicode character that cannot be represented in ascii. If you split the print in 2 lines:

for row in reader:
    val = proper_names[row[1].decode('utf-8')]
    print val

the error will occur on print line.

You must encode it back with a correct charset. the one I know best is latin1, but it cannot represent \u017b, so I use again utf8:

for row in reader:
    val = proper_names[row[1].decode('utf-8')]
    print val.encode('utf8')

or directly

for row in reader:
    print proper_names[row[1].decode('utf-8')].encode('utf8')