Why doesn't Python recognize my utf-8 encoded source file?

Question

Here is a little tmp.py with a non ASCII character:

if __name__ == "__main__":
    s = 'ß'
    print(s)

Running it I get the following error:

Traceback (most recent call last):
  File ".\tmp.py", line 3, in <module>
    print(s)
  File "C:\Python32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xdf' in position 0: character maps to <undefined>

The Python docs says:

By default, Python source files are treated as encoded in UTF-8...

My way of checking the encoding is to use Firefox (maybe someone would suggest something more obvious). I open tmp.py in Firefox and if I select View->Character Encoding->Unicode (UTF-8) it looks ok, that is the way it looks above in this question (wth ß symbol).

If I put:

# -*- encoding: utf-8 -*-

as the first string in tmp.py it does not change anything—the error persists.

Could someone help me to figure out what am I doing wrong?

@Blender: u doesn't do anything in Python 3 (and in earlier versions of Python 3 was an error until it was added back for backwards compatibility) — Wooble
More likely you have a problem setting the encoding in your editor. — LtWorf
Also it says encoding error.. not decoding error. Since cp866 is ms-dos code page, I think you are trying to print it to the console, which requires encoding. — Esailija
@Wooble Can you explain to me how does one can determine that? If I do my Firefox trick and choose ISO 8859-1 I see s = 'ÃŸ' instead of s = 'ß'. — Anton Daneyko

Martijn Pieters Martijn Pieters · Accepted Answer · 2013-01-11T18:26:53

The encoding your terminal is using doesn't support that character:

>>> '\xdf'.encode('cp866')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/encodings/cp866.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\xdf' in position 0: character maps to <undefined>

Python is handling it just fine, it's your output encoding that cannot handle it.

You can try using chcp 65001 in the Windows console to switch your codepage; chcp is a windows command line command to change code pages.

Mine, on OS X (using UTF-8) can handle it just fine:

>>> print('\xdf')
ß

Why doesn't Python recognize my utf-8 encoded source file?

1 Answers