python - UnicodeEncodeError: 'ascii' codec can't encode character in position 0: ordinal not in range(128)

Question

I'm working on a Python script that uses the scissor character (9986 - ✂) and I'm trying to port my code to Mac, but I'm running into this error.

The scissor character shows up fine when run from IDLE (Python 3.2.5 - OS X 10.4.11 iBook G4 PPC) and the code works entirely fine on Ubuntu 13.10, but when I attempt to run this in the terminal I get this error/traceback:

Traceback (most recent call last):
  File "snippets-convert.py", line 352, in <module>
    main()
  File "snippets-convert.py", line 41, in main
    menu()
  File "snippets-convert.py", line 47, in menu
    print ("|\t ",snipper.decode(),"PySnipt'd",snipper.decode(),"\t|")
UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128)

and the code that is giving me the problem:

print ("|\t ",chr(9986),"PySnipt'd",chr(9986),"\t|")

Doesn't this signal that the terminal doesn't have the capability to display that character? I know this is an old system, but it is currently the only system I have to use. Could the age of the OS is interfering with the program?

I've read over these questions:

• UnicodeEncodeError: 'ascii' codec can't encode character u'\xef' in position 0: ordinal not in range(128) - Different character

• "UnicodeEncodeError: 'ascii' codec can't encode character" - Using 2.6, so don't know if it applies

• UnicodeEncodeError: 'ascii' codec can't encode character? - Seems to be a plausible solution to my problem, .encode('UTF-8'), I don't get the error. However, it displays a character code, not the character I want, and .decode() just gives me the same error. Not sure if I'm doing this right.

• UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-6: ordinal not in range(128) - Not sure if this applies, he's using a GUI, getting input, and all in Greek.

What's causing this error? Is it the age of the system/OS, the version of Python, or some programming error?

EDIT: This error crops up later with this duplicate issue (just thought I'd add it as it is within the same program and is the same error):

Traceback (most recent call last):
  File "snippets-convert.py", line 353, in <module>
    main()
  File "snippets-convert.py", line 41, in main
    menu()
  File "snippets-convert.py", line 75, in menu
    main()
  File "snippets-convert.py", line 41, in main
    menu()
  File "snippets-convert.py", line 62, in menu
    search()
  File "snippets-convert.py", line 229, in search
    print_results(search_returned)      # Print the results for the user
  File "snippets-convert.py", line 287, in print_results
    getPath(toRead)                                             # Get the path for the snippet
  File "snippets-convert.py", line 324, in getPath
    snipXMLParse(path)
  File "snippets-convert.py", line 344, in snipXMLParse
    print (chr(164),child.text)
UnicodeEncodeError: 'ascii' codec can't encode character '\xa4' in position 0: ordinal not in range(128)

EDIT:

I went into the terminal character settings and it does in fact support that character (as you can see in this screenshot:

when I insert it into terminal it prints out this: \342\234\202 and when I press Enter I get this: -bash: ✂: command not found

EDIT Ran commands as @J.F. Sebastian asked:

python3 test-io-encoding.py:

PYTHONIOENCODING:       None
locale(False):  US-ASCII
device(stdout): US-ASCII
stdout.encoding:        US-ASCII
device(stderr): US-ASCII
stderr.encoding:        US-ASCII
device(stdin):  US-ASCII
stdin.encoding: US-ASCII
locale(False):  US-ASCII
locale(True):   US-ASCII

python3 -S test-io-encoding.py:

PYTHONIOENCODING:       None
locale(False):  US-ASCII
device(stdout): US-ASCII
stdout.encoding:        US-ASCII
device(stderr): US-ASCII
stderr.encoding:        US-ASCII
device(stdin):  US-ASCII
stdin.encoding: US-ASCII
locale(False):  US-ASCII
locale(True):   US-ASCII

EDIT Tried the "hackerish" solution provided by @PauloBu:

As you can see, this caused one (Yay!) scissor, but I am now getting a new error. Traceback/error:

+-=============================-+
✂Traceback (most recent call last):
  File "snippets-convert.py", line 357, in <module>
    main()
  File "snippets-convert.py", line 44, in main
    menu()
  File "snippets-convert.py", line 52, in menu
    print("|\t "+sys.stdout.buffer.write(chr(9986).encode('UTF-8'))+" PySnipt'd "+ sys.stdout.buffer.write(chr(9986).encode('UTF-8'))+" \t|")
TypeError: Can't convert 'int' object to str implicitly

EDIT Added results of @PauloBu's fix:

+-=============================-+
|
✂ PySnipt'd 
✂       |
+-=============================-+

EDIT:

And his fix for his fix:

+-=============================-+
✂✂|       PySnipt'd     |
+-=============================-+

Answer 1

When Python prints and output, it automatically encodes it to the target medium. If it is a file, UTF-8 will be used as default and everyone will be happy, but if it is a terminal, Python will figure out the encoding the terminal is using and will try to encode the output using that one.

This means that if your terminal is using ascii as encoding, Python is trying to encode scissor char to ascii. Of course, ascii doesn't support it so you get Unicode decode error.

This is why you always have to explicitly encode your output. Explicit is better than implicit remember? To fix your code you may do:

import sys
sys.stdout.buffer.write(chr(9986).encode('utf8'))

This seems a bit hackerish. You can also set PYTHONIOENCODING=utf-8 before executing the script. I'am uncomfortable with both solutions. Probably your console doesn't support utf-8 and you see gibberish. But your program will be behaving correctly.

What I strongly recommend if you definitely need to show correct output on your console is to set your console to use another encoding, one that support scissor character. (utf-8 perhaps). On Linux, that can be achieve by doing: export lang=UTF_8. On Windows you change the console's code page with chcp. Just figure out how to set utf8 in yours and IMHO that'll be the best solution.

---

printsys.stdout.write

sys.stdout.buffer.write(("|\t "+ chr(9986) +" PySnipt'd " + chr(9986)+" \t|").encode('utf8'))

I suggest you to take a read at the docs to see what's going on under the hood with print function and with sys.stdout: http://docs.python.org/3/library/sys.html#sys.stdin

Hope this helps!

Answer 2

test_io_encoding.py output suggests that you should change your locale settings e.g., set LANG=en_US.UTF-8.

---

The first error might be due to you are trying to decode a string that is already Unicode. Python 2 tries to encode it using a default character encoding ('ascii') before decoding it using (possibly) different character encoding. The error happens on the encode step:

>>> u"\u2702".decode() # Python 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2702' in position 0: ordinal not in range(128)

It looks like you are running your script using Python 2 instead of Python 3. You would get:

>>> "\u2702".decode() # Python 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'decode'

different error otherwise.

Just drop the .decode() call:

print("|\t {0} PySnipt'd {0} \t|".format(snipper))

The second issue is due to printing a Unicode string into a pipe:

$ python3 -c'print("\u2702")'
✂
$ python3 -c'print("\u2702")' | cat
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128)

Set appropriate for your purpose PYTHONIOENCODING environment variable:

$ PYTHONIOENCODING=utf-8 python3 -c'print("\u2702")' | cat
✂

---

the terminal is just displaying this: | b'\xe2\x9c\x82' PySnipt'd b'\xe2\x9c\x82' |

If snipper is a bytes object then leave the snipper.decode() calls.

$ python3 -c"print(b'\xe2\x9c\x82'.decode())"
✂
$ python3 -c"print(b'\xe2\x9c\x82'.decode())" | cat
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128)

The fix is the same:

$ PYTHONIOENCODING=utf-8 python3 -c"print(b'\xe2\x9c\x82'.decode())" | cat
✂

Answer 3

My locale is set to de_AT.UTF-8 but these lines in /etc/profile were missing:

export LANG=de_AT.UTF-8
export LANGUAGE=de_AT.UTF-8
export LC_ALL=de_AT.UTF-8

logout / login and your problem should be solved

To verify if all locales are set correctly type locale in your terminal

The output should be similar to this:

LANG=de_AT.UTF-8
LANGUAGE=de_AT.UTF-8
LC_CTYPE="de_AT.UTF-8"
LC_NUMERIC="de_AT.UTF-8"
LC_TIME="de_AT.UTF-8"
LC_COLLATE="de_AT.UTF-8"
LC_MONETARY="de_AT.UTF-8"
LC_MESSAGES="de_AT.UTF-8"
LC_PAPER="de_AT.UTF-8"
LC_NAME="de_AT.UTF-8"
LC_ADDRESS="de_AT.UTF-8"
LC_TELEPHONE="de_AT.UTF-8"
LC_MEASUREMENT="de_AT.UTF-8"
LC_IDENTIFICATION="de_AT.UTF-8"
LC_ALL=de_AT.UTF-8

Answer 4

in the first line of your file .py you need to add this string, :

# -- coding: utf-8 --

and you can also try this:

print ("|\t ",unichr(9986),"PySnipt'd",unichr(9986),"\t|")