Google App Engine: Non-western characters in datastore, how to print to browser

Question

This is a follow up question to this: Google Analytics Character encoding of __utm cookies

I have the following string stored in a datastore db.TextProperty - Test%D0%AFTest. This was read from a utf-8 encoded cookie value.

I'm trying to simply print this string to the browser as TestЯTest (Notice the backwards R) but instead all I can get to display is: TestЯTest

I've tried all kinds of unicode, encode('utf-8') decode('utf-8') solutions but nothing seems to work. I either get a UnicodeDecode Error or the above text.

My HTTP headers contain the line: Content-Type: text/html; charset=utf-8 and the HTML itself contains the meta tag: http-equiv="Content-Type" content="text/html; charset=utf-8"

Nothing seems to work.

Answer 1

Since it is being stored as part of a URL, you will have to unparse it using urllib.unparse. You have the right idea with the encoding, you just need to do the extra step of "unparsing" the URL.

>>> import urlparse
>>> s = 'Test%D0%AFTest'
>>> print urlparse.unquote(s).decode('utf8')
TestЯTest