Default encoding of Python 3's open() under Windows. sys.getdefaultencoding() setting not respected

5
votes

I've been pulling my hair out over this issue for several hours now.

I have a message file I want to generate using django's makemessages command, which works just fine in a Linux environment which actually handles locale settings in a sane way. However, when I try the same thing under Windows, every time python tries to open a file, it assumes it is encoded in cp932 (SHIFT-JIS), which causes all sorts of havoc.

Manually adding encoding='utf-8' to every open call works, but that's hardly a good way of fixing the problem. Is there any way to force open to use a specific default encoding?

  • sys.getdefaultencoding() returns 'utf-8', for some arcane reason this setting is not respected
  • PYTHONIOENCODING and PYTHONENCODING are both set to 'utf-8'
  • My code page is set to cp65001

This is my python version string:

Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32

EDIT: I've noticed that locale.getpreferredencoding() returns cp932, so I guess finding a Windows locale with utf-8 as its default would do the trick. Does such a thing even exist?

1
pythonpython-3.xencodingutf-8
It may be a bug in 3.3 on Windows if you really can't find a solution. On the other hand, being explicit about the encoding when calling open() is not a bad idea. - Simeon Visser
Well, I know that being explicit about the encoding is never a bad idea, the issue is that replacing all calls to open in the relevant parts of Django's utilities isn't really feasible. I've discovered that locale.getpreferredencoding() does return cp932, so that might be the issue. Are there any Windows locales that set this to utf-8? - mval
Possibly, I'm not too familiar with that. But the fact that it returns cp932 may indeed indicate that it's the locale setting on your machine (or Windows in general) that interferes here. - Simeon Visser
I should add that this problem can (in principle) be solved by overriding locale.getpreferredencoding (which I temporarily did to get past the issue), but I really hope there's an alternative way to accomplish this. - mval
You might suggest to Django developers that they let you, rather than Microsoft, specify the encoding that you want Django to use in its internal open calls. In the meanwhile, upgrade at least to 3.3.5 for its bugfixes; there might be some related to unicode. - Terry Jan Reedy

1 Answers

2
votes

Try this

import locale
locale.setlocale(locale.LC_ALL, 'en_US.utf-8')