UnicodeDecodeError on attempt to save file through django default filebased backend

Question

When i attempt to add a file with russian symbols in name to the model instance through default instance.file_field.save method, i get an UnicodeDecodeError (ascii decoding error, not in range (128) from the storage backend (stacktrace ended on os.exist). If i write this file through default python file open/write all goes right. All filenames in utf-8. I get this error only on testing Gentoo, on my Ubuntu workstation all works fine.

class Article(models.Model):
    file = models.FileField(null=True, blank=True, max_length = 300,
                            upload_to='articles_files/%Y/%m/%d/')

Traceback:
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
  100.                     response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  24.                 return view_func(request, *args, **kwargs)
File "/var/www/localhost/help/wiki/views.py" in edit_article
  338.                 new_article.file.save(fp, fi, save=True)
File "/usr/lib/python2.6/site-packages/django/db/models/fields/files.py" in save
  92.         self.name = self.storage.save(name, content)
File "/usr/lib/python2.6/site-packages/django/core/files/storage.py" in save
  47.         name = self.get_available_name(name)
File "/usr/lib/python2.6/site-packages/django/core/files/storage.py" in get_available_name
  73.         while self.exists(name):
File "/usr/lib/python2.6/site-packages/django/core/files/storage.py" in exists
  196.         return os.path.exists(self.path(name))
File "/usr/lib/python2.6/genericpath.py" in exists
  18.         st = os.stat(path)

Exception Type: UnicodeEncodeError at /edit/
Exception Value: ('ascii', u'/var/www/localhost/help/i/articles_files/2010/03/17/\u041f\u0440\u0438\u0432\u0435\u0442', 52, 58, 'ordinal not in range(128)')

It's usual class Article(models.Model): ... file = models.FileField(null=True, blank=True, max_length = 300, upload_to='articles_files/%Y/%m/%d/') — Ivan Kuznetsov
I think that problem in os difference, on all my ubuntu workstation all works fine. — Ivan Kuznetsov
Full stacktrace on my office pc, but it's not so helpful in this case (i call save method, it calls storage save method, i'll post it tomorrow). — Ivan Kuznetsov

Ivan Kuznetsov Ivan Kuznetsov · Accepted Answer · 2010-03-17T10:26:44

The solution is quite simple:

In revision 12659 this bug was fixed. http://code.djangoproject.com/ticket/11030

But revision 12661 reverted it

"(In [12661]) Fixed #11030: Reverted a change that assumed the file system encoding was utf8, and changed a test to demonstrate how that assumption corrupted uploaded non-ASCII file names on systems that don't use utf8 as their file system encoding (Windows for one, specifically). Thanks for the report to vrehak."

So all I need to do is revert to 12659

UnicodeDecodeError on attempt to save file through django default filebased backend

2 Answers