Python zipfile module can't extract filenames with Chinese characters

Question

I'm trying to use a python script to download files from a Chinese service provider (I'm not from China myself). The provider is giving me a .zip file which contains a file which seems to have Chinese characters in its name. This seems to be causing the zipfile module to barf.

Code:

import zipfile

f = "/path/to/zip_file.zip"

if zipfile.is_zipfile(f):
    fz = zipfile.ZipFile(f, 'r')

The zipfile itself doesn't contain any non-ASCII characters but the file inside it does. When I run the above script i get the following exception:

Traceback (most recent call last):   File "./temp.py", line 9, in <module>
    fz = zipfile.ZipFile(f, 'r')   File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()   File "/usr/lib/python2.7/zipfile.py", line 859, in _RealGetContents
    x.filename = x._decodeFilename()   File "/usr/lib/python2.7/zipfile.py", line 379, in _decodeFilename
    return self.filename.decode('utf-8')   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xbd in position 30: invalid start byte

I've tried looking through the answers to many similar questions:

• Read file with Chinese Characters
• Extract zip files with non-unicode filenames
• Extract files with invalid characters

Please correct me if I'm wrong, but it looks like an open issue with the zipfile module.

How do I get around this? Is there any alternative module for dealing with zipfiles that I should use? Or any other solution?

TIA.

Edit: I can access/unzip the same file perfectly with the linux command-line utility "unzip".

Answer 1

The way of Python 2.x(2.7) and Python 3.x dealing with non utf-8 filename in module zipfile are a bit different.

First, they both check ZipInfo.flag_bits of the file, if ZipInfo.flag_bits & 0x800, name of the file will be decode with utf-8.

If the check of above is False, in Python 2.x, the byte string of the name will be returned; in Python 3.x, the module will decode the file with encoding cp437 and return decoded result. Of course, the module will not know the true encoding of the filename in both Python versions.

So, suppose you have got a filename from a ZipInfo object or zipfile.namelist method, and you have already know the filename is encoded with XXX encoding. Those are the ways you get the correct unicode filename:

# in python 2.x
filename = filename.decode('XXX')


# in python 3.x
filename = filename.encode('cp437').decode('XXX')

Answer 2

Recently I met the same problem. Here is my solution. I hope it is useful for you.

import shutil
import zipfile
f = zipfile.ZipFile('/path/to/zip_file.zip', 'r')
for fileinfo in f.infolist():
    filename = fileinfo.filename.encode('cp437').decode('gbk')
    outputfile = open(filename, "wb")
    shutil.copyfileobj(f.open(fileinfo.filename), outputfile)
    outputfile.close()
f.close()

UPDATE: You can use the following simpler solution with pathlib:

from pathlib import Path
import zipfile

with zipfile.ZipFile('/path/to/zip_file.zip', 'r') as f:
    for fn in f.namelist():
        extracted_path = Path(f.extract(fn))
        extracted_path.rename(fn.encode('cp437').decode('gbk'))

Answer 3

The ZIP file is invalid. It has a flag that signals that filenames inside it are encoded as UTF-8, but they're actually not; they contain byte sequences that aren't valid as UTF-8. Maybe they're GBK? Maybe something else? Maybe some unholy inconsistent mixture? ZIP tools in the wild are unfortunately very very poor at handling non-ASCII filenames consistently.

A quick workaround might be to replace the library function that decodes the filenames. This is a monkey-patch as there isn't a simple way to inject your own ZipInfo class into ZipFile, but:

zipfile.ZipInfo._decodeFilename = lambda self: self.filename

would disable the attempt to decode the filename, and always return a ZipInfo with a byte string filename property that you can proceed to decode/handle manually in whatever way is appropriate.

Answer 4

What about this code?

import zipfile

with zipfile.ZipFile('/path/to/zip_file.zip', 'r') as f:
    zipInfo = f.infolist()
    for member in zipInfo:
        member.filename = member.filename.encode('cp437').decode('gbk')
        f.extract(member)