Extract files with invalid characters in filename with Python

Question

I use python's zipfile module to extract a .zip archive (Let's take this file at http://img.dafont.com/dl/?f=akvaleir for example.)

f = zipfile.ZipFile('akvaleir.zip', 'r')
for fileinfo in f.infolist():
    print fileinfo.filename
    f.extract(fileinfo, '.')

Its output:

Akval�ir_Normal_v2007.ttf
Akval�ir, La police - The Font - Fr - En.pdf

Both files are unaccessable after extraction because there are invalid encoded characters in their filenames. The problem is zipfile module doesn't have an option to specify output filenames.

However, "unzip akvaleir.zip" escapes the filename well:

root@host:~# unzip akvaleir.zip 
Archive:  akvaleir.zip
  inflating: AkvalВir_Normal_v2007.ttf  
  inflating: AkvalВir, La police - The Font - Fr - En.pdf  

I tried capturing output of "unzip -l akvaleir.zip" in my python program and these two filenames are:

Akval\xd0\x92ir_Normal_v2007.ttf
Akval\xd0\x92ir, La police - The Font - Fr - En.pdf

How can I get the correct filename like what unzip command does without capturing output of "unzip -l akvaleir.zip"?

Answer 1

Instead of the extract method, use the open method and save the resulting pseudofile to disk under whatever name you wish, for example with shutil.copyfileobj.

Answer 2

It took some time but I think I found the answer.

I assumed the word was supposed to be Akvaléir. I found a page description about that, in French. When I used your code snippet I had a string like

>>> fileinfo.filename
'Akval\x82ir, La police - The Font - Fr - En.pdf'
>>> 

That didn't work at UTF8, Latin-1, CP-1251 or CP-1252 encodings. I then found that CP863 was a possible Canadian encoding, so perhaps this was from French Canada.

>>> print unicode(fileinfo.filename, "cp863").encode("utf8")
Akvaléir, La police - The Font - Fr - En.pdf
>>> 

However, I then read the Zip file format specification which says

The ZIP format has historically supported only the original IBM PC character encoding set, commonly referred to as IBM Code Page 437.

...

If general purpose bit 11 is set, the filename and comment must support The Unicode Standard, Version 4.1.0 or greater using the character encoding form defined by the UTF-8 storage specification.

Testing that out gives me the same answer as the Canadian code page

>>> print unicode(fileinfo.filename, "cp437").encode("utf8")
Akvaléir, La police - The Font - Fr - En.pdf
>>>

I don't have a Unicode encoded zip file and I'm not going to create one to find out, so I'll just assume that all zip files have the cp437 encoding.

import shutil
import zipfile

f = zipfile.ZipFile('akvaleir.zip', 'r')
for fileinfo in f.infolist():
    filename = unicode(fileinfo.filename, "cp437")
    outputfile = open(filename, "wb")
    shutil.copyfileobj(f.open(fileinfo.filename), outputfile)

On my Mac that gives

 109936 Nov 27 01:46 Akvale??ir_Normal_v2007.ttf
  25244 Nov 27 01:46 Akvale??ir, La police - The Font - Fr - En.pdf

which tab-completes to

ls Akvale\314\201ir

and shows up with a nice 'é' in my file browser.

Answer 3

I ran into a similar issue while running my application using Docker. Adding this lines to the Dockerfile, fixed everything for me:

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

So, I guess if you're not using Docker, give it a try and make sure locales are properly generated and set.