How do you unzip very large files in python?

Question

Using python 2.4 and the built-in ZipFile library, I cannot read very large zip files (greater than 1 or 2 GB) because it wants to store the entire contents of

Answer 1

As of Python 2.6, you can use ZipFile.open() to open a file handle on a file, and copy contents efficiently to a target file of your choosing:

import errno
import os
import shutil
import zipfile

TARGETDIR = '/foo/bar/baz'

with open(doc, "rb") as zipsrc:
    zfile = zipfile.ZipFile(zipsrc)
    for member in zfile.infolist():
       target_path = os.path.join(TARGETDIR, member.filename)
       if target_path.endswith('/'):  # folder entry, create
           try:
               os.makedirs(target_path)
           except (OSError, IOError) as err:
               # Windows may complain if the folders already exist
               if err.errno != errno.EEXIST:
                   raise
           continue
       with open(target_path, 'wb') as outfile, zfile.open(member) as infile:
           shutil.copyfileobj(infile, outfile)

This uses shutil.copyfileobj() to efficiently read data from the open zipfile object, copying it over to the output file.