How to read from a text file compressed with 7z?

Question

I would like to read (in Python 2.7), line by line, from a csv (text) file, which is 7z compressed. I don't want to decompress the entire (large) file, but to stream the lines.

I tried pylzma.decompressobj() unsuccessfully. I get a data error. Note that this code doesn't yet read line by line:

input_filename = r"testing.csv.7z"
with open(input_filename, 'rb') as infile:
    obj = pylzma.decompressobj()
    o = open('decompressed.raw', 'wb')
    obj = pylzma.decompressobj()
    while True:
        tmp = infile.read(1)
        if not tmp: break
        o.write(obj.decompress(tmp))
    o.close()

Output:

    o.write(obj.decompress(tmp))
ValueError: data error during decompression

Answer 1

This will allow you to iterate the lines. It's partially derived from some code I found in an answer to another question.

As far as I can tell, at this point in time the py7zlib doesn't provide an API that would allow archive members to be read as a stream of bytes or characters—its ArchiveFile class only provides a read() function that decompresses and returns all the uncompressed data at once that comprised the member. Given that, about the best you can do is return bytes or lines iteratively using that as a buffer. The following does that, but many not help if the problem is the archive member file itself is huge.

I've modified the code below to work in both Python 2.7 and 3.x.

import io
import os
import py7zlib

class SevenZFileError(py7zlib.ArchiveError):
    pass

class SevenZFile(object):
    @classmethod
    def is_7zfile(cls, filepath):
        """ Determine if filepath points to a valid 7z archive. """
        is7z = False
        fp = None
        try:
            fp = open(filepath, 'rb')
            archive = py7zlib.Archive7z(fp)
            _ = len(archive.getnames())
            is7z = True
        finally:
            if fp: fp.close()
        return is7z

    def __init__(self, filepath):
        fp = open(filepath, 'rb')
        self.filepath = filepath
        self.archive = py7zlib.Archive7z(fp)

    def __contains__(self, name):
        return name in self.archive.getnames()

    def readlines(self, name):
        """ Iterator of lines from an archive member. """
        if name not in self:
            raise SevenZFileError('archive member %r not found in %r' %
                                  (name, self.filepath))

        for line in io.StringIO(self.archive.getmember(name).read().decode()):
            yield line

Sample usage:

import csv

if SevenZFile.is_7zfile('testing.csv.7z'):
    sevenZfile = SevenZFile('testing.csv.7z')

    if 'testing.csv' not in sevenZfile:
        print('testing.csv is not a member of testing.csv.7z')
    else:
        reader = csv.reader(sevenZfile.readlines('testing.csv'))
        for row in reader:
            print(', '.join(row))

Answer 2

If you were using Python 3.3+, you might be able to do this using the lzma module which was added to the standard library in that version.

See: lzma Examples