gzip raised OverflowError: Size does not fit in an unsigned int

问题

Environment: Windows, Python 3.4.1, 64-bit version.

I tried to save data with pickle and gzip, simply like this:

with gzip.open(filename, 'rb') as f:
    pickle.dump(data,f)

The data can be successfully dumped without gzip, but with gzip, exception raised as:

File "C:\Python34\lib\gzip.py", line 344, in write
  self.fileobj.write( self.compress.compress(data) )
OverflowError: Size does not fit in an unsigned int

I traced back the code, and found that gzip is actually built upon zlib. And after googling this problem, I came across this page http://bugs.python.org/file32715/zlib_64bit-4.patch. There seems a length limit of unsigned int type is imposed.

So, my question is, is there any way to make up for this bug or pass around it?

回答1:

You could try wrapping the gzip file in a writer that splits the data into chunks of a given maximum size. Here is a sketch:

class ChunkedWriter(object):
    def __init__(self, file, chunksize=65536):
        self.file = file
        self.chunksize = chunksize

    def write(self, data):
        mdata = memoryview(data)
        for i in range(0, len(mdata), self.chunksize):
            self.file.write(bytes(mdata[i:i+self.chunksize]))

I'm not sure whether this will actually solve your problem since I was unable to reproduce it on my own computer.

来源：https://stackoverflow.com/questions/33562394/gzip-raised-overflowerror-size-does-not-fit-in-an-unsigned-int