_pickle in python3 doesn't work for large data saving

Question

I am trying to apply _pickle to save data onto disk. But when calling _pickle.dump, I got an error

OverflowError: cannot serialize          


        

Answer 1

Yes, this is a hard-coded limit; from save_bytes function:

else if (size <= 0xffffffffL) {
    // ...
}
else {
    PyErr_SetString(PyExc_OverflowError,
                    "cannot serialize a bytes object larger than 4 GiB");
    return -1;          /* string too large */
}

The protocol uses 4 bytes to write the size of the object to disk, which means you can only track sizes of up to 2³² == 4GB.

If you can break up the bytes object into multiple objects, each smaller than 4GB, you can still save the data to a pickle, of course.