Pickle Cython Class with C pointers

Question

I am trying to write a __reduce__() method for a cython class that contains C pointers but have so far found very little information on the best way to go about

Answer 1

One approach is to serialise the data in your array into a Python bytes array. The __reduce__ method first calls the get_data method which casts the data pointer to <char*> then to <bytes> (if you try to go there directly Cython doesn't know how to do it). __reduce__ returns this object, along with a reference to the rebuild function (a module-level function, not a method!) which can be use to recreate the instance using the set_data method. If you need to pass more than one array, as in your example, you just need to accept more arguments to rebuild and extend the tuple returned by __reduce__.

I haven't done much testing on this but it seems to work. It would probably explode if you passed it malformed data.

from cpython.mem cimport PyMem_Malloc, PyMem_Realloc, PyMem_Free
from libc.string cimport memcpy

cdef int length = 40

cdef class MyClass:
    cdef long *data

    def __cinit__(self):
        self.data = <long*>PyMem_Malloc(sizeof(long)*length)
        if not self.data:
            raise MemoryError()

    cdef bytes get_data(self):
        return <bytes>(<char *>self.data)[:sizeof(long)*length]

    cdef void set_data(self, bytes data):
        memcpy(self.data, <char*>data, sizeof(long)*length)

    def set_values(self):
        # assign some dummy data to the array 0..length
        for n in range(0, length):
            self.data[n] = n

    def get(self, i):
        # get the ith value of the data
        return self.data[i]

    def __reduce__(self):
        data = self.get_data()
        return (rebuild, (data,))

    def __dealloc__(self):
        PyMem_Free(self.data)

cpdef object rebuild(bytes data):
    c = MyClass()
    c.set_data(data)
    return c

Example usage (assuming MyClass is in hello.pyx):

import hello
import pickle

c1 = hello.MyClass()
c1.set_values()
print('c1', c1)
print('fifth item', c1.get(5))

d = pickle.dumps(c1)
del(c1)  # delete the original object

c2 = pickle.loads(d)
print('c2', c2)
print('fifth item', c2.get(5))