Pickling a django model with a binary field in cacheops

Question

I have a simple Django model with a binary field that I would like to pickle.

class MyModel(models.Model):
    bin_data = models.BinaryField()

From the context of my unittests, I do the following:

import pickle
tmp_obj = MyModel.objects.create(bin_data="12345")
obj = MyModel.objects.get(pk=tmp_obj.pk)  # load from DB
data = pickle.dumps(obj)
obj2 = pickle.loads(data)

However the pickle.dumps() fails with:

TypeError: can't pickle buffer objects

When I use the following command to pickle:

data = pickle.dumps(obj, protocol=-1)

The dump succeeds but pickle.loads() fails with:

TypeError: buffer() takes at least 1 argument (0 given)

This actually relates to a problem I'm having with the django-cacheops library which I'm using in order to cache my queryset.

Under the hood django-cacheops uses pickle.dumps(obj, protocol=-1), and I receive the same error as described above for the pickle.loads()

I would appreciate an answer for both the pickle issue and the django-cacheops issue.

Thanks

Answer 1

Fixed in cacheops 2.1.1.

Using external pickling function for buffer this way:

import copy_reg
copy_reg.pickle(buffer, lambda b: (buffer, (bytes(b),)))

Answer 2

I managed to solve the mystery so I might as well help anyone else who might encounter it.

This issue is apparently related to a bug in the pickle module in python 2.7 that will not be fixed... http://bugs.python.org/issue8323

In a nutshell, the pickle library (when using the latest protocol) is able to pickle buffer types but not to unpickle them.

When using a BinaryField in a django model, the field type in the model instance when loaded from the DB is 'buffer' which causes the problem.

A simple workaround would be to cast the 'buffer' field into an 'str'.

As to my example, this can be easily done using a post_init signal:

class MyModel(models.Model):
    bin_data = models.BinaryField()


from django.db.models.signals import post_init

def on_model_load(sender, **kwargs):
    model_obj = kwargs.get('instance', None)
    if model_obj and model_obj.bin_data is not None:
        model_obj.bin_data = str(model_obj.bin_data)

post_init.connect(on_model_load, sender=MyModel)

The workaround will allow pickling the model instance and also fix the behavior of the django-cacheops module.