How to check if an object is pickleable

Question

I have a list of objects of various types that I want to pickle. I would like to pickle only those which are pickleable. Is there a standard way to check if an object is of

Answer 1

I would propose duck testing in this case. Try to pickle into a temporary file or a memory file, as you find suitable, then if it fails discard the result, if it succeeds rename.

Why?

In python you can check if the object has some properties in two ways.

Check if object is an instance of some Abstract Base Class. E.g. Number "The root of the numeric hierarchy. If you just want to check if an argument x is a number, without caring what kind, use isinstance(x, Number)."

Or try it and then handle exceptions. This occurs during many occasions. The pythonic philosopy is based around the duck. Duck typing, duck test, and EAFP are the keywords.

I even believe the 1st one has been properly introduced with python3 under the pressure from the part of the community, while many still strongly believe duck is the way to go with python.

AFAIK there is no special preconditions that can be checked, nor any ABC that object can be checked against in case of pickling. So all that is left is duck.

Maybe something else could be attempted but probably it is not worth of it. It would be very hard to do manual introspection of the object to find out preliminarily if it's suitable for pickling.

Answer 2

There's the dill.pickles method in dill package that does just that.

>>> class Foo(object):
...   x = iter([1,2,3])
... 
>>> f = Foo()     
>>> 
>>> dill.pickles(f)
False

We can use methods in dill to look for what causes the failure.

>>> dill.detect.badtypes(f)
<class '__main__.Foo'>
>>> dill.detect.badtypes(f, depth=1)
{'__setattr__': <type 'method-wrapper'>, '__reduce_ex__': <type 'builtin_function_or_method'>, '__reduce__': <type 'builtin_function_or_method'>, '__str__': <type 'method-wrapper'>, '__format__': <type 'builtin_function_or_method'>, '__getattribute__': <type 'method-wrapper'>, '__class__': <type 'type'>, '__delattr__': <type 'method-wrapper'>, '__subclasshook__': <type 'builtin_function_or_method'>, '__repr__': <type 'method-wrapper'>, '__hash__': <type 'method-wrapper'>, 'x': <type 'listiterator'>, '__sizeof__': <type 'builtin_function_or_method'>, '__init__': <type 'method-wrapper'>}
>>> dill.detect.badtypes(f, depth=1).keys()
['__setattr__', '__reduce_ex__', '__reduce__', '__str__', '__format__', '__getattribute__', '__class__', '__delattr__', '__subclasshook__', '__repr__', '__hash__', 'x', '__sizeof__', '__init__']

So, the only thing that's failing that's not a "built-in" method of the class is x… so that's a good place to start. Let's check 'x', then replace it with something else if it's the problem.

>>> dill.pickles(Foo.x)
False
>>> Foo.x = xrange(1,4)
>>> dill.pickles(Foo.x)
True

Yep, x was causing a failure, and replacing it with an xrange works because dill can pickle an xrange. What's left to do?

>>> dill.detect.badtypes(f, depth=1).keys()
[]
>>> dill.detect.badtypes(f, depth=1)       
{}
>>> dill.pickles(f)                 
True
>>> 

Apparently (likely because references to x in the class __dict__ now pickle), f now pickles… so we are done.

dill also provides trace to show the exact path in pickling the object.

>>> dill.detect.trace(True)
>>> dill.pickles(f)
T2: <class '__main__.Foo'>
F2: <function _create_type at 0x10e79b668>
T1: <type 'type'>
F2: <function _load_type at 0x10e79b5f0>
T1: <type 'object'>
D2: <dict object at 0x10e7c6168>
Si: xrange(1, 4)
F2: <function _eval_repr at 0x10e79bcf8>
D2: <dict object at 0x10e7c6280>
True

Answer 3

dill allows for pickling more things that the builtin pickle.

This should do what you what, I think:

def is_picklable(obj):
  try:
    pickle.dumps(obj)

  except pickle.PicklingError:
    return False
  return True