Store object using Python pickle, and load it into different namespace

Question

I\'d like to pass object state between two Python programs (one is my own code running standalone, one is a Pyramid view), and different namespaces. Somewhat related questio

Answer 1

Solution 1

On pickle.load, the module __main__ needs to have a function or class called MyClass. This does not need to be the original class with the original source code. You can put other methods in it. It should work.

class MyClass(object):
    pass

with open("my_c.pik", "rb") as f :
    c = pickle.load(f)

Solution 2

Use the copyreg module which is used to register constructors and pickle functions to pickle specific objects. This is the example given by the module for a complex number:

def pickle_complex(c):
    return complex, (c.real, c.imag)

copyreg.pickle(complex, pickle_complex, complex)

Solution 3

Override the persistent_id method of the Pickler and Unpickler. pickler.persistent_id(obj) shall return an identifier that can be resolved by unpickler.persistent_id(id) to the object.

Answer 2

The easiest solution is to use cloudpickle:

https://github.com/cloudpipe/cloudpickle

It enabled me to easily send a pickled class file to another machine and unpickle it using cloudpickle again.

Answer 3

Use dill instead of pickle, because dill by default pickles by serializing the class definition and not by reference.

>>> import dill
>>> class MyClass:
...   def __init__(self): 
...     self.data = set()
...     self.more = dict()
...   def do_stuff(self):
...     return sorted(self.more)
... 
>>> c = MyClass()
>>> c.data.add(1)
>>> c.data.add(2)
>>> c.data.add(3)
>>> c.data
set([1, 2, 3])
>>> c.more['1'] = 1
>>> c.more['2'] = 2
>>> c.more['3'] = lambda x:x
>>> def more_stuff(self, x):  
...   return x+1
... 
>>> c.more_stuff = more_stuff
>>> 
>>> with open('my_c.pik', "wb") as f:
...   dill.dump(c, f)
... 
>>> 

Shut down the session, and restart in a new session…

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> with open('my_c.pik', "rb") as f:
...   c = dill.load(f)
... 
>>> c.data
set([1, 2, 3])
>>> c.more
{'1': 1, '3': <function <lambda> at 0x10473ec80>, '2': 2}
>>> c.do_stuff()
['1', '2', '3']
>>> c.more_stuff(5)
6

Get dill here: https://github.com/uqfoundation/dill