Python serialize lexical closures?

Question

Is there a way to serialize a lexical closure in Python using the standard library? pickle and marshal appear not to work with lexical closures. I don\'t really care about

Answer 1

If you simply use a class with a __call__ method to begin with, it should all work smoothly with pickle.

class foo(object):
    def __init__(self, bar, baz):
        self.baz = baz
    def __call__(self,waldo):
        return self.baz * waldo

On the other hand, a hack which converted a closure into an instance of a new class created at runtime would not work, because of the way pickle deals with classes and instances. pickle doesn't store classes; only a module name and class name. When reading back an instance or class it tries to import the module and find the required class in it. If you used a class created on-the-fly, you're out of luck.

Answer 2

PiCloud has released an open-source (LGPL) pickler which can handle function closure and a whole lot more useful stuff. It can be used independently of their cloud computing infrastructure - it's just a normal pickler. The whole shebang is documented here, and you can download the code via 'pip install cloud'. Anyway, it does what you want. Let's demonstrate that by pickling a closure:

import pickle
from StringIO import StringIO

import cloud

# generate a closure
def foo(bar, baz):
    def closure(waldo):
        return baz * waldo
    return closure
closey = foo(3, 5)

# use the picloud pickler to pickle to a string
f = StringIO()
pickler = cloud.serialization.cloudpickle.CloudPickler(f)
pickler.dump(closey)

#rewind the virtual file and reload
f.seek(0)
closey2 = pickle.load(f)

Now we have closey, the original closure, and closey2, the one that has been restored from a string serialisation. Let's test 'em.

>>> closey(4)
20
>>> closey2(4)
20

Beautiful. The module is pure python—you can open it up and easily see what makes the magic work. (The answer is a lot of code.)

Answer 3

Yes! I got it (at least I think) -- that is, the more generic problem of pickling a function. Python is so wonderful :), I found out most of it though the dir() function and a couple of web searches. Also wonderful to have it [hopefully] solved, I needed it also.

I haven't done a lot of testing on how robust this co_code thing is (nested fcns, etc.), and it would be nice if someone could look up how to hook Python so functions can be pickled automatically (e.g. they might sometimes be closure args).

Cython module _pickle_fcn.pyx

# -*- coding: utf-8 -*-

cdef extern from "Python.h":
    object PyCell_New(object value)

def recreate_cell(value):
    return PyCell_New(value)

Python file

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# author gatoatigrado [ntung.com]
import cPickle, marshal, types
import pyximport; pyximport.install()
import _pickle_fcn

def foo(bar, baz) :
    def closure(waldo) :
        return baz * waldo
    return closure

# really this problem is more about pickling arbitrary functions
# thanks so much to the original question poster for mentioning marshal
# I probably wouldn't have found out how to serialize func_code without it.
fcn_instance = foo("unused?", -1)
code_str = marshal.dumps(fcn_instance.func_code)
name = fcn_instance.func_name
defaults = fcn_instance.func_defaults
closure_values = [v.cell_contents for v in fcn_instance.func_closure]
serialized = cPickle.dumps((code_str, name, defaults, closure_values),
    protocol=cPickle.HIGHEST_PROTOCOL)

code_str_, name_, defaults_, closure_values_ = cPickle.loads(serialized)
code_ = marshal.loads(code_str_)
closure_ = tuple([_pickle_fcn.recreate_cell(v) for v in closure_values_])
# reconstructing the globals is like pickling everything :)
# for most functions, it's likely not necessary
# it probably wouldn't be too much work to detect if fcn_instance global element is of type
# module, and handle that in some custom way
# (have the reconstruction reinstantiate the module)
reconstructed = types.FunctionType(code_, globals(),
    name_, defaults_, closure_)
print(reconstructed(3))

cheers,
Nicholas

EDIT - more robust global handling is necessary for real-world cases. fcn.func_code.co_names lists global names.

Answer 4

Recipe 500261: Named Tuples contains a function that defines a class on-the-fly. And this class supports pickling.

Here's the essence:

result.__module__ = _sys._getframe(1).f_globals.get('__name__', '__main__')

Combined with @Greg Ball's suggestion to create a new class at runtime it might answer your question.

Answer 5

#!python

import marshal, pickle, new

def dump_func(f):
    if f.func_closure:
        closure = tuple(c.cell_contents for c in f.func_closure)
    else:
        closure = None
    return marshal.dumps(f.func_code), f.func_defaults, closure


def load_func(code, defaults, closure, globs):
    if closure is not None:
        closure = reconstruct_closure(closure)
    code = marshal.loads(code)
    return new.function(code, globs, code.co_name, defaults, closure)


def reconstruct_closure(values):
    ns = range(len(values))
    src = ["def f(arg):"]
    src += [" _%d = arg[%d]" % (n, n) for n in ns]
    src += [" return lambda:(%s)" % ','.join("_%d"%n for n in ns), '']
    src = '\n'.join(src)
    try:
        exec src
    except:
        raise SyntaxError(src)
    return f(values).func_closure




if __name__ == '__main__':

    def get_closure(x):
        def the_closure(a, b=1):
            return a * x + b, some_global
        return the_closure

    f = get_closure(10)
    code, defaults, closure = dump_func(f)
    dump = pickle.dumps((code, defaults, closure))
    code, defaults, closure = pickle.loads(dump)
    f = load_func(code, defaults, closure, globals())

    some_global = 'some global'

    print f(2)