Is there an easy way to pickle a python function (or otherwise serialize its code)?

问题

I'm trying to transfer a function across a network connection (using asyncore). Is there an easy way to serialize a python function (one that, in this case at least, will have no side affects) for transfer like this?

I would ideally like to have a pair of functions similar to these:

def transmit(func):
    obj = pickle.dumps(func)
    [send obj across the network]

def receive():
    [receive obj from the network]
    func = pickle.loads(s)
    func()

回答1:

You could serialise the function bytecode and then reconstruct it on the caller. The marshal module can be used to serialise code objects, which can then be reassembled into a function. ie:

import marshal
def foo(x): return x*x
code_string = marshal.dumps(foo.func_code)

Then in the remote process (after transferring code_string):

import marshal, types

code = marshal.loads(code_string)
func = types.FunctionType(code, globals(), "some_func_name")

func(10)  # gives 100

A few caveats:

marshal's format (any python bytecode for that matter) may not be compatable between major python versions.
Will only work for cpython implementation.
If the function references globals (including imported modules, other functions etc) that you need to pick up, you'll need to serialise these too, or recreate them on the remote side. My example just gives it the remote process's global namespace.
You'll probably need to do a bit more to support more complex cases, like closures or generator functions.

回答2:

Check out Dill, which extends Python's pickle library to support a greater variety of types, including functions:

>>> import dill as pickle
>>> def f(x): return x + 1
...
>>> g = pickle.dumps(f)
>>> f(1)
2
>>> pickle.loads(g)(1)
2

It also supports references to objects in the function's closure:

>>> def plusTwo(x): return f(f(x))
...
>>> pickle.loads(pickle.dumps(plusTwo))(1)
3

回答3:

Pyro is able to do this for you.

回答4:

The most simple way is probably inspect.getsource(object) (see the inspect module) which returns a String with the source code for a function or a method.

回答5:

It all depends on whether you generate the function at runtime or not:

If you do - inspect.getsource(object) won't work for dynamically generated functions as it gets object's source from .py file, so only functions defined before execution can be retrieved as source.

And if your functions are placed in files anyway, why not give receiver access to them and only pass around module and function names.

The only solution for dynamically created functions that I can think of is to construct function as a string before transmission, transmit source, and then eval() it on the receiver side.

Edit: the marshal solution looks also pretty smart, didn't know you can serialize something other thatn built-ins

回答6:

The cloud package (pip install cloud) can pickle arbitrary code, including dependencies. See https://stackoverflow.com/a/16891169/1264797.

回答7:

The basic functions used for this module covers your query, plus you get the best compression over the wire; see the instructive source code:

y_serial.py module :: warehouse Python objects with SQLite

"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."

http://yserial.sourceforge.net

回答8:

code_string = '''
def foo(x):
    return x * 2
def bar(x):
    return x ** 2
'''

obj = pickle.dumps(code_string)

Now

exec(pickle.loads(obj))

foo(1)
> 2
bar(3)
> 9

回答9:

Cloudpickle is probably what you are looking for. Cloudpickle is described as follows:

cloudpickle is especially useful for cluster computing where Python code is shipped over the network to execute on remote hosts, possibly close to the data.

Usage example:

def add_one(n):
  return n + 1

pickled_function = cloudpickle.dumps(add_one)
pickle.loads(pickled_function)(42)

回答10:

You can do this:

def fn_generator():
    def fn(x, y):
        return x + y
    return fn

Now, transmit(fn_generator()) will send the actual definiton of fn(x,y) instead of a reference to the module name.

You can use the same trick to send classes across network.

来源：https://stackoverflow.com/questions/1253528/is-there-an-easy-way-to-pickle-a-python-function-or-otherwise-serialize-its-cod

标签

python

function

pickle