Is it possible to programmatically construct a Python stack frame and start execution at an arbitrary point in the code?

允我心安 提交于 2019-11-28 17:04:34

The expat python bindings included in the normal Python distribution is constructing stack frames programtically. Be warned though, it relies on undocumented and private APIs.

http://svn.python.org/view/python/trunk/Modules/pyexpat.c?rev=64048&view=auto

What you generally want are continuations, which I see is already a tag on this question.

If you have the ability to work with all of the code in the system, you may want to try doing it this way rather than dealing with the interpreter stack internals. I'm not sure how easily this will be persisted.

http://www.ps.uni-sb.de/~duchier/python/continuations.html

In practice, I would structure your workflow engine so that your script submits action objects to a manager. The manager could pickle the set of actions at any point and allow them to be loaded and begin execution again (by resuming the submission of actions).

In other words: make your own, application-level, stack.

Stackless python is probably the best… if you don't mind totally going over to a different python distribution. stackless can serialize everything in python, plus their tasklets. If you want to stay in the standard python distribution, then I'd use dill, which can serialize almost anything in python.

>>> import dill
>>> 
>>> def foo(a):
...   def bar(x):
...     return a*x
...   return bar
... 
>>> class baz(object):
...   def __call__(self, a,x):
...     return foo(a)(x)
... 
>>> b = baz()
>>> b(3,2)
6
>>> c = baz.__call__
>>> c(b,3,2)
6
>>> g = dill.loads(dill.dumps(globals()))
>>> g
{'dill': <module 'dill' from '/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <module '__builtin__' (built-in)>, 'baz': <class '__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x4d39d30>, '__doc__': None}

Dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

Here's dill pickling the whole interpreter session...

>>> # continuing from above
>>> dill.dump_session('foobar.pkl')
>>>
>>> ^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foobar.pkl')
>>> c(b,3,2)
6

dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

You also asked for where it's used to save interpreter state?

IPython can use dill to save the interpreter session to a file. https://nbtest.herokuapp.com/github/ipython/ipython/blob/master/examples/parallel/Using%20Dill.ipynb

klepto uses dill to support in-memory, to-disk, or to-database caching that avoids recomputation. https://github.com/uqfoundation/klepto/blob/master/tests/test_cache_info.py

mystic uses dill to save the checkpoints for large optimization jobs by saving the state of the optimizer as it's in progress. https://github.com/uqfoundation/mystic/blob/master/tests/test_solver_state.py

There are a couple other packages that use dill to save state of objects or sessions.

You could grab the existing stack frame by throwing an exception and stepping back one frame along in the traceback. The problem is there is no way provided to resume execution in the middle (frame.f_lasti) of the code block.

“Resumable exceptions” are a really interesting language idea, although it's tricky to think of a reasonable way they could interact with Python's existing ‘try/finally’ and ‘with’ blocks.

For the moment, the normal way of doing this is simply to use threads to run your workflow in a separate context to its controller. (Or coroutines/greenlets if you don't mind compiling them in).

With standard CPython this is complicated by the mixture of C and Python data in the stack. Rebuilding the call stack would require the C stack to be reconstructed at the same time. This really puts it in the too hard basket as it could potentially tightly couple the implementation to specific versions of CPython.

Stackless Python allows tasklets to be pickled, which gives most of the capability required out of the box.

I have the same type of problem to solve. I wonder what the original poster decided to do.

stackless claims it can pickle tasklets as long as there's no associated 'encumbered' C stack (encumbered is my choice of phrasing).

I'll probably use eventlet and figure out some way of pickling 'state', I really don't want to write an explicit state machine though..

How about using joblib?

I'm not quite sure this is what you want but it seems to fit the idea of having a workflow of which stages can be persisted. Joblib's use case seems to be to avoid recomputation, I'm not sure if this is what you are trying to do here or something more complicated?

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!