What are the arguments to the types.CodeType() python call?

Question

I'm currently trying to roll my own "marshal" code for python so i can store compiled python code on Google App Engine to serve scripts on a dynamic way. As you all can verify, "marshal" isn't supported on GAE and "pickle" can't serialize code objects.

I found out i can construct a code object with types.CodeType() but it expects 12 arguments.

As much as i've tried, i can't find any documentation on this call and i really need to construct the code object so i can exec() it. My question is, does anyone know what are the parameters for this types.CodeType() "constructor" or any way to introspect it? i have used the info() function defined here but it spits out just generic info!

Quick FAQ:

• Q: Why compile the code?
• A: CPU time costs real money on Google App Engine, and every bit of CPU cycles i can save counts.
• Q: Why not use "marshal"?
• A: That's one of the unsupported modules in Google App Engine.
• Q: Why not use "pickle"?
• A: Pickle doesn't support serialization of code objects.

UPDATE

Google App Engine infrastructure doesn't allow the instantiation of code objects as of 7th July 2011, so my argument here is moot. Hope this gets fixed in the future on GAE.

Answer 1

I went and took the code found here and removed the dependency for the deprecated "new" module.

import types, copy_reg
def code_ctor(*args):
    # delegate to new.code the construction of a new code object
    return types.CodeType(*args)
def reduce_code(co):
    # a reductor function must return a tuple with two items: first, the
    # constructor function to be called to rebuild the argument object
    # at a future de-serialization time; then, the tuple of arguments
    # that will need to be passed to the constructor function.
    if co.co_freevars or co.co_cellvars:
        raise ValueError, "Sorry, cannot pickle code objects from closures"
    return code_ctor, (co.co_argcount, co.co_nlocals, co.co_stacksize,
        co.co_flags, co.co_code, co.co_consts, co.co_names,
        co.co_varnames, co.co_filename, co.co_name, co.co_firstlineno,
        co.co_lnotab)
# register the reductor to be used for pickling objects of type 'CodeType'
copy_reg.pickle(types.CodeType, reduce_code)
if __name__ == '__main__':
    # example usage of our new ability to pickle code objects
    import cPickle
    # a function (which, inside, has a code object, of course)
    def f(x): print 'Hello,', x
    # serialize the function's code object to a string of bytes
    pickled_code = cPickle.dumps(f.func_code)
    # recover an equal code object from the string of bytes
    recovered_code = cPickle.loads(pickled_code)
    # build a new function around the rebuilt code object
    g = types.FunctionType(recovered_code, globals( ))
    # check what happens when the new function gets called
    g('world')

Answer 2

The C API function PyCode_New is (minimally) documented here: http://docs.python.org/c-api/code.html ­— the C source code of this function (Python 2.7) is here: http://hg.python.org/cpython/file/b5ac5e25d506/Objects/codeobject.c#l43

PyCodeObject *
PyCode_New(int argcount, int nlocals, int stacksize, int flags,
           PyObject *code, PyObject *consts, PyObject *names,
           PyObject *varnames, PyObject *freevars, PyObject *cellvars,
           PyObject *filename, PyObject *name, int firstlineno,
           PyObject *lnotab)

However, in the Python constructor, the last six arguments appear to be swapped around a little. This is the C code that extracts the arguments passed in by Python: http://hg.python.org/cpython/file/b5ac5e25d506/Objects/codeobject.c#l247

if (!PyArg_ParseTuple(args, "iiiiSO!O!O!SSiS|O!O!:code",
                      &argcount, &nlocals, &stacksize, &flags,
                      &code,
                      &PyTuple_Type, &consts,
                      &PyTuple_Type, &names,
                      &PyTuple_Type, &varnames,
                      &filename, &name,
                      &firstlineno, &lnotab,
                      &PyTuple_Type, &freevars,
                      &PyTuple_Type, &cellvars))
    return NULL;

Pythonized:

def __init__(self, argcount, nlocals, stacksize, flags, code,
                   consts, names, varnames, filename, name, 
                   firstlineno, lnotab, freevars=None, cellvars=None): # ...

Answer 3

The question asked:

what are the parameters for this types.CodeType() "constructor"

From the python docs about the inspect module:

co_argcount: number of arguments (not including * or ** args)
co_code: string of raw compiled bytecode
co_consts: tuple of constants used in the bytecode
co_filename: name of file in which this code object was created
co_firstlineno: number of first line in Python source code
co_flags: bitmap: 1=optimized | 2=newlocals | 4=*arg | 8=**arg
co_lnotab: encoded mapping of line numbers to bytecode indices
co_name: name with which this code object was defined
co_names: tuple of names of local variables
co_nlocals: number of local variables
co_stacksize: virtual machine stack space required
co_varnames: tuple of names of arguments and local variables

This blog post has much more detailed explanation: http://tech.blog.aknin.name/2010/07/03/pythons-innards-code-objects/

Note: the blog post talks about python 3 while the quoted python docs above is python 2.7.

Answer 4

Answering the question you need answered rather than the one you asked:

You can't execute arbitrary bytecode in the App Engine Python environment, currently. Although you may be able to access the bytecode or code objects, you can't load one.

You have an alternative, however: per-instance caching. Store a global dict mapping datastore keys (for your datastore entries that store the Python code) to the compiled code object. If the object doesn't exist in the cache, compile it from source and store it there. You'll have to do the compilation work on each instance, but you don't have to do it on each request, which should save you a lot of work.