Lambdify works with Python, but throws an exception with Cython

Question

My website runs this Python script that would be way more optimized if Cython is used. Recently I needed to add Sympy with Lambdify, and this is not going well with Cython.

So I stripped the problem to a minimum working example. In the code, I have a dictionary with string keys with values that are lists. I would like to use these keys as variables. In the following simplified example, there's only 1 variable, but generally I need more. Please check the following example:

import numpy as np
from sympy.parsing.sympy_parser import parse_expr
from sympy.utilities.lambdify import lambdify, implemented_function
from sympy import S, Symbol
from sympy.utilities.autowrap import ufuncify


def CreateMagneticFieldsList(dataToSave,equationString,DSList):

    expression  = S(equationString)
    numOfElements = len(dataToSave["MagneticFields"])

    #initialize the magnetic field output array
    magFieldsArray    = np.empty(numOfElements)
    magFieldsArray[:] = np.NaN

    lam_f = lambdify(tuple(DSList),expression,modules='numpy')
    try:
        # pass
        for i in range(numOfElements):
            replacementList = np.zeros(len(DSList))


            for j in range(len(DSList)):
                replacementList[j] = dataToSave[DSList[j]][i]

            try:
                val = np.double(lam_f(*replacementList))

            except:
                val = np.nan
            magFieldsArray[i] = val
    except:
        print("Error while evaluating the magnetic field expression")
    return magFieldsArray


list={"MagneticFields":[1,2,3,4,5]}

out=CreateMagneticFieldsList(list,"MagneticFields*5.1",["MagneticFields"])

print(out)

Let's call this test.py. This works very well. Now I would like to cythonize this, so I use the following script:

#!/bin/bash

cython --embed -o test.c test.py
gcc -pthread -fPIC -fwrapv -Ofast -Wall -L/lib/x86_64-linux-gnu/ -lpython3.4m -I/usr/include/python3.4 -o test.exe test.c

Now if I execute ./test.exe, it throws an exception! Here's the exception:

Traceback (most recent call last):
  File "test.py", line 42, in init test (test.c:1811)
    out=CreateMagneticFieldsList(list,"MagneticFields*5.1",["MagneticFields"])
  File "test.py", line 19, in test.CreateMagneticFieldsList (test.c:1036)
    lam_f = lambdify(tuple(DSList),expression,modules='numpy')
  File "/usr/local/lib/python3.4/dist-packages/sympy/utilities/lambdify.py", line 363, in lambdify
    callers_local_vars = inspect.currentframe().f_back.f_locals.items()
AttributeError: 'NoneType' object has no attribute 'f_locals'

So the question is: How can I get lambdify to work with Cython?

Notes: I would like to point out that I have Debian Jessie, and that's why I'm using Python 3.4. Also I would like to point out that I don't have any problem with Cython when not using lambdify. Also I would like to point out that Cython is updated to the last version with pip3 install cython --upgrade.

Answer 1

This is a something of a workround to the real problem (identified in the comments and @jjhakala's answer) that Cython doesn't generate full tracebacks/introspection information for compiled functions. I gather from your comments that you'd like to keep most of your program compiled with Cython for speed reasons.

The "solution" is to use the Python interpreter for only the individual function that needs to call lambdify and leave the rest in Cython. You can do this using exec.

A very simple example of the idea is

exec("""
def f(func_to_call):
    return func_to_call()""")

# a Cython compiled version    
def f2(func_to_call):
    return func_to_call())

This can be compiled as a Cython module and imported, and upon being imported the Python interpreter runs the code in the string, and correctly adds f to the module globals. If we create a pure Python function

def g():
    return inspect.currentframe().f_back.f_locals

calling cython_module.f(g) gives me a dictionary with key func_to_call (as expected) while cython_module.f2(g) gives me the __main__ module globals (but this is because I'm running from an interpreter rather than using --embed).

---

Edit: Full example, based on your code

from sympy import S, lambdify # I'm assuming "S" comes from sympy
import numpy as np

CreateMagneticFieldsList = None # stops a compile error about CreateMagneticFieldsList being undefined

exec("""def CreateMagneticFieldsList(dataToSave,equationString,DSList):

    expression  = S(equationString)
    numOfElements = len(dataToSave["MagneticFields"])

    #initialize the magnetic field output array
    magFieldsArray    = np.empty(numOfElements)
    magFieldsArray[:] = np.NaN

    lam_f = lambdify(tuple(DSList),expression,modules='numpy')
    try:
        # pass
        for i in range(numOfElements):
            replacementList = np.zeros(len(DSList))


            for j in range(len(DSList)):
                replacementList[j] = dataToSave[DSList[j]][i]

            try:
                val = np.double(lam_f(*replacementList))

            except:
                val = np.nan
            magFieldsArray[i] = val
    except:
        print("Error while evaluating the magnetic field expression")
    return magFieldsArray""")


list={"MagneticFields":[1,2,3,4,5]}

out=CreateMagneticFieldsList(list,"MagneticFields*5.1",["MagneticFields"])
print(out)

When compiled with your script this prints

[ 5.1 10.2 15.3 20.4 25.5 ]

Substantially all I've done is wrapped your function in an exec statement, so it's executed by the Python interpreter. This part won't see any benefit from Cython, however the rest of your program still will. If you want to maximise the amount compiled with Cython you could divide it up into multiple functions so that only the small part containing lambdify is in the exec.

Answer 2

It is stated in cython docs - limitations that

Stack frames

Currently we generate fake tracebacks as part of exception propagation, but don’t fill in locals and can’t fill in co_code. To be fully compatible, we would have to generate these stack frame objects at function call time (with a potential performance penalty). We may have an option to enable this for debugging.

f_locals in

AttributeError: 'NoneType' object has no attribute 'f_locals'

seems to point towards this incompability issue.