Passing big complex arrays from Python to C++ - what's my best option?

Question

2017/06/13 EDIT: I tried using boost as was suggested, but after spending more than 3 days trying to get it to compile and link, and failing, I decided that the stupid painf

Answer 1

Note added in edit. As mentioned in the comments, python itself, being an interpreted language, has little potential for computational efficiency. So in order to make python scripts efficient, one must use modules which aren't all interpreted, but under the hood call compiled (and optimized) code written in, say, C/C++. This is exactly what numpy does for you, in particular for operations on whole arrays.

Therefore, the first step towards efficient python scripts is the usage of numpy. Only the second step is to try to use your own compiled (and optimized) code. Therefore, I have assumed in my example below that you were using numpy to store the array of complex numbers. Everything else would be ill-advised.

---

There are various ways in which you can access python's original data from within a C/C++ program. I personally have done this with boost.Python, but must warn you that the documentation and support are lousy at best: you're pretty much on your own (and stack overflow, of course).

For example your C++ file may look like this

// file.cc
#include 
#include 

namespace p = boost::python;
namespace n = p::numpy;

n::ndarray func(const n::ndarray&input, double control_variable)
{
  /* 
     your code here, see documentation for boost python
     you pass almost any python variable, doesn't have to be numpy stuff
  */
}

BOOST_PYTHON_MODULE(module_name)
{
  Py_Initialize();
  n::initialize();   // only needed if you use numpy in the interface
  p::def("function", func, "doc-string");
}

to compile this, you may use a python script such as

# setup.py

from distutils.core import setup
from distutils.extension import Extension

module_name = Extension(
    'module_name',
    extra_compile_args=['-std=c++11','-stdlib=libc++','-I/some/path/','-march=native'],
    extra_link_args=['-stdlib=libc++'],
    sources=['file.cc'],
    libraries=['boost_python','boost_numpy'])

setup(
    name='module_name',
    version='0.1',
    ext_modules=[module_name])

and run it as python setup.py build, which will create an appropriate .so file in a sub-directory of build, which you can import from python.