How to handle C++ return type std::vector in Python ctypes?

Question

I cannot find how ctypes will bridge the gap between std::vector and Python; no where on the internet is the combination mentioned. Is this bad practice, does i

Answer 1

The particular reason is that speed is important. I'm creating an application that should be able to handle big data. On 200,000 rows the missings have to be counted on 300 values (200k by 300 matrix). I believe, but correct me if I'm wrong, that C++ will be significantly faster.

Well, if you're reading from a large file, your process is going to be mostly IO-bound, so the timings between Python and C probably won't be significantly different.

The following code...

result = []
for line in open('test.txt'):
    result.append(line.count('NA'))

...seems to run just as fast as anything I can hack together in C, although it's using some optimized algorithm I'm not really familiar with.

It takes less than a second to process 200,000 lines, although I'd be interested to see if you can write a C function which is significantly faster.

---

Update

If you want to do it in C, and end up with a Python list, it's probably more efficient to use the Python/C API to build the list yourself, rather than building a std::vector then converting to a Python list later on.

An example which just returns a list of integers from 0 to 99...

// hack.c

#include 

PyObject* foo(const char* filename)
{
    PyObject* result = PyList_New(0);
    int i;

    for (i = 0; i < 100; ++i)
    {
        PyList_Append(result, PyInt_FromLong(i));
    }

    return result;
}

Compiled with...

$ gcc -c hack.c -fPIC
$ ld -o hack.so -shared hack.o -lpython2.7

Example of usage...

>>> from ctypes import *
>>> dll = CDLL('./hack.so')
>>> dll.foo.restype = py_object
>>> dll.foo('foo')
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...]