Cython: understanding a typed memoryview with a indirect_contignuous memory layout

Question

I want to understand more about Cython\'s awesome typed-memoryviews and the memory layout indirect_contiguous.

According to the documentation indi

Answer 1

Let's set the record straight: typed memory view can be only used with objects which implement buffer-protocol.

Raw C-pointers obviously don't implement the buffer-protocol. But you might ask, why something like the following quick&dirty code works:

%%cython    
from libc.stdlib cimport calloc
def f():
    cdef int* v=<int *>calloc(4, sizeof(int))
    cdef int[:] b = <int[:4]>v
    return b[0] # leaks memory, so what?

Here, a pointer (v) is used to construct a typed memory view (b). There is however more, going under the hood (as can be seen in the cythonized c-file):

• a cython-array (i.e. cython.view.array) is constructed, which wraps the raw pointer and can expose it via buffer-protocol
• this array is used for the creation of typed memory view.

Your understanding what view.indirect_contiguous is used for is right - it is exactly what you desire. However, the problem is view.array, which just cannot handle this type of data-layout.

view.indirect and view.indirect_contiguous correspond to PyBUF_INDIRECT in protocol-buffer parlance and for this the field suboffsets must contain some meaningful values (i.e >=0 for some dimensions). However, as can be see in the source-code view.array doesn't have this member at all - there is no way it can represent the complex memory layout at all!

Where does it leave us? As pointed out by @chrisb and @DavidW in your other question, you will have to implement a wrapper which can expose your data-structure via protocol-buffer.

There are data structures in Python, which use the indirect memory layout - most prominently the PIL-arrays. A good starting point to understand, how suboffsets are supposed to work is this piece of documenation:

void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
                       Py_ssize_t *suboffsets, Py_ssize_t *indices) {
    char *pointer = (char*)buf;    // A
    int i;
    for (i = 0; i < ndim; i++) {
        pointer += strides[i] * indices[i]; // B
        if (suboffsets[i] >=0 ) {
            pointer = *((char**)pointer) + suboffsets[i];  // C
        }
    }
    return (void*)pointer;  // D
}

In your case strides and offsets would be

• strides=[sizeof(int*), sizeof(int)] (i.e. [8,4] on usual x86_64 machines)
• offsets=[0,-1], i.e. only the first dimension is indirect.

Getting the address of element [x,y] would then happen as follows:

• in the line A, pointer is set to buf, let's assume BUF.
• first dimension:
  • in line B, pointer becomes BUF+x*8, and points to the location of the pointer to x-th row.
  • because suboffsets[0]>=0, we dereference the pointer in line C and thus it shows to address ROW_X - the start of the x-th row.
• second dimension:
  • in line B we get the address of the y element using strides, i.e. pointer=ROW_X+4*y
  • second dimension is direct (signaled by suboffset[1]<0), so no dereferencing is needed.
• we are done, pointer points to the desired address and is returned in line D.

---

FWIW, I have implemented a library which is able to export int** and similar memory layouts via buffer protocol: https://github.com/realead/indirect_buffer.