Getting data from ctypes array into numpy

Question

I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and spec

Answer 1

Using np.ndarrays as ctypes arguments

The preferable approach is using ndpointer, as mentioned in the numpy-docs.

This approach is more flexible than using, for example, POINTER(c_double), since several restrictions can be specified, which are verified upon calling the ctypes function. These include data type, number of dimensions, shape and flags. If a given array does not satisfy the specified restrictions, a TypeError is raised.

Minimal, Reproducible Example

Calling memcpy from python. Eventually the filename of the standard C-library libc.so.6 needs to be adjusted.

import ctypes
import numpy as np

n_bytes_f64 = 8
nrows = 2
ncols = 5

clib = ctypes.cdll.LoadLibrary("libc.so.6")

clib.memcpy.argtypes = [
    np.ctypeslib.ndpointer(dtype=np.float64, ndim=2, flags='C_CONTIGUOUS'),
    np.ctypeslib.ndpointer(dtype=np.float64, ndim=1, flags='C_CONTIGUOUS'),
    ctypes.c_size_t]
clib.memcpy.restype = ctypes.c_void_p

arr_from = np.arange(nrows * ncols).astype(np.float64)
arr_to = np.empty(shape=(nrows, ncols), dtype=np.float64)

print('arr_from:', arr_from)
print('arr_to:', arr_to)

print('\ncalling clib.memcpy ...\n')
clib.memcpy(arr_to, arr_from, nrows * ncols * n_bytes_f64)

print('arr_from:', arr_from)
print('arr_to:', arr_to)

Output

arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0.0e+000 4.9e-324 9.9e-324 1.5e-323 2.0e-323]
 [2.5e-323 3.0e-323 3.5e-323 4.0e-323 4.4e-323]]

calling clib.memcpy ...

arr_from: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
arr_to: [[0. 1. 2. 3. 4.]
 [5. 6. 7. 8. 9.]]

If you modify the ndim=1/2 arguments of ndpointer to be inconsistent with the dimensions of arr_from/arr_to, the code fails with an ArgumentError.

As the title of this question is quite general, ...

Constructing a np.ndarray from a ctypes.c_void_p result

Minimal, Reproducible Example

In the following example, some memory is allocated by malloc and filled with 0s by memset. Then a numpy array is constructed, to access this memory. Of course the occur some ownership issues, as python will not free memory, which was allocated in c. To avoid memory leaks, one has to free the allocated memory again by ctypes. The copy method can be used for the np.ndarray to acquire ownership.

import ctypes
import numpy as np

n_bytes_int = 4
size = 7

clib = ctypes.cdll.LoadLibrary("libc.so.6")

clib.malloc.argtypes = [ctypes.c_size_t]
clib.malloc.restype = ctypes.c_void_p

clib.memset.argtypes = [
    ctypes.c_void_p,
    ctypes.c_int,
    ctypes.c_size_t]
clib.memset.restype = np.ctypeslib.ndpointer(
    dtype=np.int32, ndim=1, flags='C_CONTIGUOUS')

clib.free.argtypes = [ctypes.c_void_p]
clib.free.restype = ctypes.c_void_p


pntr = clib.malloc(size * n_bytes_int)
ndpntr = clib.memset(pntr, 0, size * n_bytes_int)
print(type(ndpntr))
ctypes_pntr = ctypes.cast(ndpntr, ctypes.POINTER(ctypes.c_int))
print(type(ctypes_pntr))
print()
arr_noowner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,))
arr_owner = np.ctypeslib.as_array(ctypes_pntr, shape=(size,)).copy()
# arr_owner = arr_noowner.copy()


print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

print('\nfree allocated memory again ...\n')
_ = clib.free(pntr)

print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

print('\njust for fun: free some python-memory ...\n')
_ = clib.free(arr_owner.ctypes.data_as(ctypes.c_void_p))

print('arr_noowner (at {:}): {:}'.format(arr_noowner.ctypes.data, arr_noowner))
print('arr_owner (at {:}): {:}'.format(arr_owner.ctypes.data, arr_owner))

Output

arr_noowner (at 104719884831376): [0 0 0 0 0 0 0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]

free allocated memory again ...

arr_noowner (at 104719884831376): [ -7687536     24381 -28516336     24381         0         0         0]
arr_owner (at 104719884827744): [0 0 0 0 0 0 0]

just for fun: free some python-memory ...

arr_noowner (at 104719884831376): [ -7687536     24381 -28516336     24381         0         0         0]
arr_owner (at 104719884827744): [ -7779696     24381 -28516336     24381         0         0         0]