Python ctypes: How to modify an existing char* array

Question

I'm working on a Python application that makes use of libupnp which is a C library. I'm using CTypes to use the library which is easy enough. The problem I'm having is when I'm registering a callback function for read requests. The function has a prototype of the following form:

int read_callback(void *pFileHandle, char *pBuf, long nBufLength);

pFileHandle is just some file handle type. pBuf is a writable memory buffer. This is where the data is output. nBufLength is the number of bytes to read from the file. A status code is returned.

I have a Python function pointer for this. That was easy enough to make but when I define a Python function to handle this callback I've found that pBuf doesn't get written to because Python strings are immutable and when you assign or modify them they create new instances. This poses a big problem because the C library expects the char pointer back when the function finishes with the requested file data. The buffer ends up being empty every time though because of the way Python strings are. Is there some way around this without modifying the C library?

The handler should modify the buffer parameter that is given which is my problem.

So what I want to have happen is that the Python function gets called to perform a read of some file (could be in memory, a file system handle, or anything in between). The pBuf parameter is populated by a read of the stream (again in Python). The callback then returns to the C code with pBuf written to.

Answer 1

ctypes can allocate a buffer object that your C library should be able to write to:

import ctypes
init_size = 256
pBuf = ctypes.create_string_buffer(init_size)

See: http://docs.python.org/2/library/ctypes.html#ctypes.create_string_buffer

Answer 2

The callback is invoked with pBuf and nBufLength. pBuf is already allocated with writable memory, but if you ask for pBuf.value, this is converted to an immutable python string.

Instead convert pBuf to an object that can be modified directly:

## If pBuf is c_char_p, then convert to writable object
c = ctypes.cast(pBuf, ctypes.POINTER(ctypes.c_char))
## At this point, you can set individual bytes
## with c[i] = x, but this is dangerous and there's a safer way:

## get address of string
addr = ctypes.addressof(c.contents)

## convert to char[] for safe writing
c2 = (c_char*nBufLength).from_address(addr)

## see how many bytes to write
nb = min(len(msg), nBufLength-1)

c2[:nb] = msg[:nb]
c2[nb+1] = '\0'

Answer 3

Don't declare pBuf as c_char_p. ctypes converts that type to an immutable Python string. You'll want to declare it as POINTER(c_char) instead. A quick and dirty Windows example:

DLL code (compiled on MSVC as cl /LD test.c)

typedef int (*CALLBACK)(void *pFileHandle, char *pBuf, long nBufLength);
char g_buf[10] = "012345678";
CALLBACK g_callback;
__declspec(dllexport) void set_callback(CALLBACK callback) { g_callback = callback; }
__declspec(dllexport) void call_callback() { g_callback(0,g_buf,10); }
__declspec(dllexport) const char* get_buf() { return g_buf; }

Python 3.X code:

from ctypes import *

# Declare the callback type, argument types and return types
CALLBACK = CFUNCTYPE(c_int,c_void_p,POINTER(c_char),c_long)
dll = CDLL('x')
dll.set_callback.argtypes = [CALLBACK]
dll.set_callback.restype = None
dll.call_callback.argtypes = []
dll.call_callback.restype = None
dll.get_buf.argtypes = []
dll.get_buf.restype = c_char_p

@CALLBACK
def callback(handle,buf,length):
    for i in range(9):
        buf[i] = ord('A') + i
    buf[9] = 0
    return 0

dll.set_callback(callback)
dll.call_callback()
print(dll.get_buf())

Output:

b'ABCDEFGHI'