ctypes variable length structures

Question

Ever since I read Dave Beazley\'s post on binary I/O handling (http://dabeaz.blogspot.com/2009/08/python-binary-io-handling.html) I\'ve wanted to create a Python library for

Answer 1

This question is really, really, old:

I have a simpler answer, which seems strange, but avoids metaclasses and resolves the issue that ctypes doesn't allow me to directly build a struct with the same definition as I can in C.

The example C struct, coming from the kernel:

struct some_struct {
        __u32   static;
        __u64   another_static;
        __u32   len;
        __u8    data[0];
};

With ctypes implementation:

import ctypes
import copy

class StructureVariableSized(ctypes.Structure):
    _variable_sized_ = []

    def __new__(self, variable_sized=(), **kwargs):
        def name_builder(name, variable_sized):
            for variable_sized_field_name, variable_size in variable_sized:
                name += variable_sized_field_name.title() + '[{0}]'.format(variable_size)
            return name

        local_fields = copy.deepcopy(self._fields_)
        for variable_sized_field_name, variable_size in variable_sized:
            match_type = None
            location = None
            for matching_field_name, matching_type, matching_location in self._variable_sized_:
                if variable_sized_field_name == matching_field_name:
                    match_type = matching_type
                    location = matching_location
                    break
            if match_type is None:
                raise Exception
            local_fields.insert(location, (variable_sized_field_name, match_type*variable_size))
        name = name_builder(self.__name__, variable_sized)
        class BaseCtypesStruct(ctypes.Structure):
            _fields_ = local_fields
            _variable_sized_ = self._variable_sized_
        classdef = BaseCtypesStruct
        classdef.__name__ = name
        return BaseCtypesStruct(**kwargs)


class StructwithVariableArrayLength(StructureVariableSized):
    _fields_ = [
        ('static', ctypes.c_uint32),
        ('another_static', ctypes.c_uint64),
        ('len', ctypes.c_uint32),
        ]
    _variable_sized_ = [
        ('data', ctypes.c_uint8)
    ]

struct_map = {
    1: StructwithVariableArrayLength
} 
sval32 = struct_map[1](variable_sized=(('data', 32),),)
print sval32
print sval32.data
sval128 = struct_map[1](variable_sized=(('data', 128),),)
print sval128
print sval128.data

With sample output:

machine:~ user$ python svs.py 
<__main__.StructwithVariableArrayLengthData[32] object at 0x10dae07a0>
<__main__.c_ubyte_Array_32 object at 0x10dae0830>
<__main__.StructwithVariableArrayLengthData[128] object at 0x10dae0830>
<__main__.c_ubyte_Array_128 object at 0x10dae08c0>

This answer works for me for a couple reasons:

1. The argument to the constructor can be pickled, and has no references to types.
2. I define all of the structure inside of the StructwithVariableArrayLength definition.
3. To the caller, the structure looks identical as if I had just defined the array inside of _fields_
4. I have no ability to modify the underlying structure defined in the header file, and accomplish my goals without changing any underlying code.
5. I don't have to modify any parse/pack logic, this only does what I'm trying to do which is build a class definition with a variable length array.
6. This is a generic, reusable container that be sent into the factory like my other structures.

I would obviously prefer the header file took a pointer, but that isn't always possible. That answer was frustrating. The others were very tailored to the data structure itself, or required modification of the caller.