How to work with UTF-16 in python ctypes?

问题

I have a foreign C library which uses utf-16 in API: as function arguments, return values and structure members.

On Windows its OK with ctypes.c_wchar_p, but under OSX ctypes uses UCS-32 in c_wchar and I could not find the way to support utf-16.

Here is my research:

1. Use _SimpleCData subclassing to redefine _check_retval_.

   • it allows a transparent conversion of utf-16 to Python string.
   • can be placed as C structure member
   • But it doesn't allow to handle strings as arguments, its from_param() method never been called (Why?): func('str', b'W\x00B\x00\x00\x00') # passed without conversion

2. Use own type with from_param() method.

   • advantages: can be initialized either with constructor or encoded on the fly when passing strings to function:
   • disadvantages: cannot be used as function return type or structure member.

Here it is:

ustr = myutf16('hello')
func(ustr)
func('hello')   # calls myutf16.from_param('hello')

回答1:

You can override from_param in a c_char_p subclass to encode a unicode string as UTF-16. You can add a _check_retval_ method to decode a UTF-16 result as a unicode string. For struct fields you can use a descriptor class that handles setting and getting the attribute. Make the field a private _name of type c_char_p, and set the descriptor as the public name. For example:

import sys
import ctypes

if sys.version_info[0] > 2:
    unicode = str

def decode_utf16_from_address(address, byteorder='little',
                              c_char=ctypes.c_char):
    if not address:
        return None
    if byteorder not in ('little', 'big'):
        raise ValueError("byteorder must be either 'little' or 'big'")
    chars = []
    while True:
        c1 = c_char.from_address(address).value
        c2 = c_char.from_address(address + 1).value
        if c1 == b'\x00' and c2 == b'\x00':
            break
        chars += [c1, c2]
        address += 2
    if byteorder == 'little':
        return b''.join(chars).decode('utf-16le')
    return b''.join(chars).decode('utf-16be')

class c_utf16le_p(ctypes.c_char_p):
    def __init__(self, value=None):
        super(c_utf16le_p, self).__init__()
        if value is not None:
            self.value = value

    @property
    def value(self,
              c_void_p=ctypes.c_void_p):
        addr = c_void_p.from_buffer(self).value
        return decode_utf16_from_address(addr, 'little')

    @value.setter
    def value(self, value,
              c_char_p=ctypes.c_char_p):
        value = value.encode('utf-16le') + b'\x00'
        c_char_p.value.__set__(self, value)

    @classmethod
    def from_param(cls, obj):
        if isinstance(obj, unicode):
            obj = obj.encode('utf-16le') + b'\x00'
        return super(c_utf16le_p, cls).from_param(obj)

    @classmethod
    def _check_retval_(cls, result):
        return result.value

class UTF16LEField(object):
    def __init__(self, name):
        self.name = name

    def __get__(self, obj, cls,
                c_void_p=ctypes.c_void_p,
                addressof=ctypes.addressof):
        field_addr = addressof(obj) + getattr(cls, self.name).offset
        addr = c_void_p.from_address(field_addr).value
        return decode_utf16_from_address(addr, 'little')

    def __set__(self, obj, value):
        value = value.encode('utf-16le') + b'\x00'
        setattr(obj, self.name, value)

Example:

if __name__ == '__main__':
    class Test(ctypes.Structure):
        _fields_ = (('x', ctypes.c_int),
                    ('y', ctypes.c_void_p),
                    ('_string', ctypes.c_char_p))
        string = UTF16LEField('_string')

    print('test 1: structure field')
    t = Test()
    t.string = u'eggs and spam'
    print(t.string)

    print('test 2: parameter and result')
    result = None

    @ctypes.CFUNCTYPE(c_utf16le_p, c_utf16le_p)
    def testfun(string):
        global result
        print('parameter: %s' % string.value)
        # callbacks leak memory except for simple return
        # values such as an integer address, so return the
        # address of a global variable.
        result = c_utf16le_p(string.value + u' and eggs')
        return ctypes.c_void_p.from_buffer(result).value

    print('result: %s' % testfun(u'spam'))

Output:

test 1: structure field
eggs and spam

test 2: parameter and result
parameter: spam
result: spam and eggs

来源：https://stackoverflow.com/questions/35500018/how-to-work-with-utf-16-in-python-ctypes