GDAL GDALRATSetValueAsString() how to save Chinese characters (c#)?

Question

I need help with GDAL. The string value with Chinese symbols is not readed/saved correctly (C#).

For SAVING grid value we using:
private static extern void GDALRATSetValueAsString(IntPtr handle, int row, int field, [In][MarshalAs(UnmanagedType.LPStr)] string value); method (c#) to save string value, it seems that this method saves string as ANSI string.

For READING:

private static extern IntPtr GDALRATGetValueAsString(IntPtr handle, int row, int field);

In. Example my string "银行Flamwood C2" There is for methods to get value by pointer (use in GDALRATGetValueAsString metho):

var pointer = GDALRATGetValueAsString(GDALRasterAttributeTableH, row, field);
    a)    var b = Marshal.PtrToStringUni(pointer);       // value: "㼿汆浡潷摯䌠2"
    b)    var a = Marshal.PtrToStringAnsi(pointer);      // value: "??Flamwood C2"
    c)    var c = Marshal.PtrToStringAuto(pointer);      // value: "㼿汆浡潷摯䌠2"
    d)    var d = Marshal.PtrToStringBSTR(pointer);       //Throws an error out of memory.

Q: So how I can get Unicode string with was saved (so I can get using this Marshal.PtrToStringUni(pointer)) or most likely how to save the Unicode string to GDALRAT (GDAL RAT - GDAL Raster Attribute Table)?

GDAL version: 1.11.1

I tried to set CharSet = CharSet.Unicode but id does not helped, still get not correct string:

[DllImport(GdalWrapper.GdalDLL, CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Unicode)] 
private static extern void GDALRATSetValueAsString(IntPtr handle, int row, int field, [In][MarshalAs(UnmanagedType.LPStr)] string value); 

Thanks for any help.

P.S. If the GDAL source files need to be build again to save string as unicode string, then what build parameters and where has to be set?

Answer 1

GDAL uses UTF-8 encoding internally when working with strings. That means strings must be converted to UTF-8 before passing them to GDAL. The same is valid for GDAL output strings - have to be converted from UTF-8 to local encoding before using.

C# uses UTF-16 strings so conversions to UTF-8 and back must be introduced:

public class EncodingConverter
{
    public static string Utf16ToUtf8(string utf16String)
    {
        byte[] utf16Bytes = Encoding.Unicode.GetBytes(utf16String);
        byte[] utf8Bytes = Encoding.Convert(Encoding.Unicode, Encoding.UTF8, utf16Bytes);
        return Encoding.Default.GetString(utf8Bytes);
    }

    public static string Utf8ToUtf16(string utf8String)
    {
        byte[] utf8Bytes = Encoding.Default.GetBytes(utf8String);
        byte[] utf16Bytes = Encoding.Convert(Encoding.UTF8, Encoding.Unicode, utf8Bytes);
        return Encoding.Unicode.GetString(utf16Bytes);
    }
}

Going back to your problem, Japanese characters will be processed correctly if encoding conversion will be applied.

    public void SetValueAsString(int row, int field, string value)
    {
        string utf8Value = EncodingConverter.Utf16ToUtf8(value);
        GDALRATSetValueAsString(GDALRasterAttributeTableH, row, field, utf8Value);
    }

    public string GetValueAsString(int row, int field)
    {
        string value = null;

        var pointer = GDALRATGetValueAsString(GDALRasterAttributeTableH, row, field);
        if (pointer != IntPtr.Zero)
        {
            string utf8Value = Marshal.PtrToStringAnsi(pointer);
            value = EncodingConverter.Utf8ToUtf16(utf8Value);
        }
        return value;
    }

Answer 2

Read this first Specifying a Character Set. Make sure there is a unicode version of GDALRATGetValueAsString. Unicode version ends with a W e.g. GDALRATGetValueAsStringW. ANSI version ends with a A e.g. GDALRATGetValueAsStringA. If you import GDALRATGetValueAsString the charset is auto. It is not clear which version of the function you are referring to.