How can I return a StringBuilder or other string buffer from a PInvoke native callback

Question

I would like a clean way to increase the size of a StringBuilder() as required for population by native code, the callback method below seems clean, but somehow we get a copy of the buffer instead of the actual buffer - I'm interested in explanations and solutions (preferably sticking to the callback type allocation as it would be nice and clean if only it could be made work).

using System;
using System.Runtime.InteropServices;
using System.Text;

namespace csharpapp
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var buffer = new StringBuilder(12);
            // straightforward, we can write to the buffer but unfortunately
            // cannot adjust its size to whatever is required
            Native.works(buffer, buffer.Capacity); 
            Console.WriteLine(buffer);

            // try to allocate the size of the buffer in a callback - but now
            // it seems only a copy of the buffer is passed to native code
            Native.foo(size =>
                           {
                               buffer.Capacity = size;
                               buffer.Replace("works", "callback");
                               return buffer;
                           });
            string s = buffer.ToString();
            Console.WriteLine(s);
        }
    }

    internal class Native
    {
        public delegate StringBuilder AllocateBufferDelegate(int bufsize);
        [DllImport("w32.dll", CharSet = CharSet.Ansi)]
        public static extern long foo(AllocateBufferDelegate callback);
        [DllImport("w32.dll", CharSet = CharSet.Ansi)]
        public static extern void works(StringBuilder buf, int bufsize);
    }
}

native header

#ifdef W32_EXPORTS
#define W32_API __declspec(dllexport)
#else
#define W32_API __declspec(dllimport)
#endif

typedef char*(__stdcall *FnAllocStringBuilder)(int);
extern "C" W32_API long foo(FnAllocStringBuilder fpAllocate);
extern "C" W32_API void works(char *buf, int bufsize);

native code

#include "stdafx.h"
#include "w32.h"
#include <stdlib.h>

extern "C" W32_API long foo(FnAllocStringBuilder fpAllocate)
{
    char *src = "foo       X";
    int len = strlen(src) + 1;

    char *buf = fpAllocate(len);
    return strcpy_s(buf,len,src);
}

extern "C" W32_API void works(char *buf, int bufsize)
{
    strcpy_s(buf,bufsize,"works");
}

Answer 1

I have a theory for why this happens. I suspect that the marshalling of StringBuilder involves making a copy of the data, passing it to the P/Invoke call, and then copying back into the StringBuilder. I couldn't actually verify this though.

The only alternative to this would require the StringBuilder to be flattened first (it is internally a linked list of char[]'s), and the char[] pinned, and even then this would only work for marshalling to pointer-to-Unicode-chars strings, but not to ANSI or COM strings.

Thus, when you pass a StringBuilder as an argument, there's an obvious place for .NET to copy any changes back: right after the P/Invoke returns.

The same isn't true for when you pass a delegate returning a StringBuilder. In this case .NET needs to create a wrapper which converts an int => StringBuilder function into an int => char* function. This wrapper will create the char* buffer and populate it, but obviously can't copy any changes back yet. It also can't do this after the function that takes the delegate returns: it's still too early!

In fact, there is no obvious place at all where the reverse copy could occur.

So my guess is that this is what happens: when marshalling a StringBuilder-returning delegate, .NET can only perform a one-way conversion, hence any changes you make aren't reflected in the StringBuilder. This is slightly better than being completely unable to marshal such delegates.

---

As for solutions: I would recommend first asking the native code how large the buffer needs to be, and then passing a buffer of the appropriate size in a second call. Or, if you need better performance, guess a large enough buffer, but allow the native method to communicate that more space is required. This way most calls would involve only one P/Invoke transition.

This can be wrapped into a more convenient function that you can just call from the managed world without worrying about buffers.

Answer 2

In addition to the input provided by romkyns I will share the minimal changes solution I came up with. If anyone uses this be careful of your encodings!

the principal modification is:

    private static void Main(string[] args)
    {            
        byte[] bytes = null;
        var gcHandle = new GCHandle();

        Native.foo(size =>
                        {
                            bytes = new byte[size];
                            gcHandle = GCHandle.Alloc(bytes,GCHandleType.Pinned);
                            return gcHandle.AddrOfPinnedObject();
                        });

        if(gcHandle.IsAllocated)
            gcHandle.Free();

        string s = ASCIIEncoding.ASCII.GetString(bytes);

        Console.WriteLine(s);
    }

with the delegate signature changeing to:

public delegate IntPtr AllocateBufferDelegate(int bufsize);