Unmanaged DLL causing AccessViolationException

Question

This one is really starting to give me a headache :(

I have an unmanaged DLL that I'm trying to interop with and it's not going well. The application will sometimes work...but most times, randomly through an AccessViolationException and crash horribly.

I think I've narrowed it down to my mishandling of a single DllImport:

C++ function:

HTMLRENDERERDLL_REDIST_API void SetDataBuffer( int windowHandle, unsigned char* dataSource, int format, int stride, int totalBufferSize );

C# DllImport:

[DllImport("MyDll.dll", ExactSpelling = true, CallingConvention = CallingConvention.Cdecl)]
    static private extern unsafe void SetDataBuffer(Int32 windowHandle, [MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] byte[] dataSource, Int32 format, Int32 stride, Int32 totalBufferSize);

Call to said function:

var buffer = new byte[windowWidth * windowHeight * bytesPerPixel];
SetDataBuffer(windowHandle, buffer, (Int32)0, (Int32)(windowWidth * bytesPerPixel), (Int32)(windowWidth * windowHeight * bytesPerPixel));

Is there anything glaringly wrong with this? I suspect that dataSource is a culprit but...not sure how to prove it!

Thanks

Answer 1

Your problem can be inferred from the name of the function. When you "set a buffer", it is likely that the native code is going to use that buffer later. That's incompatible with the garbage collector, it is going to move the array when it compacts the heap. That's a big Kaboom when the native code then writes to the buffer, it is going to write to memory that isn't there anymore. The most typical outcome is a FatalExecutionEngineException when the garbage collector detects that the heap integrity got compromised.

The array needs to be pinned, something that the pinvoke marshaller does when it calls the function, but it unpins the array after the call.

You can pin a managed array with GCHandle.Alloc(), but that's pretty detrimental to the garbage collector if you leave it pinned for a long time. By far the best solution is to use Marshal.AllocHGlobal to allocate a chunk of unmanaged memory, it never moves.

If you still have trouble then worry about the size of the buffer. And just plain misery in the native code, it rarely needs much help to fall over on an AccessViolation. That's the standard failure mode for native code. Very hard to diagnose, impossible if you don't have the source code for it. Contact the code owner for support, have a small repro snippet available to help him locate the problem.

Answer 2

I agree that it's likely that datasource is the problem.

Try changing the dllimport so that it's using an IntPtr rather than the byte[].

[DllImport("MyDll.dll", ExactSpelling = true, CallingConvention = CallingConvention.Cdecl)]
static private extern unsafe void SetDataBuffer(Int32 windowHandle, IntPtr dataSource, Int32 format, Int32 stride, Int32 totalBufferSize);

Then when you're calling it, explicitly allocate the memory for the buffer like so:

IntPtr buffer = Marshal.AllocHGlobal(windowWidth * windowHeight * bytesPerPixel);
SetDataBuffer(windowHandle, buffer, (Int32)0, (Int32)(windowWidth * bytesPerPixel), (Int32)(windowWidth * windowHeight * bytesPerPixel));

And don't forget to call Marshal.FreeHGlobal(buffer) when you're done done with it.

I take it that the buffer is being used internally by the Update method?