C++ from C#: C++ function (in a DLL) returning false, but C# thinks it's true!

Question

I'm writing a little C# app that calls a few functions in a C++ API. I have the C++ code building into a DLL, and the C# code calls the API using DllImport. (I am using a .DEF file for the C++ DLL so I don't need extern "C".)

So far the API has one function, which currently does absolutely nothing:

bool Foo()
{
  return false;
}

In C#, I have the following:

public class FooAPI
{
    [DllImport("Foo.dll")]
    public static extern bool Foo();
}

...

bool b = FooAPI.Foo(); 
if (!b) 
{ 
    // Throw an exception 
} 

My problem is that, for some reason, b is always evaluating to TRUE. I have a breakpoint on if (!b) and the debugger reports it as 'true', irrelevant of whatever the C++ function is returning.

Is the C# bool the same as the C++ bool? Though even if this wasn't the case, I still don't get how it would find the return value to be 'true' :)

Can anyone help me with this bizarre discrepancy?

Thanks in advance!

Answer 1

Try [return: MarshalAs (UnmanagedType.I1)]. By default, C# interop marshals C# bool as the Win32 BOOL, which is the same as int, while C++ bool is one byte AFAIR. Thus, C#'s default marshaling expects the return value to be a BOOL in the eax register, and picks up some non-zero garbage because C++ bool is returned in al.

Answer 2

Your code snippet as posted cannot work. If this was compiled with a C++ compiler then the function name would be ?Foo@@YA_NXZ and your C# code could never find it. The calling convention is important too, it is not the default (CallingConvention.StdCall) unless you tell the compiler. And somewhere you should have told the linker to export the function.

Start by declaring the exported function so it is compatible with default P/Invoke attributes:

extern "C" __declspec(dllexport) 
bool __stdcall Foo() {
    return false;
}

Next problem is that the C++ compiler uses only one byte to store a bool. The machine code generated for your return statement is:

013D138E  xor         al,al

The P/Invoke marshaller will however assume it is a 32-bit integer and check the value of the eax register. Either declare the return type of the function as int or BOOL or use a [return: MarshalAs(UnmanagedType.U1)] attribute in the C# declaration.

Answer 3

I used signed int.

C++ code

   extern "C" __declspec(dllexport) signed int TestDouble(double* output)
    {   
        *output = 1.3;
        return true;
    }

C# code

 [DllImport("abc.dll", EntryPoint = "TestDouble", CallingConvention = CallingConvention.Cdecl)]
 public static extern bool TestDouble(out double output);