I am using WiX 3.5.1930 in Visual Studio 2010, targeting the .NET Framework 3.5. (Later weekly builds of WiX seem to be very broken with respect to their custom action temp
It sounds like you are using DTF. If you see:
using Microsoft.Deployment.WindowsInstaller;
then you certainly are. Be sure to read the following for how it all works:
Deployment Tools Foundation (DTF) Managed Custom Actions
Also you'll find a DTF help chm in the start menu under WiX.
Basically it sounds like to me you are wiring the .NET assembly into the installer instead of the unmanged wrapper dll. Read the above article for an overview of how to look at it in Depends and to know what to expect. The WiX | C# Custom Action project should output Foo.dll and Foo.CA.dll. You want the later in your installer.
For people who land on this page in the future (the answer was originally for the poster ) there is a whole list of things to check:
Any one of these can cause an 1154 error. This is the reason I wrote a comprehensive blog article on the subject and linked to it in this answer. It's important to fully understand how managed code is presented to the unmanaged Windows Installer service and to know how to use Depends to validate that the public static method is exported as a stdcall function in the resulting .CA.dll that WiX/DTF produces.
Try putting your custom action call in
<InstallExecuteSequence/>
in hopes of getting a better error message. I have received different error messages depending on how the action was called. Also, try using fuslogvw.exe. It might give you a pretty nice error message too.