I have a basic MSI InstallShield installation with a managed EXE custom action running from the Binary table. I tried a simple test that just runs a console and that works
I couldn't find a decent way to use .Net custom actions that allowed me to do what I was trying to do. I ended up using the DTF (Deployment Tools Foundation) section of WiX to create the assembly and it worked great.