I have a basic MSI InstallShield installation with a managed EXE custom action running from the Binary table. I tried a simple test that just runs a console and that works
Please add your DLL as a dependency in 'isclrwrap' binary table. You can find that binary table from direct editor. This will solve your problem.