I have a basic MSI InstallShield installation with a managed EXE custom action running from the Binary table. I tried a simple test that just runs a console and that works
Custom Actions only extract a single file to a temporary location under a temporary name. For the dependency on the .dll to work, they need to both be extracted, and at least the .dll must have the expected name. Typically this is easiest to do by adding both to "setup files" and referencing [SUPPORTDIR]\your.exe for the custom action.