In VS2012, I resolved a BadImageException from running a unit test by going to: TEST-> TEST SETTINGS -> DEFAULT PROCESSOR ARCHITECTURE -> x64.
This works in the built in
We had a similar issue when trying to setup MSTEST for Sharepoint 2007 - it would keep running in x86 instead of x64. Instantiating SPSite kept saying "The Web application at http://server:port/ could not be found".
I started with replacing the EXEs suggested by @valentin-kuzub to confirm it would work - but then realized that we had the wrong Test Settings file active - there were (2) Local.testsettings and TraceAndTestImpact.testsettings. One of them was set to use x86 instead of targeting x64.
Another option is switching from testSettings to runSettings and assigning TargetPlatform.
x64
Framework40
The last option (which you already tried) is in the XML comments from option #2 -> You can also change it from menu Test->Test Settings->Default Processor Architecture->x64.
