Running VS2012 unit tests from TFS2010 Team Build

Question

I have a Visual Studio 2012 solution using .NET 4.5 framework. The rest of our development system is TFS2010. Recently some of our unit tests have been failing on the build server only. After going through several rounds of debugging I was able to determine that the cause of the failed tests was most likely due to the version of MSTest being used on the build server. The tests in question pass when run from VS2012 on my local machine and when run from VS2012 on the actual build server.

I have followed the steps described in the following articles (I have not found any useful articles directly relevant to VS2012 and TFS2010) without success: Running VS2010 UnitTests project from TFS2008 Team Build and VS2010, TFS 2008 and Unit Tests

On the build server I now have the following:

• VS2010 Ultimate SP1 installed
• VS2012 Update 3 installed
• .NET 4.5 framework SDK installed
• Changed Microsoft.TeamFoundation.Build.targets file in "C:\Program Files (86)\MSBuild\Microsoft\VisualStudio\TeamBuild\" and "C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\" so the AssemblyFile attribute of the TestToolsTask build task pointed to "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies\Microsoft.TeamFoundation.Build.ProcessComponents.dll"

I have 2 build definitions that I have been testing with:

1. In the first build definition I have left the ToolPath of the "run MS Test" activity blank so it then uses "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE"
2. In the other build definition I have changed the "Run MS Test" activity ToolPath parameter to "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE"

Both build definitions appear to be building correctly and are using the following MSBuild.exe: C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe

The first build defintion runs the VS2010 version of MStest with any errors, but 5 of the approx. 2030 tests fail. These tests do not fail when run from within Visual Studaio 2012. An example of the command line is (actual paths and server names removed):

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe 
/nologo /usestderr /searchpathroot:"d:\Builds\xxxxx\Binaries" 
/resultsfileroot:"d:\Builds\xxxxx\TestResults" /testcontainer:"d:\Builds\xxxxx.dll" 
/testcontainer:"d:\Builds\xxxxx.dll" /publish:"http://xxxxx 
/publishbuild:"vstfs:///Build/Build/xxxxx" /teamproject:"xxxxx" /platform:"x86" /flavor:"Debug"

When running the second build definition MSTest fails to run with the output below. If I run this command from the command line on the build server without the switches that are causing the Invalid switch error then MSTest runs and the tests in question are passed:

The MSTestActivity was invoked without a value for Platform or Flavor. The values x86 and Debug were used.

    C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe 
    /nologo /usestderr /searchpathroot:"d:\Builds\xxxxx\Binaries" 
    /resultsfileroot:"d:\Builds\xxxxx\TestResults" /testcontainer:"d:\Builds\xxxxx.dll" 
    /publish:"http://xxxxx" /publishbuild:"vstfs:///Build/Build/xxxxx" /teamproject:"xxxxx" 
    /platform:"x86" /flavor:"Debug" 
    Invalid switch "/publish".
    Invalid switch "/publishbuild".
    Invalid switch "/teamproject".
    Invalid switch "/platform".
    Invalid switch "/flavor".
    For switch syntax, type "MSTest /help"

I have done a lot of runs of these builds but I hvae not been able to resolve the problem. Is there anything else that I am missing that needs to be done either on the build server or in the build definition?

In relation to the actual tests that are failing they have in common that they are comparing objects in lists, but as mentioned they pass when run from within VS2012.

Answer 1

Installing VS2012 Ultimate on the build server caused the Invalid switch errors when running MSTest to go away and the MSTest could run successfully. I had earlier installed VS2012 Professional and VS2012 Team Explorer, but these editions "obviously" (discovered after lots of debug runs!!) do not contain the required version of MSTest. The information about MSTest command line options does not mention this (MSDN Command-Line options for publishing tests), only mentioning that Team Explorer must be installed.