Coded UI Test now gets error about Newtonsoft.json not being found

Question

I have a simple Coded UI test to log into an application. The test is data driven and uses data from TFS. We are using a TfsTestAgent user (that has admin privileges) that is on both the server and the agent systems. When I execute the test, I see the following error:

    The unit test adapter failed to connect to the data source or to read the data.
 For more information on troubleshooting this error, see "Troubleshooting Data-Driven 
Unit Tests" (http://go.microsoft.com/fwlink/?LinkId=62412) in the MSDN Library.Error 
details: Could not load file or assembly 'Newtonsoft.Json, Version=4.5.0.0, 
Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The 
system cannot find the file specified.

The test run log (UITestLog.html) shows this:

I, 2524, 81, 2015/03/24, 13:27:09.815, 14494724802, QTAgent32_40.exe, ExtensionFramework : Reading extensions from custom dir 'C:\Program Files (x86)\Common Files\Microsoft Shared\VSTT\12.0\UITestExtensionPackages'
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, UnitTestExecuter.RunClassInitializeMethod: Acquiring m_runner.SyncRoot.
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, UnitTestExecuter.RunClassInitializeMethod: Acquired m_runner.SyncRoot.
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, CodedUITest : CodedUITestExtension.BeforeClassInitialize()
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, CodedUITest : CodedUITestExtension.AfterClassInitialize()
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, UnitTestExecuter.RunClassInitializeMethod: Released m_runner.SyncRoot.
V, 2524, 81, 2015/03/24, 13:27:09.830, .\QTAgent32_40.exe, UnitTestRunner.ExecuteDataDrivenTest: started.
V, 2524, 8, 2015/03/24, 13:27:10.377, .\QTAgent32_40.exe, CodedUITest : CodedUITestExtension.BeforeClassCleanup
V, 2524, 8, 2015/03/24, 13:27:10.377, .\QTAgent32_40.exe, CodedUITest : CodedUITestExtension.AfterClassCleanup
V, 2524, 8, 2015/03/24, 13:27:10.393, .\QTAgent32_40.exe, CodedUITest : CodedUITestExtension.Dispose()
I, 2524, 8, 2015/03/24, 13:27:10.393, .\QTAgent32_40.exe, UnitTestRunner.Dispose.

The interesting thing is that Newtonsoft.Json (aka Json.Net) isn't used in the project at all or any referenced libraries. From the above log, it appears to me that this is outside of the coded UI test since it isn't registered as a script exception.

The question then is, why did this start after changing the user that the Coded UI system runs tests under? I have tried reverting it back to the original user via a snapshot (and change the host environment in Test Manager). Doing so gives me the same result above.

I'm looking for any ways to debug this further or ideas for fixes. I have tried the following as failed fixes:

• including Newtonsoft.Json in the solution
• copy over the library to c:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies
• revert to a known working build and run it (still get same error)

Big thing to note is that last item. I had builds that I know worked and worked on multiple machines. Using those builds on any machine produces the same result. What could cause that error message and be machine/test code independent?

Answer 1

The error

The error message is

Could not load file or assembly 'Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies.

One thing to know here is that this error has two options:

1. It cannot find Newtonsoft.Json
2. It cannot find a dependency

The analysis

The typical way to analyze this sort of problem is using Process Monitor [Microsoft SysInternals].

If you want to do it fully manually, follow these steps:

1. Run Process Monitor
2. Add a filter for the executable which has the problem
3. Reproduce the issue
4. Stop recording events once the problem has been reproduced
5. The hard task: From the bottom, look for errors Name not found, Path not found or Access denied. You have to find a DLL that has a problem and is not followed by Success later (which may happen if the DLL is found late in the search paths).

I have automated this process and provide Process Monitor Log Analyzer for download. Disclaimer: I'm the author of this free tool.

With that tool, the steps are:

1. Run Process Monitor
2. Add a filter for the executable which has the problem
3. Reproduce the issue
4. Stop recording events once the problem has been reproduced
5. Save Process Monitor log as XML
6. Open the XML in Process Monitor Log Analyzer
7. Go though the items from top to bottom

Your questions

The question then is, why did this start after changing the user that the Coded UI system runs tests under?

Maybe the "old" user had sufficient access rights and the "new" user doesn't. This should not be the reason for an admin user (though still possible).

Maybe the "old" user had different folders in the PATH and the "new" user doesn't.

The interesting thing is that Newtonsoft.Json (aka Json.Net) isn't used in the project at all or any referenced libraries.

I wonder how you checked that. Note that you cannot simply use Dependency Walker for .NET. You need something .NET specific, e.g. dotPeek [JetBrains], but dotPeek does not show all dependencies at once, you'd need to go through all references manually. However, this will not find missing native DLLs. Therefore, the most reliable way for me is Process Monitor as described before.

What could cause that error message and be machine/test code independent?

Same as before: user specific configurations.