How to get passed and fail test case count in xunit using cake(c# make) script

Question

I try to use the cake script for running the test cases written in Xunit using the cake script , I need to know the number of passed and failed test cases count.

#tool "nuget:?package=xunit.runner.console"
var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies);

Reference : http://www.cakebuild.net/dsl/xunit-v2

Can anyone please suggest how to get the number of passed and failed test cases?

Answer 1

You'll have to use XUnit2Aliases​.XUnit2(IEnumerable < FilePath >, ​XUnit2Settings) + XmlPeekAliases for reading the XUnit output.

var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies,
     new XUnit2Settings {
        Parallelism = ParallelismOption.All,
        HtmlReport = false,
        NoAppDomain = true,
        XmlReport = true,
        OutputDirectory = "./build"
    });

The xml format is:(XUnit documentation, the example source, more information in Reflex)

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="1" errors="1" failures="0" skip="0">
    <testcase classname="path_to_test_suite.TestSomething"
              name="test_it" time="0">
        <error type="exceptions.TypeError" message="oops, wrong type">
        Traceback (most recent call last):
        ...
        TypeError: oops, wrong type
        </error>
    </testcase>
</testsuite>

Then the following snippet should bring you the information:

var file = File("./build/report-err.xml");
var failuresCount = XmlPeek(file, "/testsuite/@failures");
var testsCount = XmlPeek(file, "/testsuite/@tests");
var errorsCount = XmlPeek(file, "/testsuite/@errors");
var skipCount = XmlPeek(file, "/testsuite/@skip");

Answer 2

Like most test runners, XUnit returns the number of failed tests in the return code from the console runner. Out of the box, Cake throws an exception, and therefore fails the build, when the return code of a tool is NOT zero.

This can be seen in the XUnit Runner Tests here:

https://github.com/cake-build/cake/blob/08907d1a5d97b66f58c01ae82506280882dcfacc/src/Cake.Common.Tests/Unit/Tools/XUnit/XUnitRunnerTests.cs#L145

Therefore, in order to know whether:

simply it is passed or failed in code level

This is known implicitly by whether or not the build succeeded or not. I typically use a strategy similar to this:

Task("Tests")
.Does(() =>
{
    var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
    XUnit2(testAssemblies,
        new XUnit2Settings {
            Parallelism = ParallelismOption.All,
            HtmlReport = false,
            NoAppDomain = true,
            XmlReport = true,
            OutputDirectory = "./build"
    });
})
.ReportError(exception =>
{
    Information("Some Unit Tests failed...");
    ReportUnit("./build/report-err.xml", "./build/report-err.html");
});

This is making use of the Exception Handling capabilities in Cake:

http://cakebuild.net/docs/fundamentals/error-handling

To take action when an error occurs. On top of that, I am then using the ReportUnit alias to convert the XML Report into a human readable HTML Report.