Unable to get hudson to parse JUnit test output XML

Question

EDIT: This issue has been fixed by google in gtest 1.4.0; see the original bug report for more information.

I've recently switched to gtest for my C++ testing framework, and one great feature of it which I am presently unable to use is the ability to generate JUnit-style XML test reports, which could then be read in by our hudson build server.

The XML output generated by the gtest test suite all looks legit:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="370" failures="0" disabled="0" errors="0" time="45.61" name="AllTests">
    <testsuite name="application" tests="7" failures="0" disabled="0" errors="0" time="8.953">
        <testcase name="zero_tasks_on_bootup" status="run" time="0" classname="application" />
...etc.
    </testsuite>
</testsuite>

I've also tried adding a JUnitReport task to my ant build script, which works fine, and generates XML like so:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="370" failures="0" disabled="0" errors="0" time="45.61" name="AllTests">
    <testsuite name="application" tests="7" failures="0" disabled="0" errors="0" time="8.953">
        <testcase name="zero_tasks_on_bootup" status="run" time="0" classname="application" />
    ...etc.
    </testsuite>
 </testsuite>

The problem is, whenever I tell ant to publish the JUnit test results, and then point it to either the raw test result XML, or the compiled result generated in the ant JUnitReport task, hudson always complains about finding no test results there.

I'm not a java guy, so I can't tell what's going on here, and I can't find an example of how the JUnit XML ought to look like. Can someone help to point me in the right direction?

Answer 1

Here's how I do it:

    <target name="junit" depends="compile-tests" description="run all unit tests">
      <mkdir dir="${reports}"/>
      <junit haltonfailure="false">
         <jvmarg value="-Xms128m"/>
         <jvmarg value="-Xmx128m"/>
         <classpath>
            <path refid="project.classpath"/>
         </classpath>
         <formatter type="xml"/>
         <batchtest fork="yes" todir="${reports}">
            <fileset dir="${test}/classes">
                <include name="**/*Test*.class"/>
            </fileset>
         </batchtest>
      </junit>
  </target>

  <target name="generate-reports" depends="junit" description="create JUnit test HTML reports">
      <mkdir dir="${reports}"/>
      <junitreport todir="${reports}">
          <fileset dir="${reports}">
              <include name="TEST-*.xml"/>
          </fileset>
          <report format="frames" todir="${reports}"/>
      </junitreport>
  </target>

Answer 2

Edit: Google test has fixed this issue, which is included in the gtest 1.4.0 release. See the original bug report for more info.

Bah! I've finally found the cause of this problem -- it's because gtest produces one giant XML file for all test results, and hudson expects one XML test report per class. I've written a perl script as a workaround for this issue. To use it, you would make a target in your ant xml script which looks something like this:

<target name="runtests">
  <exec executable="wherever/${ant.project.name}Test" failonerror="false" dir="tests">
    <arg value="--gtest_output=xml:${build.dir}\reports\${ant.project.name}.xml"/>
  </exec>
  <!-- Workaround for broken gtest output -->
  <mkdir dir="${build.dir}/reports/output"/>
  <exec executable="perl" failonerror="false" dir="tests">
  <arg value="gtest-hudson.pl"/>
    <arg value="${build.dir}/reports/${ant.project.name}.xml"/>
    <arg value="${build.dir}/reports/output"/>
  </exec>
</target>

For some reason, gtest also doesn't like the wrong style of slashes being passed to it from ant, so I made my exec for windows only, as my hudson is running on a windows server. Change to '/' for unix, obviously.

I've also filed an issue for this on the gtest page, and also one on hudson's issue tracker, so hopefully one of the two teams will pick up on the issue, as I don't have enough time to jump in and make a patch myself.... though if this doesn't get fixed in the near future, I might just have to. ;)

Answer 3

I'm almost certain that this is not a problem parsing the XML but rather a problem finding the XML files. If you are using a relative path in the Hudson config, make sure you are clear which directory it is relative to (I seem to remember it being non-obvious under certain circumstances).

As for examples of what the JUnit XML files are supposed to look like, good luck with that. It's not precisely specified anywhere. Different tools have differing dialects. That said, Hudson does a good job of recognising all of them. I believe it was the developers of JUnitReport who first introduced the XML format, so if you're using that, that's about as canonical as you are going to get.