XCode 5.1 Unit Test Coverage Analysis Fails On Files Using Blocks

Question

Today I was tasked with adding unit test coverage analysis to our code base. Today is also the day iOS 7.1 is released along with XCode 5.1. From the release notes:

The gcov tool for code coverage testing has been reimplemented. The new version uses the llvm-cov tool from the LLVM project. It is functionally equivalent to the old version for all significant features. The location of gcov within Xcode has also moved, use xcrun to invoke it. If you find problems, please file bug reports. For this release, you can still use the old version of gcov from GCC, which is available as gcov-4.2. 11919694 updated

I realized this only after following several instructional blog posts, getting my environment set up properly - generating .gcda/.gcno files in the simulator's build folders upon testing - and having the report generating tools here try to parse them into a report. (that is a ./getcov script which gathers your environment variables to pass to lcov-1.10 scripts to generate the report)

The first hurdle was that the new bundled gcov program doesn't support the -v argument to get the version, which is the first step of lcov's initialization. Seemed like a non-starter already, but reading the release notes above I modified the lcov script to use the old gcov-4.2 version and got that solved.

However, lcov errored out very early in processing my coverage data files. This generated a report with maybe the first 10 or so files alphabetically in my project. Not particularly useful. The error output was minimal and unhelpful as well:

geninfo: ERROR: GCOV failed for (build_artifacts)/(class_that_errored).gcda!

I modified the lcov script to print the error it was getting (which only yielded 11 unfortunately, couldn't find any reference in the gcov(-io).c code) and to continue operation instead of quitting, so I was left with a lot more files in the report, but still probably 85% of my source files had errored out as above.

The only pattern I could discern between the files that successfully wound up in the report and the ones that threw an error was that any file that used an in-line block declaration failed. None of the files that passed used blocks in any fashion, and all the files I've checked that failed contain blocks. Strange.

Then I figured out I could open the individual .gcda files in CoverStory, including the ones that had errored in the lcov script. In the message window beneath the coverage report, all the files that had errored had the warning messages:

(class_that_errored).gcno:no lines for '__copy_helper_block_'

(class_that_errored).gcno:no lines for '__destroy_helper_block_'

My best hypothesis at this point is that the new XCode 5.1 is generating .gcda files that the old gcov-4.2 program isn't equipped to deal with regarding block declarations.

But I've exhausted everything I can think to try, so I'm here to ask if anybody has a piece of knowledge that I've missed, or has any ideas to further the debugging effort. Or if anyone is successfully measuring test coverage since today's XCode 5.1 update with the new gcov, I'd love to hear about any changes you had to make as well.

Answer 1

The problem is in the LCOV 1.10 geninfo script. It tests for the current version of gcov. It does this by parsing the version string. Since gcov now points to llvm-cov, the version string is parsed incorrectly.

The solution is to modify geninfo’s get_gcov_version() subroutine. In line 1868, change -v to --version. Then replace line 1874 with:

if ($version_string =~ m/LLVM/)
{
    info("Found llvm-cov\n");
    $result = 0x40201;
}
elsif ($version_string =~ /(\d+)\.(\d+)(\.(\d+))?/)

 

The modified subroutine should look like this:

sub get_gcov_version()
{
    local *HANDLE;
    my $version_string;
    my $result;

    open(GCOV_PIPE, "-|", "$gcov_tool --version")
        or die("ERROR: cannot retrieve gcov version!\n");
    $version_string = <GCOV_PIPE>;
    close(GCOV_PIPE);

    $result = 0;
    if ($version_string =~ m/LLVM/)
    {
        info("Found llvm-cov\n");
        $result = 0x40201;
    }
    elsif ($version_string =~ /(\d+)\.(\d+)(\.(\d+))?/)
    {
        if (defined($4))
        {
            info("Found gcov version: $1.$2.$4\n");
            $result = $1 << 16 | $2 << 8 | $4;
        }
        else
        {
            info("Found gcov version: $1.$2\n");
            $result = $1 << 16 | $2 << 8;
        }
    }
    return ($result, $version_string);
}

 

^{NOTE: Make sure --gcov-tool is not set to gcov-4.2.}

Answer 2

Rather than modifying geninfo I created an llvm-cov-wrapper script and used lcov's --gcov-tool command line option:

#!/bin/bash
# llvm-cov wrapper to make it behave more like gcov

if [ "$1" = "-v" ]; then
    echo "llvm-cov-wrapper 4.2.1"
    exit 0
else
    /usr/bin/gcov $*
fi

Answer 3

Actually I noticed that lcov is invoking gcov with -b option and gcov-4.2 will crash on this option (raise segmentation fault). If I remove the -b option from getinfo, then even though it still shows some error info, the gcov file can still be generated.

That is probably why coverstory can still give coverage output. So I guess the workaround is to remove -b option from lcov. And also as you suggested, ignore the error in getinfo

Answer 4

For anyone new to this thread, note that lcov-1.11 is out. This eliminates the issue with gcov -v compatibility. However, I am using XCode 6.1 and still getting errors on some files.

geninfo: WARNING: /Users/XXX/MyFile.gcno: found unrecognized record format - skipping

Note that you can add --ignore-errors graph to skip over these errors, but the problem isn't really solved.