Breaking the build in TeamCity if .NET Core unit tests running under Docker have code-coverage less than 90%

Question

I have recently been looking at Docker, and how I can use TeamCity to run .NET Core unit tests in Docker containers as part of my build pipe-line. I add this as the final line in my Dockerfile to be able to run tests:

ENTRYPOINT ["dotnet", "test", "--verbosity=normal"]

These Dockerfiles are then referenced in compose files which TeamCity builds and runs using docker-compose in the command line.

I have this working successfully now. Next challenge is to break the build if unit/integration test coverage is less than 90% - or some other value - no arguments about this please!

I'm successfully using the coverlet.msbuild NuGet dependency to measure code-coverage as part of my build. This works fine in TeamCity too, and I see the output in my TeamCity build.

I got this working by adding coverlet.msbuild to each of my test projects, and changing the Dockerfile entry-point to:

ENTRYPOINT ["dotnet", "test", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=90", "/p:ThresholdType=line"]

The TeamCity build output shows the ASCII tables with the results in, but as of yet I've not been able to find a good way to break the build if the code-coverage isn't high enough. Left to its own devices, TeamCity doesn't mark builds as failing if the code-coverage is too low, which if fair enough as it's not psychic!

I naively thought I could create a failure condition in TeamCity which would detect for the presence of the following text:

'[Assemnbly]' has a line coverage '9.8%' below specified threshold '95%'

...using a regular expression like this:

has a line coverage '((\d+(\.\d*)?)|(\.\d+))%' below specified threshold '((\d+(\.\d*)?)|(\.\d+))%'

However, when DLLs being tested reference other DLLs that are tested separately, it gets tricky because coverlet.msbuild reports coverage metrics for all "touched" DLLs. For instance, I have a test project called Steve.Core.Files.Tests which tests Steve.Core.Files. However, Steve.Core.Files in turn references Steve.Core.Extensions. I test Steve.Core.Extensions separately in its own test DLL so I don't care about the results for that DLL when testing files. The output in TeamCity looks like this:

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Steve.Core.Extensions | 23.5%  | 40%    | 40%    |
+-----------------------+--------+--------+--------+
| Steve.Core.Files      | 100%   | 100%   | 100%   |
+-----------------------+--------+--------+--------+

...so it fails based on the 23.5% bit, even though the DLL in question is 100%. This actually makes it very difficult to check the using a Regex failure condition.

To complicate things further I'm running all tests in all assemblies using a single dynamic Dockerfile, for two reasons:

1. I don't want to have to change the Dockerfile and docker-compose file (and TeamCity) each time I add more projects and tests.

2. There are many dependencies between the DLLs so it makes sense to build them once and test them all together.

This means I'm loath to split the tests up so that each has its own Dockerfile - I know that this would allow me to use the Exclude/Include flags to get the desired behaviour.

Does anyone have any other ideas how I can solve this please?

I'm hoping I can add a file at the level of each test project to tell it which DLLs to do coverage for - that would be the best solution. Failing that, as I use a strict naming convention between projects and test projects, can I add a switch to my dotnet test command to test only the assembly that has the same name as the test assembly minus the .Tests bit on the end?

Thanks in advance; help appreciated!

Cheers,

Steve.

Update 7th September 2018:

So, my Dockerfiles are now specific to each unit test project. They look like this and exist in next to the test project files:

FROM microsoft/dotnet:2-sdk

# Set the working directory:
WORKDIR /src

# Copy the solution file and the NuGet.config across to the src directory:
COPY *.sln NuGet.config ./

# Copy the main source project files to the root level:
COPY */*.csproj ./

# Make directories for each project file and move the project file to the correct place:
RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done

# Restore dependencies:
RUN dotnet restore

# Copy all files so that we have all everything ready to compile:
COPY . .

# Set the flag to tell TeamCity that these are unit tests:
ENV TEAMCITY_PROJECT_NAME = ${TEAMCITY_PROJECT_NAME}

# Run the tests:
ENTRYPOINT ["dotnet", "test", "Steve.Core.Configuration.Tests/Steve.Core.Configuration.Tests.csproj", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=95", "/p:ThresholdType=line", "/p:Exclude=\"[Steve.Core.Testing]*\""]

Note the exclude switch which is supposed to stop the coverage results for the Steve.Core.Testing DLL being included in the results for Steve.Core.Configuration, which is the main dependency of the tests, and the project being unit tested.

My compose file looks like this and exists next to the solution file:

version: '3.6'

services:
  # Dependencies:
  steve.core.ldap.tests.ldap:
    image: osixia/openldap
    container_name: steve.core.ldap.tests.ldap
    environment:
      LDAP_ORGANISATION: Steve
      LDAP_DOMAIN: steve.com
      LDAP_ADMIN_PASSWORD: Password1
  steve.core.data.mysql.tests.database:
    image: mysql
    container_name: steve.core.data.mysql.tests.database
    command: mysqld --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD=Password1
      - MYSQL_DATABASE=testdb
  steve.core.data.sqlserver.tests.database:
    image: microsoft/mssql-server-linux
    container_name: steve.core.data.sqlserver.tests.database
    environment:
      - MSSQL_SA_PASSWORD=Password1
      - ACCEPT_EULA=Y
      - MSSQL_PID=Developer  
  steve.core.email.tests.smtp:
    image: mailhog/mailhog 
    container_name: steve.core.email.tests.smtp  

  # Steve.Core.Configuration:
  steve.core.configuration.tests:
    image: steve.core.configuration.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Configuration.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data.MySql:
  steve.core.data.mysql.tests:
    image: steve.core.data.mysql.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.MySql.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data.SqlServer:
  steve.core.data.sqlserver.tests:
    image: steve.core.data.sqlserver.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.SqlServer.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Data:
  steve.core.data.tests:
    image: steve.core.data.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Data.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Email:
  steve.core.email.tests:
    image: steve.core.email.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Email.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Encryption:
  steve.core.encryption.tests:
   image: steve.core.encryption.tests:tests
   build:
     context: .
     dockerfile: Steve.Core.Encryption.Tests/Dockerfile
   environment:
     - TEAMCITY_PROJECT_NAME

  # Steve.Core.Execution:
  steve.core.execution.tests:
    image: steve.core.execution.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Execution.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Extensions:
  steve.core.extensions.tests:
    image: steve.core.extensions.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Extensions.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Files:
  steve.core.files.tests:
    image: steve.core.files.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Files.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Ldap:
  steve.core.ldap.tests:
    image: steve.core.ldap.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Ldap.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Maths:
  steve.core.maths.tests:
    image: steve.core.maths.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Maths.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

  # Steve.Core.Time:
  steve.core.time.tests:
    image: steve.core.time.tests:tests
    build:
      context: .
      dockerfile: Steve.Core.Time.Tests/Dockerfile
    environment:
      - TEAMCITY_PROJECT_NAME

When it runs in TeamCity, it reports only 7 tests from two projects (for some strange reason) even though there are 236 tests in 12 projects.

I'd be happy to email the output from the TeamCity build if it will help.

Does anyone know how I can get my tests all running again please?

Thanks,

Steve.

Answer 1

So, the only solution was to split each unit test project up into its own compose file which includes the dependencies needed for just that test DLL. (e.g. mailhog for testing email DLLs, SQL Server for testing database DLLs, etc...). TeamCity then runs them all individually using a single script like this:

docker-compose -f docker-compose-configuration-tests.yml up --force-recreate --abort-on-container-exit --build
docker-compose -f docker-compose-configuration-tests.yml down --volumes --remove-orphans

docker-compose -f docker-compose-data-mysql-tests.yml up --force-recreate --abort-on-container-exit --build
docker-compose -f docker-compose-data-mysql-tests.yml down --volumes --remove-orphans

...

Each one has its own Dockerfile which builds the test DLL and sets the DLL exceptions for unit test coverage. TeamCity spits out the results of all tests in a single build step, and the regex code coverage failure condition mentioned in my question above then correctly detects test projects that don't achieve x% coverage and breaks the build.

Now to work out how to integrate code checking (e.g. the modern equivalents of FxCop and StyleCop) into my build process...