GitLab CI - How to verify that Unity3D Tests actually passed

问题

Here is my gitlab-ci.yml:

variables:
  GIT_STRATEGY: fetch
  GIT_CLONE_PATH: $CI_BUILDS_DIR
  GIT_DEPTH: 10
  GIT_CLEAN_FLAGS: none

stages:
  - test
  - build
unit-test:
  script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\'2019.4.3f1'\\Editor\\Unity.exe \
    -runTests \
    -batchmode \
    -projectPath . \
    -logFile ./log.txt \
    -testResults ./unit-tests.xml"
  stage: test
  tags: 
    - unity
unity-build:
  stage: build
  script: echo 'Building...'
  tags: 
    - unity

Here is the output of the pipeline:

 Running with gitlab-runner 13.1.1 (6fbc7474)
   on Desktop ryBW4ftU
Preparing the "shell" executor
00:00
 Using Shell executor...
Preparing environment
00:01
 Running on DESKTOP-G62KPLQ...
Getting source from Git repository
00:06
 Fetching changes with git depth set to 10...
 Reinitialized existing Git repository in X:/ScarsOFHonor_CI/.git/
 Checking out 8ca0d362 as master...
 Encountered 2 file(s) that should have been pointers, but weren't:
    ProjectSettings/EditorSettings.asset
    ProjectSettings/XRSettings.asset
 git-lfs/2.4.2 (GitHub; windows amd64; go 1.8.3; git 6f4b2e98)
 Skipping Git submodules setup
Executing "step_script" stage of the job script
01:18
 $ C:\'Program Files'\Unity\Hub\Editor\'2019.4.3f1'\Editor\Unity.exe -runTests -batchmode -projectPath . -logFile ./log.txt -testResults ./unit-tests.xml
 . is not a valid directory name. Please make sure there are no unallowed characters in the name. 
 (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 218)
 . is not a valid directory name. Please make sure there are no unallowed characters in the name. 
 (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 218)
 Job succeeded

As you can see pipeline passed with success. However that should not be true. I intentionally created an pushed a failing test.

See the content of unit-tests.xml:

<?xml version="1.0" encoding="utf-8"?>
<test-run id="2" testcasecount="1" result="Failed(Child)" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0" engine-version="3.5.0.0" clr-version="4.0.30319.42000" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.105692">
  <test-suite type="TestSuite" id="1007" name="ScarsOFHonor" fullname="ScarsOFHonor" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.105692" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
    <properties />
    <failure>
      <message><![CDATA[One or more child tests had errors]]></message>
    </failure>
    <test-suite type="Assembly" id="1010" name="EditMode.dll" fullname="X:/ScarsOFHonor_CI/Library/ScriptAssemblies/EditMode.dll" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.079181" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
      <properties>
        <property name="_PID" value="27144" />
        <property name="_APPDOMAIN" value="Unity Child Domain" />
        <property name="platform" value="EditMode" />
      </properties>
      <failure>
        <message><![CDATA[One or more child tests had errors]]></message>
      </failure>
      <test-suite type="TestSuite" id="1011" name="Tests" fullname="Tests" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.076239" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
        <properties />
        <failure>
          <message><![CDATA[One or more child tests had errors]]></message>
        </failure>
        <test-suite type="TestFixture" id="1008" name="DummyTest" fullname="Tests.DummyTest" classname="Tests.DummyTest" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.067615" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
          <properties />
          <failure>
            <message><![CDATA[One or more child tests had errors]]></message>
          </failure>
          <test-case id="1009" name="DummyTestSimplePasses" fullname="Tests.DummyTest.DummyTestSimplePasses" methodname="DummyTestSimplePasses" classname="Tests.DummyTest" runstate="Runnable" seed="785721540" result="Failed" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.032807" asserts="0">
            <properties />
            <failure>
              <message><![CDATA[  Expected: 1
  But was:  2
]]></message>
              <stack-trace><![CDATA[at Tests.DummyTest.DummyTestSimplePasses () [0x00001] in X:\ScarsOFHonor_CI\Assets\Tests\EditMode\DummyTest.cs:15
]]></stack-trace>
            </failure>
          </test-case>
        </test-suite>
      </test-suite>
    </test-suite>
  </test-suite>
</test-run>

As you can see there is a failing test. Why then my pipeline passes ? Is there any way I can print my results in the pipeline itself and mark the pipeline as failed when actually I have failing test?

回答1:

Hm a bit (actually extremely :D ) dirty but you could simply check if the result file contains the word <failure> using find like e.g.

find "<failure>" ./unit-tests.xml > nul && exit 1 || exit 0

> nul don't print to the console

so if the word <failure> was found it exits with failure (1) otherwise with success (0)

Not a yml expert but I think you could e.g. add it to

script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\'2019.4.3f1'\\Editor\\Unity.exe -runTests -batchmode -projectPath . -logFile ./log.txt -testResults ./unit-tests.xml && find '<failure>' ./unit-tests.xml > nul && exit 1 || exit 0"

来源：https://stackoverflow.com/questions/62954146/gitlab-ci-how-to-verify-that-unity3d-tests-actually-passed

标签

unity3d

continuous-integration

gitlab