Best practice for debug Asserts during Unit testing

Question

Does heavy use of unit tests discourage the use of debug asserts? It seems like a debug assert firing in the code under test implies the unit test shouldn\'t exist or the d

Answer 1

Assertions in your code are (ought to be) statements to the reader that say "this condition should always be true at this point." Done with some discipline, they can be part of ensuring that the code is correct; most people use them as debug print statements. Unit Tests are code that demonstrates that your code correctly performs a particular test case; don'e well, they can both document the reuirements, and raise your confidence that the code is indeed correct.

Get the difference? The program assertions help you make it correct, the unit tests help you develop someone else's confidence that the code is correct.

Answer 2

Does heavy use of unit tests discourage the use of debug asserts?

No. The opposite. Unit testing makes Debug asserts much more valuable, by double-checking the internal state while running the white-box tests you've written. Enabling Debug.Assert during unit test is essential, because you rarely ship DEBUG-enabled code (unless performance is not important at all). The only two times DEBUG code is run is when you are either 1) doing that tiny bit of integration testing you really do, all good intentions aside, and 2) running unit tests.

It is easy to instrument code with Debug.Assert tests to check invariants as you write it. These checks serve as sanity checks when unit tests run.

The other things Assert does is point to exactly the first point in the code where things went wrong. This can greatly reduce debugging time when your unit test does find the problem.

This increases the value of unit tests.

It seems like a debug assert firing in the code under test implies the unit test shouldn't exist or the debug assert shouldn't exist.

Case in point. This question is about a real thing that happens. Right? Therefore you need debug asserts in your code, and you need them to trigger during unit tests. The possibility that a debug assert might fire during a unit test clearly demonstrates that debug asserts should be enabled during unit tests.

An assert firing means that either your tests are using your internal code incorrectly (and should be fixed), or some of the code under test is calling other internal code incorrectly, or that somewhere a fundamental assumption is wrong. You do not write tests because you think your assumptions are wrong, you... actually, you do. You write tests because at least some of your assumptions are probably wrong. Redundancy is OK in this situation.

"There can be only one" seems like a reasonable principle. Is this the common practice? Or do you disable your debug asserts when unit testing, so they can be around for integration testing?

Redundancy hurts nothing but the runtime of your unit tests. If you really have 100% coverage, runtime might be an issue. Otherwise, no I strongly disagree. There is nothing wrong with checking your assumption automatically in the middle of a test. That's practically the definition of "testing".

Also here is an example that I believe shows the dilema: A unit test passes invalid inputs for a protected function that asserts it's inputs are valid. Should the unit test not exist? It's not a public function. Perhaps checking the inputs would kill perf? Or should the assert not exist? The function is protected not private so it should be checking it's inputs for safety.

Usually, it is not the purpose of a unit test framework is to test the behavior of your code when the invariant assumptions have been violated. In other words, if the documentation you wrote says "if you pass null as a parameter, results are undefined", you do not need to verify that results are indeed unpredictable. If the failure results are clearly defined, they are not undefined, and 1) it should not be a Debug.Assert, 2) you should define exactly what the results are, and 3) test for that result. If you need to unit test the quality of your internal debug assertions, then 1) Andrew Grant's approach of making Assertion frameworks a testable asset should probably be checked as the answer, and 2) wow you have awesome test coverage! And I think this is largely a personal decision based on the project requirements. But I still think debug asserts are essential and valuable.

In other words: Debug.Assert() greatly increases the value of unit tests, and the redundancy is a feature.

Answer 3

Do you mean C++/Java asserts for "programming by contract" assertions, or CppUnit/JUnit asserts? That last question leads me to believe that it's the former.

Interesting question, because it's my understanding that those asserts are often turned off at runtime when you deploy to production. (Kinda defeats the purpose, but that's another question.)

I'd say that they should be left in your code when you test it. You write tests to ensure that the pre-conditions are being enforced properly. The test should be a "black box"; you should be acting as a client to the class when you test. If you happen to turn them off in production, it doesn't invalidate the tests.

Answer 4

It has been a while since this question has been asked, but I think I have a different way of verifying Debug.Assert() calls from within a unit test using C# code. Note the #if DEBUG ... #endif block, which is needed for skipping the test when not running in debug configuration (in which case, Debug.Assert() won't be fired anyway).

[TestClass]
[ExcludeFromCodeCoverage]
public class Test
{
    #region Variables              |

    private UnitTestTraceListener _traceListener;
    private TraceListenerCollection _originalTraceListeners;

    #endregion

    #region TestInitialize         |

    [TestInitialize]
    public void TestInitialize() {
        // Save and clear original trace listeners, add custom unit test trace listener.
        _traceListener = new UnitTestTraceListener();
        _originalTraceListeners = Trace.Listeners;
        Trace.Listeners.Clear();
        Trace.Listeners.Add(_traceListener);

        // ... Further test setup
    }

    #endregion
    #region TestCleanup            |

    [TestCleanup]
    public void TestCleanup() {
        Trace.Listeners.Clear();
        Trace.Listeners.AddRange(_originalTraceListeners);
    }

    #endregion

    [TestMethod]
    public void TheTestItself() {
        // Arrange
        // ...

        // Act
        // ...
        Debug.Assert(false, "Assert failed");



    // Assert

#if DEBUG        
    // NOTE This syntax comes with using the FluentAssertions NuGet package.
    _traceListener.GetWriteLines().Should().HaveCount(1).And.Contain("Fail: Assert failed");
#endif

    }
}

The UnitTestTraceListener class looks like follows:

[ExcludeFromCodeCoverage]
public class UnitTestTraceListener : TraceListener
{
    private readonly List<string> _writes = new List<string>();
    private readonly List<string> _writeLines = new List<string>();

    // Override methods
    public override void Write(string message)
    {
        _writes.Add(message);
    }

    public override void WriteLine(string message)
    {
        _writeLines.Add(message);
    }

    // Public methods
    public IEnumerable<string> GetWrites()
    {
        return _writes.AsReadOnly();
    }

    public IEnumerable<string> GetWriteLines()
    {
        return _writeLines.AsReadOnly();
    }

    public void Clear()
    {
        _writes.Clear();
        _writeLines.Clear();
    }
}

Answer 5

IMHO debug.asserts rock. This great article shows how to stop them from interrupting your unit test by adding an app.config to your unit testing project, and disabling the dialog box:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.diagnostics>
    <assert assertuienabled="false"/>
</system.diagnostics>

Answer 6

First to have both Design by Contract assertions and unit tests, your unit testing framework shall be able to catch the assertions. If your unit tests abort because of a DbC abort, then you simply cannot run them. The alternative here is to disable those assertions while running (read compiling) your unit tests.

Since you're testing non-public functions, what is the risk of having a function invoked with invalid argument ? Don't your unit tests cover that risk ? If you write your code following the TDD (Test-Driven Development) technique, they should.

If you really want/need those Dbc-type asserts in your code, then you can remove the unit tests that pass the invalid arguments to the methods having those asserts.

However, Dbc-type asserts can be useful in lower level functions (that is not directly invoked by the unit tests) when you have coarse-grained unit tests.