What's the difference between failure and error in JUnit?

Question

I\'m running JUnit tests on a large code base, and I\'ve been realizing that sometimes I get \"Errors\" while other times I get \"Failures\". What\'s the difference?

Answer 1

From "Pragmatic Unit Testing in Java 8 with JUnit":

Assertions (or asserts) in JUnit are static method calls that you drop into your tests. Each assertion is an opportunity to verify that some condition holds true. If an asserted condition does not hold true, the test stops right there, and JUnit reports a test failure.

(It’s also possible that when JUnit runs your test, an exception is thrown and not caught. In this case, JUnit reports a test error.)

Answer 2

If your test throws an exception which does not get bubbled up through the Assertion framework in Junit, it gets reported as an error. For example, a NullPointer, or a ClassNotFound exception will report an error:

String s = null;
s.trim();

or,

try {

    // your code
} catch(Exception e) {
    // log the exception
    throw new MyException(e);
}

Having said that, the following will report a failure:

Assert.fail("Failure here");

or,

Assert.assertEquals(1, 2);

or even:

throw new AssertionException(e);

It depends on the Junit version you are using. Junit 4- will make the distinction between a failure and an error, but Junit 4 simplifies it as failures only.

Following link provides more interesting inputs:

http://www.devx.com/Java/Article/31983/1763/page/2

Answer 3

Ok, I've just noticed a pattern and think I've figured it out (correct me if I'm wrong). It seems to me that failures are when your test cases fail - i.e. your assertions are incorrect. Errors are unexpected errors that occur while trying to actually run the test - exceptions, etc.

Answer 4

Source class : JUnitReportReporter.java

public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String defaultOutputDirectory) {
//......

            for (ITestResult tr : (Set) entry.getValue()) {
                TestTag testTag = new TestTag();

                boolean isSuccess = tr.getStatus() == 1;
                if (!(isSuccess)) {
                    if (tr.getThrowable() instanceof AssertionError)
                        ++errors;
                    else {
                        ++failures;
                    }
                }
}

As you can see below line in above method

tr.getThrowable() instanceof AssertionError

errors count is increased when it is instance of AssertionError otherwise(any Throwable) is counted as failures.

Answer 5

You are right that failures come from the AssertionErrors thrown by the JUnit assertion methods, or by throwing an AssertionError, or by throwing an exception that you declared in your @Test annotation, and Errors come from other, unexpected Exceptions. But there's an important distinction between them:

A failure means your test ran correctly, and identified a defect in your code.

An error could mean a bug in your code, but one that you weren't even testing for. It could also mean that the bug is in the test itself.

In short, a failure means you need to rewrite the code that is being tested. An error means that it may be the unit test that you need to rewrite. It may mean this even if the failure was in your code, such as a NullPointerException, because you detected a flaw that you weren't even testing for, so it might be wise to test for that.

Answer 6

Below test explains the difference between Test Error vs Test failure.

I have commented the line which throws test error and test failure.

    @Test
    public void testErrorVsTestFailure() {

        final String sampleString = null;

        assertEquals('j', sampleString.charAt(0) );
        //above line throws test error as you are trying to access charAt() method on null reference

        assertEquals(sampleString, "jacob");
        //above line throws Test failure as the actual value-a null , is not equal to expected value-string "jacob"
        }

So Junit shows test error whenever you get an exception , and test failure when your expected result value doesn't match your actual value