Mark unit test as an expected failure in JUnit

Question

How can I mark a test as an expected failure in JUnit 4?

In this case I want to continue to run this test until something is patched upstream. Ignoring the test goes

Answer 1

I'm assuming here that you want the test to pass if your assert fails, but if the assert succeeds, then the test should pass as well.

The easiest way to do this is to use a TestRule. TestRule gives the opportunity to execute code before and after a test method is run. Here is an example:

public class ExpectedFailureTest {
    public class ExpectedFailure implements TestRule {
        public Statement apply(Statement base, Description description) {
            return statement(base, description);
        }

        private Statement statement(final Statement base, final Description description) {
            return new Statement() {
                @Override
                public void evaluate() throws Throwable {
                    try {
                        base.evaluate();
                    } catch (Throwable e) {
                        if (description.getAnnotation(Deprecated.class) != null) {
                            // you can do whatever you like here.
                            System.err.println("test failed, but that's ok:");
                        } else {
                            throw e;
                        }
                    }
                }
            };
        }
    }

    @Rule public ExpectedFailure expectedFailure = new ExpectedFailure();

    // actually fails, but we catch the exception and make the test pass.
    @Deprecated
    @Test public void testExpectedFailure() {
        Object o = null;
        o.equals("foo");
    }

    // fails
    @Test public void testExpectedFailure2() {
        Object o = null;
        o.equals("foo");
    }
}

First, note that the first method is marked as @Deprecated. I'm using this as a marker for the method for which I want to ignore any assertion failures. You can do whatever you like to identify the methods, this is just an example.

Next, in the ExpectedFailure#apply(), when I do the base.evaluate(), I'm catching any Throwable (which includes AssertionError) and if the method is marked with the annotation @Deprecated, I ignore the error. You can perform whatever logic you like to decide whether you should ignore the error or not, based on version number, some text, etc. You can also pass a dynamically determined flag into ExpectedFailure to allow it to fail for certain version numbers:

public void unmarshalledDocumentHasExpectedValue() {
    doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));

    expectedFailure.setExpectedFailure(doc.getVersionNumber() < 3000);

    final ST title = doc.getTitle();
    assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}

For further examples, see ExternalResource, and ExpectedException

Ignoring an expected failure test rather than passing it

If you want to mark you tests as Ignored rather than Success, it becomes a bit more complex, because tests are ignored before they are executed, so you have to retrospectively mark a test as ignored, which would involve constructing your own Runner. To give you a start, see my answer to How to define JUnit method rule in a suite?. Or ask another question.