JUnit test with dynamic number of tests

Question

In our project I have several JUnit tests that e.g. take every file from a directory and run a test on it. If I implement a testEveryFileInDirectory method in t

Answer 1

If TestNG is an option, you could use Parameters with DataProviders.

Each individual file's test will have its result shown in the text-based report or Eclipse's TestNG plugin UI. The number of total tests run will count each of your files individually.

This behavior differs from JUnit Theories, in which all results are lumped under one "theory" entry and only count as 1 test. If you want separate result reporting in JUnit, you can try Parameterized Tests.

Test and inputs

public class FileTest {

    @DataProvider(name="files")
    public File[][] getFiles(){
        return new File[][] {
            { new File("file1") },
            { new File("file2") }
        };
        // or scan a directory
    }

    @Test(dataProvider="files")
    public void testFile(File file){
        //run tests on file
    }
}

Example output

PASSED: testFile(file1)
PASSED: testFile(file2)

===============================================
    Default test
    Tests run: 2, Failures: 0, Skips: 0
===============================================

Answer 2

Junit 5 Parameterized Tests

JUnit 5 parameterized tests support this by allowing the use of a method as data source:

@ParameterizedTest
@MethodSource("fileProvider")
void testFile(File f) {
    // Your test comes here
}

static Stream<File> fileProvider() {
    return Arrays.asList(new File(".").list()).stream();
}

JUnit 5 DynamicTests

JUnit 5 also supports this through the notion of a DynamicTest, which is to be generated in a @TestFactory, by means of the static method dynamicTest.

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.util.stream.Stream;

@TestFactory
public Stream<DynamicTest> testFiles() {
    return Arrays.asList(new File(".").list())
            .stream()
            .map((file) -> dynamicTest(
                    "Test for file: " + file,
                    () -> { /* Your test comes here */ }));
}

The tests run in your IDE (IntelliJ here) will be displayed like this:

Answer 3

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

Answer 4

JUnit 3

public class XTest extends TestCase {

    public File file;

    public XTest(File file) {
        super(file.toString());
        this.file = file;
    }

    public void testX() {
        fail("Failed: " + file);
    }

}

public class XTestSuite extends TestSuite {

    public static Test suite() {
        TestSuite suite = new TestSuite("XTestSuite");
        File[] files = new File(".").listFiles();
        for (File file : files) {
            suite.addTest(new XTest(file));
        }
        return suite;
    }

}

JUnit 4

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestY {

    @Parameters
    public static Collection<Object[]> getFiles() {
        Collection<Object[]> params = new ArrayList<Object[]>();
        for (File f : new File(".").listFiles()) {
            Object[] arr = new Object[] { f };
            params.add(arr);
        }
        return params;
    }

    private File file;

    public TestY(File file) {
        this.file = file;
    }

    @Test
    public void testY() {
        fail(file.toString());
    }

}

Answer 5

Take a look at Parameterized Tests in JUnit 4.

Actually I did this a few days ago. I'll try to explain ...

First build your test class normally, as you where just testing with one input file. Decorate your class with:

@RunWith(Parameterized.class)

Build one constructor that takes the input that will change in every test call (in this case it may be the file itself)

Then, build a static method that will return a Collection of arrays. Each array in the collection will contain the input arguments for your class constructor e.g. the file. Decorate this method with:

@Parameters

Here's a sample class.

@RunWith(Parameterized.class)
public class ParameterizedTest {

    private File file;

    public ParameterizedTest(File file) {
        this.file = file;
    }

    @Test
    public void test1() throws Exception {  }

    @Test
    public void test2() throws Exception {  }

    @Parameters
    public static Collection<Object[]> data() {
        // load the files as you want
        Object[] fileArg1 = new Object[] { new File("path1") };
        Object[] fileArg2 = new Object[] { new File("path2") };

        Collection<Object[]> data = new ArrayList<Object[]>();
        data.add(fileArg1);
        data.add(fileArg2);
        return data;
    }
}

Also check this example

Answer 6

I had a similar problem and ended up writing a simple JUnit 4 runner that allows med to dynamically generate tests.

https://github.com/kimble/junit-test-factory