What mutation-testing frameworks exist? [closed]

Answer 1

I took a look at Jester (the actual source code) and it seems to me that it does not support too many mutations. There is a file in there where these mutations are specified. I might be wrong about the above but what I definitely did not like was the mix between launching the tool from command line and the little GUI feedback interface. Why not give feedback in the command line like JUnit does when run outside an IDE?

Jumble is another thing :). It has a simple command line interface and comes with an Eclipse plugin too. The feedback is all text in the console. I am happy with this tool and I plan to write some ANT target to add it in my project continuous integration.

I am also looking at Javalanche but did not try it yet.

I'll have news in a few weeks.

Answer 2

There is also PIT which can be hooked into your build via a maven plugin or command line interface.

It provides much nicer reports than the other available tools with combined mutation and line coverage. It also runs considerably faster than the source based tools for Java such as Jester, and about twice as fast as Jumble.

Unlike the Jumble and Javalanche it also works with all the major mocking frameworks (Mockito, JMock, EasyMock, PowerMock and JMockit).

(disclosure I'm the author).

Answer 3

For Ruby there is Heckle, and a newcomer called Boo_hiss.

Answer 4

You can try µJava. I haven't used it, but it looks like mutation testing might be an interesting way to evaluate test suites.

Answer 5

Jester does provide a file for the mutations and they are limited. To some degree, you can add your own mutations to the file.

I've experimented with Jumble and Jester and I found that Jumble provides more mutations and better documentation. Additionally, I've had quick responses from the project owners when I've emailed them. One drawback to Jumble is that it operates on the bytecode using BCEL. That presents something of a learning curve for many developers.

My company, State Farm, wrote an Ant task that we may contribute back to the Jumble project. Based on what I've read in their mailing lists, others are working on an Ant task for Jumble too.

I'm looking at Javalanche as well. I’ll be glad to share what I know when I’m done.