Unit testing a multithreaded application?

Question

Does anyone have any advice for a consistent way to unit test a multithreaded application? I have done one application where our mock \"worker threads\" had a thread.sleep

Answer 1

TypeMock (commercial) has a unit testing framework that automatically tries to find deadlocks in multithreaded applications and I think can be set up to run threads with predictable context switching.

I saw a demo this week at a show -- apparently it's in Alpha (called Racer)

http://www.typemock.com/Typemock_software_development_tools.html

Answer 2

Step one is to realise that often much of the code you need to unit test is orthogonal to the threading. This means that you should try and break up the code that does the work from the code that does the threading. Once that's done, you can easily test the code that does the work using normal unit testing practices. But of course, you knew that.

The problem is then one of testing the threading side of the problem, but at least now you have a point where this threading interfaces with the code that does the work and hopefully you have an interface there that you can mock. Now that you have a mock for the code that the threading code calls into, I find the best thing to do is add some events to the mock (this may mean you need to hand roll your mock). The events will then be used to allow the test to synchronise with and block the threading code under test.

So, for example, let's say we have something really simple, a multi-threaded queue that processes work items. You'd mock the work item. The interface might include a 'Process()' method that the thread calls to do the work. You'd put two events in there. One that the mock sets when Process() is called and one that the mock waits on after it has set the first event. Now, in your test you can start up your queue, post a mock work item and then wait on the work item's "I'm being processed" event. If all you're testing is that process gets called, then you can set the other event and let the thread continue. If you're testing something more complex, like how the queue handles multiple dispatch or something, then you might do other things (like post and wait for other work items) before releasing the thread. Since you can wait with a timeout in the test, you can make sure that (say) only two work items get processed in parallel, etc, etc. The key thing is that you make the tests deterministic using events that the threaded code blocks on so that the test can control when they run.

I'm sure your situation is more complex, but this is the basic method that I use to test threaded code and it works pretty well. You can take a surprising amount of control over multi-threaded code if you mock out the right bits and put synchronisation points in.

Here is some more info on this kind of thing, though it's talking about a C++ codebase: http://www.lenholgate.com/blog/2004/05/practical-testing.html

Answer 3

Not quite a unit test, but you could write some test code which repeatedly calls the code that will execute on different threads. Trying to create maximal interleaving between threads with a periodic or final consistency check. Of course this approach has the downside of not being reproducable, so you would need to use extensive logging to figure out what went wrong. This approach would best be coupled with unit tests for each threads individual tasks.