How to detect and debug multi-threading problems?

Question

This is a follow up to this question, where I didn\'t get any input on this point. Here is the brief question:

Is it possible to detect and debug problems co

Answer 1

In addition to the other good answers you already got: Always test on a machine with at least as many processors / processor cores as the customer uses, or as there are active threads in your program. Otherwise some multithreading bugs may be hard to impossible to reproduce.

Answer 2

Tess Ferrandez's blog has good examples of using WinDbg to debug deadlocks in .NET.

Answer 3

I thought that the answer you got to your other question was pretty good. But I'll emphasis these points.

Only modify shared state in a critical section (Mutual Exclusion)

Acquire locks in a set order and release them in the opposite order.

Use pre-built abstractions whenever possible (Like the stuff in java.util.concurrent)

Also, some analysis tools can detect some potential issues. For example, FindBugs can find some threading issues in Java programs. Such tools can't find all problems (they aren't silver bullets) but they can help.

As vanslly points out in a comment to this answer, studying well placed logging output can also very helpful, but beware of Heisenbugs.

Answer 4

Sometimes, multithreaded solutions cannot be avoided. If there is a bug,it needs to be investigated in real time, which is nearly impossible with most tools like Visual Studio. The only practical solution is to write traces, although the tracing itself should:

1. not add any delay
2. not use any locking
3. be multithreading safe
4. trace what happened in the correct sequence.

This sounds like an impossible task, but it can be easily achieved by writing the trace into memory. In C#, it would look something like this:

public const int MaxMessages = 0x100;
string[] messages = new string[MaxMessages];
int messagesIndex = -1;

public void Trace(string message) {
  int thisIndex = Interlocked.Increment(ref messagesIndex);
  messages[thisIndex] = message;
}

The method Trace() is multithreading safe, non blocking and can be called from any thread. On my PC, it takes about 2 microseconds to execute, which should be fast enough.

Add Trace() instructions wherever you think something might go wrong, let the program run, wait until the error happens, stop the trace and then investigate the trace for any errors.

A more detailed description for this approach which also collects thread and timing information, recycles the buffer and outputs the trace nicely you can find at: CodeProject: Debugging multithreaded code in real time 1

Answer 5

Assuming I have reports of troubles that are hard to reproduce I always find these by reading code, preferably pair-code-reading, so you can discuss threading semantics/locking needs. When we do this based on a reported problem, I find we always nail one or more problems fairly quickly. I think it's also a fairly cheap technique to solve hard problems.

Sorry for not being able to tell you to press ctrl+shift+f13, but I don't think there's anything like that available. But just thinking about what the reported issue actually is usually gives a fairly strong sense of direction in the code, so you don't have to start at main().

Answer 6

I faced a thread issue which was giving SAME wrong result and was not behaving un-predictably since each time other conditions(memory, scheduler, processing load) were more or less same.

From my experience, I can say that HARDEST PART is to recognize that it is a thread issue, and BEST SOLUTION is to review the multi-threaded code carefully. Just by looking carefully at the thread code you should try to figure out what can go wrong. Other ways (thread dump, profiler etc) will come second to it.