Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Reading the comments to this answer, specifically:

Just because you can't write a test doesn't mean it's not broken. Undefined behaviour which usually happens to work as expected (C and C++ are full of that), race conditions, potential reordering due to a weak memory model... – CodesInChaos 7 hours ago

@CodesInChaos if it cant be reproduced then the code written to 'fix' cant be tested either. And putting untested code into live is a worse crime in my opinion – RhysW 5 hours ago

...has me wondering if there are any good general ways to consistently trigger very infrequently occurring in production problems caused by race conditions in test case.

share|improve this question
step through (assembly) instruction by instruction on both ends –  ratchet freak Apr 25 '13 at 14:26
Static analysis can often show potential UB, not clear if this is counted as test though –  jk. Apr 25 '13 at 14:36
Sorry to ask, but what does 'UB' mean? –  Doug Apr 25 '13 at 14:49
Nice question, i would be interesting in seeing the potential solutions to this. –  RhysW Apr 25 '13 at 14:53
@Doug Undefined Behaviour, which can include, but is not limited to, race conditions –  jk. Apr 25 '13 at 14:54

5 Answers 5

up vote 60 down vote accepted

After having been in this crazy business since about 1978, having spent almost all of that time in embedded real-time computing, working multitasking, multithreaded, multi-whatever systems, sometimes with multiple physical processors, having chased more than my fair share of race conditions, my considered opinion is that the answer to your question is quite simple.


There's no good general way to trigger a race condition in testing.

Your ONLY hope is to design them completely out of your system.

When and if you find that someone else has stuffed one in, you should stake him out an anthill, and then redesign to eliminate it. After you have designed his faux pas (pronounced f***up) out of your system, you can go release him from the ants. (If the ants have already consumed him, leaving only bones, put up a sign saying "This is what happens to people who put race conditions into XYZ project!" and LEAVE HIM THERE.)

share|improve this answer
I completely agree. In other words, this is much like the joke - Patient: "Doctor, it hurts when I do this..." Doctor: "Then stop doing it!" –  Mark Rushakoff Apr 25 '13 at 14:43
Nice answer. If something causes an un-testable problem, try to work around it to start with, avoid the problem altogether! –  RhysW Apr 25 '13 at 14:58
My only question is: How big an anthill should I use? (+1 BTW). –  Peter K. Apr 25 '13 at 15:16
+1 for correct pronunciation of faux pas. (And the rest of the answer.) –  Blrfl Apr 25 '13 at 15:29
@PeterK., this is one of those few cases in software development, along with monitors, RAM, and disk drives, where bigger IS better. –  John R. Strohm Aug 31 '14 at 13:35

There is no way to be absolutely sure various kinds of undefined behavior (in particular race conditions) don't exist.

However, there are a number of tools that show up a good number of such situations. You may be able to prove that a problem exists currently with such tools, even though you cannot prove that your fix is valid.

Some interesting tools for this purpose:

Valgrind is a memory checker. It finds memory leaks, reads of uninitialized memory, uses of dangling pointers and out-of-bounds accesses.

Helgrind is a thread safety checker. It finds race conditions.

Both work by dynamic instrumentation, i.e. they take your program as-is and execute it in a virtualized environment. This makes them unintrusive, but slow.

UBSan is an undefined behavior checker. It finds various cases of C and C++ undefined behavior, such as integer overflows, out-of-range shifts and similar stuff.

MSan is a memory checker. It has similar goals as Valgrind.

TSan is a thread safety checker. It has similar goals as Helgrind.

These three are built into the Clang compiler and generate code at compile time. This means that you need to integrate them into your build process (in particular, you have to compile with Clang), which makes them much harder to initially set up than *grind, but on the other hand they have a much lower runtime overhead.

All the tools I listed work on Linux and some of them on MacOS. I don't think any work on Windows reliably yet.

share|improve this answer

The best tool I know for these sort of problems is an extension of Valgrind called Helgrind.

Basically Valgrind simulates a virtual processor and runs your binary (unmodified) on top of it, so it can check every single access to memory. Using that framework, Helgrind watch system calls to infer when an access to a shared variable is not properly protected by a mutual exclusion mechanism. That way it can detect a theorical race condition even if it has not actually happened.

Intel sells a very similar tool called Intel Inspector.

These tools give great results but your program will be considerably slower during analysis.

share|improve this answer
is Valgrind still a *nix only tool? –  Dan Neely Apr 25 '13 at 20:20
Yes, Linux, MacOSX, android and some BSD: valgrind.org/info/platforms.html –  Julien Apr 25 '13 at 20:35
ThreadSanitizer is a similar tool. It works differently than Helgrind, which gives it the advantage of being much faster, but requires integration into the toolchain. –  Sebastian Redl Apr 26 '13 at 17:16

If you are in the ms tool chain. Ms research has created a tool which will force new interlevings for each run and can recreated failed runs its called chess.

here is a video showing it in use.

share|improve this answer
That looks impressive; I'll have to find time to try it out at some point. –  Dan Neely Apr 25 '13 at 17:13

Exposing a multi-threading bug requires forcing different threads of execution to perform their steps in a particular interleaved order. Usually this is hard to do without manual debugging or manipulating the code to get some kind of "handle" to control this interleaving. But changing code that behaves unpredictably will often influence that unpredictability, so this is hard to automate.

A nice trick is described by Jaroslav Tulach in Practical API Design: if you have logging statements in the code under question, manipulate the consumer of those logging statements (e.g. an injected pseudo-terminal) so that it accepts the individual log messages in a particular order based on their content. This allows you to control the interleaving of steps in different threads without having to add anything to production code that isn't already there.

share|improve this answer
I have done similar before using injected repository's to sleep the threads that call it in specific orders to force the interleave I want. Having written code that does it, I'm inclined to +1 @John's answer above. Seriously, this stuff is so painful to employ correctly, and still gives only best guess guarantees because there could be slightly different interleaves with different results; the better approach is to just eliminate all possible race conditions through static analysis and or careful combing of code for any and all shared state –  Jimmy Hoffa Apr 25 '13 at 14:48

protected by GlenH7 Sep 1 '14 at 13:02

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.