Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a chunk of code that looks something like this:

function bool PassesBusinessRules()
{
    bool meetsBusinessRules = false;

    if (PassesBusinessRule1 
         && PassesBusinessRule2
         && PassesBusinessRule3)
    {
         meetsBusinessRules= true;
    }

    return meetsBusinessRules;
}

I believe there should be four unit tests for this particular function. Three to test each of the conditions in the if statement and ensure it returns false. And another test that makes sure the function returns true.

Question: Should there actually be ten unit tests instead? Nine that checks each of the possible failure paths. IE:

  • False False False
  • False False True
  • False True False

And so on for each possible combination.

I think that is overkill, but some of the other members on my team do not. The way I look at it is if BusinessRule1 fails then it should always return false, it doesn't matter if it was checked first or last.

share|improve this question
    
Does the compiler use greedy evaluation for the && operator? –  suszterpatt Jul 6 '12 at 19:48
10  
If you wrote 10 unit tests, you would be testing && operator, not your methods. –  Mert Jul 6 '12 at 20:32
2  
Wouldn't there only be eight tests if you tested all possible combinations? Three boolean parameters turned either on or off. –  Kris Harper Jul 6 '12 at 20:32
2  
@Mert: Only if you can guarantee that the && will always be there. –  Chris Jul 6 '12 at 21:55
    
Hickey: If we spend it writing tests, that is time we are not spending doing something else. Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design. –  Job Jul 7 '12 at 4:40
show 2 more comments

7 Answers

up vote 14 down vote accepted

Formally, those types of coverage have names.

First, there's predicate coverage: you want to have a test case that makes the if statement true, and one that makes it false. Having this coverage met is probably a basic requirement for a good test suite.

Then there Condition Coverage: Here you want to test that each sub-condition in the if has the value true and false. This obviously creates more tests, but it usually catches more bugs, so it's often a good idea to include in your test suite if you have time.

The most advanced coverage criteria is usually called Combinatorial Condition Coverage: Here the goal is to have a test case that goes through all possible combinations of boolean values in your test.

Is this better than simple predicate or condition coverage? In terms of coverage, of course. But it's not free. It comes at a a very high cost in test maintenance. For this reason, most people don't bother with full combinatorial coverage. Usually testing all branches (or all conditions), will be good enough for catching bugs. Adding the extra tests for combinatorial testing won't usually catch more bugs, but requires a lot of effort to create and maintain. The extra effort usually makes this not worth the very very small payoff, so I wouldn't recommend this.

Part of this decision should be based on how risky you think that code will be. If it has a lot of room to fail, it's worth testing. If it's somewhat stable, and won't change much, you should consider focusing your testing efforts elsewhere.

share|improve this answer
2  
If the boolean values are passed from external sources (meaning they aren't always validated), then combinatorial conditional coverage is often necessary. First make a table of the combinations. Then, for each entry, decide if that entry represents a meaningful use case. If not, there should be code somewhere (either software assertions or validation clause) to prevent that combination from being executed. It is important not to lump all parameters in a single combinatorial test: try to partition parameters into groups that interact with each other i.e. shares the same boolean expression. –  rwong Jul 7 '12 at 4:57
add comment

Ultimately, it depends on you(r team), the code and the specific project environment. There is no universal rule. You(r team) should write as many tests as you need to feel comfortable that the code is indeed correct. So if your teammates aren't convinced by 4 tests, maybe you need more.

OTOH time to write unit tests is usually a scarce resource. So strive to find the best way to spend the limited time you have. E.g. if you have another important method with 0% coverage, it may be better to write a couple of unit tests to cover that one, rather than to add extra tests for this method. Of course, it also depends on how fragile the implementation of each is. Planning a lot of changes to this particular method in the foreseeable future may justify extra unit test coverage. So may being on a critical path inside the program. These are all factors which only you(r team) can assess.

I personally would usually be happy with the 4 tests you outline, that is:

  • true false false
  • false true false
  • false false true
  • true true true

plus maybe one:

  • true true false

to ensure that the only way to get a return value of true is to satisfy all 3 business rules. But in the end, if your teammates insist on having combinatorial paths covered, it may be cheaper to add those extra tests than to continue the argument a lot longer :-)

share|improve this answer
add comment

If you want to be safe, you would need eight unit tests using the conditions represented by a three variable truth table (http://teach.valdosta.edu/plmoch/MATH4161/Spring%202004/and_or_if_files/image006.gif).

You can never be sure that the business logic will always stipulate that the checks are performed in that order and you want the test to know as little about the actual implementation as possible.

share|improve this answer
1  
Unit testing is white-box testing. –  Péter Török Jul 6 '12 at 19:49
    
Well order should not matter, && is communitive, or at least should be –  Zachary K Feb 16 at 14:17
add comment

Yes, there should be the full combination in an ideal world.

When doing the unit test, you really should try to ignore how the method does its work. Simply provide the 3 inputs and verify that the output is correct.

share|improve this answer
1  
Unit testing is white-box testing. And we don't live in an ideal world. –  Péter Török Jul 6 '12 at 19:48
    
@PéterTörök - We don't live in an ideal world to be sure, but stackexchange disagrees with you on the other point. Especially for TDD, the tests are written to the specifications, not the implementation. I personally take 'specification' to include all inputs (including member variables) and all outputs (including side effects). –  Telastyn Jul 6 '12 at 20:05
1  
It's only one specific thread on StackOverflow, about a specific case, which shouldn't be overgeneralized. Especially as this current post is obviously about testing code which is already written. –  Péter Török Jul 6 '12 at 20:53
add comment

State is evil. The following function does not need a unit test because it has no side effects and it is well understood what it does and what it does not do. Why test it? Do you not trust your own brain??? Static functions are great!

static function bool Foo(bool a, bool b, bool c)
{
    return a && b && c;
}
share|improve this answer
1  
No, I don't trust my own brain - I learned the hard way to always double-check what I do :-) So I would still need unit tests to ensure that I haven't e.g. mistyped anything, and that noone is going to break the code in the future. And more unit tests to verify the caller method which calculates the state represented by a, b and c. You can move business logic around any way you want, in the end you still need to test it somewhere. –  Péter Török Jul 7 '12 at 11:41
    
@Péter Török, you can also make typos in your tests and thus end up with false positives, so where do you stop? Do you write unit tests for your unit tests? I do not trust my brain 100% either, but at the end of the day writing code is what I do for a living. It is possible to have a bug inside this function but it is important to write code in such a manner that a bug will be easy to trace to the source and so that once you isolated the problem and have made a fix, you are better off. Well written code can rely on integration tests mostly infoq.com/presentations/Simple-Made-Easy –  Job Jul 9 '12 at 3:37
1  
Indeed the tests can be faulty too. (TDD adresses this by making the tests fail first.) However, making the same kind of error twice (and overlooking it) has a much lower probability. In general, no amount and type of testing can prove that the software is bug-free, just reduce the probability of bugs to an acceptable level. And in speed of tracing bugs to the source, IMO nothing can beat unit tests - fast feedback rulez :-) –  Péter Török Jul 9 '12 at 8:50
    
"The following function does not need a unit test" I think you're being sarcastic here, but it's not clear. Do I trust my own brain? NO! Do I trust the brain of the next guy who touches the code? EVEN MORE NO! Do I trust that all the assumptions behind the code will be true a year from now? ...you get my drift. Also, static functions kill OO... if you want to do FP, then use an FP language. –  Rob Y Feb 16 at 7:12
add comment

This is one of those cases where Something like quickcheck (http://en.wikipedia.org/wiki/QuickCheck) will be your friend. Instead of writing out all N cases by hand have the computer generate all (or at least a large number) of possible test cases and validate that all return a sensible result.

We program computers for a living here, why not program the computer to generate your test cases for you?

share|improve this answer
add comment

You could refactor the conditions into guard conditions:

if (! PassesBusinessRule1) {
    return false;
}

if (! PassesBusinessRule2) {
    return false;
}

if (! PassesBusinessRule3) {
    return false;
}

I don't think that reduces the number of cases, but my experience is that it's easier to break them out this way.

(Note that I'm a big "single point of exit" fan, but I do make an exception for guard conditions. But there are other ways to structure the code so you don't have separate returns.)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.