Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Should unit tests test passing all cases as well as all failing cases?

For example, imagine I have a test Widget_CannotActiveWidgetIfStateIsCancelled.

And let's say there are 25 possible states.

Can I get away with testing only that I cannot activate my widget when State == Cancelled, or do I have to also test that I CAN activate it in each of the other 24 states?

share|improve this question
1  
I'm wondering if it wouldn't be quicker to write those tests, instead of asking a question about it and waiting for a definitive answer ;-) –  phi Sep 4 '13 at 18:49
1  
@phi: You're not wrong, but you're also ignoring the cost to maintain the tests. If it was the wrong thing to do (and I'm not saying it is - I wrote a much more detailed answer) then all those extra tests would just add overhead to future development efforts. –  Aaronaught Sep 4 '13 at 19:02
1  
While ideally, you don't need to, you are allowed to look at the code when writing a unit test. If you have a statement that says if(State == Cancelled) do 1 thing else do something else then there's no reason to test all 24 other possible states. It won't be testing anything extra. Of course, this also depends upon what it means to "activate" and if any differing processing is performed because of the state. –  Dunk Sep 4 '13 at 19:12

3 Answers 3

Normally you would let your test names do the talking. What I mean is, if you can have these two tests:

  • When_state_is_canceled_then_cannot_activate_widget
  • When_state_is_not_canceled_then_can_activate_widget

And if both of the above tests are an accurate description of the behaviour, then no, you don't need to write tests for all 25 possible states, because there are only 2 states that actually matter, "canceled" and "not canceled". In the actual implementation of "not canceled", of course, you will have to pick some specific other state. From a pure code coverage standpoint, you'll get full coverage this way.

On the other hand, you should also try to predict how the SUT might change. Maybe today there are 24 states that all behave exactly the same way, but three weeks from now the rules may change in a very subtle way that either breaks or isn't covered by the test you wrote today.

One thing I've sometimes done is, in the "negative" test, have a loop that enumerates all possible states, excludes the ones that don't apply, and verifies them each in sequence. That's the lazy way, but it's a little less flaky than just picking a state at random.

But most of the time I prefer to write Context/Specification style tests, which just remove this ambiguity altogether by making each possible state a separate test class, and yes, that does mean testing all 25 states, although there are numerous ways you could use inheritance or composition in the tests to reduce code bloat while still maintaining the tests for all states.

There's no rule, you can "get away with" whatever you want, it's just a question of how thorough you want your tests to be, which is probably going to depend on such things as how many other developers are working on the code, what the cost of each defect is to the business, how often you release, etc. If you're trying to do TDD or BDD, use context/specification for each state. If you're just trying to get half-decent code coverage, then just test one "on" and one "off" state.

share|improve this answer

While you certainly should test that your unit fails correctly in addition to passing correctly, there is no reason to write a test that does not add to your confidence in the code.

However, if your "unit" has twenty-five possible states, it may be better to think of each state as a distinct unit, and construct your tests appropriately. (don't forget that your tests can share code where appropriate.)

share|improve this answer

It depends on the implementation, but most likely you'd write two tests: one for Cancelled and one for any of the other 24 states.

Why is this enough? Because all 24 other states form an equivalence class. Testing these two cases should cover all branches in the implementation of Activate.

Also note that this approach is what you would end up with if you used Test driven development. The other approach - writing 25 test cases - is a form of black box testing that generally leads to multiple tests testing the same thing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.