Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We test our code to make it more correct (actually, less likely to be incorrect). However, the tests are also code -- they can also contain errors. And if your tests are buggy, they hardly make your code better.

I can think of three possible types of errors in tests:

  1. Logical errors, when the programmer misunderstood the task at hand, and the tests do what he thought they should do, which is wrong;

  2. Errors in the underlying testing framework (eg. a leaky mocking abstraction);

  3. Bugs in the tests: the test is doing slightly different than what the programmer thinks it is.

Type (1) errors seem to be impossible to prevent (unless the programmer just... gets smarter). However, (2) and (3) may be tractable. How do you deal with these types of errors? Do you have any special strategies to avoid them? For example, do you write some special "empty" tests, that only check the test author's presuppositions? Also, how do you approach debugging a broken test case?

share|improve this question
5  
+1 Excellent question –  Toby Oct 12 '10 at 23:06
2  
Every introductory piece I've read about mocking seems to hit this problem. Once you start mocking things the tests always seem to be more complicated than the code they're testing. Obviously this is less likely to be the case when testing real-world code, but its quite disheartening when you're trying to learn. –  Carson63000 Oct 13 '10 at 3:52
    
@Carson63000 If it's a simple test testing something with a tested mock, the complexity is split and under control (, I think). –  mlvljr Nov 12 '10 at 10:43
5  
But then how do you test the test tests? –  Slomojo Feb 20 '11 at 21:30
    
+1. Item 1 could be a requirements error. Can only be prevented by reviewing the requirements. Probably out of the programmer's hands unless they are also the requirements analyst –  MarkJ Feb 27 '12 at 17:30
add comment

9 Answers 9

up vote 10 down vote accepted
  1. For this problem: If you have an interface to the component a tester is the client of the interface and the developer is the implementer. You have multiple solutions: a) The client should be different person than the implementer. b) You could apply a technique Test Driven Development (TDD) in order to think about the client first and write tests, and after that you add functionality.

  2. You don't need to test the underlying platform. The tests not only exercise the code written by you, but they run the code from the platform two. While you don't necessary want to catch bugs in the testing platform, is very hard to write code and tests that always hides a bug in the platform, in other words is very hard to have a systematic bug in both your tests/code and in the platform, and the probability is lowered with each test that you create. Even if you would try to do this you would have a very hard task.

  3. While you could have bugs in tests they usually are easy to be caught. Because the tests are tested by the code developed. Between the code and the tests you have a self enforcement feedback. Both make prediction about how a specific call of an interface should behave. If the response is different you don't necessary have a bug in the code. You could have a bug in the test as well.

So, as you can see from my answers this is a non problem. Tests are by design protected from the bugs, because the testing only detects differences between code and our expectations. If there are problems we have an error. The error could be in the code or with the same probability in tests.

share|improve this answer
    
Very nice answer. I like the idea of a self-reinforcing loop between the tests and the code and the observation that it would be difficult to write tests consistently hiding bugs in the underlying platform. –  Ryszard Szopa Oct 13 '10 at 0:15
add comment

Short answer: The production code tests the tests.

Compare this to the credit / debit model used in economics. The mechanics are very simple - If the credit differs from the debit there is something wrong.

he same goes for unit tests - If a test fails it indicates something is wrong. It might be the production code, but it might aswell be the test code! This last part if important.

Note that your type (1) bugs cannot be found by unit tests. To avoid this types of bugs you need other tools.

share|improve this answer
add comment

Not an answer (I do not have the privilege to comment), but was wondering if you forgot other reasons for developing test cases...
Once you figure out all the bugs in the tests, you can regression test your application easily. Automated test suites would help you find problems earlier, before integration. The changes to requirements are relatively easier to test, as the changes can become newer, altered version of older test cases that pass, and older cases stay to pick up failures.

share|improve this answer
add comment

Hey.
You have to applications:

  • Your product
  • Your test for that product.

When you are running tests against your product, you actually are not intrested in test itself, but in interaction between your product and your tests. If test fails it doesn't say that application has a bug. It says that interaction between product and test was not successful. Now it is your job to determine what went wrong. It can be either:

  • application is not behaving as you expected (this expectation is expressed in your test)
  • application is behaving correctly, you just haven't documented this behavior correctly (in your tests)

For me tests failing are not simple feedback, that this and that is wrong. It is indicator that there is inconsistency, and I need to examine both to check want went wrong. In the end I am responsible for verifying that application is correct, tests are just a tool to highlight areas that may worth checking.

Tests are only checking some parts of application. I test the application, I test the tests.

share|improve this answer
add comment

One tactic is to write the test before the code it tests, and ensure the test fails first for the right reason. If you use TDD you should get at least this level of testing of tests.

A more exhaustive way to test the quality of a test suite is to use mutation testing.

share|improve this answer
1  
And that your test fails for the right reason. –  Frank Shearar Oct 13 '10 at 4:33
    
@Frank - Yes. I'll add that to the answer. –  Don Roby Oct 13 '10 at 9:35
1  
+1 for mentioning mutation testing. –  Ryszard Szopa Oct 13 '10 at 17:02
    
And you're adding a new test for the new behavior to be tested. Don't add to existing tests. –  Huperniketes Oct 14 '10 at 17:00
    
@DonRoby, Have you found mutation testing useful in practice? What deficiencies you've found in your test cases with that? –  dzieciou Nov 25 '12 at 21:42
add comment

For #1 and #3: Unit tests should not contain any logic, if you do then you are probably testing more than one thing in your unit test. One best practice for unit testing is to only have one test per unit test.

Watch this video by Roy Osherove to learn more on how to write unit tests well.

share|improve this answer
    
ad #3 - I agree that the tests should be as simple as possible, and should not contain any logic. However, think about the setup phase of the test, when you create the objects that it will need. You may create slightly wrong objects. This is the kind of problems I am thinking about. –  Ryszard Szopa Oct 13 '10 at 0:57
    
When you say 'slightly wrong objects' do you mean the object state is not correct or the actual design of the object is not correct? For object state you could probably write tests to check it's validity. If the design is wrong then the test should fail. –  Piers Myers Oct 13 '10 at 1:08
add comment

There must be a point when one should stop trying to unit test. Should know when to draw the line. Should we write test cases to test test cases? What about the new test cases written to test test cases? How will we test them?

if (0 > printf("Hello, world\n")) {
  printf("Printing \"Hello, world\" failed\n");
}

Edit: Updated with explanation as suggested by comment.

share|improve this answer
    
-1 What? This seems to have no relevance. –  alternative Oct 12 '10 at 23:38
2  
There must be a point when one should stop trying to unit test. Should know when to draw the line. Should we write test cases to test test cases? What about the new test cases written to test test cases? How will we test them? –  aufather Oct 12 '10 at 23:52
2  
Process Brain raised EInfiniteRecursion while attempting to extrapolate your statement... –  Mason Wheeler Oct 13 '10 at 0:10
    
Replace your answer with your comment and you'll get a +1 –  Note to self - think of a name Oct 13 '10 at 0:13
3  
In all fairness, your example is a straw man. You're testing the printf() subsystem in a C library, not the actual program calling printf(). I do agree, however, that at some point one must break the recursive testing of tests. –  Tim Post Oct 13 '10 at 2:33
add comment

In terms of #1 - I think it's a good idea to pair / code review for this side of things. It's easy to make presuppositions or just get things wrong but if you have to explain what your test is doing, what the point is, you're more likely to pick up if you're aiming at the wrong target.

share|improve this answer
add comment

Try making the individual tests as small (short) as possible.

This should reduce the chances of creating a bug in the first place. Even if you manage to create one, it's easier to find. Unit tests are supposed to be small and specific, with low tolerance for failure and deviation.

In the end, it's probably just a matter of experience. The more tests you write, the better you become at it, the less chance you have to make crappy tests.

share|improve this answer
2  
What if the tests need some rather complicated setup? Sometimes these sort of things are not under your control. –  Ryszard Szopa Oct 12 '10 at 23:38
    
Well, I'm guessing the complicated setup are "initial condition" of the tests. If that fails, all your tests should fail. In fact, I'm working on such a project right now, and people who never used unit tests constantly asked the same thing..until we explained what unit tests really are :) Then they realized it can be done, in spite of tremendous complexity of the project. –  dr Hannibal Lecter Oct 13 '10 at 7:21
    
What's the best way to check that this "initial condition" is met is exactly the point of my question. Do you write a separate test for that? Or just assume that the tests will break if this condition is not true? What about the situation when the setup is not "catastrophically" bad, just slightly off? –  Ryszard Szopa Oct 13 '10 at 14:52
1  
Your tests should fail if the initial conditions are not right, that's the whole point. When in state A, you expect result B. If you don't have state A, a test should fail. At that point you can investigate why it failed, bad initial conditions or a bad test, but it should fail in both cases. Even if it is, as you say, "slightly off" (i.e. "A" => "B", "a" => "b", but never "a" => "B" or your test is bad). –  dr Hannibal Lecter Oct 13 '10 at 15:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.