Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

One of the few things that most software developers agree on is that you shouldn't rely on code to work correctly unless you test it. If you don't test it, it may have hidden bugs that are only going to cause you more work down the road.

I understand how to test my normal code, but how should I test my test code to make sure it can effectively find and report errors when they are present? I personally have been stupid enough to write erroneous test cases that would pass when they should not have, thus defeating the purpose of my writing tests in the first place. Fortunately, I found and fixed the errors in time, but according to testing mantra it seems like no test suite would be complete without having its own set of tests to make sure it worked.

It seems to me that the best way to do this would be to make sure the test fails for buggy code.* If I spend 2 minutes alternately adding bugs to the code and making sure it fails, I should have an acceptable degree of confidence that the tests 'work'. This brings me to my second question: What are good ways to introduce bugs to make sure that they are caught by the test cases? Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition, and change the execution order of code with side-effects, etc., until I'm satisfied my tests will catch most common bugs? How do professional developers validate that their tests actually do what they're supposed to do? Do they just assume the tests work, or do they take the time to test them as well? If so how do they test the tests?

I'm not suggesting the people should spend so much time testing their tests and then testing the tests for their tests that they never actually write the real code, but I've done stupid enough things that I feel like I could benefit from a bit of 'meta-testing', and was curious about the best way to go about it. :D

* I could check to see if the test passes when testing 'bug-free' code, but using the code as a spec for the test seem quite backwards...

share|improve this question
    
Sounds like you don't have confidence in your unit tests - most likely because you lack a lot of experience in writing tests? In that case it would be unreasonable to write more tests and expect a different outcome. Just keep doing what you're doing, be as thorough as you can (test for failure as well as success) and soon your unit tests will start to pay for themselves - and your confidence in them will grow. –  MattDavey Jul 24 '12 at 7:08
add comment

4 Answers

The standard flow for TDD is:

  1. Write a failing test. (Red)
  2. Make the smallest code change that makes it pass (Green)
  3. Refactor (Keeping it green)

The test for your tests in this case is step 1 - making sure that the test fails before you make any code changes.

Another test that I like is whether you can delete some code and re-implement it a different way, and your tests fail after deletion but work with a different algorithm in place.

As with all things, there is no magic bullet. Forgetting to write a required test is just as easy for a developer to do as forgetting to write the code. At least if you're doing both, you have twice as many opportunities to discover your omission.

share|improve this answer
2  
Double bookkeeping: The unit tests test the production code and vice versa. They are two different ways to state the same problem. Like in double bookkeeping, where you record your transactions in two different ways and both must get the same totals. –  EricSchaefer Jul 24 '12 at 18:21
add comment

Tests for tests? Don't go that road. Then you'll probably need tests for tests for tests, and then tests for tests for tests for tests... where do you stop?

Usual testing flow goes like this, and as a developer, you'll spend majority of your time on points 1-3:

  1. Code
  2. Unit tests
  3. Integration tests
  4. System/other automated
  5. QA/human testers

If I spend 2 minutes alternately adding bugs to the code (...)

Your code will eventually "grow" its own bugs, don't waste time introducing them by hand. Not to mention, is a thing you knew about upfront really a bug? Bugs will come, I wouldn't worry about that.

Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition (...)

This is actually a viable approach to verify whether you actually test what you think you do. I don't think it is always that good as it suffers from the same problem as "test for test for test ..." thing: when do you stop altering code knowing code you're testing 100% works?

It's also good to remember about all time classic pragmatic programmer advice - you ain't gonna need it. Be agile, write tests and code for actual bugs, instead for those hypothetical that might or might not appear.

share|improve this answer
2  
I'm not worried about my code growing its own bugs, I'm worried about my tests catching them when they happen. If my tests are faulty, they won't do their job and I'll think that I've got rid of a certain class of bug when they actually still exist, just making my job harder because I'm looking at inaccurate test results (pass when they should fail). –  Gordon Gustafson Jul 23 '12 at 21:49
    
@CrazyJugglerDrummer: your tests won't catch all the bugs that's for sure. They don't serve that purpose - this is where QA comes in. If they would do so, it would mean software is bug-free, unless source code changes, which I've never seen. –  jimmy_keen Jul 23 '12 at 21:54
add comment

One approach is a tool like Jester:

Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed

share|improve this answer
add comment

By construction, functional code and test code are tested one against the other. One problem remains: the case of common mode bugs, when a bug in functional code is hidden by a bug in test code. TDD is not immune to this effect. This is why testing is usually performed at multiple levels by different people in order to decrease this probability.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.