Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In particular, I'm curious about the following aspects:

  1. How do you know that your test cases are wrong (or out-of-date) and needed to be repaired (or discarded)? I mean, even if a test case became invalid, it might still pass and remain silent, which could let you falsely believe that your software works okay. So how do you realize such problems of your test suite?

  2. How do you know that your test suite is no longer sufficient and that new test cases should be added? I guess this has something to do with the requirement changes, but is there any systematic approach to check the adequacy of test suite?

share|improve this question
4  
To paraphrase: who tests the tests? –  Konrad Rudolph Oct 8 '12 at 9:11
    
A good question for Software Quality Assurance and Testing.sx –  Martin Schröder Oct 14 '12 at 12:56
add comment

2 Answers

Short answer: use known tools that help maintain the quality of test cases such as the following code coverage and code quality tools: Cobertura, PMD, Sonar, etc. that will help you notice when a critical component of the program is not tested enough. Also, write integration tests, that are most likely to break first whenever something goes wrong.

Long answer:

How do you know that your test cases are wrong (or out-of-date) and needed to be repaired (or discarded)? I mean, even if a test case became invalid, it might still pass and remain silent, which could let you falsely believe that your software works okay. So how do you realize such problems of your test suite?

  • Using code coverage tools such as Cobertura, you are able to detect that test cases for a given class, or complex methods, are no longer sufficient. You don't need to reach 100% code coverage everywhere and in most cases it will be hard to achieve and not necessarily useful ; but tests for the most critical aspects of a program should be maintained with a goal of at least 80% of code coverage.
  • Using continuous build and integration tools, such as Jenkins which I'm very fond of, in combination with the Sonar plugin, you are able to set triggers that send emails and other types of alerts to persons responsible for the latest changes. Various graphs and statistics (Sonar also uses Cobertura among many other tools) help code reviewers and test case developers to focus on what's critical.

How do you know that your test suite is no longer sufficient and that new test cases should be added? I guess this has something to do with the requirement changes, but is there any systematic approach to check the adequacy of test suite?

What I've wrote for the first question is part of the answer for your second question. I'll also add the following points here:

  • Write integration test cases (or "business" cases if you prefer) in addition to test cases. These are most likely to change/break first because they often depend on multiple classes/methods. And since they break often, it's less likely that will be forgotten. The only approach/methodology that, from my personal experience, helps writing good tests is Test-Driven Development. Particularly if the person writing the test case is NOT the same person writing the code for it. Writing good test cases using TDD also takes time, but the results, at least for me, were extremely satisfactory.
  • Whenever a bug comes out, write the fix and the test case that comes with it. The test case should only cover this particular bug. Since you've completely covered the code that responsible for the bug, it should not come out again.
share|improve this answer
1  
+1 for TDD alone –  mouviciel Oct 8 '12 at 7:57
    
I agree with it all except for person writing the test is not the same person writing code. This sounds good in theory, and would be good if it weren't so inefficient. No matter how awesome your codebase is, if it's of any size, it takes a couple hours just to get familiar with how a portion of it works.. So basically, instead of the test-writer already being familiar with the cdoe and how it works, someone else has to come in and learn it's in and outs for a bit, and then write a test. If code quality isn't the best, it could take days to write a comprehensive test –  Earlz Oct 8 '12 at 14:46
    
@Earlz I agree with you if the two persons don't work on the same project. If the two developers work on the same project, which arguably uses in a consistent way the same framework, libraries and development methodology, he should not have any trouble, EXCEPT if it's a complex business requirement. –  Jalayn Oct 8 '12 at 15:42
    
@Jalayn for my case, the product is just very complex. Code quality isn't the best, but definitely isn't the worst(we do regular refactoring). We enforce having a separate tester, but for unit tests the person who completed the work does it. Our product consists of hundreds(maybe thousands?) of classes dealing with a complex subject, obfuscation. –  Earlz Oct 8 '12 at 15:45
1  
Well, if you have a continuous build server, your applications may be built and tested every time something is committed to the repository (we do that at work). It is configurable, you may also for example build every 15 minutes. As for the code coverage, it is enabled during the test cases and does not add much overhead. However, builds with full code quality checks enabled, like Sonar, usually take a very long time, these are run nightly for example. Ideally, you should not have to run these tools manually. –  Jalayn Oct 9 '12 at 8:56
show 1 more comment

There really isn't any way to make sure your test cases are correct, except by concentrating really well when creating them - understanding the requirement, understanding the code, and making sure that they agree. The point of having a test suite is that you only have to do this once, and from then on you can just re-execute the tests and check that they pass, whereas without a test suite you would have to concentrate really hard all the time, i.e. whenever you do anything to your code base. But the fundamental problem of having to make sure that you were doing the right thing in the first place remains - computers simply are not intelligent enough to relieve us of that task.

Therefore, (1) if your test suite is incomplete, there is no simple way of seeing that. Code coverage analysis can prove that some lines of code are never executed, i.e. that the suite is deficient in some way, but not how severe that deficiency is, and it can never prove that it is sufficient. Even with 100% code coverage you have no guarantee that all relevant states of the system are exercised, and complete state coverage is inachievable for any realistic system because of the combinatorial number of states that could exist. One good technique of making sure that you test case is at least correct for checking the thing you want checked is to write the test, verify that it does indeed fail, write/change the code, and then verify that it now passes. Hence the enthusiasm for test-driven development: it allows you to be quite sure that an individual test does the right thing, and if you create your entire code base that way, you can get a similar level of confidence even in a large system.

(2) A test suite normally becomes insufficient whenever requirements change - you don't have to guess. If the customer wants some particular behaviour changed, and your tests would succeed both before and after the change, then clearly they weren't exercising that particular input/output relationship.

As for legacy systems that don't have test coverage, or where you don't know what the coverage is, there is no formal proof, but (parental advisory: personal opinion follows!) speaking from experience it is overwhelmingly likely that the tests are not adequate. When testing is viewed as an after-the-fact, optional, quality-enhancing-but-not-really-necessary activity, it tends to be incomplete and unsystematic because the incentive for making sure the tests keep up with the code base just isn't there.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.