How do people maintain their test suite?

Question

In particular, I'm curious about the following aspects:

1. How do you know that your test cases are wrong (or out-of-date) and needed to be repaired (or discarded)? I mean, even if a test case became invalid, it might still pass and remain silent, which could let you falsely believe that your software works okay. So how do you realize such problems of your test suite?

2. How do you know that your test suite is no longer sufficient and that new test cases should be added? I guess this has something to do with the requirement changes, but is there any systematic approach to check the adequacy of test suite?

Answer 1

Short answer: use known tools that help maintain the quality of test cases such as the following code coverage and code quality tools: Cobertura, PMD, Sonar, etc. that will help you notice when a critical component of the program is not tested enough. Also, write integration tests, that are most likely to break first whenever something goes wrong.

Long answer:

How do you know that your test cases are wrong (or out-of-date) and needed to be repaired (or discarded)? I mean, even if a test case became invalid, it might still pass and remain silent, which could let you falsely believe that your software works okay. So how do you realize such problems of your test suite?

• Using code coverage tools such as Cobertura, you are able to detect that test cases for a given class, or complex methods, are no longer sufficient. You don't need to reach 100% code coverage everywhere and in most cases it will be hard to achieve and not necessarily useful ; but tests for the most critical aspects of a program should be maintained with a goal of at least 80% of code coverage.
• Using continuous build and integration tools, such as Jenkins which I'm very fond of, in combination with the Sonar plugin, you are able to set triggers that send emails and other types of alerts to persons responsible for the latest changes. Various graphs and statistics (Sonar also uses Cobertura among many other tools) help code reviewers and test case developers to focus on what's critical.

How do you know that your test suite is no longer sufficient and that new test cases should be added? I guess this has something to do with the requirement changes, but is there any systematic approach to check the adequacy of test suite?

What I've wrote for the first question is part of the answer for your second question. I'll also add the following points here:

• Write integration test cases (or "business" cases if you prefer) in addition to test cases. These are most likely to change/break first because they often depend on multiple classes/methods. And since they break often, it's less likely that will be forgotten. The only approach/methodology that, from my personal experience, helps writing good tests is Test-Driven Development. Particularly if the person writing the test case is NOT the same person writing the code for it. Writing good test cases using TDD also takes time, but the results, at least for me, were extremely satisfactory.
• Whenever a bug comes out, write the fix and the test case that comes with it. The test case should only cover this particular bug. Since you've completely covered the code that responsible for the bug, it should not come out again.

Answer 2

There really isn't any way to make sure your test cases are correct, except by concentrating really well when creating them - understanding the requirement, understanding the code, and making sure that they agree. The point of having a test suite is that you only have to do this once, and from then on you can just re-execute the tests and check that they pass, whereas without a test suite you would have to concentrate really hard all the time, i.e. whenever you do anything to your code base. But the fundamental problem of having to make sure that you were doing the right thing in the first place remains - computers simply are not intelligent enough to relieve us of that task.

Therefore, (1) if your test suite is incomplete, there is no simple way of seeing that. Code coverage analysis can prove that some lines of code are never executed, i.e. that the suite is deficient in some way, but not how severe that deficiency is, and it can never prove that it is sufficient. Even with 100% code coverage you have no guarantee that all relevant states of the system are exercised, and complete state coverage is inachievable for any realistic system because of the combinatorial number of states that could exist. One good technique of making sure that you test case is at least correct for checking the thing you want checked is to write the test, verify that it does indeed fail, write/change the code, and then verify that it now passes. Hence the enthusiasm for test-driven development: it allows you to be quite sure that an individual test does the right thing, and if you create your entire code base that way, you can get a similar level of confidence even in a large system.

(2) A test suite normally becomes insufficient whenever requirements change - you don't have to guess. If the customer wants some particular behaviour changed, and your tests would succeed both before and after the change, then clearly they weren't exercising that particular input/output relationship.

As for legacy systems that don't have test coverage, or where you don't know what the coverage is, there is no formal proof, but (parental advisory: personal opinion follows!) speaking from experience it is overwhelmingly likely that the tests are not adequate. When testing is viewed as an after-the-fact, optional, quality-enhancing-but-not-really-necessary activity, it tends to be incomplete and unsystematic because the incentive for making sure the tests keep up with the code base just isn't there.