How To Know When To Stop Testing?

Question

I know this is a very very basic question. For some software applications there are a large almost infinitely high number of test cases for an application. It is not practical to test all those test cases. How do we decide when to stop testing? (other than "when money runs out").

Answer 1

Glenford Myers's book The Art of Software Testing has a simple but well principled rule for this: Testing is complete when you have stopped finding bugs. Or, more practically, when the rate at which you find new bugs greatly slows down.

Bugs tend to "cluster" in certain modules and certain functions: The moment you find a bug in one, you know that you should look in it further for more bugs. To find bugs, you can use the techniques of blackbox testing, whitebox testing, and mutation testing. As long as you are finding bugs, you know that your testing process is working!

To visualize your progress, chart the number of bugs your team has found per day. If the chart slopes down, then you know the techniques your team is using aren't going to find them anyway. Of course, if you believe your techniques aren't up to par, then please read Myers's book an apply the principles.

Now, there is a chance that you might just be short of missing a new patch of bugs, and the rate of finding bugs would have greatly increased had you continued testing for a little bit more. However, if you believe your techniques are sound, this is unlikely.

Answer 2

The simple answer is it depends on the system. If you're writing embedded software for a heart monitor or safety monitoring tools for a nuclear reactor then the standard is far higher than if you're writing a blogging platform.

This is really a question for a good system tester (and I'm not one) but I'll give it a shot.

Your basic measure is going to be test coverage: How much of the application has actually been tested (both by unit test and functionally).

You need to assess each potential use case (and parameters for that use case) for likelihood of it actually being used (so you may drop edge cases), complexity (simpler things being less likely to contain bugs, or rather less likely to contain hard to find bugs), cost to test (in terms of time) and potential impact of a defect if discovered in that area (this is where the nuclear reactor vs. blogging platform comes in).

Based on that assessment you need to work out which of those are going to be tested and in how much detail. Once you have a list like that the team (including a product manager / project manager / user representative) can go through that list and prioritise based on the constraints you have.

One useful technique to think about is that you may also vary the use cases that are tested with each release. For instance you might have a list of non-critical test cases and test half of them with one release and half with the next (then alternate). This way you're increasing the total test coverage you get for the effort (though at the risk of regression bugs being introduced).

This could also extend to platform testing - if you support two database back ends (or multiple browsers) test half the app on one, the other half on the other and then swap next release.

(I think this is referred to as striping but don't quote me on that.)

And then the final thing to think about is not what you test but what you actually fix when issues are discovered. It's common to say "fix all bugs" but the reality is that there are time pressures and not all bugs are equal. Again, regular bug scrubs with all the relevant parties are the best way forward. This is particularly relevant where a bug fix may be particularly intrusive as the additional work in retesting and regression testing it generates may outweigh the benefit of the fix.

Answer 3

When the risk associated with the software's use has been reduced to an acceptable level.

Answer 4

If you wait till the project finishes, you'll indeed have a very large numbers of test cases. If you deliver continously, focusing on small deliveries, you'll have less test cases at each iteration and you'll be able to test everything. If you can't do small deliveries, then prioritize and start testing from the greatest priority and go testing untill you have to stop.

Answer 5

If you're talking about unit testing and you're doing TDD (writing the tests first) then this is a non-issue: you just stop testing when the features are done.

In incremental TDD, you write a test that fails, then implement the smallest amount of code that can make it pass, then refactor. Keep adding tests in this manner until the method is feature complete.

Here's a great example.

Answer 6

I think you'd find Michael Bolton's blog post about stopping heuristics for testing useful to read:

http://www.developsense.com/blog/2009/09/when-do-we-stop-test/

You may recognise some of the heuristics people have suggested in this thread.

Answer 7

When the date of shipping has arrived. There is no end to testing for a software. But then again there is something known as schedule. You will have to test most of your functionality in the scheduled time and fix the bugs that you encounter. There is no way you can guarantee that the software is perfect.

Answer 8

Never, I think you will never finish testing in a system.. there are sooo many variables you can't manage.

But, as we know, you can not test "for ever", so i think the limit deppendes basically on:

• When the risk associated with the software's use has been reduced to an acceptable level. (as @Graham Lee says)
• Who is the system user? it can be you or the president of united states. In the first case you don't care a lot if a bug appears because you solve it and its done. In the second case you don't want ANY bug to appear.
• What's your relationship with your client? Maybe the client its your dad, so it is not so terrible, or maybe it is a big company.
• How serious is for the system users a bug? Will it cause the third world war or just an uggly message?

Answer 9

The first things to test would be the "happy path", edge cases, and invalid inputs. If there will be more than one concurrent user, you'll need to test for concurrency issues like locking and race conditions. If the app uses external resources, you'll need to test how the app behaves when those resources are unavailable. After that, you can use the code to look for things that might cause it to break and test for those. When all of those tests pass, the cost/benefit ratio of further testing starts to go up, so it's reasonable to stop at that point.

Answer 10

It all boils down to a matter of confidence. Do you feel confident that the system is tested enough?

Obviously, "confidence level" is highly subjective since you can never feel completely certain, but certain enough -- and that is what we are looking for. For that, you need to create a list of indicators, commonly known as definition of done and should be something your whole team agrees upon.

Here are a few test related "Done-indicators":

• Is your build and installation completely automated and all tests (unit, gui, integration) executed automatically?
• Do you write your tests while (or preferably before) writing the code, rather than after?
• Do you feel safe enough to make large code refactoring without introducing bugs?
• Is your code coverage level high enough?
• Do you have a dedicated tester in your team? Is he/she involved daily throughout the whole development and not just at the end?
• Has your tester manually (exploratory) tried to break it without success?

If you can check these points, then you probably can say you have tested enough.

Answer 11

"Program testing can be used to show the presence of bugs, but never to show their absence!" --Edsger Dijkstra

Something good to keep in mind when doing any testing, automated or otherwise. You can only prove that you haven't found any more bugs, not that there aren't any more.

But the more eyes you put on a section of code, the more confident you can be of its proper operation. Its a lot like Knuth's quote on optimization in that regard: you can test the wrong stuff very easily, and you can test at the wrong times in your development.

Essentially, you want to be covered in two big places:

1. Does the software pass the BDD tests that demonstrate it meets the specified requirements. The software can't even be called done if this isn't true.

2. Do the most critical, complex, and unsure segments have adequate tests to provide confidence? If it's a core loop, or something you had to optimize or hack at: put a test on it. If it's complicated and has a lot of logical splits: put a lot of tests on it. If you can't unit test it, or it is embedded too deep to practically test directly: ensure the code has been reviewed, and the code indirectly tested by hand.

Answer 12

When the people that have to sign off on the deploy are satisfied.

or in some cases the majority of the responsible parties are satisfied.

Answer 13

In my experience it has been enough by applying Pareto principle.