Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I know this is a very very basic question. For some software applications there are a large almost infinitely high number of test cases for an application. It is not practical to test all those test cases. How do we decide when to stop testing? (other than "when money runs out").

share|improve this question
2  
when it fails.. –  Javier Nov 30 '10 at 15:59
    
I think you'd find Michael Bolton's blog post about stopping heuristics for testing useful to read: http://www.developsense.com/blog/2009/09/when-do-we-stop-test/ You may recognise some of the heuristics people have suggested in this thread. –  testerab Nov 30 '10 at 21:47
    
In my experience it has been enough by applying Pareto principle. –  Amir Rezaei Nov 30 '10 at 22:06

11 Answers 11

up vote 2 down vote accepted

Glenford Myers's book The Art of Software Testing has a simple but well principled rule for this: Testing is complete when you have stopped finding bugs. Or, more practically, when the rate at which you find new bugs greatly slows down.

Bugs tend to "cluster" in certain modules and certain functions: The moment you find a bug in one, you know that you should look in it further for more bugs. To find bugs, you can use the techniques of blackbox testing, whitebox testing, and mutation testing. As long as you are finding bugs, you know that your testing process is working!

To visualize your progress, chart the number of bugs your team has found per day. If the chart slopes down, then you know the techniques your team is using aren't going to find them anyway. Of course, if you believe your techniques aren't up to par, then please read Myers's book an apply the principles.

Now, there is a chance that you might just be short of missing a new patch of bugs, and the rate of finding bugs would have greatly increased had you continued testing for a little bit more. However, if you believe your techniques are sound, this is unlikely.

share|improve this answer
    
The rate of finding new bugs is highly dependent on external factors, and sadly - some project managers will game this. Cem Kaner cites examples of the test team being sent to the movies so that the bug discovery rate would drop, and the PM could ship. –  testerab Feb 20 '11 at 14:25

The simple answer is it depends on the system. If you're writing embedded software for a heart monitor or safety monitoring tools for a nuclear reactor then the standard is far higher than if you're writing a blogging platform.

This is really a question for a good system tester (and I'm not one) but I'll give it a shot.

Your basic measure is going to be test coverage: How much of the application has actually been tested (both by unit test and functionally).

You need to assess each potential use case (and parameters for that use case) for likelihood of it actually being used (so you may drop edge cases), complexity (simpler things being less likely to contain bugs, or rather less likely to contain hard to find bugs), cost to test (in terms of time) and potential impact of a defect if discovered in that area (this is where the nuclear reactor vs. blogging platform comes in).

Based on that assessment you need to work out which of those are going to be tested and in how much detail. Once you have a list like that the team (including a product manager / project manager / user representative) can go through that list and prioritise based on the constraints you have.

One useful technique to think about is that you may also vary the use cases that are tested with each release. For instance you might have a list of non-critical test cases and test half of them with one release and half with the next (then alternate). This way you're increasing the total test coverage you get for the effort (though at the risk of regression bugs being introduced).

This could also extend to platform testing - if you support two database back ends (or multiple browsers) test half the app on one, the other half on the other and then swap next release.

(I think this is referred to as striping but don't quote me on that.)

And then the final thing to think about is not what you test but what you actually fix when issues are discovered. It's common to say "fix all bugs" but the reality is that there are time pressures and not all bugs are equal. Again, regular bug scrubs with all the relevant parties are the best way forward. This is particularly relevant where a bug fix may be particularly intrusive as the additional work in retesting and regression testing it generates may outweigh the benefit of the fix.

share|improve this answer

When the risk associated with the software's use has been reduced to an acceptable level.

share|improve this answer
5  
Well that's the problem statement, just rephrased, isn't it? –  Martin Wickman Nov 30 '10 at 13:54
    
@Martin: apparently not. Rather than starting with test case 1 and ending with test case ∞, this answer should lead the questioner to start with the most important test case and finish when they no longer add value. –  user4051 Nov 30 '10 at 14:11
1  
While philosophically correct (and thoughtful), I think the OP is looking for something a bit more hands-on. –  Martin Wickman Nov 30 '10 at 14:14
    
"acceptable" can be defined beforehand. That helps quite a bit. –  user1249 Nov 30 '10 at 17:31
    
@Thorbjørn: "Can be defined". Yes but how? That is what the OP is looking for. –  Martin Wickman Nov 30 '10 at 18:04

If you wait till the project finishes, you'll indeed have a very large numbers of test cases. If you deliver continously, focusing on small deliveries, you'll have less test cases at each iteration and you'll be able to test everything. If you can't do small deliveries, then prioritize and start testing from the greatest priority and go testing untill you have to stop.

share|improve this answer
    
+1 for continuous small deliveries and early testing. This also has the effect that defects are easier to fix, as the original programmer is still in context and hasn't moved on to another area. I'm now working in an environment where we do this and it's scary how much more productive everyone is. –  testerab Feb 20 '11 at 14:32

"Program testing can be used to show the presence of bugs, but never to show their absence!" --Edsger Dijkstra

Something good to keep in mind when doing any testing, automated or otherwise. You can only prove that you haven't found any more bugs, not that there aren't any more.

But the more eyes you put on a section of code, the more confident you can be of its proper operation. Its a lot like Knuth's quote on optimization in that regard: you can test the wrong stuff very easily, and you can test at the wrong times in your development.

Essentially, you want to be covered in two big places:

  1. Does the software pass the BDD tests that demonstrate it meets the specified requirements. The software can't even be called done if this isn't true.

  2. Do the most critical, complex, and unsure segments have adequate tests to provide confidence? If it's a core loop, or something you had to optimize or hack at: put a test on it. If it's complicated and has a lot of logical splits: put a lot of tests on it. If you can't unit test it, or it is embedded too deep to practically test directly: ensure the code has been reviewed, and the code indirectly tested by hand.

share|improve this answer

When the date of shipping has arrived. There is no end to testing for a software. But then again there is something known as schedule. You will have to test most of your functionality in the scheduled time and fix the bugs that you encounter. There is no way you can guarantee that the software is perfect.

share|improve this answer
3  
So even if you've not tested half of it you ship? This is everything that's wrong with software development. You should no more ship with incomplete testing that you would having not coded half of it. –  Jon Hopkins Nov 30 '10 at 11:28
2  
This will only produce a certain psychological resignation in the tester. I am going to think "no matter what I do, I can't test this thing completely since it is going to ship on the xth of January anyway so Ill just do whatever I can until then". Not the way we should be building software is it? –  rsman Nov 30 '10 at 11:44
    
As a system tester, I've rarely been in the position where the release date got delayed for more testing. I know I will never test anything completely - what I try to do is prioritise. Obviously, the quality of the calls I make about which areas to test first are dependent on the information I get about technical risk and business importance. The most important thing is that it should ALWAYS be a business decision and not a dev/test decision about what level of risk the company is willing to take on. We can advise, but it's the business that must decide. –  testerab Feb 20 '11 at 14:30
    
Though I'd completely agree: it's not done until it's tested. (I tend to agree with the idea that we'd be better off using the term "fixing phase" than a testing phase.) My disagreement is just that I think testing is inherently open-ended - you can never draw a line and say "there are no more tests we could possibly do now", just "there are no more tests we think it's worth doing now". –  testerab Feb 20 '11 at 16:57

If you're talking about unit testing and you're doing TDD (writing the tests first) then this is a non-issue: you just stop testing when the features are done.

In incremental TDD, you write a test that fails, then implement the smallest amount of code that can make it pass, then refactor. Keep adding tests in this manner until the method is feature complete.

Here's a great example.

share|improve this answer

The first things to test would be the "happy path", edge cases, and invalid inputs. If there will be more than one concurrent user, you'll need to test for concurrency issues like locking and race conditions. If the app uses external resources, you'll need to test how the app behaves when those resources are unavailable. After that, you can use the code to look for things that might cause it to break and test for those. When all of those tests pass, the cost/benefit ratio of further testing starts to go up, so it's reasonable to stop at that point.

share|improve this answer

It all boils down to a matter of confidence. Do you feel confident that the system is tested enough?

Obviously, "confidence level" is highly subjective since you can never feel completely certain, but certain enough -- and that is what we are looking for. For that, you need to create a list of indicators, commonly known as definition of done and should be something your whole team agrees upon.

Here are a few test related "Done-indicators":

  • Is your build and installation completely automated and all tests (unit, gui, integration) executed automatically?
  • Do you write your tests while (or preferably before) writing the code, rather than after?
  • Do you feel safe enough to make large code refactoring without introducing bugs?
  • Is your code coverage level high enough?
  • Do you have a dedicated tester in your team? Is he/she involved daily throughout the whole development and not just at the end?
  • Has your tester manually (exploratory) tried to break it without success?

If you can check these points, then you probably can say you have tested enough.

share|improve this answer

Never, I think you will never finish testing in a system.. there are sooo many variables you can't manage.

But, as we know, you can not test "for ever", so i think the limit depends basically on:

  • When the risk associated with the software's use has been reduced to an acceptable level. (as @Graham Lee says)
  • Who is the system user? it can be you or the president of united states. In the first case you don't care a lot if a bug appears because you solve it and its done. In the second case you don't want ANY bug to appear.
  • What's your relationship with your client? Maybe the client its your dad, so it is not so terrible, or maybe it is a big company.
  • How serious is for the system users a bug? Will it cause the third world war or just an ugly message?
share|improve this answer

When the people that have to sign off on the deploy are satisfied.

or in some cases the majority of the responsible parties are satisfied.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.