Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Any non-trivial commercial application/program/system/what-have-you requires testing (QA/QC).

Not counting things like automated tests, how much time/money relative to the development investment in the project should be spent testing (e.g. with live people)?

I know there's no magic percentage of time or budget, and individual projects will differ in size and scope of testing, but is there something that's considered good practice? An application may be perfect in every way, but still the only means of ensuring that is to test it.

Basically my employer wants to cut down on testing of over-seas-developed software, but I have strong reservations against that (being in charge of QA and having worked with the vendor before, I don't think that our directive of telling them to "make fewer bugs" is exactly going to get us results, no matter what is stipulated in the contract).

Is there any kind of best-practice or guideline that I can look to?

share|improve this question
4  
There's no guideline. You need enough to ship software that works. Also, remember Garbage In, Garbage Out. If your outsourced developers are giving you an unfit product that requires extra work and expense to bring into compliance with your quality needs, how much money are you really saving by using them? –  Rein Henrichs May 31 '11 at 18:33
    
This is a good question for Blizzard –  Tim May 31 '11 at 18:42
add comment

closed as too broad by gnat, Corbin March, Jimmy Hoffa, Dan Pichelman, MichaelT Sep 11 '13 at 2:35

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

2 Answers

up vote 8 down vote accepted

If there's one law that I've stumbled on after working software projects for more than a decade, it's this:

The longer you wait to fix a problem, the more expensive and troublesome it will be to fix.

Over the years, I've been able to experiment with different process strategies. The one that consistently increased the quality of the software the most was where we had 50% calendar days testing. An iteration would work like this:

  • Define/refine the set of features/bug fixes that will go in the current sprint.
  • Implement the features/bug fixes
  • Deploy to testing.
  • Testers begin testing the newly deployed internal release, while developers start the next iteration (first bullet)
  • Testers report problems, which get prioritized for future iterations (most of the time they have higher priority than new features)

Essentially I had two teams: the development team and the test team. The test team, by definition, was always one iteration behind the development team. This helped provide early feedback and catch significant problems before they grew up to insurmountable problems.

It also meant that an iteration could never be shorter than the time it takes to test the application. That helps avoid a cycle that was too rapid for the teams to keep up. The sweet spot for my teams seemed to be about 1-2 weeks per iteration/sprint (all iterations are the same length of time).

My test team was usually smaller than the development team, so the man hours worked out differently. With a team of 6 developers, we could keep up with 1 or 2 testers.

share|improve this answer
    
I think this is a very reasonable answer and fits with what I want to do. I've basically been told to test a few hundred software products in two languages (staggered, but with about 50 simultaneous products in QA at the peak, which will last a month or so) with a QA staff of about five or six part-time people/freelancers/interns over the course of the next year or more. Not a terribly exciting prospect. –  Kyle Lowry May 31 '11 at 23:15
add comment

In short, theres no answer to the question. Whats is your domain? Web games for kids, Incubaters for premature babies, iPhone iapps or Avionics? How much would you spend on each testing of those applications? How much does it cost to fix if you let a defect though, can it even be fixed?

Testing is not equal to QA/QC. QA starts at the first contact with the customer and ends when the software is decommissioned. Unfortunately you cannot test in quality, the best you can do is test for the absence of quality. If you detect no bugs in testing, is the program perfect or the test imperfect? Methods exist to establish this, but based on you post, the budget won't be.

You can cut down on testing if thats what your boss demands, without cutting quality - spend the savings elsewhere in the life cycle.

If he's wanting to cut costs altogether, thats a different questions with a different answer.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.