Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have a large team of developers and testers. The ratio is one tester for every one developer.

We have full bug tracking and reporting systems in place.

We have test plans in place.

Every change to the product, the testing team is involved in the design of the feature and are included in the development process as much as possible.

We build in small iterative blocks, using scrum methodology and every scrum they are included in, including the grooming sessions etc.

But every release of the product, they miss even the most simple and obvious defects.

How can we improve this?

share|improve this question

closed as too broad by gnat, MichaelT, GlenH7, Kilian Foth, Bart van Ingen Schenau Feb 17 at 12:37

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Are you trying to fix this from the developer side? Testing side? Or project management side? –  MichaelT Apr 15 '13 at 7:11
    
@MichaelT - Primarily as Developer –  SetiSeeker Apr 15 '13 at 7:13
    
"large team of developers and testers" Scrum teams should be no more than 5-7 people. If you have more, it will become problematic. Also how are testers motivated to release high-quality product? –  Euphoric Apr 15 '13 at 10:57
    
Also, how much automated your tests are? Agile development assumes massive use of automatic testing with manual being rare and only for most obvious problems. –  Euphoric Apr 15 '13 at 10:58
add comment

8 Answers

up vote 7 down vote accepted

Any methodology is only as good as your application of it. Somewhere in the process (or perhaps at multiple places), something is not getting done adequately. Hypothetical examples:

  • Tester X goes and tells his friend developer Y about a bug, rather than putting it into the bug tracking system. It doesn't really get fixed, and the bug is forgotten.
  • The testing plan for feature A is not detailed enough. It says "user can successfully open the file". But it doesn't say what kind of file to use, so the tester only tries it with a small test file, rather than a 1,000,000 line file that might be used in real life. The program passes the test, but doesn't work with real-world data.
  • The testing is rigorous, but after the testing process, someone notices small feature Z that is missing. This tiny feature is added without retesting (it is a trivial feature anyway, what could it break?) and the release is rushed out the door. But this breaks the release version.

And so on. It is impossible to tell where your specific process is going wrong. But it should be easy get at the root cause by looking at the bug reports and asking simple questions:

  • Was the bug in functionality covered in the testing plan?

    • Yes. Was the specific failure covered by a test case?
      • Yes. Was this test run and passed in the version that shipped?
        • Yes. The tests themselves are inadequate and need to be improved.
        • No. The process for releasing the software is broken if software that doesn't pass your own testing requirements was shipped.
      • No. Test cases are not rigorous enough or not specific enough.
    • No. The tests do not cover all functionality.

I do find one thing suspicious. You say:

the testing team is involved in the design of the feature and are included in the development process as much as possible.

What kind of input are the testers giving here?

One essential feature of good testing is that the testers are separate from the process of creating the software. They are not making decisions about the design and functionality of the program. And they certainly should not be writing code. If they are doing things like this on your project, then your testers are not really testers, and your testing process is broken. Why is this such a problem?

  • Testers who help design features are likely to make the same assumptions made during the design process and less likely to discover problems due to those assumptions being broken. Also, due to human nature, they will be less free to criticize the way things work if they helped design the way it works.
  • Testers who also write the code are far less productive at finding bugs. None of us want our own code to break, so subconsciously we will make the testing less rigorous, avoiding things that might be a problem, rationalizing to ourselves "that doesn't really need to be tested, it would never happen anyway, etc...". And the same assumptions used in writing the code will also be used in the testing, again preventing finding bugs that break those assumptions.
share|improve this answer
    
your input is invaluable. thank you. regarding where the testers are involved in the design, it is more they are aware of the end goal. ie the customer requires this new functionality and it will do Y. –  SetiSeeker May 1 '13 at 10:00
    
@SetiSeeker, that makes sense. Obviously the testers need to be thinking about how the customers are going to use it. I just wasn't sure from the question what exactly that meant. I'm glad to help. –  dan1111 May 1 '13 at 11:17
    
Excellent answer! It would be neat to see this answer summarized in a decision tree, so I can print it out and pin it to the cube walls of our QA team ;-) –  maple_shaft Jun 30 '13 at 13:28
    
"One essential feature of good testing is that the testers are separate from the process of creating the software". I have to disagree with that. While I agree that they shouldn't be writing any code that they are responsible to test, they should ideally be involved in all other aspects of the process. –  Bryan Oakley Jun 30 '13 at 16:15
add comment

The first thing you need to do, before this question can be answered any further, is to do a root cause analysis - keep asking "why?". Then you'll hopefully know what needs to be improved/fixed.

share|improve this answer
    
+1: United Technologies has what they call "Relentless Root Cause Analysis". It is very useful for getting to the bottom of things. –  Peter K. Apr 15 '13 at 11:35
    
while, a very simple Idea, this is brilliant and something we willa tempt to do. –  SetiSeeker May 1 '13 at 10:02
add comment

Are the test plans correct?

Do your requirements change a lot. Do the test team have enough time to update their test plans? Who validates that the test plans are correct?

Are there certain testers whose sections end up with more bugs than others?

Looking at it another way, there will always be bugs, but it's the "simple and obvious defects" phrase that sticks out.

Are you sure your dev team have enough unit tests in place? There is nothing I hate more as a dev when someone finds an "simple and obvious defect". Why didn't I catch that? Sure some will get through, but if there is a lot, then I am doing something wrong.

share|improve this answer
    
In my opinion, in there current form the test plans are not comprehensive enough and have way to many holes. we have been attempting to correct these and make them correct and complete. –  SetiSeeker May 1 '13 at 10:03
add comment

Just guessing... could you be suffering from a Test Pyramid with feet of clay (poor or no unit test suite) ?

"Simple and obvious defects" sounds like something that should be taken care of right from early development stages.

Also, it might be a good idea (if not already the case) to include quality assurance as part of the definition of Done for your user stories. Having the testing team test during the sprint and reject any non-compliant story is much more efficient than testing once per release.

share|improve this answer
add comment

As a lone developer, there is not much you can do to improve the number/quality of the defects found by the testers.

As a team, you can try to encourage the testers to try to break the system in as many and as creative ways as they can. That is the role of the tester, to break the system, not to prove that it works.

To get the testers to think in terms of breaking the system, you might try to put out a small reward for the tester that, during the last sprint, broke the system in the most creative way. It does not even have to have monetary value or physical form.

share|improve this answer
    
That 'reward' can even be non-material, if you manage to set up a (friendly) competition between testers and developers; the developers boasting about their flawless code and the testers about their ability to find errors in there. With a big hurrah for the tester finding the most spectacular bug (e.g. a program crash, a very cunning test approach, etc) –  Jan Doggen Apr 15 '13 at 8:18
    
@JanDoggen: I agree. The idea is that finding bugs gets rewarded in some way. –  Bart van Ingen Schenau Apr 15 '13 at 8:50
    
I can't let the talk about rewards tied into bugs go by without bringing up the classic Dilbert Minivan comic from Nov. 13, 1995. search.dilbert.com/comic/Minivan –  Dunk Apr 15 '13 at 15:03
add comment

But every release of the product, they miss even the most simple and obvious defects.

The root of any good process (no matter how it look's in detail - be it scrum or traditional) is always:

  • maximum constructive quality assurance: good productive tools, coding standards, solid requirements etc. -> preventing defects
  • good analytical quality assurance: such as manual/automated blackbox testing, code reviews --> finding defects

"maximum constructive" means: before you improve finding defects, improve that you create less defects.

Are you sure you have a good analytical QA mix? Such as unit tests, manual tests, but also code reviews etc?

Do your testers use different techniques such as Exploratory Testing and not only Plan-based testing? Is your tooling sufficent for testing? (I hope you don't use Excel-lists...)

Takes your testing different view angles?

Do your testers have good test data?

In the end: budget? Would it be possible to test everything as desired? If not, set priorities.

That are only some points. Probably no team or company is good at everything and your team is probably also not bad at everything.

Try to brainstorm over the issue and priorize things that might help. But always ensure that you have a balance between constructive and analytical measures.

And some not so nice truth: some people are not suited for complex projects - they are sloppy and will nevery change. Get rid of them.

share|improve this answer
add comment

We experienced this a lot.

The main problem, which I have experienced a lot is that the feature gets tested, it seems OK, you push it to production and then you experience problems.

My approach to this (I am a one-man QA operations with 4 developers!) is a good mix of automated regression tests combined with unit testing the current feature.

I don't have the time or capacity to write (selenium in my case) scripts for all the various features being delivered, so what I do is tested them manually. That's usually fairly straight-forward. I focus on multiple browsers, version and devices at this point to see how things look. I try to put in unusual values, click all links, that kinda stuff.

On a parallel track I develop automated tests using Selenium. These serve as good Integration tests and are really helpful at getting at those 'unintended consequences' that happen when one thing is changed and it affect others. These tests also represent true user workflow. I have each suite start from scratch and build up the entire system. Then the data gets deleteed (also through the ui). I have had to learn about selenium variables, xpath and css and javascript call-out to achieve some of the more difficult things but it has had great reward. The suites focus on the core functionality and workflow of the product, not the latest features.

Also 'when' QA gets involved is important. I have seen two approaches to this:

  1. When the feature has been coded and delivered and is deemed 'ready for QA'
  2. During the design session

It is easiest to do 1) but this leads to QA being too late in the process and - critically - not being involved in the design or layout. This in turn leads to low motivation and low job satisfaction which are the two biggest killers to getting high-quality work. Under approach 2) when QA is involved up-front during design, they are then more likely to be motivated to do good QA when the time comes.

Also - make sure (and humor can help here) that QA knows that it's really good when they find bug(s). It is their job and it saves the company from putting it in front of the client. "Many QA testers quickly feel they become Product Owners themselves. Find a ay to work with this, not against it."

share|improve this answer
add comment

You're asking the wrong questions. For one, the whole team is responsible for software quality, so it's not just that the testers are missing the defects. The developers are, too.

The more important question is, why are your teams adding the "simple and obvious" defects to begin with? Don't focus your attention on catching more bugs, focus your attention on writing fewer bugs.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.