Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

During last year, several of the projects of the company where I work failed.

We develop web software, that are not especially complicated (the most are CRUDS and search form about this data).

One of the problems that management has identified is the test phase.

Putting simple, our protocol is:

  1. Analysis of the requeriments
  2. Development
  3. When construction is over, upload the executable to a server
  4. Execute test plan
  5. Repair the errors
  6. Execute the test plan, to check if errors persist.
  7. If all is ok -> Send to the client.

The problem is that when we finish phase 3 and start phase 4, the number of errors is to damn high, that the execution plan takes to much time, and we get an interminable list of things that we have to repair.

Until arrive to the test plan we are on time, but after this we spend more hours that were planned and then "fail" the project. We deliver with enought quality for the client but spend more hours repairing bugs.

The management is looking for a solution, and is planned a meeting to ask us for ideas to improve the quality before the first test plan. Any idea?

share|improve this question
    
English is not my main languaje, so if there are something that is hard to understand, please let me know, and I try to explain better. –  user674887 Oct 19 '11 at 17:11
1  
Do you have formal testers or who does testing? Can testers read and write code? –  P.Brian.Mackey Oct 19 '11 at 20:04
2  
Write flawless code ;) –  MattyD Oct 19 '11 at 21:43
6  
#1 problem I see is that you are waiting until the end of the SDLC to do any testing. You should be testing iteratively throughout the process. I'm willing to be that the schedules also tend to assume that the code will always pass the tests, or include only a single round of test/fix. Very unrealistic. –  JohnFx Oct 19 '11 at 21:52
    
@P.Brian.Mackey Nope. It's a small company and the people who test usually are non technical, and are similar to our clients. –  user674887 Oct 20 '11 at 16:12
show 3 more comments

7 Answers

up vote 14 down vote accepted

P. Brian Mackey is right about TDD being useful - and any unit tests will improve things dramatically. As a tester, I can tell you that a thorough unit test suite dramatically decreases the number of bugs, and limits the remaining functional bugs to mostly a few predictable areas where I can focus my testing (mostly integration points / environmental interactions). This simplifies test coverage and makes testing go faster.

Other ideas:

  • Get QA involved ASAP. If you don't have at least one QA specialist sitting on all or nearly all of the meetings, starting with the design stage, you are missing a lot of value. Senior testers can help find design flaws, bring up small features that will greatly speed up testing, and can start planning test strategy sooner when they are involved in design meetings.
  • Get good, solid requirements, get them early, and get QA to review them. A good QA specialist will find holes in the requirements that no one thought to address. Sometimes things that are minor details to devs are huge blockers to QA (e.g., the specifics of business rules; dev can build the framework for rules and then add the actual rules later with little effort, but tests can't be written until the specifics are known). The worst test cycles I've seen have been when the BA folks didn't get clear requirements, got them late, or communicated modifications to requirements verbally to developers without notifying QA. Also, test automation can't be developed concurrently with product development without good requirements up front.
  • Develop more incrementally. Don't throw the entire product to QA, but deliver pieces throughout the development cycle as they are finished for component testing and component integration testing. Agile development techniques are great for reducing the test cycles and distributing them more smoothly throughout the project so you don't have a huge snarl at the end.
  • If you have developers or testers who read code working in QA, have them do code reviews as well as developers (so one dev, one QA reviews each major code change). This both helps QA learn the code base and system, and also provides a quality-focused perspective on the code.
  • Use more automation in your QA process. Limit manual testing to a few well-defined areas. Automated tests can be developed before the development on the component is finished, can be run at night, and require relatively few resources once developed. Although you will probably invest more test resources overall into automation, a lot of the tester time will happen before the 'test bottleneck', when tester time is cheap.
  • Ensure that developers make working with and unblocking testers a high priority through the entire development cycle. When you discourage a tester from asking questions, learning the system, and generally doing their job, you acquire technical debt. Developers might not realize this because you don't pay interest on that debt until you get to the test phase; the cost is externalized from the developers.
  • Automate test environment setup. It's time-consuming, and most testers need to do it at least daily to keep up with bug fixes. A tester should be able to install the entire product(s) with a single command.
share|improve this answer
    
The problem with using TDD without knowing where you defects come from are that it will only help with finding problems in the code. Testing the code doesn't do anything to verify or validate your requirements and design nor does it compare your implementation against the design. –  Thomas Owens Oct 19 '11 at 18:40
    
All of this, particularly using testers at all points of the SDLC. Great testers can find bugs in more than a finished GUI –  joshin4colours Oct 19 '11 at 18:50
    
+1 I agree with most of your points, especially automating tests so they can be run nightly. However, I feel that a non-developer should not be reading code, even if they could understand it. Code reviews should be left for developers to do as part of their team development process. –  Bernard Oct 19 '11 at 19:08
    
This is a great idealistic scenario. The question is, how advanced is the OP's team as a whole? Do they have formal testers that are also advanced enough to read and write code? If they did, I don't see how they could be failing so badly. That's why I believe this to be an impractical suggestion to improve by leaps and bounds rather than incrementally. Management won't buy into leaps and bounds. –  P.Brian.Mackey Oct 19 '11 at 20:06
1  
@P. Brian, wanted to add that, actually, it sounds like the test team is doing pretty well . . . the problem isn't that it takes forever to find the bugs, it's that "the number of errors is to damn high" and "we get an interminable list of things that we have to repair". That's not a test problem - that's a code quality problem that is upstream of test. There might also be test problems that are resulting in slow turn-around, etc., but they weren't mentioned (although some of my suggestions were targeted at that scenario). –  Ethel Evans Oct 19 '11 at 23:39
show 2 more comments

I suggest that you require developers to unit test and to commit their tests to Source control along with the code. Nothing goes to Qa without a set of unit tests that have been certified as run by the developer. TDD is a great idea (where you write the tests before you write the code) but clearly your devs weren't testing their code at all, so start there. Just becuase you have QA department does not relieve the developer of the responsibilty to test his code.

Next code review should be a requirement in your company. No code is realesed until reviewed and coded reviewers will be instructed to not only review the code for techinical accuracy but for compliance with the requirements. Working code that does the wrong thing is a bug too.

Next sit down with the players (devs and qa) from the failed projects and go through the bugs found and see if there are patterns you can see. Are certain types of errors common? Then maybe your developers need training on those topics. This is not a blame session, you don't care who created the bug, you are looking to improve process and the first step is gathering data on what went wrong so you can prevent it in the future.

Next have the managers take a good hard look at the skill level of the developers you have. It is less expensive to train existing personnel to improve their skills than to fire them and hire new people. However, if you have some people who repeatedly make the same mistakes and never learn from their mistakes, it may be time to consider if they are contributing to the company. I would start with providing training (I suspect your corporate culture probably hasn't given them a chance to improve, I've seen devs have their stuff fixed by someone else and didn't even know it was bad.) and then see if the same mistakes continue. Then in about 6 months, review performance and see if some poeple just aren't improving to level you need to have.

share|improve this answer
add comment

There are quit a few challenges in your story.

Things that spring to mind are a more agile way of development (TDD is a great one but there will be a massive gap in testcases since you already have a product out there). Continuous integration and continuous delivery will help resolving build and integration issues. Also a quick growing amount of unit tests will enable you to improve quality and maintain a regression base that will indicate wheter new features break existing code.

The hardest thing to change will be the actual process, things like continuous integration require deployment of a set of tools which is a walk in the park compared to changing the process.

Cheers, Carlo

share|improve this answer
add comment

Evidence supposedly says that one of the best things you can do to decrease errors, even more than testing itself, is formal code inspection.

The article also covers the effectiveness of many other techniques you might want to consider (though pre-dates TDD so doesn't include that - personally I'd be surprised if it beat formal code review but that's personal opinion and not supported by anything).

share|improve this answer
add comment

Where it's possible for the developers to write unit tests, during them development stages, I would definitely agree with P.Brian.Mackey's advice to use Test Driven Development.

However, we've recently also got a lot of value from having users write down what they would do to test the system before development even starts: these are known as User Acceptance Tests.

Each UAT starts off as a short bulleted list, describing what the user does, and saying what will happen. They'll also record things like where to find sample input files.

As the functionality is implemented, the UAT gets fleshed out with more detail, so they are a sort of "living document". For this reason, we've ended up storing them on simple internal wiki pages, rather than any document management system.

These have turned out to be valuable in a surprisingly wide range of situations:

  1. They tell the developers exactly what the expected behaviour is, before writing the code
  2. This saves time being spent on implementing incorrect functionality, only to be told that "no, that wasn't what was wanted - please fix it"
  3. They serve as a kind of instruction manual during development:
    • for real users, to allow software to be tried out during the development stage
    • for developers who are new to a project
  4. They prompt all sorts of useful questions about the what's required, before work even starts - allowing us to work out what decisions we need to make, before development starts
  5. They give developers a better understanding of the chunk of work, earlier on - this making it much easier to estimate roughly time will be required for each stage.
  6. And of course, they make it easier for the system to be tested, as well.
share|improve this answer
add comment

The first thing to do is figure out where you defects are being injected and where they are being detected. Although you know that too many are slipping through into the testing phase, you don't know where they are coming from. Applying Defect Removal Effectiveness tracking techniques can help here. Simply, create a matrix of phases (given what you said, your phases would probably be requirements, design, implementation, testing, and field). Whenever you find a defect, track where you found it and where it came from. For each phase, track the percentage of defects that have been identified and removed.

Once you know which specific phases are problematic, you can introduce changes. Examples of quality techniques include requirements, design, and code reviews. Cause and effect analysis can also be used to determine why a defect was injected and what can be done to prevent it. Checklists can be employed to ensure repeatibility across your processes. There are a number of quality tools out there, but until you know where your problems are and what is causing them, doing anything might be a waste of resources.

I would recommend reading Stephen Kan's Metrics and Models in Software Quality Engineering. This was the textbook used in my university's Software Engineering Process and Product Quality course, and it's a very good book when it comes to understanding and improving quality at both the product level and the process level.

share|improve this answer
add comment

Use Test Driven Development. This way you continually test and integrate functionality. In CRUD based development, I like to use a iCRUD interface. This makes it easy to setup my tests. Just create ConcreteItemX which implements iCRUD and setup the 4 tests (Create, Read Update Delete). Implemenent until all tests pass.

Rinse and repeat.

Its fine to have testing later down the pipe. You just cannot wait until that late in the process to actually begin any real testing due to the fact that the cost to fix a bug increases dramatically over time.

share|improve this answer
1  
The problem with using TDD without knowing where you defects come from are that it will only help with finding problems in the code. Testing the code doesn't do anything to verify or validate your requirements and design nor does it compare your implementation against the design. –  Thomas Owens Oct 19 '11 at 18:40
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.