Are developers are in the US, our testers are in India.
How do we measure their ability, quality, success?
Some possibilities: - Coverage at the UI level - Total time needed to run regression tests
Tell me more
×
Programmers Stack Exchange is a question and answer site for
professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
|
|
migrated from stackoverflow.com Jan 29 '11 at 1:16
|
Well I tend to use the fingers metric. I hold up my fingers to count and say:
Then the problem of defining other metrics goes away. Now, before that gets marked down... The only reason I would use an offshore testing team, is if I was testing a localised version of my software in that region. Otherwise, it will only cause trouble, testing requires a lot of communication to impart exact details of bugs and errors, small nuances must be made where otherwise perfect functionality has subtle issues. Many interfaces and functionalities require assumptions in general use to understand fully their operation, assumptions and approaches tend to vary by region, offshore people might miss these. Please do not have me down as a racist or some other imbecile going on about giving jobs away to other countries; there are good reasons why testing should be done at home, preferably by people who work with you, but not actually on the same project. There are first-class, highly skilled people in all countries, not knocking India (the subject of the question), the country has a very high quality of CS graduates and a talented programmer base. Just saying, outsourcing long-distance such things can cause issues. You can probably get away with it in low-level unit testing and such like, but wider project testing, not so much. If that argument is unpersuasive, then other than fluency of communication, I do not think there are any special metrics for an offshore team as opposed to an onshore team. |
|||
|
|
|
Simple answer: There is no metric that you can put in place that cannot be gamed by intelligent motivated people. If you pick a metric to judge your test team by, they will give you the numbers you seek. They might not give you all the other stuff you really wanted, but you'll certainly get your numbers. Be careful of getting exactly what you ask for. To add complication: If you're trying to measure your test team's ability, quality, and success, you've already got a big problem because of the distance, and you need to take into account that the communication problem in splitting distributed teams by function (as opposed to by component, or project), is going to play a major part in how effective your test team can be. When you measure their effectiveness, by whatever means, you are also measuring your own ability to communicate with them - they may be a great team, but you aren't using them well because you just aren't keeping them in the loop. I'd recommend stepping back and asking yourself: what are you hoping to get out of these metrics? Is there another way of achieving the same goal, which isn't prone to all the measurement errors that you'll get with this path? What are you hoping to get out of your distant test team? Are they giving you the information you need to improve the quality of your product? You would be better off putting qualitative measures in place - they may seem more subjective, but you've a better chance of actually capturing useful information. For more information about the sort of problems you'll need to consider setting up a metrics programme, Kaner & Bond's "Software Engineering Metrics: What Do They Measure and How Do We Know?" is the classic paper: http://www.kaner.com/pdfs/metrics2004.pdf There's also an article Kaner wrote a few years before that that's a bit quicker & easier to read if you just want to get a general overview to start with: http://www.kaner.com/pdfs/rethinking_sw_metrics.pdf |
|||
|
|
|
This will probably sound a little bit tongue-in-cheek, but IMO the only valid metrics for a testing team are quantity, severity and frequency of bugs reported by users. |
|||||||||||||||||
|
|
The most important metric is of course bugs, found by end-users. Do they create automated tests? Try using various code coverage metrics and code reviews. Do they handle bugs? Calculate share of rejected or incomplete bugs, or incorrectly processed bugs, review their bug reports, research severity of bugs found by them. Do they create test documentation? Review their test documentation, look at traceability matrix. |
|||
|
