Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Are developers are in the US, our testers are in India.

How do we measure their ability, quality, success?

Some possibilities: - Coverage at the UI level - Total time needed to run regression tests

share|improve this question

migrated from stackoverflow.com Jan 29 '11 at 1:16

This question came from our site for professional and enthusiast programmers.

4 Answers 4

Well I tend to use the fingers metric.

I hold up my fingers to count and say:

Me: "How many offshore testing teams would I consider working with?"

Me: "Umm... None."

Then the problem of defining other metrics goes away.


Now, before that gets marked down... The only reason I would use an offshore testing team, is if I was testing a localised version of my software in that region. Otherwise, it will only cause trouble, testing requires a lot of communication to impart exact details of bugs and errors, small nuances must be made where otherwise perfect functionality has subtle issues. Many interfaces and functionalities require assumptions in general use to understand fully their operation, assumptions and approaches tend to vary by region, offshore people might miss these.

Please do not have me down as a racist or some other imbecile going on about giving jobs away to other countries; there are good reasons why testing should be done at home, preferably by people who work with you, but not actually on the same project. There are first-class, highly skilled people in all countries, not knocking India (the subject of the question), the country has a very high quality of CS graduates and a talented programmer base. Just saying, outsourcing long-distance such things can cause issues. You can probably get away with it in low-level unit testing and such like, but wider project testing, not so much.


If that argument is unpersuasive, then other than fluency of communication, I do not think there are any special metrics for an offshore team as opposed to an onshore team.

share|improve this answer

Simple answer: There is no metric that you can put in place that cannot be gamed by intelligent motivated people. If you pick a metric to judge your test team by, they will give you the numbers you seek. They might not give you all the other stuff you really wanted, but you'll certainly get your numbers. Be careful of getting exactly what you ask for.

To add complication: If you're trying to measure your test team's ability, quality, and success, you've already got a big problem because of the distance, and you need to take into account that the communication problem in splitting distributed teams by function (as opposed to by component, or project), is going to play a major part in how effective your test team can be. When you measure their effectiveness, by whatever means, you are also measuring your own ability to communicate with them - they may be a great team, but you aren't using them well because you just aren't keeping them in the loop.

I'd recommend stepping back and asking yourself: what are you hoping to get out of these metrics? Is there another way of achieving the same goal, which isn't prone to all the measurement errors that you'll get with this path? What are you hoping to get out of your distant test team? Are they giving you the information you need to improve the quality of your product? You would be better off putting qualitative measures in place - they may seem more subjective, but you've a better chance of actually capturing useful information.

For more information about the sort of problems you'll need to consider setting up a metrics programme, Kaner & Bond's "Software Engineering Metrics: What Do They Measure and How Do We Know?" is the classic paper: http://www.kaner.com/pdfs/metrics2004.pdf

There's also an article Kaner wrote a few years before that that's a bit quicker & easier to read if you just want to get a general overview to start with: http://www.kaner.com/pdfs/rethinking_sw_metrics.pdf

share|improve this answer

This will probably sound a little bit tongue-in-cheek, but IMO the only valid metrics for a testing team are quantity, severity and frequency of bugs reported by users.

share|improve this answer
3  
Generally agreed, but I would add number of bugs found and time given to test. I've seen teams deliver code that just wouldn't run. By the time the test team was given code that met basic sanity checking criteria, three quarters of the time left for testing had expired. –  Jim Rush Jan 29 '11 at 13:33
    
Unfortunately, testers don't put the bugs in. They only find them. If the system is chock-full of bugs in the first place (and being fixed, cack-handedly by the same folks who created the original bugs, then your testing team will look bad by this metric, no matter how good they are, as each bug they report and get fixed will introduce several more. –  testerab Jan 30 '11 at 15:10
1  
Jim +1 for time given to test. It's a classic problem in a waterfall environment - test comes at the end, and always gets squeezed. To my mind, if you're 75% of the way through testing time before you get code that actually runs - this should be a huge red flag for the project manager that either the programmers did not get enough time to develop the product, or they did such a bad job that attempting to patch up the mess will only get you so far. –  testerab Jan 30 '11 at 16:12
1  
@Testerab: I've never really understood the "each bug that gets fixed will introduce several more" idea. That has not been my experience at all as a software developer. The vast majority of bugs are caused by the coder overlooking something, which becomes obvious and easy to fix once you isolate it with a debugger. Less than 10% of fixes end up creating new bugs, usually just one new bug to replace the one old bug that got fixed. If this isn't the case where you work, something's very wrong with your development process. –  Mason Wheeler Jan 30 '11 at 23:12
    
I agree absolutely that it doesn't make sense to say bug fixes generally introduce a lot more bugs - and some of that >10% was likely just revealed by the fix, not introduced. However, I was responding to Jim's outlined scenario of code delivered that doesn't even run - in that case, yes, something is very wrong with that development process. I wouldn't expect those teams to be doing a great or even adequate job of bug fixing, as whatever caused the original problem is likely to apply just as much to the bug fixing. Sorry if that wasn't clearer in my original comment. –  testerab Feb 1 '11 at 0:05

The most important metric is of course bugs, found by end-users.

Do they create automated tests? Try using various code coverage metrics and code reviews.

Do they handle bugs? Calculate share of rejected or incomplete bugs, or incorrectly processed bugs, review their bug reports, research severity of bugs found by them.

Do they create test documentation? Review their test documentation, look at traceability matrix.

share|improve this answer
    
Traceability matrix is only applicable to some very specific environments: if you're being asked to confirm compliance to requirements, then you would use it. It doesn't tell you anything however, about other quality aspects of the software, and should be regarded as a bad sign if used in an inappropriate project context. –  testerab Mar 13 '11 at 12:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.