Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Has anyone had experience with out-sourced testing? Is it worth the fairly high premium you have to pay?

Can you simply hire some off-shore 'mechanical turks' (who offer the same services, but a lot cheaper) with a testing script as a cheaper alternative to professional software testing services?

share|improve this question
3  
To be honest, I don't see how it could possibly work. If anything, testers have to know your product even better than you (the developer) do. That's not the kind of knowledge that you can out-source. –  Benjol Jan 7 '11 at 12:20
    
The best experience I ever had with testing was when the testing was outsourced to a company on the other side of the world. We would make a final build at the end of the day, go home, and when we got back in in the morning we had a list of bugs to fix and things to look at. Of course, then it was hard to contact them if they had poor documentation for the bug, so... –  Ryan Hayes Feb 13 '11 at 13:06

3 Answers 3

Sure, you can hire people who don't know anything about testing to do it but I'd expect it to be about as successful as hiring people who don't know anything about programming to do your development.

Testing services are essentially just consultancies comprising of professional software testers.

Testing is a skill with all the associated knowledge, best practice and so on. Professional systems testers with experience are generally people with the right aptitude for testing who've spent years learning about what they're doing. That simply isn't the case for a random individual.

A good tester will know how to develop test strategies, test plans, and test scripts based on the requirements presented. They know how to accurately assess test coverage and progress, they'll understand how and when regression tests need to be carried out and to what level, they understand why tests need to be repeatable and can log defects in a way that allow developers to reproduce them easily, and they know when there isn't time to test everything (which is pretty much always), how to balance the various pressures in ways which have the best chance of maintaining quality (techniques such as striping).

Ranty Bit

Developers get rightly annoyed when people belittle the skills they have and suggest that it's easily learned but to me this is just the same thing. When programmers ask questions like this showing little thought for what other professionals do or know, what right do they have to expect others to behave differently when it comes to their skills?

Seriously, do a bit of research into what testing really is and what's involved.

Ranty Bit Ends

In terms of the specifics of off-shore testing, it shares many of the same problems as off-shore anything, largely arising from two things: communication and cultural set up.

Communication is an obvious one. I won't go into it much other than to say you need a strategy to deal with it and the main question should be "how will the testers show developers the bugs when the developers are saying they can't see them and wandering over to their desk isn't an option?". Any answer to that question involving e-mail or other text descriptions should be kicked into touch immediately.

Cultural set up is less often talked about but is for me a far more significant factor. By culture I don't mean race, I mean business culture. Most companies I've worked for don't have business processes geared around team members being hundreds of miles away. They don't write specifications in enough detail to allow people to work from them without questioning the person who wrote it (who is now in a different time zone). They operate on the assumption that quick meetings to flesh things out which are missing or unclear are easy to organise.

That's the thing I'd really look at - are your processes really good enough to work with people who can't easily just come and ask a question, and if they're not can you change them so they will work (and do you want to).

share|improve this answer
    
For the ranty part, maybe you could explain the difference. Have you had experience with testing services? –  Chris S Jan 7 '11 at 12:31
1  
@Chris S - I've worked with professional testers some of whom have worked for that sort of company yes. If your comparison is between professional testers within your organisation and external testers then skill level shouldn't be an issue, but if it's between professionals and those who have no knowledge of testing it's the world of difference. –  Jon Hopkins Jan 7 '11 at 12:43
    
Does that answer mean you have worked with testing companies? I've worked with testers too, but I'm interested in the differences between the cheaper and more expensive out sourcing companies, rather than hiring your own. –  Chris S Jan 11 '11 at 11:37
    
@Chris - I've looked into it (including fairly advanced discussions with these companies) and decided against it (in favour of a proper in-house test team) and I've worked with testers who've worked for these companies and have told me what it was like. –  Jon Hopkins Jan 11 '11 at 11:39
1  
@Chris S - Fair enough. It would be better than nothing but a long way short of getting it done properly (I've worked at companies who've taken on untrained temps to do it - they found some stuff but not much and not the sort of detailed stuff that really comes back to bite you when it gets into production). –  Jon Hopkins Jan 11 '11 at 11:58

The problem with getting someone from outside the company to test your software is that they will not know anything about your software. So you need to make sure you have good documentation that they can test the system with: e.g. requirements specification, test specification, user guides, etc.

They also won't care about your software. So you'd need to have a decent contract in place to ensure the system is tested property.

More importantly you need to have a good rapport with the testers - as you and your developers will be dealing with them a lot.

Ideally they need to be involved as the system is developed. Things change, specs change. They can make impartial and very useful contributions here.

From personal experience outsourcing testing can be a lot more hassle than it's worth. The turn around time between testing, finding and fixing bugs can be huge - days instead of hours. At crunch times in your project - i.e. system test! - you may not want to deal with anybody outside the company, in another time zone and culture.

share|improve this answer

There are some good answers already on the sort of problems you'll need to handle - so I'm going to focus in on this part: "Can you simply hire some off-shore 'mechanical turks' [...] with a testing script as a cheaper alternative to professional software testing services?" - specifically, the idea of using a testing script as a replacement for experience. I'd strongly recommend that you don't try to hire in an external firm to "just run" test scripts you have written.

It's a common misconception, usually most popular amongst the Standard School, that you can essentially split the "hard bits" out, by getting a test lead to write really detailed test scripts that cheap untrained test monkeys faithfully follow. This works about as well as getting an architect to write really detailed instructions that an army of cheap untrained code monkeys faithfully follow - i.e. not very, but non-technical managers like to see it, because it looks like lots of work is happening.

My experience of watching unskilled testers following scripts is - painful. People really do walk right past the most glaring errors without even seeing them when they're focused on script execution. However, it isn't just a matter of being untrained - detailed procedural test scripts are well known to be an ineffective way of testing. Skilled testers also walk right past defects when they're heavily focused on a test script - the reason many are still effective with that approach is that they don't follow the scripts, they just use them as a rough guide - I know they do, because in many years of working in heavily scripted environments, this is what I've observed. (However, most will deny it if you challenge them directly, because they believe they'll get in trouble with their managers. With justification in some cases.)

In short: scripting won't solve problems with lack of skill, or lack of knowledge of your product.

share|improve this answer
    
+1 Very interesting presentation in the link you provided. –  Jimmy C Feb 13 '11 at 13:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.