Talking from personal experience of having managed off-shore projects from an on-shore base the most significant challenges are:
As far as "does it EVER work" concerned: it does. It doesn't work well though. Most people can run, but it doesn't mean that most people can run as fast as Usain Bolt.
Normally this is brought forward as the biggest challenge, but it is not. It is the least of all problems, which doesn’t make it less significant though. You need to communicate using the same language and it is almost always the language of the client. It is needless to say that off-shore developers with a good command of, let's say, English are more expensive and on many occasions are hard to attract.
Hence off-shore shops tend to have a few more expensive people who can communicate well and a bunch of cheaper doers (usually computer science students) struggling to comprehend even technical documentation.
Compared to a native speaker, documentation takes longer to write and their quality is lower. UI may, at times, carry awkward labels and messages and might need to be revisited on-shore, at extra cost.
Then there are some harder-to-communicate and more abstract business concepts that are hard to grasp outside the business, economic and cultural context.
It’s possible to overcome communication problems by use of collaboration software, choosing sensible communication hours and frequency, hiring an on-shore project manager who is able to speak the off-shore language, keeping a technical writer on shore, etc. But it all comes at a direct financial cost.
Security and Compliance
Apart from letting your source code sit at the off-shore hard drive, where you might have little direct control over its fate, you will inevitably hit testing and business support related issues. Both testing and support are likely to require access to live data or live systems containing sensitive information. Any personally identifiable information on individuals (at least in the European Union, but there be should similar legislation in United States), is to be considered sensitive and needs to be protected. Passing it on into countries that might have less strict data protection laws constitutes a regulatory breach.
Ways round the problem include keeping business support on-shore or bringing off-shore people over for short periods of time, de-personalising test data, defining and enforcing security policy, etc. It is needless to say that some companies struggle to manage their internal compliance and security risks, let alone off-shore: you'd need to educate and monitor developers who, because of the constant cost pressures, are very much inclined to cut corners.
A breach can attract a serious fine, negatively affect the customer base, cause loss of intellectual property — you name it!
When the off-shore team is an external organisational entity, it is very hard to control who they hire, who they put on your project, how they train developers (if at all), who they keep and whom they move to more lucrative projects, and, above all, what happens to your source code. Sometime it's even hard to control that developers put enough hours in, I don't mean any overtime, just the time you've paid for (technology factories are usually susceptible of this behaviour).
As the project matures, a lot of control and negotiation power tends to drift to the off-shore location. An alternative is to change the horses, but it might cost you dearly. You can’t just oust individuals; you have to oust the entire team.
Availability of Information
Some of the core domain information and implied requirements are not going to be available to the off-shore team, at the same time fearing to lose the contract or be seen as incompetent they might dread to ask questions and will tend to focus on acquiring solutions on their own. Physical separation from the users might also be fairly difficult to overcome. Often the off-shore team just doesn’t feel the pain when something goes wrong and can’t imagine let alone see their application being used by any breathing creature.
On the other hand, don't expect them to be quick reporting about any of their problems, since they might expect you to see it as a weakness, as a threat to securing a contract, receiving the next payment or as potentially projecting a bad image of their abilities.
Real live examples?
Estimation for the next piece of work increases threefold in comparison to the most recent module finished taking everyone on-shore by surprise and still the team does not seem to be very keen. After many questions we’re told reluctantly that the former piece took twice as long as it was planned and paid for, besides the team had to work evening and weekends. The were reluctant to tell since they perceived this as entirely their own issue.
At some point during a private conversation, it turns out that some of the worst newly discovered software issues are known and been identified during off-shore testing but the team didn’t want to highlight them since it could affect the deadline. They were hoping to fix these later, quietly.
The team didn’t have much real life exposure to technology X, but it was too late when they realised that they would need to use the technology extensively, so they didn’t wish to rock the boat and bring the issue forward, since it would affect timelines, costs, etc. but rather decided to push the solution towards using less relevant and more limited technology Y. Having had they mentioned the actual problem and the extra time and budget would have very likely been granted in the name of a better end result.
When the off-shoring team is an external entity, information just doesn’t flow as freely as it would be necessary to achieve the best outcome. It makes it difficult for both parties to validate and verify decisions made by the opposite side.
Whereas off-shoring customers are interested in lower cost, off-shoring providers want to make more or easier money than they would have made on the domestic market. They care about receiving the next contract as long as it is easy enough and gives sufficient profit margin.
Some off-shore developers are not particularly worried about quality of code as long as they get their bills paid and gain valuable work experience in a sweatshop that will enable them to finish university and move onto more exciting domestic projects where they can actually meet and talk to real users.
Some say “we will keep QA on-shore and employ local BA’s and a project manager to keep the off-shore development team on the short leash.”
Although any software where requirements evolve or subjective (so called e-type) when taken off-shore will definitely benefit from an on-shore project manager or coordinator plus an on-shore BA and QA, the extra people won’t provide a “silver bullet” solution to many problems.
These people on-shore will be very capable of explaining post-factum what went wrong and why it went wrong (same as in the case of the financial crisis or incorrect weather forecast for that matter). But most of them won’t see it coming. Because requirements evolve and cannot be plainly modelled on a piece of paper much smaller than the printout of the software source itself.
Because no sane sweatshop is going to hand the code over until they received the payment (source is their only serious leverage!) and without the source QA is fairly superficial. Because the off-shore is not going to proactively share information that will help invalidate their solution (the interests are misaligned, remember, when you want quality they want payment). Because being able to point out an error or misspelled label is quite different from being able to write robust and elegant software with perfect English UI and no amount of errors will teach the off-shore team overnight how to produce quality. And the developers who can do robust software with a perfect English UI don't need to work in off-shore sweatshop, they can chase much more attractive jobs worldwide.
Large companies that open their subsidiaries off-shore can tackle most of the issues and still get the benefit of cost saving. Often however for these companies it is worth to move the development of software closer to users, that is, do localisation in the office actually based in the target country or move developing of manufacturing software closer to the actual factories, etc.
Smaller companies looking at hiring 2-15 developers or these companies that have a strategically important software to produce, would actually be much better off hiring as locally as possible.