Why do large IT projects tend to fail or have big cost/schedule overruns?

Question

I always read about large scale transformation or integration project that are total or almost total disaster. Even if they somehow manage to succeed the cost and schedule blow out is enormous. What is the real reason behind large projects being more prone to failure. Can agile be used in these sort of projects or traditional approach is still the best.

One example from Australia is the Queensland Payroll project where they changed test success criteria to deliver the project.
See some more failed projects in this SO question (on Wayback Machine)

Have you got any personal experience to share?

Answer 1

The main reason is an increase in scope, which the book "The Pragmatic Programmer" describes as:

• feature bloats
• creeping featurism
• requirement creep

It is an aspect of the boiled-frog syndrome.

The idea of the various "agile" method is to accelerate feedback and - hopefully - correct the evolution of the project in time.

But the other reason is release management: if you aren't geared toward releasing the project (however imperfect it may be), chances are it will fail (because released too late, with too many buggy features, and harder to fix/update/upgrade).

That does not mean you have to have a fixed release date, but that means you must be able at all time to build a running version of your program, in order to test/evaluate/release it.

---

The blog post "Late projects are late one day at a time" contains many more examples:

I know the ‘Getting Real’ thing to do would be to Flex the scope and keep the launch date fixed, but that doesn’t work if there is agreed upon functionality that cannot be completed in time.

That’s why we don’t advocate specs or “agreed upon functionality.” That’s the root of the problem — saying you know everything about what you need and how its going to be implemented even before the first pixel is painted or line of code is written.

When you predict a rigid future on a flexible present you’re in trouble. Rigid futures are among the most dangerous things. They don’t leave room for discovery, emergence, and mistakes that open new doors.

Answer 2

Usually the complexity of project is underestimated.

Answer 3

I blame the bidding process. It rewards the group that can make the deal look cheapest/quickest on paper.

The people putting together bids don't want to waste their time if they have no chance of winning, so their normal estimations get put on hold. I know people who have specified normal switches instead of POE switches to save $80. But the project needed POE because it had IP cameras. That $80 needs to be spent, but now it is outside of the spec.

I have a firm belief that a 2 month $2,000,000 project will still take 2 months $2,000,000 no matter how many bids you get. If you think doing it right is expensive, wait and see how expensive it is to do it wrong.

Answer 4

Larger Project = More Complexity
More Complexity = More Uncertainties
More Uncertainties = Harder to Estimate
Harder to Estimate = Bad Estimates
Bad Estimates = Cost Overruns

Answer 5

Steve McConnell (of "Code Complete" fame) has a list of the classic mistakes.

Answer 6

One possible reason is that estimates are based on smaller projects, assuming a linear growth in cost with the project size, when in fact cost growth is e.g. quadratic due to increasing complexity, longer project duration (more time for requirement changes) etc. Estimating is hard, and the bigger the project, the harder it gets to estimate correctly.

Another reason are optimizm biased estimates: To win the bidding, best-case estimates are used to calculate the price. The larger the project, the less likely is a best-case scenario. Bidding rules make it likely that the most optimistic offerer gets the acceptance, so even if 5 vendors make a realistic estimate and the 6th is too optimistic, the optimistic one wins the bidding and fails later. So this is kind of a negative selection.

Answer 7

Cost does not preclude schedule in the eyes of 'management' which is an important distinction to make. As we know, "nine women can't make a baby in one month", yet you'd be surprised at how many people think that problems decrease in depth in relation to the amount of money that is thrown at them. Bad project management, often manifesting itself in the form of micro management is the leading cause of most projects tanking (in my experience). Micro management kicks in when 'management' realizes that something is getting out of control and they are clueless as to why.

When that isn't the cause, the expected outcome of the project was probably not tenable to begin with. In my experience, if the time frame of a project is too short, people will be so afraid of making mistakes that result in 'double work' that they don't get much of anything done at all.

This is why management should be populated with seasoned programmers who have a history of leading teams that produced successful projects. Such a person might say "No way could we do that responsibly" despite the possible revenue, and would not be in management for long, which is why many of us (ultimately) answer to MBA's instead of PHD's.

I lost count of the number of companies that I've worked for where a non-programmer was in charge of hiring programmers. I had an interview once where the hiring manager wanted to do nothing but discuss a recent sporting event (I think it was a football game). If the person you have in charge draws more inspiration from an NFL coach than Knuth, the project is going to tank.

Once in a while, you run into something that was well planned, well understood, realistic and seemingly straight forward. For whatever reason, six months into development, everything reversed itself. It happens. Rarely, however is that the underlying cause of a project becoming a glorified pork barrel.

Still, I have to admit .. if you watch the news, you might see an occasional motorcycle accident or train wreck. You never hear about the millions of motorcycles or trains that arrive on time every day without incident. The same goes with projects. Sure, it's interesting to see a public autopsy of something that went really, really bad, but you almost never hear about stuff that went really, really well. I think that tanked projects are still the exception, not the norm.

Answer 8

Most of the time a combination of two or more of the following:

• collaboration problem between departments
• politics... too much politics...
• wrong team
• unrealistic scheduling
• changing scope without the appropriate methodology
• missing informations

Nice book on the subject: Death March.

Answer 9

The larger the project, the more likely you are working for a large organization. The larger the organization, the more layers of management. The more layers of management, the harder it is for bad news ("we can't have what we want for what we can afford") to make it up the chain of command. The less likely bad news can make it up the chain of command, the more likely a fantasy plan will be accepted and then held to long after it is known to be untenable.

Answer 10

Perception of Reality

Here is the best description of this problem I have ever found. Tree Swing From businessballs.com

I was introduced to the concept of "Perception of Reality" early on im my programming carreer. For this I am truely grateful. I believe that this is the biggest reason that any project fails, not just IT projects.

Answer 11

People tend to think that software development is a predictive process, trying to measure and estimate things one year ahead. This is not possible! Building software is not bolt manufacture.

Following the same "trend", they try to do a huge analisys (again one year ahead) thinking that they'll cover all the possibilities, and later, turning programmer into mere typists. How come one think that this could work? This kind of behavior just leads to bad estimates and lots of bureaucracy.

Answer 12

Elaborating a bit on VonC's answer:

This big projects tend to have an "all or nothing" mentality. The project as defined has to be released in one go - often as it's a change over from an existing system.

This means that the problems of feature/requirement creep are harder to address so when the project comes to fruition it's often seen as no longer meeting requirements. This can be exacerbated if the existing system has been updated or technology has moved on in the meantime.

What's the solution to this?

I don't really know as no one wants to have two systems running in parallel with a changing set of functions split between the two.

Answer 13

Software projects of all sizes "tend to fail" or "have cost overruns." You don't hear about the cost overrun at the business around the corner, but you do hear about things like the FBI Virtual Case system, or the Denver Airport baggage handling system. So I will make the claim that not all large systems fail, nor do all large systems have cost/schedule overruns.

I've come across large systems that came in on time (the schedule moved once and only once during the project) and on spec (I had no access to budgetary information as we were just 1 of many suppliers). One that impresses me still (and I've written a bit about it on this site) was a large integrated customer management system for a large (in the first 100 of the fortune 500) financial client. I estimate that they blew about $100k/day (for more than a year) on peoples' salaries during conference calls.

In the case of the baggage handling system, the software managers said "based on projects of this size and complexity, it will take 4 years to build and debug this system." The sales and executive managers said "the airport opens in 2 years, we told the client it will take 2 years, so you have 2 years to do it." The test to see if you are a programmer or a mismanager is a simple answer to the following question: "was the baggage handling system late or on time?"

If the customer knows exactly what they want (and very few do), they will be very far along the path to keeping costs and time under control (and these are the folks who tend to do quite well offshoring). If your project has to meet every single possible feature that your customers can possibly dream up, and every single department has veto power over when their pet goldbricks get added to the project, then you are doomed to abject failure from the start (like the FBI's VCF project).

Answer 14

A factor that has been touched on but not addressed yet:

Just about all the dramatic failures are contracts that were bid out. What happens to a competent company in such a situation? They make a realistic estimate and thus are almost certainly underbid by someone who made a bad estimate.

If the company can't estimate properly is it surprising they can't build a system properly also?

Answer 15

They forgot Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.

Answer 16

One reason for failures is that a big project is usually a high-profile, important-to-the business project. When projects and tasks are high-profile, it encourages people to lie.

Your boss wants you to estimate your completion status on the high side. He wants to estimate overruns and delays on the low side. When you encounter a problem, he doesn't want to hear that it will add three weeks to the task; he wants to hear you can work it into a couple of hours tonight.

And so on and so forth.

I was on one project several years ago, for a client. I was brought in after the bid and project plan were completed. There was constant pressure to go faster, faster, and ridiculous cost cutting decisions, heavy overloading of staff, no resources for them; no desks, computers, anything.

Finally, I discovered the project was bid at 7 months and 16 million dollars. I estimated on the back of an envelope it should be 24 months and 50 to 100 million. I set up a meeting with my manager and his manager, and presented my case, and how we were NOT coming anywhere near delivering on time or budget; they downplayed all the problems. At the end of the meeting the CIO called and told both these managers essentially what I said, with the exception of the flaw in the original bid.

I had a chance to roll off the project when they changed technologies to one I wasn't skilled at. I spoke with someone much later. The project ended up being cancelled when it was about half done...at 12 months and 35 million dollars.

High profile big projects discourage people saying, "this is a mistake". Mistakes aren't tolerated.

Answer 17

Michael Krigsman over on ZDNet has a blog devoted to "IT Project Failures," that may be of interest here.

Another point is that with long projects that span years, there will generally either be upgrades to consider or alternative solutions,e.g. could a project now be done in the cloud versus on-site for something starting to come up more and more, that in considering these this wasn't a given when the project was started. For example, while one could start while something is at 6.0, by the time the first phase is done there may well be a 6.3 or 6.4 out and the question of when to upgrade is being asked. Changes in scope and desired functionality, either because requirements weren't gathered correctly or someone changed their mind, are another couple of points that have already been covered quite a bit.

Answer 18

Can agile be used in these sort of projects or traditional approach is still the best.

The reason why well applied agile processes don't seem to suffer from this problem as much is for a simple reason. You cannot start a large project in an agile manner. You can chose one or the other.

With an agile process, you are never really looking past one or two iterations into the future of the project. there is no 'plan' for the next 2 years, only the next two weeks. When your view is that short, you have to make a few compromises. You cannot ever start with a plan to make "The final word in widgets", for whatever sort of widget you are designing. At most, you can start with "A widget that can frob", because thats about the most work you can get done in one or two iterations.

The good thing about this, is after a few iterations, you already have a completed, working product that someone can find useful, especially that one customer who desperately needs a widget that can frob and zort.

Essentially, large projects can fail because they aim to solve all of the problems of all of the potential customers. An agile project never has this goal, instead addressing in each version just one critical issue a single customer might have. After a good long while, though, an agile project might be solving alot of critical issues for alot of customers.

Answer 19

The complexity of large project can be wildly exacerbated by external political pressures. One department may have a very clear, focused idea of what they want in the new system, but then associated departments jump in with dozens of requests along the lines of "Well, as long as you're doing that , why don't you do this little side task for us too?" You might start by saying "No, that's out of scope.", but then the political in-fighting among the deaprtments begins, and the budget for the project is threatened unless everybody gets their piece of the pie.

For years, our local police couldn't search for partial plates through the motor vehicle system, a feature that seems absurdly simple. I asked a friend what on earth was so hard about adding this feature, and they said that every time they proposed switching to a modern database, every other department in the state that had any interaction with the motor vehicle system wanted to get their portion of the system fixed too. The result was complete gird-lock in IT modernization. Finally the state put together enough capital to do a system wide modernization effort, which then floundered because it was so hideously complex.

Answer 20

• Failure to ship to actual users

Big projects have a nasty tendency to get into "infrastructure" mode for years, and forget about building real end-user features and shipping them. By the time it ships, it's very expensive to change, and usually the biggest conceptual changes end up being asked after the first real beta testing occurs.

• Failure to accurately estimate cost

If projects seem like they will outgrow their return on investment, they get canceled.

• Failure to control quality

With enough defects it is possible for forward momentum to fall to zero, or below. Without any progress at all, it's hard to argue for the continued existence.

Answer 21

Things I've Seen In Web Development:

• Too many cooks - Major B&M Retailer where the emphasis suddenly shifted to the web. Suddenly 20 Department heads are dogpiling every initiative to impress the new head cheese. I once had to implement new icons because legal didn't like the look of the old ones.

• Excess focus on matching specs over achieving goals - "IE6's icons are slightly faded compared to IE7's. Please drop that launch-date-critical work you're doing and attend to .05% of our customer base to do awful things that will take days to implement and slow down the IE experience even more."

• Bad Tools picked by non-devs who couldn't even be bothered to ask their in-house devs for advice.

• Bad devs picked by tools - "Why pay 20 competent Java devs a decent salary when we can outsource 200 barely code literate guys who know too little to use version control?" as they simultaneously and with people in different countries, work on the back-ends primarily for 3 large apps.

• Bad/Broken Architecture - Layers upon layers of panicked, get-it-done-yesterday code, by people who were fired or are now managers.