Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In a blog post by Andrew Hay, the following axiom was posited:

It costs significantly more to fix a bug at the end of the project that it does to fix the same bug earlier in the project.

However, this doesn't seem certain, especially after reading a blog post on Less Wrong, and the data I've seen to back it up is extremely old.

Is this still axiom accurate today?

share|improve this question
    
@StefanHendriks The comments in your linked to article from Morendil really cover everything you could ask; IMHO. Great information there. –  Aaron McIver Feb 7 '12 at 20:26
    
@AaronMcIver my intention is to have more people get to know about this. Like spreading the word and increasing chances of getting real data out. I do not search for a real discussion; great ones are being held at the article (as you also have found out :)). –  Stefan Hendriks Feb 7 '12 at 21:24
2  
Hi Stefan, this is not a discussion board, nor is it a soapbox: I've removed your commentary from the question. We can help explain aspects of software development, but if you're looking to use this site as a means to promote your ideas or share blog posts you liked, you're in the wrong place. –  user8 Feb 7 '12 at 21:36
    
Although this certainly has to do with programming, the nature of the question may actually make it more appropriate on critics.stackexchange.com. –  StriplingWarrior Feb 7 '12 at 22:49
add comment

11 Answers 11

No statistical data, but personal experience:

The rocket motor control code I was working on had a line like powerCutoff = someCondition && debugOptions.cutoffIsAllowed;. The default option was no cutoff allowed. The 'final' build was supposed to remove all the debug options, so the line got modified to powerCutoff = someCondition;.

Did you catch the bug during code review? We didn't. The first time the trigger condition occurred in test causing an unexpected cutoff was just a few months before the first flight.

This bug would have cost less than an hour had it been caught during review. It might have cost a day or two if it was caught during integration, causing a single test repeat. If it was caught during formal qualification, it could have cost a week or two by causing a complete test series restart with a new build.

As it was, the cost balooned. First we designed and ran tests to determine if the flight unit could even trigger the condition. After it was determined to be a real possibility, there was cost for engineering, management and customer analysis of the best fix, releasing the new build, creating and executing a new regression test plan, system testing in multiple units and simulators. All in all it cost thousands if not tens of thousands of man-hours. Plus the original 15 minutes to actually make the right code change.

share|improve this answer
add comment

The only hard data I've ever seen is Boehm and Papaccio, Understanding and Controlling Software Costs.

This stems back to 1988 and was a study of around 80 software projects. They concluded that a decision made early and corrected late could cost 50-200 times what it would have if it had been corrected early. But the kind of very early decisions they're talking about are which OS to run on and which language and database to use.

So those figures may be overwrought with respect to today's software development. However, now we have a lot of experience in the field and we know instinctively that it still holds true to a point.

In the extreme, we know that if a failure in requirements is caught just before we go to production, this causes a lot of rework and get the project delayed or indeed cancelled, where if it had been caught before any work had been done, we'd have been fine.

Edit: Doc Brown makes a good point in his comment.

Boehm's research was done on COBOL and FORTRAN projects in a time when compile and run times were ridiculously slow. I started my career in the early 90s on COBOL and the compile and test cycle used to take so long that it was worth the effort to dry-test the code before going through the cycle (or at least during the build phase, just in case you could catch something and cancel it early, saving yourself an hour or so).

In turn, our bosses used to laugh at our complaints because it wasn't so long ago they used to have to carry a box of hand-sorted punch cards to the server room and leave it there for a day.

So it was definitely more true then than it is now.

And yet, very recently, I've seen blogs reusing Steve McConnell's visualisation of this problem (ref, dated 1996) as if that graph was actually based on hard numbers. It isn't. It's a visualisation, to explain his point simply.

I think that Morendil's premise, in the article the OP quoted, is a good one. The science we have on this subject is poor and outdated and yet treated as canon. But I also think that it holds up well and sounds true because we know from bitter experience that it is still true, at least to a point. And I think his dramatic "diseased discipline" phrasing does him no favours.

share|improve this answer
1  
+1. I think one should add that Boehm's research has been done at a time where the costs of building and deploying bugfix releases was much higher than it is today. –  Doc Brown Feb 8 '12 at 10:39
    
@DocBrown: Good point. Added. Along with a further ramble. –  pdr Feb 8 '12 at 11:23
    
+1 for reference indeed, +1 for the visualisation (too bad I can only give one point.) Great answer, thanks! –  Stefan Hendriks Feb 9 '12 at 10:22
add comment

I once read an article that had two interesting points (unfortunately the references I had is long gone, so I'll have to just postulate here). The first point they made was that around 50% of all errors were introduced in the requirements specification and that around 90% of all errors where found during UAT or System tests.

The second point they had was that for each phase in the V-model the cost was increased 10-fold. Whether or not the factor is correct I find kind of irrelevant but the most costly errors are when your design is based on an incorrect assumption. This leads to a massive amount of rewritting. All the code that works because of that assumption but fails when the correct assumption is applied will have to be rewritten.

I've experienced the entire domain model having to be rewritten because of one incorrect assumption in the requirements specifications. If such a bug is caught early, Ie when reviewing the requirements specifications the cost is very low. In this particular case it would have taken ten lines of text. In the case where it's found during UAT (as this was) the cost is sumstantial (in the given example the project cost was increased by 50%)

share|improve this answer
add comment

I don't have access to hard data or facts, so I can only offer you anecdotal observations gleaned from my last 20 years in IT.

I believe there is a vast difference between the that way most developers create software today as compared to 20 years ago. With the Agile movement having gained so much momentum, particularly in the last 5-6 years, I've seen a real shift in attitudes in the workplace. So much so that the quality of what we do just seems to grow in leaps and bounds every year, and with every project as we apply the lessons we've learned from project to project. Leaner processes combined with a focus on test-first development has grown from being very controversial to commonplace. So much so that to walk into many companies today, if you are not comfortable with Agile you'll be lucky if they don't show you the door.

So what impact has this had. First of all, I have noticed that problems do often get identified much earlier. Often it is the case that if the problem doesn't look to be too great, it can sometimes be put on hold indefinitely. In a rare handful of cases, I have seen bugs that were thought to be trivial become serious problems when addressed later, as some fundamental issue becomes apparent that wasn't considered at the time. Sometimes this can lead to a drawn out fix cycle, and that can be costly to a degree, but that cost is often measured less in terms of resourcing, and more often in terms of the impact on the relationship between customer and developer. Customers are growing used to this Agile way of thinking, which returns results to them much faster than it did in the old days, with highly iterative development sprints and fast turnaround between requests and implementation, so they have come to expect a great deal of us. And as far as the actual bugs are concerned, the time to get a bug fixed is more often greatly diminished as a result of having a solid suite of tests to support changes, and the ability to create new tests from which to provide insight and solutions to the problems reported.

So on the whole, it appears that the overall effort to fix bugs has been in most cases reduced if there is a robust suite of tests in place, and procedures to ensure that testing remains the focus of what the developer does, but the actual cost has in some ways shifted in part at least from the implementation, to other areas of the business, because in some ways, the focus has also shifted from pure supply and demand to relationship management.

Another thing that has become apparent, is that our gut instincts of a few years ago which suggested that being Agile would reduce our maintenance cycles has been proven to a degree both right and wrong. Right in the sense that solid testing has made it easier to debug and fix our code to a large degree, and to reduce overall the number of bugs released into production code, and wrong in the sense that we are now working harder to avoid needing to maintain legacy code, by constantly refactoring code and improving architecture such that it is becoming rarer that we need to develop new products completely from scratch.

So in the end, what does this mean with regards to the OP's question? Well, it means that the answer really isn't as cut-and-dry as we once might have thought it to be. 15 years ago, I would have probably answered the question as a Yes, but now I feel it's more realistic to say that it really is too difficult to measure empirically, because the nature of what we do to develop software has changed greatly from when we first started asking ourselves the OP's question back then. In some ways, the more we advance our techniques and skills as an industry, the further the question grows from a definitive yes, to a point where I suspect that in a short number of years we'll be saying that it doesn't matter when we fix bugs, because our tests and processes will be so much more robust, that the timing of the bug fixes will be less driven by efforts to save our budgets, and more by priorities to satisfy our customers needs, and the relative cost will become virtually meaningless contextually.

But as I say, this isn't hard data-supported evidence, just my observations of the past few years, and my gut telling me that there will be more ground-shaking wisdom to come that will improve the way that we do things.

share|improve this answer
add comment

This is basic accepted stuff from the systems engineering - and it applies to any form of technical development (be that building bridges, missiles, battleships or software).

Essentially, the cost of things goes up approximately an order of magnitude as you move through stages of development.

Something that costs $10 to fix at the point of conceiving the idea...

Will cost about $100 if you need to go an update the specification....

Or cost about $1000 if something was implemented and you need to make changes at that point (and update the spec, and get the approvals and so on) but it had not gone through some kind of formal acceptance / sell off test

Or cost about $10000 if something was implemented and customer accepted, and you need to make changes at that point (and update the spec, and get the approvals, and re-test and re-run customer acceptance and qualification, and so on)

And the cost after deployment / roll out / putting in service is even more again.

Examples abound and its easy to understand them: a banking system with a serious scope change made after you have 25,000 employees using it will cost a packet in re-training time... before you even consider the scoping, coding, test, regression, etc etc etc.

OBVIOUSLY your mileage will vary: the costs and impacts of changing Fred Nurke's electronic sock warming e-commerce web site are somewhat different to the costs of changing software on an aircraft flight control computer.

share|improve this answer
    
Say, you deliver every month a new release of a software to your users (or just patches, like MS does it for Windows). Now two bugs show up, one which has been introduced into the software two years ago, one introduced last month. The cost of fixing those two bugs and deploying the new release can be virtually the same. The cost of fixing the problems caused by any of those bugs may be a different thing, but that depends a lot on the bug itself. –  Doc Brown Feb 8 '12 at 12:41
    
Not quite the same thing - thats after shipping. Costs after shipping are similar in magnitude (they all need updates, testing, deployment.) What I point out above is that costs escalate dramatically post-release. –  quickly_now Feb 8 '12 at 23:12
    
"post-release" is a state valid for embedded software, to a degree for shrink-wrap software, and also for software developed in a (misguided!) waterfall model. Other types of software are developed and released incrementally, the "post-release" time is virtually small compared to the lifetime of the product. This is specifically the case for web applications. –  Doc Brown Feb 9 '12 at 8:59
    
It might be the case for web applications but thats not the whole universe. What about washing machines? Cars? Missiles? PC operating systems? Power stations? PLCs running cement plants? On and on and on the list goes. –  quickly_now Feb 9 '12 at 10:25
add comment

Ask Intel how much the Pentium Bug cost them, The Ariane 5 Rocket is another good example. These bugs were fixed at the end oft he project. I have worked on a system where an "attempt" at a software release has a budget of 6 figures. In these extreme cases, it's easy to see the cost. In other (most?) cases, the cost gets hidden by the noise, but it is still there.

There is no doubt that bugs cost money while they exist. One item, Defect reports, take time to compile, triage and close as dup, time is money- therefore an open bug creates ongoing cost. therefore, it must be that deferring bug fixes costs more than fixing them sooner.

If a bug escapes into the wild, the cost has a step wise jump...... Is "The end of the project" before or after the release of the software?

share|improve this answer
    
Software bugs in the embedded world is definitely more expensive to fix at the end of the project. Imagine having to do a recall on car because of some software bug in the engine control module. –  tehnyit Feb 8 '12 at 8:23
    
The bugs you mention were not found early and therefore not fixed early. –  user1249 Feb 8 '12 at 12:19
    
@Thorbjørn You are indeed correct - although not found early the defects we inserted early (In the case of The Ariane Rocket, the bug was inserted before the project was even started, as they reused existing code.). The cost is proportional to the time between insertion and deployed correction, nothing to do with when it gets found or fixed (Most developers consider it fixed once the patch is in the code base. A defect is not fixed until the end users have it installed). All this is merely IMHO though - I have no evidence to support it. –  mattnz Feb 8 '12 at 22:28
add comment

If you fix the bug after release, then you have the cost of finding and fixing the bug - which may or may not take more time/cost to do post release. However you have a round of Integration testing, regression testing, UA testing, release activities, etc that will need to be accounted for. Unless the bug fix goes in with a number of other fixes or a version update you will have additional expense for the testing and release activities that would be avoided by including the fix in the initial release - because these costs would be shared over a number of fixes/updates/features.

Also consider the cost that the bug will cause in use, if is just cosmetic, then it probably does not matter, but a function or performance bug could create a cost with support activities or reduced productivity or incorrect calculations.

share|improve this answer
add comment

I doubt that it's at all possible to come up with a scientifically rigid way of measuring this - there's just too many other factors involved, and no two projects are comparable enough to serve as more than case studies. Logical thinking should get you a long way though. A few arguments:

  • The total amount of code in a project tends to grow toward the end. The longer you wait fixing a bug, the larger the codebase that you have to touch.
  • The quality of new code added to a project decreases toward the end, at least if there is pressure (which is usually a given): a looming deadline makes people throw best practices overboard just to ship on time. This means that the later you fix a bug, the more bad code you have to sift through.
  • Even though code duplication is generally frowned upon, it happens all the time, and since it is easy to copy-paste, but hard to re-merge duplicate code sections, the amount of once copy-pasted code typically increases over the lifetime of a project. More copy-pasted code means a higher chance that your bug is going to be duplicated and needs to be found and fixed several times (and consequently, a higher chance that some of its occurrences go unnoticed).
  • Fixing a bug is a change to a codebase; you hope to make things better, but a change always carries a risk. A change that causes a serious problems in a project with months to go should leave plenty of headroom for damage management, but two days before shipping, you're in for serious trouble.
  • The longer the bug itself exists, the more likely it becomes that other parts of the application start relying on its misbehavior. Then when you fix it, you suddenly unleash a bunch of secondary bugs in code that doesn't expect your function to actually provide the correct documented results.
share|improve this answer
    
+1. Great answer. Copy and pasting code which has bugs generate more bugs depending on modules dependent on it. –  Karthik Sreenivasan Feb 8 '12 at 4:21
add comment

Well, I probably can't give you the definitive proof you're asking for, but I can relate a fairly recent incident from my work.

We added a feature that provided workflow management capabilities to our product. Typical BDUF stuff, specs signed off and approved by the client. Implemented to spec. Complaints from Day 1 on deployment.

We hadn't done a real usability walkthrough with the client, just taken their word for what they wanted. Result: hundreds of hours of rework - analysis, design, implementation, and QA had to be redone. All because the spec missed particular use cases. A bug in the specification, if you will.

I've seen similar things in prior jobs when someone in the chain makes assumptions that are different from the end-user assumptions. Straight-up coding bugs are relatively easy to deal with if caught close to when they happen, but design bugs can kill entire systems.

share|improve this answer
add comment

Early bugs will propagate to other parts of the system so when you fix the bug you may be forced to rewrite some parts of the system that relied on the bug itself.

Beside over time you'll foget how some parts of the program are built and you'll need to remind yourself. It's some form of technical debt (if you rush the project in early stage you'll have problems with finishing it because of shortcuts you have taken).

It's as simple as that and there's nothing to proof.

I think you're trying to rush the project as fast as possible to present some working solution to your employee. Good news is that you'll have it very fast, bad news is that you probably will never finish it without complete rewrite if you just keep writing crap as fast as possible and plan to fix everything in a couple of months. You probably won't even be able to refactor this.

share|improve this answer
    
Yes that all makes sense. Although, I do wonder if this is significantly different from fixing later. Yes, you need to re-learn stuff a bit. However, perhaps by not releasing earlier you have lost more money than the cost it would be of fixing this issue. Would that make this issue cheap or expensive to fix? Even when it is less work because it was at the beginning? –  Stefan Hendriks Feb 7 '12 at 20:19
2  
Fixing already released system is much more complicated. You can't just rewrite data structures for example. You'll need to provide users of some way to migrate their data. Again if you release too early you'll end up in a mess, instead of fixing bugs you'll be wasting time writing migration code. Maybe you loose some money, it's better than loosing clients because of selling them lousy software. –  Slawek Feb 7 '12 at 20:22
    
>> ...you need to re-learn stuff a bit... Edge cases in particular make this tricky and non-trivial. Interactions outside the immediate get forgotten quickly unless you have an exhaustive, correct, and maintained specification. –  DaveE Feb 7 '12 at 21:56
add comment

While I'm not aware of any hard data or other evidence to support this claim, at a bare minimum, I'd figure it's common sense.

Think about it this way.. If you have a complex system with inter-dependent subsystems (as most non-trivial applications do), think about knock-on issues that may be a result of changes made to any one of the systems. If subsystems are proven to be correct (via unit tests and the like) and fixed early, the number of bugs that will be caused due to knock-ons alone are mitigated simply by fixing early.

Also, if you're fixing bugs early, the implementation is still fresh in the developer's mind. Depending on the length of any given project, if you're fixing bugs at the end, the developer will need to spend time figuring out what they wrote and (perhaps) how the subsystems that their code depends on works. Time spent re-learning this = $.

share|improve this answer
1  
So basically we're talking about the amount of effort it takes to fix a bug later (or earlier). I can think of other factors that make bugs more expensive if fixed later. But that depends on your definition of bug. Perhaps thats something to be agreed upon first. In my book it is also a 'not matching expectation in this release'. Like, a missing functionality. This could cost real money, so its more obvious then. Some features though might not cost more (like for websites, CSS changes?) now than early. Or, not that significantly more. Still, I don't have data. –  Stefan Hendriks Feb 7 '12 at 20:16
    
@StefanHendriks: We're talking about both the amount of effort it takes as well as new bugs incurred by claimed fixes. You'd probably have to dig into project post mortems (ones that have employed both methods) to get actual data. –  Demian Brecht Feb 7 '12 at 20:18
2  
@AaronMcIver: My takeaway from the article isn't which method is better, but that the research and hard data used in backing up the claim and misinterpretations in subsequent reports. While my answer isn't based on public data, it's based on 10+ years of professional experience dealing with highly complex systems. –  Demian Brecht Feb 7 '12 at 20:38
1  
BTW I disagree that CSS changes do not suffer from this. Try fixing a layout issue once you have all the other stuff pixel perfect and you will find that you may have to break a lot of things –  Andrea Feb 7 '12 at 20:39
1  
@DemianBrecht Your answer is very subjective, which is why I ask. It's speculation and a gut feeling. While those can certainly hold weight, the problem is that it may more often than not be an inaccurate representation of reality as the article points out. Using common sense as your criteria on why it can cost more can also be reversed. You could argue that the number of individuals involved in a bug fix may be the real factor in it costing more or not, disregarding the developers actual effort. –  Aaron McIver Feb 7 '12 at 20:50
show 5 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.