Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In a meeting the other day a claim was made that agile was only 60% as efficient in development time when compared to waterfall. I am not looking to validate or refute this claim. I am interesting in finding out if there have been any studies comparing the 2 methodologies.

Are there any studies out there comparing the two?

share|improve this question
4  
Agile doesn't mean delivering better software. Quality software can be shipped regardless of the methodology. Agile is typically about delivering high quality value adding software in less time, while responding to a customer's changing needs. –  Thomas Owens Dec 15 '11 at 16:17
6  
Ask for source of the claim. –  Loki Astari Dec 15 '11 at 16:20
    
Well if the original person has a source then it may include links to other studies. –  Loki Astari Dec 15 '11 at 16:27
    
@LokiAstari - It is not really my place to ask for that from the claimant. It was a business meeting and involved outside vendors. I am not trying to start an argument or discussion about validity of the claim. I am just hoping to find out if there is any research into the statistics available –  Chad Dec 15 '11 at 16:30
4  
@Chad Why was it not your place? Who was saying this? If it was an outside vendor, what a good opportunity to understand their project management ability before you work with them. –  Thomas Owens Dec 15 '11 at 16:33
show 7 more comments

5 Answers 5

up vote 11 down vote accepted

The book "Making Software: What Really Works and Why we Believe It" has some chapters on Agile methods including XP, Scrum, Dynamic Software Development, and Lean, with good scientific backing. It's high quality, as you would expect from O'Reilly. One of the editors was the excellent Greg Wilson, a trusted computer science author, editor and presenter.

The book itself summarizes multiple research studies, including many on agile. One section summarizes research including "Are Two Heads Better Than One? On the Effectiveness of Pair Programming" by Dybå, T.; Arisholm, E.; Sjøberg, D.I.K.; Hannay, J.E.; Shull, F.; and "Empirical Studies of agile software development: A systematic review" by Tore Dybå and Torgeir Dingsøyr.

The general sense is that most agile practices are beneficial, but that the effects of pair programming and TDD and other agile tenants are not as strong as one might hope. There's even a disturbing footnote that TDD may in fact be somewhat addictive*.

The book is a great way to get access to a lot of research that's been done, all in one cohesive whole. There are a few blogs and other sites on the web that review the book.


*This isn't necessarily my opinion!

share|improve this answer
1  
Any chance that you could add some quotes and references? It might help work out whether it's worthy of one of my safari bookshelf slots. *8') –  Mark Booth Dec 15 '11 at 18:11
    
The Nook version too :) Thank you will check it out tonight. –  Chad Dec 15 '11 at 18:22
    
Added. Let me know if this was what you had in mind. If someone wants to edit this post and transcribe the book's text, that would be welcomed as well. –  Kyle Hodgson Dec 16 '11 at 6:01
    
Thanks Kyle, but I think a summary would have been better that what looks like a screen grab. It's a little difficult to get what they are talking about without more context, for instance, what do they mean by effort? Are we talking about developer hours per project? –  Mark Booth Dec 16 '11 at 16:42
1  
The book answers the question as I should have asked it though I think that it would have been too broad. Thank you for the link. –  Chad Jan 24 '12 at 17:55
add comment

As much as I dislike the title, I believe that Balancing Agility and Discipline: A Guide for the Perplexed might contain some information that is relevant to you. This book by two software engineering process and software project management experts - Barry Boehm and Richard Turner. This book looks at various aspects of the agile and plan-driven methodologies, compares and contrasts them, and also discusses integrating them to achieve a "best of both worlds" situation.

Appendix E of Balancing Agility and Discipline contains a wealth of empirical information regarding the costs and benefits of various agile and plan-driven methods. However, there doesn't appear to be any data regarding time effectiveness. But glancing through the data, it appears (as I suspected) that this isn't an either/or choice - some projects experienced decreased effort, faster schedules, and lower defects when applying agile methods. However, other projects that used. The section discusses a number of different projects in different industries, the type of process they used, and what they experienced over the course of the project.

There are plenty of case studies cited in Appendix E that yields this data. There are far too many for me to just start naming randomly, since many are focused in a particular industry or even within a particular organization. If you are going to look at cases, I would suggest finding those that are similar in nature to your team, project, organization, and industry to draw reasonably valid conclusions.

In Rapid Development: Taming Wild Software Schedules, Steve McConnell identifies a number of factors to consider when choosing a lifecycle methodology: level of understanding of the requirements, level of understanding of the architecture, desired reliability, risk management, schedule constraints, amount of process overhead, mid-project "course corrections", ability to provide the customer with visibility, ability to provide management with visibility, and sophistication of the development team and management. There are others, as well, such as organizational culture, so there probably isn't an exhaustive list anywhere.

Even given the exact same project, there is also the team factor. If you take a team that has consistantly delivered software using the plan-driven spiral methodology and throw them into Scrum, they are going to experience a decrease in productivity, an increase in thrashing, and have to overcome a new process model before they can come around to being successful. Even though another methodology might be more suited, there's always the business need to actually deliver the software. That's why process improvement efforts are frequently long-term efforts and not overnight - major changes are shocking to a team and (even if the methodology might be better suited on paper) can cause a decrease in productivity.

There's a lot more than simply effeciency or effectiveness of the process, and you can't simply look at a snapshot of the same team working in a plan-driven environment and an agile environment. You need to consider the industrial and organizational context, the attributes of the project, the team, the customer, and so on when making a decision.


Based on what I read, I'm going to have to disagree with your coworkers assessment. I'm sure that you can find some case study somewhere where an agile project was 60% less efficient with regards to some performance metric than a similar plan-driven project. However, there are also studies that show that agile yields 80% less effort, 50% less time, and high customer satisfaction with the product.

share|improve this answer
1  
@ Thomas sorry, I couldn't resists :) –  Morons Dec 15 '11 at 16:36
    
+1 for being such a good sport. (I just purchased the book) –  Morons Dec 15 '11 at 16:44
    
I ordered it Via amazon.. So its not on hand –  Morons Dec 15 '11 at 20:34
add comment

agile was only 60% as efficient in development time

True.

But, that's a lame measurement.

Agile methods usually deliver real value sooner.

Waterfall simply sticks to a schedule irrespective of what's delivered and often delivers nothing of value until a huge span of time has been passed.

Further.

You can measure "development time" separate from "development and test time".

Agile usually includes testing. So it seems slower.

Waterfall development can be cleanly separated from testing. So code is "ready to test" sooner. But isn't "done" until much later.

So. They're totally right. For what they measured.

share|improve this answer
7  
I don't know if it's always true - it depends on how (at what level) you measure efficiency. If I waterfall through a project that takes me 2 years, I just spent two years developing everything. But if I use an iterative/incremental approach, I might learn that only 40% of my requirements need to be implemented and the project successfully concludes after implementing 40% of the product backlog in 15 months. That's 9 months of development time on a different project. To me, that's an increase in efficiency - I not only met all business needs of one customer, but am already supporting a second. –  Thomas Owens Dec 15 '11 at 16:28
3  
Another case of "You get what you measure". Measure the wrong thing and it does not help. –  Loki Astari Dec 15 '11 at 16:34
3  
In my experience agile methods are definitely slower when you have a really good spec. But when your spec sucks (which is often the case), then agile methodologies save the project. Agile/SCRUM sucks when your product owner sucks. So it's pretty much the same. If you've got someone who can envision a really good product both approaches are probably equally fast. It's less dependent on methodology than on the analysts. –  Falcon Dec 15 '11 at 17:22
3  
Re-asserting the original assertion doesn't actually answer the question. Do you have any evidence, other than anecdotal, that the assertion is correct? –  Mark Booth Dec 15 '11 at 17:46
1  
You get what you measure, that's the risk you take. –  Scott Dec 15 '11 at 18:39
show 5 more comments

I don't have a study but I'd like to communicate my experience.

The effectiveness of any of the mentioned methodologies highly depends on the analysts.

When you've got a great product owner, then SCRUM for example is certainly faster than a waterfall approach with a bad spec.

Agile with a bad product owner is certainly slower than waterfall with a great specification.

However, more often than not, we don't know the exact requirements early enough and agile methodologies have faster feedback cycles. This means, that in uncertain terrain agile is a better method to deliver a high quality product within reasonable costs. There're numerous other advantages, for example, agile projects are easier to cancel when they don't work out and thus can reduce loss to a minimum.

One could say that agile methodologies reduce risk, while waterfall, even if it may be faster sometimes, can be quite a monetary gamble.

share|improve this answer
add comment

You might like the references/links in this article about the FBI Sentinel case mgmt project.

http://agilecomplexificationinverter.blogspot.com/2014/06/case-studies-software-systems-failure.html

In general the Sentinel project was a $300M waterfall project with 400 devs running years and never delivered. It was halted - vendor fired - replaced with a 40 person Scrum

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.