Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've been reading the book "The Drunkard's Walk: How Randomness Rules Our Lives" by Leonard Mlodinow and it's a truly enlightening read. The book deals with probabilities and human reasoning. And let's just say, for the record, that while some things sometimes work, there's a chance that the things you thought made it work, are unrelated to what actually made it work.

Probabilities are unintuitive.

However, it gave me an idea. There ought to be studies on this already, that have attempted to quantify the results of software engineering endeavors (which of course is a hard problem in itself). And these studies should point to what kind of software engineering practices are really important in terms of quantifiable success.

i.e.

  • A team that employs TDD is this much less likely to have this kind of problem.
  • A team that employs SOLID principles is this much less likely to have this kind of problem.
  • etc. etc.

What I'm looking for here is software engineering practices that show strong correlation between implementation and success. I'm confident that these things exist but that they are hard to come by and that's why I pose this question.

What studies or what practices do you know of that have strong correlation between implementation and success (where success is somewhat arbitrary but I think you get the idea)?

If we're gonna sell that idea that software engineering is better than cowboy coding, I think we need proofs.

share|improve this question
2  
There's no "proof". There's only evidence. –  S.Lott Feb 7 '12 at 11:01
add comment

1 Answer 1

The problem with this kind of quantification is that it's almost impossible to get good enough data on the effectiveness of software engineering practices to make any meaningful conclusion.

Most importantly, correlation does not imply causation - for example, it could just be that good programmers are quick to jump on and implement new ideas, so you see a general correlation between project success and adoption of new software engineering techniques. But that proves nothing about the effectiveness of the techniques themselves, as the entire effect might be explained by the higher talent level of the programmers who adopt them.

And then it's hard to control the independent variables. How do you ensure a fair experiement unless you are able to control for all the following?

  • Experience / skill / motivation level of team
  • Actual extent of adoption of claimed methodology (are they really doing TDD properly?)
  • Presence / absence of major design mistakes unrelated to the software engineering methodology (e.g. those requiring a major re-architecture during the project)
  • Difficulty level of projects being compared
  • Impact of externally imposed problems (e.g. major requirements changes)
  • Selection bias (e.g. did people tend to share data more often about successful projects?)
  • Confirmation bias (e.g. did people exaggerate the success of projects which use their favourite methodology?)

Even if you decide to tackle the above by giving multiple carefully selected teams the same problem under the same carefully controlled conditions then your experiment is likely to be prohibitively expensive if you want to create enough data to be statistically significant.

And finally, it's almost impossible to measure success:

  • A quantity metric like source lines of code (SLOC) is appallingly bad. The incentive on developers is to create million-line monstrosities with copy / paste coding in order to look more "productive"
  • An on-time / on-budget metric depends mainly on the level of ambition in the estimates used to create the plan / budget
  • A ROI-type metric depends as much on market situation and the commercial management of the product as it does on the quality of the engineering output (look at the history of Microsoft Windows!)
  • Story Points are useful to get a feel of velocity in an agile team but aren't really comparable across teams
  • Functionality based metrics like Function points or Use Case Points are perhaps the best of a bad bunch, but they are bureaucratic to collect and don't reflect the difference in engineering effort required to create each unit of functionality.
  • Quality metrics like bugs in production / app availability are notoriously difficult to calculate and compare on an equal basis - it depends significantly on things like platform chosen, size of user base and various operational / deployment factors.

In conclusion: trying to quantify the impact of software development tasks is an extremely difficult task, and despite many years following the topic nobody has yet come up with a truly effective approach. As a result, evaluation of software development methodologies remains more of an art than a science, and probably will remain so for many years to come.

Interestingly, there is one approach that I think has promise: application of lean principles. This isn't a panacea and won't directly solve the problem of evaluating software development methodologies, but it does have one key insight: A process with a particular element of waste is unambiguously less efficient than the same process without that element of waste, all other things being equal. So if you focus on eliminating waste in the software development process, you can at least be sure you are moving in the right direction. In addition, waste is often quantifiable so you should also get some idea of how much more efficient you are getting, at least in rough percentage terms.

share|improve this answer
2  
+1: It's not merely "hard" to control the independent variables. It's impossible. You can't do the same project twice varying one thing any more than you can eat the same sandwich twice. –  S.Lott Feb 7 '12 at 11:00
    
+1. Good explanation with proper conclusion by giving practical approach towards effective software development process by eliminating waste. –  Karthik Sreenivasan Feb 7 '12 at 14:46
    
Wait, hold on. While I do appreciated mikera's answer, it's a bit of a non-answer. I did specifically point out that I was looking for quantifiable studies. And neither do I believe as @S.Lott says that this is impossible nor do I presume to conclude causation, by simply asking for correlation. What I do know is that when dealing with a complex situation, one must first break down the problem and address individual pieces of the problem first... –  John Leidegren Feb 7 '12 at 17:41
    
You can apply statistical principles as long as you have meaningful metrics. As for the carefully controlled conditions, on average, these summarize to extraneous factors. This is a simplification, but it get's you started. I my self, firmly believe, that writing automated tests reduces the number of bugs that are found after release. Similar, I believe when an ISV goes out of business, it's more likely to find that they don't employ automated testing. These things are readily quantifiable. We just need to collect the data and get to work. –  John Leidegren Feb 7 '12 at 17:41
    
@JohnLeidegren: You can try. You can't measure very well (since the "outcome" of software is so difficult to quantify). And you can't control the degrees of freedom at all. The "quantifiable" studies are remarkably few. This answer explains why there are so few and why they don't provide much information. –  S.Lott Feb 7 '12 at 17:43
show 7 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.