Are there any scientific studies done on TDD that use total cost of ownership for a product as the measurement?

Question

When I was reading the summary of previous work in Dogsa T, Batic D. The effectiveness of test-driven development: an industrial case study. Software Quality Journal. 2011;19(4):643-661. it struck me that the measurements used in a lot of studies around TDD are based on things like lines of code, defects and time spent on development.

Are there any studies out there that focus on the total cost of ownership for products that have been developed using TDD vs ex traditional development or test-last?

I am especially interested in the total cost of acquisition and operating costs.

Answer 1

There are some studies about the implications and benefits of doing TDD, but the results are contradictory. Some projects (this is in my experience) have a lower bug rate and cost to ownership as a result of using TDD, as the cost of changing a feature reduces drastically. Some others are halted to a stop.

Some studies (here is one - check slide n50) show that the number of bugs increased with the coverage. I'm assuming that greater coverage implies TDD and that a higher number of bugs implies a higher cost of ownership.

From my point of view, no metric or practice on its own can be related to better quality or lower cost of ownership. There's a combination of factors that can lead to some correlation. And those factors change between teams and projects.

I think we've all heard stories of teams, which just started doing TDD, writing 100-line test methods, which (in my view) increases the cost of ownership, as updating that test will be expensive.

My pragmatic rule is that people who care and are eager to learn, working in an environment that supports them and their ideas have better quality and cost of ownership.

Answer 2

I don't have any specific studies but I can tell you from personal experience, and from the experiences of other developers I know that when applied correctly for medium and larger projects TDD reduces time to market, reduces bugs and defects, and improves code quality.

Having said that their are no silver bullets, can you write good code with out TDD? yes, can you write bad code using TDD yes. Also depending on your project TDD can greatly increase your cost of ownership for the code, a good example is NASA, where the cost per line of code is huge, but the cost of ownership isn't the focus, it's lack of defects.

When applied correctly TDD will increase your starting costs, and your code base, but you get the long term benefits of regression testing, early bug detection, and better code design which should reduce defects and testing costs and maintenance time, thus reducing overall cost of ownership.