Has test driven development (TDD) actually benefited a real world project?

Question

I am not new to coding. I have been coding (seriously) for over 15 years now. I have always had some testing for my code. However, over the last few months I have been learning test driven design/development (TDD) using Ruby on Rails. So far, I'm not seeing the benefit.

I see some benefit to writing tests for some things, but very few. And while I like the idea of writing the test first, I find I spend substantially more time trying to debug my tests to get them to say what I really mean than I do debugging actual code. This is probably because the test code is often substantially more complicated than the code it tests. I hope this is just inexperience with the available tools (RSpec in this case).

I must say though, at this point, the level of frustration mixed with the disappointing lack of performance is beyond unacceptable. So far, the only value I'm seeing from TDD is a growing library of RSpec files that serve as templates for other projects/files. Which is not much more useful, maybe less useful, than the actual project code files.

In reading the available literature, I notice that TDD seems to be a massive time sink up front, but pays off in the end. I'm just wondering, are there any real world examples? Does this massive frustration ever pay off in the real world?

I really hope I did not miss this question somewhere else on here. I searched, but all the questions/answers are several years old at this point. It was a rare occasion when I found a developer who would say anything bad about TDD, which is why I have spent as much time on this as I have. However, I noticed that nobody seems to point to specific real-world examples. I did read one answer that said the guy debugging the code in 2011 would thank you for have a complete unit testing suite (I think that comment was made in 2008).

So, I'm just wondering, after all these years, do we finally have any examples showing the payoff is real? Has anybody actually inherited or gone back to code that was designed/developed with TDD and has a complete set of unit tests and actually felt a payoff? Or did you find that you were spending so much time trying to figure out what the test was testing (and why it was important) that you just tossed out the whole mess and dug into the code?

Answer 1

This paper demonstrates that TDD adds 15-35% development time in return for a 40-90% reduction in defect density on otherwise like-for-like projects.

The article refers full paper (pdf) - Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. “Realizing quality improvement through test driven development: results and experiences of four industrial teams“. ESE 2008.

_{Abstract Test-driven development (TDD) is a software development practice that has been used sporadically for decades.With this practice, a software engineer cycles minute-by-minute between writing failing unit tests and writing implementation code to pass those tests. Testdriven development has recently re-emerged as a critical enabling practice of agile software development methodologies. However, little empirical evidence supports or refutes the utility of this practice in an industrial context. Case studies were conducted with three development teams at Microsoft and one at IBM that have adopted TDD. The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15–35% increase in initial development time after adopting TDD.}

_{Full paper also briefly summarizes the relevant empirical studies on TDD and their high level results (section 3 Related Works), including George and Williams 2003, Müller and Hagner (2002), Erdogmus et al. (2005) , Müller and Tichy (2001), Janzen and Seiedian (2006).}

Answer 2

I see some benefit to writing tests for some things, but very few. And while I like the idea of writing the test first, I find I spend substantially more time trying to debug my tests to get them to say what I really mean than I do debugging actual code.

I have been working TDD for the last three years, and my experience is the exact opposite. I spend less time writing unit tests that I would have debugging the code, had I not written the unit tests.

Not only do I do TDD, I work outside-in, i.e. I first implement the top/gui layer TDD. Implementing the top layer defines the requirements for the next layer in the system, which I develop using TDD, etc. until all code required for the feature has been implemented. Often, I experience that after I have implemented a feature like this, and I smoke test the feature in the actual system, it works the first time. Not all the time, but often.

And given that it takes a lot longer to smoke test a feature in the actual system than it takes to execute a couple of unit tests, I save humongous amounts of time. It is actually quicker for me to implement a feature using TDD than it is to not implement the feature not writing unit tests at all.

But writing unit tests is a skill that has to be learned and mastered, just like any other programming skill. When I started doing TDD I had 12 years of professional experience with programming, and I was a very accomplished programmer. I thought that writing large test suites for the system code would be a simple thing. But as the amount of test code grew, and different parts of the system got changed, and existing tests had to be modified, I learned that structuring and writing unit test is in itself a skill that has to be learned and mastered. Also not all code is equally testable. The system code has to be very loosely coupled in order to be efficiently testable. Actually learning TDD has helped me making the system code more loosely coupled.

But my current efficiency in working TDD comes from a combination of both mastering how to write unit tests, but also mastering the technology the system is implemented in (C# in this case).

Doing TDD while learning a new technology can be difficult, e.g. though I've been doing a bit of iPhone programming, I am not writing a significant amount of unit tests, because I do not master the language, objective c, nor do I master the library. So I have no idea how to structure my unit tests, even less how to structure the system code how to make it testable.

But how well does it work on real projects?

On the project I have been working on for the last couple of years, although there is a requirement that code should be sufficiently covered by unit tests, I am the only on the team writing the tests firsts. But the large test suite does give me confidence in being able to refactor the system, and have faith the system will work correctly if the test suite passes.

But unfortunately because many of the tests are written after the system code, quite a few of the tests themselves are faulty, i.e. they don't really test what they were intended to test. This imho cannot be avoided. Every time you write a piece of code, there is a probability of the code not working as you intended, i.e. there is a bug. The same goes for test code. Therefore there is a probability that you write a test that passes even though the code that it should be testing isn't working as intended.

Writing the test first, verifying not just that that you get a test failure, but that the test fails with exactly the error message you expect before implementing the system code seriously reduces the risk of a bug in the unit test code.

So to sum it up, in my experience once you have mastered the art of TDD, you will not just save time in the long run, you will save time up front. But it takes time, even for an experienced programmer, to master the art of TDD. And it takes even longer for a team of developers of varying skill to master the art of TDD.

Answer 3

We have benefited massively.

We are Tier 1 Merchant, which means that we process over six millions of payment transactions per year.

Our payment gateway system has thousands of unit and integration tests. These tests give us confidence in our ability to process payments. You want to be confident that your car brakes work, don't you? We want to be confident that we don't lose our business because we can't process payments.

Code coverage gives you that confidence. Of course it's not enough on its own, but it's a very good start.

Most of our payment gateway system was written using TDD. Some aspects were rather difficult to test, so we have decided to cut corners by sacrificing some code coverage. We will get back and address these issues eventually.

Personally, I find it difficult to write any logic before writing tests. Having said that, it took me a little while to start thinking in TDD way.

Visa PCI Reference: http://usa.visa.com/merchants/risk_management/cisp_merchants.html

Answer 4

Though tests may be often considered just a way of doing too much by some people, I do think it's really worth the trouble in certain cases.

I've been developing a Killer Sudoku Solver for school for about 3 months, it uses many "strategies" to remove possibilities and solutions. The fact is that, an error in a possibility can be fatal and result in a problem to solve the sudoku because when some possibility is removed, you don't try it anymore, and if it was the solution, the program can't solve the grid anymore.

But it's really hard to test manually, I mean right there's a grid, I can see which stratgeies are doing what in a real world example, but I just can't check all each time a strategy apply because this represents too much data.

And strategies applied on a certain grid are quite "random", that's to say you won't use all on a particular grid.

So I wrote tests on each strategy, checking the result on every cell, using just simple situations (for instance just two cells have already removed possibilities) and it saved me hours a day when I unfortunately had a grid which couldn't be solved. Because I already knew where was the problem.

Answer 5

The advantage of TDD is that you figure out how to call your code before your write the actual code.

In other words, TDD helps with designing your API.

In my experience this results in better API's which in turn gives better code.

---

EDIT: As I wrote, this was "in my experience", i.e. when writing "real world projects" but unfortunately this is with a closed source code base I cannot let the world see. I can understand from the comments that this is what is actually asked for, and not just a confirmation of the mere existence of such projects.

I have also found - again in my personal experience - that the real benefit show when you enter maintenance mode because requirements tend to change. The cleaner API's made it much easier to express the new or altered requirements in test code, and all the tests makes it very easy to see for a future maintainer how the code is to be called and what can be expected back.

The test cases are running versions of the specification and allows you to see very, very easily how to invoke your API. This is perhaps the single most useful form of "HOW"-documentation I've seen so far (compare to the "WHY"-documentation like JavaDoc) as you are certain it is correct (otherwise the test would fail).

Lately, I've had to maintain a scriptable ftp client with a very large set of options which all influence the way the application works. TDD has been introduced lately for new functionality and the large test suite allows us to do hotfixes while being confident that the used functionality still works as expected. In other words this transition proved to pay off very quickly.

Answer 6

How valuable a particular approach to testing is depends on how mission critical the system under development is, and how much some other mission critical system depends on it. A simple guestbook script for your website could hardly be considered mission critical, but if the website it runs on could potentially be compromised by a bug that allowed unfiltered input to the database and that site offers some vital service, then it suddenly becomes a lot more important for the guestbook script to be thoroughly tested. The same is also true of framework/library code. If you develop a framework with a bug, then every application that uses that framework feature also has that same bug.

Test driven development gives you an extra layer of safety when it comes to tests. If you write the tests alongside or even after the code you want to test, then there's a real risk that you get the tests wrong. If you write all the tests first, then how the code works internally can't influence what you write tests for, and therefore there's less possibility that you inadvertently write tests that think that a particular erroneous output is correct.

Test driven development also encourages your developers to write code that's easy to test, because they don't want to give themselves even more work to do! Code that's easy to test tends to be code that's easy to understand, reuse and maintain.

And maintenance is where you will really reap the rewards of TDD. The vast majority of the programming effort expended on software is maintenance related. This means making changes to live code to give it new features, fix bugs or adapt it to new situations. When making such changes, you want to be sure that the changes you make have the desired effect, and more importantly, they have no unexpected knock on effects. If you have a full test suite for your code, then it's easy to verify that any changes you make are not breaking something else, and if the changes you make do break something else then you can quickly locate the reason why. The benefits are long term.

You said the following in your question:

I see some benefit to writing tests for some things, but very few. And while I like the idea of writing the test first, I find I spend substantially more time trying to debug my tests to get them to say what I really mean than I do debugging actual code. This is probably because the test code is often substantially more complicated than the code it tests. I hope this is just inexperience with the available tools (rspec in this case).

This seems to suggest to me that you're not quite getting testing. A unit test is supposed to be extremely simple, just a sequence of method calls, followed by an assertion to compare the expected result against the actual result. They're meant to be simple because bugs in your tests would be disastrous, and if you introduce loops, branching or other program throw control into the test then it becomes more likely that the test will have a bug introduced into it. If you're spending a lot of time debugging tests then it indicates that your tests are overly-complicated and you should simplify them.

If the tests can't be simplified, than that fact alone suggests that there's something wrong with the code under test. For example if your class has long methods, methods with a lot of if/elseif/else or switch statements or a high number of methods that have complex interactions dictated by the current state of the class then tests will by necessity have to be extremely complex to provide full code coverage and test all eventualities. If your class has hard coded dependencies on other classes then this again will increase the number of hoops you will have to jump to in order to effectively test your code.

If you keep your classes small and highly focused, with short methods with few execution paths and try to eliminate internal state then the tests can be simplified. And this is kind of the crux of the matter. Good code is inherently easy to test. If the code isn't easy to test then there's probably something wrong with it.

Writing unit tests is something that benefits you in the long run, and avoiding them is simply storing up trouble for later. You might not be familiar with the concept of technical debt, but it works a lot like financial debt. Not writing tests, not commenting code, writing in hard coded dependencies and so in are ways of going into debt. You "borrow" time by cutting corners early on and this might help you hit a tight deadline, but the time you save earlier on in the project is on loan. Each day that goes by without cleaning up the code, commenting it properly or building a test suite will cost you interest. The longer it goes on, the more interest accumulates. Eventually, you'll discover your code has become a tangled mess that you can't make changes to without triggering unintended consequences. The interest on the time loan you took out early in the project and failed to pay back later has come back to bite you and now you're effectively bankrupt - The time taken to fix the problems with the existing system exceeds the time it would take to rewrite it from scratch.

You could think of writing unit tests early and keeping them up to date as a form of "technical credit". You're putting time in the bank by spending it early on in the project on following good practice. You'll earn interest on this foresight later on when you get to the maintenance phase of the project. When you want to make a change, you can easily validate the correctness of the change and that it doesn't have any unwanted side effects, and you can get updates out the door quickly and without fuss. If bugs turn up, you can add a new unit test that exercises the bug, then fix the bug in the code. When you next run the unit test you'll be able to verify that the bug was fixed, and that it didn't cause any other problems. Further more, you'll avoid "regressions", where a previously fixed bug is reintroduced because there's nothing documenting why things have been changed.

TL:DR - Yes, they are a real world help, but they're an investment. The benefits only become apparent later.

Answer 7

I'm using TDD quite often at work. My experience is that TDD justifies itself because you don't pay additional time or effort, you save it.

• Since I use TDD I spend much less time debugging or such. It just works from the start because I don't consider productive code as written as long as the tests don't pass.

• QA reports much less bugs, so we save cost on repairing our code after a release. This is because TDD doesn't let you write code without a test so the code coverage is much better.

• I can run my (productive) code much more often and quicker because I don't need to start the whole application server. Starting the unit test is an order of magnitude faster. Of course I only benefit from it when the test is already executable when I want to try out the productive code. When the tests come afterwards this benefit is missed.

• I do much less manual testing. My colleagues who don't practice TDD spend a lot of time clicking through the application until they reach the point where the new code is executed. I only test manually once, just before I commit to version control.

• Even if I use a debugger it's much quicker to debug the execution of a test than of the whole application.

Maybe you think of unit tests as regression tests. That is one of their purposes but understanding them as a tool for development makes them much more worthwhile.

Answer 8

Another advantage (in addition to those mentioned by the other people that have answered) kicks in when customer acceptance testers, or (heaven forbid) production users discover a bug. Turn the bug report into a TDD-style test for the class that seems to be at fault. Watch it fail. Fix it. Watch it pass. Then you know that you've fixed the bug. This technique has saved me hours.

Answer 9

Well, I know that I personally benefit by being twice as fast as my fellow developers and write less than half the bugs they do because they DON'T do TDD. People that probably should be better than me even...I outperform them by at least a factor of 2.

I didn't get there right away. I'm pretty good at writing code off the cuff and without harness. Seemed like a big waste to write all this extra crap. But it does several things, including (not exclusive):

• Forcing a design that's geared toward decoupling and reuse (everything has to be reused in a unit test).
• Providing a platform to develop code in small chunks and modules so I don't have to get everything figured out and finished before running a simple "Does it even compile and take input" kind of test.
• Providing a quick test platform for making changes when people demand feature changes I wasn't expecting.

An example on this later bit is a project I am currently working on where the lead suddenly decided to TOTALLY rewrite the communication protocol it used for pretty much 0 reason. I was able to respond to that change in 2 hours as I had decoupled that from everything else already and was able to work on it entirely independently until the very last tie it together and integrate test it step. Most of my coworkers probably would have been at it for a day or more because their code would not be decoupled and they'd be changing here, there, everywhere...compiling it all...integration testing...repeat, repeat... Takes a lot longer that way and is nowhere near as stable.

Answer 10

The answer is yes. In my company we have been developing a C++ application for over 20 years. Last year, we got into TDD in some new modules, and defect rates decreased significantly. We liked it so much that some of us are even adding tests to legacy code every time we change something there.

Moreover, a whole module was completed from start to finish, going through production, without ever demonstrating a bug (and it's a critical module, too). Its development thereby was faster than usual, because usually what would happen otherwise is that a module would be "completed", only to return 4-5 times from beta testing for bug fixes. It was a substantial improvement, and the developers were more pleased of the new process as well.

I haven't done a lot of Rails TDD, but I've done much in C++, C#, Java and Python. I can tell you that it definitely works. My guess is that you're spending a lot of time thinking about test names because you're not confident enough. Write your test first, but let your creativity flow...

I've noticed that once you truly get the hang of TDD, you start caring a bit less on "How am I going to name this test... argh!", and you just flow with it, refactoring and adapting already-written tests to fit the current situation.

Time for a tip

Tip #1

So the tip I think should help you the most is to not worry so much. One of the prettiest things about TDD is that it gives you courage about changing things that are already written and working. And that includes the tests.

Tip #2

Start off new class tests by a simple "canCreate" test, just to set your mind in the right direction, as in "Ok, I'm working on this class now... right."

Then start adding more tests, but only one at a time, and make sure that each test you create, it is the next simplest case that comes to your mind at that point (think about it for no more than 30 seconds, and then timeout with the best you've got at that point).

And Remember

Don't worry about refactoring existing tests or even removing obsolete or redundant ones. Not many people realize this, but in TDD you actually get 2 safety nets for the price of 1. Your tests are a safety net for production code changes, but your production code is also a safety net for refactoring the tests. The relationship is mutual. It's actually a good case of tight coupling.

Give it another shot. And let me recommend watching Clean Code Casts, especially the ones about TDD.

Answer 11

Real world non-trivial example:

I had to write a data structure transformation function. Input would be a data structure (actually a nested data structure, much like a tree) and the output would be a similar data structure. I could not visualize the actual transformation in my mind. One of the main benefits of TDD (for me, anyways) is the enforcement of baby steps if you don't know how to proceed (see Kent Becks "TDD by Example"). Because I did not know where this was going I started with the simple base cases like empty or trivial inputs and worked my way up to more complicated cases until I figured I had them all covered. In the end I had a working algorithm and the tests that proved it. Not only do the tests prove that my implementation is working right now, they also keep me from screwing anything up later on. Because of test driving the implementation it is decoupled and the API is clean.

Answer 12

I don't like the idea of blindly following any generic advices because I don't believe there's a one-size-fits-all suggestion that will help most developers to become more productive and reduce defects in applications. From my experience, the more you worry about quality the more you'll lose in the amount of delivered new features. So the level of importance you want to give to quality vs deliverability will actually depend on your product and current strategy and most probably it will be someone else that will decide strategically what's more important for the time being: robustness or deliverability.

Even this decision isn't black or white. Most probably some parts of your application must be robust while others don't have to. Once you identify which parts should have a high degree of quality you should focus on them from the testing perspective since you want to ensure high quality to those parts.

All of what I've said so far has nothing to do with TDD specifically in the sense of writing tests before the implementation, but I believe it's important to separate the benefits of having tested code vs writing the tests first.

Once you understand the benefits of testing itself, TDD or not, you can then discuss the testing strategy for the code you want to be covered by tests. Some people will arguee that if you write the tests later you'll miss some conditions in your tests, but I believe you should be the one to evaluate whether that applies to you. It certainly doesn't apply to me.

So, here's how it works for me. There are basically two situations that will make me write tests: it will improve quality only or it will speed up my development of some feature as well. So, a situation where I'll write tests is when there's no new features in the backlog and I can then decide to improve performance of the application, simplify the code base or improve the test suite. Another situation is the need to have a solid working code where bugs would have big enough impact in the real clients. Yet another one is for testing complex code that is easy to break when working on it. As an example, there's a QueryBuilder class in my code base that takes care of lots of use cases and it would be easy to break some of them while fixing a bug or adding a new feature. I know that writing the tests in those cases are very well-worthy since it will save me lots of time in the future and in this specific case this is also a critical part of the application I maintain, so quality is very important here.

Finally, there's the case where writing the tests first enable me to write a feature faster then not writing the tests at all. That QueryBuilder was also a case where this rule has also applied, but it doesn't mean TDD will also be the best path. Another example of TDD helping on development speed is for testing Excel generation for instance, while in the real application you might have to perform several steps each time you want to test some specific condition in the generation. Or if you need to create some records to test a feature and it's hard or impossible to delete them manually after you have manually tested the code.

So, if it's easier for you to reproduce the steps to run some in-development code programatically (through tests), go for it. But if writing the test is more complicate then testing it manually, then you have to decide if this is the time to focus on quality or if you have lots of requests in your backlog and someone in the company will probably know it better and let you know where you should be focusing in accordingly to their current need and company strategy.

In an ideal world all code is tested, but one can't pretend there isn't a trade-off and assume that TDD is always the best and only path. As with all best practices out there you must always focus on what is best for the company you work for instead of what is better for you. Once you become self-employed you are free to decide to perform TDD all the time if you think it is the best path. If your company believes all code should be tested, then you must write tests for all code you write. But for most cases you have to get the big picture and understand the trade-offs before you take any decision. Sorry, but this is not an exact science and there's no easy (or hard) one-size-fits-all answer that you should follow every time.

Just the same as with design patterns. Understand how they work and why they have been created and what kind of problems they solve and what are their drawbacks as well. Understand the reasoning is way more important than remembering the proposed solutions. What is an expensive operation today may be easily achieved tomorrow with other technologies. If the premise for some well stablished solution is no longer valid most probably the solution is no longer the best one to use. When the requirements, available technology or company's strategy has changed you should always reevaluate your toolbox and when that happens you need to understand why you chose each path in the first place instead of taking them for grant as the best options.