Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What percentage of time is saved and costed doing TDD.

I assume this percentage of cost and reward changes during a projects life-cycle.

I'd imagine the initial phase has a lot more cost but little rewards attached. Further on (during re-factoring) you get the benefit of your tests.

I've heard anywhere from 30-50% of your time is writing unit tests. However that doesn't take into account the time saved from writing those tests.

What's everyones' experience with this?

Wes

EDIT Whats the time saved as well as the time cost? In bug fixing and refactorablity?

share|improve this question
    
Write tests before you code or write tests after, I would feel the overhead is negligible as either way you write tests. –  Chris Nov 23 '10 at 19:46
1  
@Chris, when you write tests first you design the API up front, instead as an afterthought. –  user1249 Nov 23 '10 at 19:54
1  
2  
@Thorbjørn: Agreed with your observation, although it's entirely possible to design an API without using TDD, hardly an afterthought. –  Robert Harvey Nov 23 '10 at 22:08
2  
@Steven: Yes, I know what TDD is. It's interesting you say design the API up-front. That strikes me as a sound approach. I've never been completely sold on the idea that you can just "grow" an API by writing a bunch of tests. –  Robert Harvey Nov 23 '10 at 23:04
show 4 more comments

5 Answers

up vote 7 down vote accepted

Each time you run your unit tests, you save yourself the amount of time it would have taken to manually test your code.

The 30% to 50% of time you quote as being required to write your tests is also offset a great deal by the benefits of having a better (testable) software design.


Let's say it takes four times as long to write an automated test as it does to manually perform the test. That means that the fourth time you run your automated test, it pays for itself. Every time you run the automated test after that, it's free.

This holds true whether the test is an automated unit test, or an automated functional test. Not all functional tests can be automated, but many of them can. Plus, the automated test is more reliable than a person; it will run the test in exactly the same way, every time.

Having unit tests means that you can refactor the underlying implementation of a method (for performance or other reasons), and the unit tests will verify that the functionality of the method has not changed. This is especially true of TDD, where the unit test specifies the functionality of the method.

share|improve this answer
    
I'm not convinced that you save yourself manual tests at all. TBH. To ensure something is functionally working you should still be using regression at least as far as I know. –  Wes Nov 23 '10 at 19:44
4  
Unit tests are regression testing. I'm not sure what you're saying. –  Robert Harvey Nov 23 '10 at 19:53
1  
Unit tests and functional tests are both forms of regression testing. I think Wes is referring to the latter. –  Phil Mander Nov 23 '10 at 21:19
    
@Phil Mander exactly right. @Robert Harvey I meant functional testing, my brain didn't find the right word. Though I'm pretty sure my subconsious did as I used the word functionally there :S Oh and good edit btw. –  Wes Nov 23 '10 at 21:57
    
I don't think running the test exactly the same way every times is actually a positive, since it's possible to very consistently miss finding problems like that. –  Peter Ajtai Nov 7 '11 at 5:50
show 1 more comment

the important long-term measures are not only code quality and code confidence, but even moreso not burning out the team doing mindless testing

the short-term measures would be the ROI of automating the tests

for example: last week I made over 1000 code changes due to an internal-architecture shift, started the automated test suite, and went to sleep.

the tests took 28 minutes to run; they all passed. manually performing the same 40+ acceptance tests would take about 6 hours.

another example: in a prior iteration i had goofed up one of the test scenarios with a subtle bug that manual testing probably would not have found (the automated tests perform db integrity checks that manual testers almost never do). i had to run that test scenario about 50 times before i managed to figure it out and fix it. manually performing the test scenario operations would take about 50 minutes. So that's 41.6 man-hours of labor saved in one day

there is no way of calculating in advance the ROI of automated testing, because you cannot know exactly how many times you will need to run the tests.

but to me, the ROI of automated tests is nearly infinite

share|improve this answer
1  
Oh that's in interesting point. I thought that DB integrity checks should be outside unit tests. What other tests apart from unit tests do you run on an automated basis? –  Wes Nov 24 '10 at 10:29
    
@Wes: the tests in TDD are referred to as "unit" tests, but don't let that unfortunate name limit their scope. Their purpose is to test features. A feature may be 'the foo function always returns null' or it may be 'the overall system latency under maximum load must be less than 12 picoseconds'. –  Steven A. Lowe Nov 24 '10 at 14:41
add comment

TDD is often measured towards code quality rather than time and cost spent. However, with better code quality, developers and any people working with them can work better (less time spent, less cost involved, happier, etc.). http://davidlongstreet.wordpress.com/2009/04/29/new-software-metric-wtfs-per-minute/

Writing tests is great for helping to automate verification of functional and non-functional requirements. One video that convinced me to adopt TDD (actually BDD, high level TDD): http://video.google.com/videoplay?docid=8135690990081075324#

  • Writing functional tests can help spotting bugs/problems earlier during development phase. Assume you have large code base. With unit tests/specs, you only need to see "All tests passed"/"2 tests failed, see line xyz". You only need a team of developers to do both developing and testing. Without unit tests/specs, you have to manually compare printed statements with expected statements, and manually trace which methods/classes have bugs. You probably need two separate teams (developers and testers) to do this.

  • Written tests help developers explain progress and problems faced.

  • TDD helps fulfilling maintainability, adaptability, and flexibility of code. It encourages developers to write small testable chunks, and put them together into bigger testable chunks. The other way round (part of refactoring practice) also works, with condition that we have written solid tests. As a result, we can have nicely-written, modular code.

With TDD, we are glad to know when:

  • a customer requests changes on requirements (satisfying requirements)
  • better ways of writing code are discovered
  • team mates have suggestions for code improvement
  • we have to explain/pass our code to other people

TDD can be boring because development process takes small steps, and thus it becomes so predictable.

share|improve this answer
add comment

I've heard anywhere from 30-50% of your time is writing unit tests. However that doesn't take into account the time saved

In my experience, it's more than 50%.

Once you've written the test, the solution has a tendency to come very easy. So I don't think it's odd to spend 70% - 75% of your time writing tests, but you're spending much less time writing the code-under-test and spending virtually no time in the debugger.

The sooner you find a bug, the cheaper it is to fix, and TDD helps with that tremendously. I've worked on projects where the last 2 months (of an 8 month project) were spent fixing bugs, and that phase would be almost entirely eliminated with TDD.

To me though, the real value is in maintenance. Inheriting a code base with tests versus one without makes you less scared to make changes (you likely didn't break anything if the tests still pass). Since you aren't scared to make changes you're willing to refactor if something isn't right. Which means the code is cleaner, the design fits better, and in theory changes can be applied faster.

share|improve this answer
    
If the tests are good tests. However, some is better than none, and generally you can tell pretty fast by looking if the test is a good one or not. –  Michael K Nov 23 '10 at 20:37
    
so you think there are real tangible savings in time. (2 months) per your example but how much time would have gone into testing? Good answer btw. –  Wes Nov 23 '10 at 21:55
    
@Wes It's so tough to know. I write the code-under-test faster, but spend a ton of time on the tests, which help me find bugs earlier, which saves time, but I don't know how much time it saved since I didn't find the bug late! I personally think TDD costs more in the short term, but saves more in the long term. The longer the project, the more it benefits. –  Brad Cupit Nov 23 '10 at 22:29
add comment

In our case, I'd estimate it's close to 40%. However, I don't think we went through a phase where it was any more than this. We have a code generator that spits out both a code skeleton that gets fleshed out by the developers and a test suite that likewise gets fleshed out. Most of our testing effort actually goes into tracking down (or creating) appropriate test data to ensure that we get complete coverage.

share|improve this answer
    
Is this a home-grown code generator, or an open-source code generator that's available in the wild? –  Robert Harvey Nov 23 '10 at 22:07
    
It's a hand-rolled solution, based on the .NET CodeDOM classes. –  TMN Nov 24 '10 at 13:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.