Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

It's all the rage nowadays. "Everyone" recommends it. That in and of itself makes me suspicious.

What are some disadvantages you have found when doing test-first (test-driven) development? I'm looking for personal experiences from knowledgeable practitioners--I can read the hypothetical musings of a hundred wannabes elsewhere on the internet.

I ask not because I am looking to hate TDD, but because it is my job to improve software development process, and the more we can learn about the problems people encounter, the better chance we have of improving the process.

share|improve this question
add comment

5 Answers

There are quite a few, but the advantages far outweigh the disadvantages.

There's a steep learning curve.

Many developers seem to expect that they can be efficient with test-first programming right from day one. Unfortunately it takes a lot of time to gain experience and program at the same speed as before. You can't get around it.

To be more specific, it's very easy to get wrong. You can very easily (with very good intentions) end up writing a whole bunch of tests which are either difficult to maintain or testing the wrong stuff. It's difficult to give examples here - these kind of issues simply take experience to solve. You need to have a good feel of separating concerns and designing for testability. My best advice here would be to do pair-programming with someone who knows TDD really well.

You do more coding up front.

Test-first means you can't skip tests (which is good) and means you'll end up writing more code up front. This means more time. Again, you can't get around it. You get rewarded with code that's easier to maintain, extend and generally less bugs, but it takes time.

Can be a tough sell to managers.

Software managers are generally only concerned with timelines. If you switch to test-first programming and you're suddenly taking 2 weeks to complete a feature instead of one, they're not gonna like it. This is definitely a battle worth fighting and many managers are enlightened enough to get it, but it can be a tough sell.

Can be a tough sell to fellow developers.

Since there's a steep learning curve not all developers like test-first programming. In fact, I would guess that most developers don't like it at first. You can do things like pair-programming to help them get up to speed, but it can be a tough sell.

In the end, the advantages outweigh the disadvantages, but it doesn't help if you just ignore the disadvantages. Knowing what you're dealing with right from the start helps you to negotiate some, if not all, of the disadvantages.

share|improve this answer
    
These are good answers, but could be more specific about #1? I am especially interested in hearing how/whether you were able to recover your speed of programming--what did you learn that you didn't know when you started doing TDD? –  Alex Feinman Sep 20 '10 at 13:41
    
Updated to give some clarification –  Jaco Pretorius Sep 20 '10 at 13:46
4  
If you are doing testing now the total time spent in development shouldn't change significantly. It only looks like things are taking longer because you are exposing the time it takes to write and maintain unit tests. –  ChrisF Sep 20 '10 at 14:02
1  
@JeffO are you familiar with the "I'm going to write myself a minivan!" school of coding? –  Alex Feinman Sep 20 '10 at 15:48
1  
@tvanfosson - because they're trying to change two things at once - both starting testing and also TDD - which can be problematical. It also adds to time estimates - quite correctly - so managers and customers only see the up front increase, not that the overall time is actually known (for once) and may even be less. If they're doing some testing then this increase will be less. –  ChrisF Sep 20 '10 at 22:26
show 7 more comments

Test-first assumes you are writing code that is:

  • testable in a unit-test manner
  • that what you are developing has an obvious approach and will not require extensive prototyping or experimentation
  • that you will not need to refactor too heavily or that you have the time to repeatedly rewrite hundreds or thousands of test cases
  • nothing is sealed
  • everything is modular
  • everything is injectable or mockable
  • that your organization places a high enough value on low-defects to justify the resource sink
  • that there is something of benefit to test at a unit test level

If your project does not meet those requirements you will have difficulty. The promoters of TDD have no good answers to this other to suggest that you redesign your product to better fall within those lines. There are situations where that is impossible or undesirable.

In practice there can also me a huge problem with people thinking the test-first tests actually prove anything about the correct function of the program. In many cases this is not true, but even in the cases where it is true it is far from a complete picture of correctness. People see hundreds of passing tests and assume it is safe to test less since before TDD they only did a few hundred test cases anyway. In my experience TDD means that you need to have even more integration tests as the developers will also have the false security and the pain of changing all the tests to do a big redactor can lead developers to make interesting work arounds.

Examples:

My personal best example is when writing security code for asp.net. If they are meant to run in a hostile environment from the machine config, they are gac'ed, signed and sealed, and because they are running against IIS god objects they are very difficult to mock very much of correctly. Add in some constraints for performance and memory use and you very quickly loose the flexibility to use placeholder objects in the remaining areas.

Any sort of micro controller or other low resource environment code may not be possible to do truly OO style design as the abstractions do not optimize out and you have low resource limits. The same can be said for high performance routines in many cases as well.

share|improve this answer
    
Can you give some counterexamples? When wouldn't I be writing something that is testable in a unit-test manner? Why wouldn't I be writing code that is mockable or injectable (other than legacy code, which is a topic unto itself)? –  Alex Feinman Sep 20 '10 at 14:19
    
edited to add examples section –  Bill Sep 20 '10 at 15:44
3  
Agreed. TDD working seems to rely on a set of assumptions about the machines you're working with; it doesn't seem to hold true for about 50% of my projects. –  Paul Nathan Sep 20 '10 at 16:57
    
+1 For the "refactor friction" –  Conrad Frix Sep 20 '10 at 21:17
    
I totally agree... Great answer –  Khelben Sep 21 '10 at 15:33
add comment

The biggest drawback I've seen is not with TDD itself but with practitioners. They take a dogmatic and zealot approach where everything must be tested. Sometimes (many times that is), that is not necessary. Also, it might not be practical (.ie. introducing an organization to TDD.)

A good engineer finds trade-offs and applies the right balance of when/where/how to apply test-first. Also, if you find yourself constantly spending much more time developing tests instead of actual code (by a factor of 2-3 or more), you are in trouble.

In other words, be pragmatic and reasonable with TDD (or anything in software development for that matter.)

share|improve this answer
add comment

I started doing TDD in early August 2009 and convinced my whole company to switch to it in September/October 2009. Presently, the whole dev team is fully converted, and committing untested code to the repo is considered a Bad Thing and thrown upon. It has been working great for us, and I cannot imagine switching back to cowboy coding.

However, there are two problems that are pretty noticeable.

The test suite has to be maintained

When you are serious about TDD, you'll end up writing lots of tests. Moreover, it takes some time and experience to realize what is the right granularity of tests (overdoing is almost as bad as underdoing it). These tests are also code, and they are susceptible to bitrot. This means that you have to maintain them as everything else: update it when you upgrade libraries they depend on, refactor from time to time... When you make big changes in your code, a lot of tests will suddenly become out of date or even plain wrong. If you are lucky, you can simply delete them, but a lot of times you'll end up extracting the useful bits and adapting them to the new architecture.

Testing abstractions leak from time to time

We are using Django, which has a pretty great testing framework. However, sometimes it makes assumptions that are slightly at odds with reality. For example, some middleware may break the tests. Or, some tests make assumptions about a caching backend. Also, if you are using a "real" db (not SQLite3), then preparing the db for the tests will take a lot of time. Sure, you can (and should) use SQLite3 and an in-memory db for tests you do locally, but some code will just behave differently depending on the database you use. Setting up a continuous integration server that runs in a realistic setup is a must.

(Some people will tell you that you should mock all the stuff like the database, or your tests aren't "pure," but that's only ideology speaking. If you make errors in your mocking code (and believe me, you will), your testsuite will be worthless.)

This all said, the problems I described start being noticeable only when you are quite advanced with TDD... When you are just starting with TDD (or working on smaller projects) test refactoring won't be an issue.

share|improve this answer
3  
+1. "has to be maintained": this is far less of a problem when testing reusable code, as its interface and behavior normally needs to stable. For this reason, I normally only do TDD for our reusable library. –  Dimitri C. Oct 11 '10 at 6:45
add comment

For me there's some deep psychological problem with tests whenever I try to apply them extensively, as in TDD: if they're there, I code sloppily because I trust that the tests will catch any problems. But if there are no tests to provide a safety net, I code carefully, and the result is invariably better than with tests.

Maybe it's just me. But I've also read somewhere that cars with all kind of safety bells & whistles tend to crash more (because the drivers know that the safety features are there), so maybe this is something to acknowledge; TDD can be incompatible with some individuals.

share|improve this answer
    
That seems strange to me, since writing testable code usually causes me to slow down and think more about what I am coding. I actually get a bit nervous coding without tests these days. –  Matt H Oct 14 '10 at 17:12
    
It just shows that different people indeed react differently. I'm not bashing TDD - obviously quite a few people find it useful - but the fact is that it's not for everybody. –  Joonas Pulakka Oct 15 '10 at 6:26
1  
I agree 100%. I write code better and faster without automated tests. Of course it would be absurd not to test, I just think automation is a bad choice (at least for me). I find manual testing to be both faster than maintaining a test suite and safer -- but I'm also an experienced developer so I'm very good at knowing what to test and where and why, so my code additions and re-factors are regression-free. –  Ben Lee Mar 12 '12 at 19:33
    
Although I should point out that the team I work with and the projects are both small enough that I have a good sense of the entire architecture -- on a large team or a very large project, I could see automated tests as more useful, because then no single developer would necessarily be able to smell where they should be testing to avoid regressions. –  Ben Lee Mar 12 '12 at 19:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.