Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

It's all the rage nowadays. "Everyone" recommends it. That in and of itself makes me suspicious.

What are some disadvantages you have found when doing test-first (test-driven) development? I'm looking for personal experiences from knowledgeable practitioners--I can read the hypothetical musings of a hundred wannabes elsewhere on the internet.

I ask not because I am looking to hate TDD, but because it is my job to improve software development process, and the more we can learn about the problems people encounter, the better chance we have of improving the process.

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

9 Answers 9

There are quite a few, but the advantages far outweigh the disadvantages.

There's a steep learning curve.

Many developers seem to expect that they can be efficient with test-first programming right from day one. Unfortunately it takes a lot of time to gain experience and program at the same speed as before. You can't get around it.

To be more specific, it's very easy to get wrong. You can very easily (with very good intentions) end up writing a whole bunch of tests which are either difficult to maintain or testing the wrong stuff. It's difficult to give examples here - these kind of issues simply take experience to solve. You need to have a good feel of separating concerns and designing for testability. My best advice here would be to do pair-programming with someone who knows TDD really well.

You do more coding up front.

Test-first means you can't skip tests (which is good) and means you'll end up writing more code up front. This means more time. Again, you can't get around it. You get rewarded with code that's easier to maintain, extend and generally less bugs, but it takes time.

Can be a tough sell to managers.

Software managers are generally only concerned with timelines. If you switch to test-first programming and you're suddenly taking 2 weeks to complete a feature instead of one, they're not gonna like it. This is definitely a battle worth fighting and many managers are enlightened enough to get it, but it can be a tough sell.

Can be a tough sell to fellow developers.

Since there's a steep learning curve not all developers like test-first programming. In fact, I would guess that most developers don't like it at first. You can do things like pair-programming to help them get up to speed, but it can be a tough sell.

In the end, the advantages outweigh the disadvantages, but it doesn't help if you just ignore the disadvantages. Knowing what you're dealing with right from the start helps you to negotiate some, if not all, of the disadvantages.

share|improve this answer
    
These are good answers, but could be more specific about #1? I am especially interested in hearing how/whether you were able to recover your speed of programming--what did you learn that you didn't know when you started doing TDD? –  Alex Feinman Sep 20 '10 at 13:41
    
Updated to give some clarification –  Jaco Pretorius Sep 20 '10 at 13:46
7  
If you are doing testing now the total time spent in development shouldn't change significantly. It only looks like things are taking longer because you are exposing the time it takes to write and maintain unit tests. –  ChrisF Sep 20 '10 at 14:02
1  
@JeffO are you familiar with the "I'm going to write myself a minivan!" school of coding? –  Alex Feinman Sep 20 '10 at 15:48
1  
@tvanfosson - because they're trying to change two things at once - both starting testing and also TDD - which can be problematical. It also adds to time estimates - quite correctly - so managers and customers only see the up front increase, not that the overall time is actually known (for once) and may even be less. If they're doing some testing then this increase will be less. –  ChrisF Sep 20 '10 at 22:26

Test-first assumes you are writing code that is:

  • testable in a unit-test manner
  • that what you are developing has an obvious approach and will not require extensive prototyping or experimentation
  • that you will not need to refactor too heavily or that you have the time to repeatedly rewrite hundreds or thousands of test cases
  • nothing is sealed
  • everything is modular
  • everything is injectable or mockable
  • that your organization places a high enough value on low-defects to justify the resource sink
  • that there is something of benefit to test at a unit test level

If your project does not meet those requirements you will have difficulty. The promoters of TDD have no good answers to this other to suggest that you redesign your product to better fall within those lines. There are situations where that is impossible or undesirable.

In practice there can also me a huge problem with people thinking the test-first tests actually prove anything about the correct function of the program. In many cases this is not true, but even in the cases where it is true it is far from a complete picture of correctness. People see hundreds of passing tests and assume it is safe to test less since before TDD they only did a few hundred test cases anyway. In my experience TDD means that you need to have even more integration tests as the developers will also have the false security and the pain of changing all the tests to do a big redactor can lead developers to make interesting work arounds.

Examples:

My personal best example is when writing security code for asp.net. If they are meant to run in a hostile environment from the machine config, they are gac'ed, signed and sealed, and because they are running against IIS god objects they are very difficult to mock very much of correctly. Add in some constraints for performance and memory use and you very quickly lose the flexibility to use placeholder objects in the remaining areas.

Any sort of micro controller or other low resource environment code may not be possible to do truly OO style design as the abstractions do not optimize out and you have low resource limits. The same can be said for high performance routines in many cases as well.

share|improve this answer
    
Can you give some counterexamples? When wouldn't I be writing something that is testable in a unit-test manner? Why wouldn't I be writing code that is mockable or injectable (other than legacy code, which is a topic unto itself)? –  Alex Feinman Sep 20 '10 at 14:19
    
edited to add examples section –  Bill Sep 20 '10 at 15:44
4  
Agreed. TDD working seems to rely on a set of assumptions about the machines you're working with; it doesn't seem to hold true for about 50% of my projects. –  Paul Nathan Sep 20 '10 at 16:57
1  
+1 For the "refactor friction" –  Conrad Frix Sep 20 '10 at 21:17
1  
It's like anything in this game - appropriate for many situations, inappropriate for others. Beware of anyone advocating a One True Path in any area of software development. –  Alan B Oct 6 at 10:59

The biggest drawback I've seen is not with TDD itself but with practitioners. They take a dogmatic and zealot approach where everything must be tested. Sometimes (many times that is), that is not necessary. Also, it might not be practical (.ie. introducing an organization to TDD.)

A good engineer finds trade-offs and applies the right balance of when/where/how to apply test-first. Also, if you find yourself constantly spending much more time developing tests instead of actual code (by a factor of 2-3 or more), you are in trouble.

In other words, be pragmatic and reasonable with TDD (or anything in software development for that matter.)

share|improve this answer
    
Is that, perhaps, where the Michael Feathers' "new" definition of Legacy Code (i.e. "Code without Tests") comes from? –  Phill W. Oct 6 at 11:34
    
That definition wouldn't work for me :) To me, any code that runs in production and that is subject of change is legacy code, independently of code or test quality. We typically associate "legacy code" with "bad code" or "obsolete code" when in reality bad code and obsolete code is already present in code under development that hasn't seen production usage yet. Our aim should be for our code to be legacy from the get go, and to be of such quality and usefulness that it remains in usage for years, decades to come. –  luis.espinal Oct 6 at 15:14

I started doing TDD in early August 2009 and convinced my whole company to switch to it in September/October 2009. Presently, the whole dev team is fully converted, and committing untested code to the repo is considered a Bad Thing and thrown upon. It has been working great for us, and I cannot imagine switching back to cowboy coding.

However, there are two problems that are pretty noticeable.

The test suite has to be maintained

When you are serious about TDD, you'll end up writing lots of tests. Moreover, it takes some time and experience to realize what is the right granularity of tests (overdoing is almost as bad as underdoing it). These tests are also code, and they are susceptible to bitrot. This means that you have to maintain them as everything else: update it when you upgrade libraries they depend on, refactor from time to time... When you make big changes in your code, a lot of tests will suddenly become out of date or even plain wrong. If you are lucky, you can simply delete them, but a lot of times you'll end up extracting the useful bits and adapting them to the new architecture.

Testing abstractions leak from time to time

We are using Django, which has a pretty great testing framework. However, sometimes it makes assumptions that are slightly at odds with reality. For example, some middleware may break the tests. Or, some tests make assumptions about a caching backend. Also, if you are using a "real" db (not SQLite3), then preparing the db for the tests will take a lot of time. Sure, you can (and should) use SQLite3 and an in-memory db for tests you do locally, but some code will just behave differently depending on the database you use. Setting up a continuous integration server that runs in a realistic setup is a must.

(Some people will tell you that you should mock all the stuff like the database, or your tests aren't "pure," but that's only ideology speaking. If you make errors in your mocking code (and believe me, you will), your testsuite will be worthless.)

This all said, the problems I described start being noticeable only when you are quite advanced with TDD... When you are just starting with TDD (or working on smaller projects) test refactoring won't be an issue.

share|improve this answer
3  
+1. "has to be maintained": this is far less of a problem when testing reusable code, as its interface and behavior normally needs to stable. For this reason, I normally only do TDD for our reusable library. –  Dimitri C. Oct 11 '10 at 6:45

For me there's some deep psychological problem with tests whenever I try to apply them extensively, as in TDD: if they're there, I code sloppily because I trust that the tests will catch any problems. But if there are no tests to provide a safety net, I code carefully, and the result is invariably better than with tests.

Maybe it's just me. But I've also read somewhere that cars with all kind of safety bells & whistles tend to crash more (because the drivers know that the safety features are there), so maybe this is something to acknowledge; TDD can be incompatible with some individuals.

share|improve this answer
    
That seems strange to me, since writing testable code usually causes me to slow down and think more about what I am coding. I actually get a bit nervous coding without tests these days. –  Matt H Oct 14 '10 at 17:12
1  
It just shows that different people indeed react differently. I'm not bashing TDD - obviously quite a few people find it useful - but the fact is that it's not for everybody. –  Joonas Pulakka Oct 15 '10 at 6:26
2  
I agree 100%. I write code better and faster without automated tests. Of course it would be absurd not to test, I just think automation is a bad choice (at least for me). I find manual testing to be both faster than maintaining a test suite and safer -- but I'm also an experienced developer so I'm very good at knowing what to test and where and why, so my code additions and re-factors are regression-free. –  Ben Lee Mar 12 '12 at 19:33
1  
Although I should point out that the team I work with and the projects are both small enough that I have a good sense of the entire architecture -- on a large team or a very large project, I could see automated tests as more useful, because then no single developer would necessarily be able to smell where they should be testing to avoid regressions. –  Ben Lee Mar 12 '12 at 19:36
    
Are you leaving out the refactoring step? –  Cwan Oct 6 at 8:04

One situation in which test-first really gets in my way is when I want to quickly try out some idea and see if it can work before I write a proper implementation.

My approach is normally:

  1. Implement something that runs (proof of concept).
  2. If it works, consolidate by adding tests, improving design, refactoring.

Sometimes I do not get to step 2.

In this case, using TDD has turned out to have more disadvantages than advantages for me:

  • Writing tests during the implementation of the proof of concept just slows me down and interrupts my flow of thoughts: I want to understand an idea and I do not want to waste time testing details of my first rough implementation.
  • It may take longer to find out if my idea is worth something or not.
  • If it turns out that the idea was useless, I have to throw away both my code and my nicely written unit tests.

So, when I have to explore some new ideas, I do not use TDD and only introduce unit tests when I get a feeling the new code is getting somewhere.

share|improve this answer
1  
It sounds like you are confusing prototype code with usable code. Prototype code is test code. It doesn't need to be tested and you should not create tests that run against it. The step you are missing is between 1. and 2.: you say "consolidate by writing tests". The problem is you do not have something to consolidate, but something to write. Plan to rewrite prototype code, don't plan to reuse it. Reusing it leaves lots of place for compromise. Rewriting formalizes the split between your exploration phase and your "quality code" phase. –  utnapistim Oct 6 at 8:18
2  
@utnapistim: I am not confusing prototype code with usable code, rather, TDD zealots confuse them and suggest that you should use TDD for prototype code as well. Or rather, they assume that there is no prototype code at all. Also, I agree with you that often you have to rewrite when you move from prototype to the real implementation. Sometimes you can reuse parts of the prototype code, but you must be ready to rewrite. You really have to decide from case to case. –  Giorgio Oct 6 at 9:33
2  
@utnapistim: See also luis.espinal's answer: "The biggest drawback I've seen is not with TDD itself but with practitioners. They take a dogmatic and zealot approach where everything must be tested.". –  Giorgio Oct 6 at 9:41

The benefits of TDD is that it forces you to guard your code against people who don't understand it. Yes, this often includes yourself. But, what happens when the code isn't worth guarding? There is a lot of code that shouldn't even be there in the first place! So the problem with TDD is when it comes to developers who write bad code. TDD probably wont help them write good code, it is much more likely that they will write horrible tests as well. Thus in their case TDD will only add to the mess; badly written and/or redundant tests aren't any more fun than other forms of bad code.

share|improve this answer
1  
If you yourself don't understand your own code, how can a handful among billions of possible testcases possibly guard against the code being wrong? –  Michael Shaw Oct 6 at 17:48
2  
Because you understood it when you wrote it but forgot about it along the way? –  Johan Oct 6 at 18:09
    
+1 TDD doesn't protect against a developer who has misunderstood a business requirement. This is where BDD comes in... –  Robbie Dee Oct 7 at 10:10

Disadvantages or Costs of TDD

Note: There is a range of different kinds of TDD. Regardless of unit, BDD, ATDD, or other variants many of the difficulties remain

Side effects

Whether it's mocking, fixtures or functional tests, dependencies on external states or systems are often the source of most complexity in tests, confusion in how to test, and biggest risk in getting it wrong. A few issues I've seen:

  • Mocking: forget to assert the order of calls
  • Mocking: mock doesn't match real call or response
  • Fixture: test relies on unrealistic data, masking other issues
  • Fixture: test an impossible state in production
  • Functional: false build breaks because of dependent system being temporarily unavailable
  • Functional: speed of test is very slow

You will have to change your approach to coding, for some it will be a drastic change.

Different people code in wildly different ways. In TDD you need to start with a test that asserts a specific behavior, and then implement so the test passes. I've seen and was a programmer whose programming was not conducive to TDD. It took me about 2 months when I initially started to get used to changing my development approach.

It takes time to understand what you care about testing and what you don't care about testing.

Every team should make an explicit decision on where they want to draw the line in testing. What things they value that they want tested, and what they don't. It is often a painful process learning how to write good tests, and what you actually care about testing. Meanwhile the code will continue to be in a state of flux until there is consistency in both style and approach.

Unit Test specific: large refactors

A large or fundamental refactor of a significant codebase with tens of thousands of unit tests will generate a huge cost in order to update all the tests. This will often manifest in pushback against doing a refactor even if it is the correct thing to do simply because the cost associated with doing it.

share|improve this answer

My analogy is barriers on a Scalextric track. If you put them on, you get far less cautious.

People also get a little bit space cadetish about their tests - because they run fine, they believe the code is fully tested whereas it is only the beginning of the testing process.

To my mind TDD is a stepping stone to BDD. A raft of tests that run doesn't really help support developers without knowing what the tests do. With BDD, the test output is in English which documents the testing and thereby builds understanding of the system.

share|improve this answer

protected by maple_shaft Oct 6 at 23:35

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.