Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What is a negative side of your TDD experience? Do you find baby steps (the simplest fix to make test green) annoying and useless? Do you find no-value tests (when test has sense initially but in final implementation checks the same logic as other test) maintanence critical? etc.

The questions above are about things which I am uncomfortable with during my TDD experience. So I am interested whether other developers have similar feelings and what do they think about them.

Would be thankful for the links to articles describing negative sides of TDD (Google is fullfilled by positive and often fanatic articles).

share|improve this question

closed as primarily opinion-based by GlenH7, World Engineer Jul 29 '13 at 14:24

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

7  
My guess is that you won't hear a lot of honest answers about people's negative experiences, because TDD is still in somewhat of an "early adopter" state and most people who are interested and committed enough to try it aren't objective enough to evaluate it on its relative merits. It usually takes several years for the industry to really establish the long-term effects of new processes and techniques, and even then it's hard to get straight answers because of the lack of controls. Good question nevertheless, and good luck on getting helpful answers! –  Aaronaught Aug 4 '11 at 15:13
13  
Are you having trouble finding negativity on the internet?! –  Eric Wilson Aug 4 '11 at 15:47
3  
@Job (and possibly other): don't forget he asked about TDD, not unit testing. TDD != unit testing. –  n1ckp Aug 4 '11 at 20:27
2  
I'm tempted to answer this question, but I don't really want to get started because it'll take up half my day. –  Rei Miyasaka Aug 4 '11 at 20:40
2  
When I find myself spending enough time on bugs that it seems I could spend less time writing tests for every single thing that I write before I write it, I will not adopt test-first-all-things-TDD. I will instead hang my head in shame and start searching for a new career. Not about bugs you say? Design? Yes. That's it. That's it exactly. And you haven't learned a damned thing about robust design by giving yourself a safety net and a license to continue working stupid. Put the IDE away and try to read your code if you want to discover the real problem. –  Erik Reppen Mar 9 '13 at 5:31

16 Answers 16

up vote 70 down vote accepted

Like everything that comes under the "Agile" banner, TDD is something that sounds good in theory, but in practice it's not so clear how good it is (and also like most "Agile" things, you are told that if you don't like it, you are doing it wrong).

The definition of TDD is not etched in stone: guys like Kent Beck demand that a non-compiling test must be written before a single line of code and every single line of code should be written to pass a failing test. Up front design is minimal and everything is driven by the tests. It just doesn't work. I've seen a big enterprise app developed using that methodology and I hope that it is the worse code I see in my career (it won't be far off; and that was despite having some talented developers working on it). From what I've seen it results in a huge number of poorly thought out tests that mainly validate that function calls occur, that exceptions are thrown when variables are null and the mocking framework gets a thorough workout (whoop-de-whoop); your production code gets heavily coupled to these tests and the dream of constant and easy refactoring does not appear - in fact people are even less likely to fix bad code because of all the test it will break. In this kind of environment software managers would rather have bad software with passing tests and high code coverage than good software with less tests.

Conversely I've heard people argue that TDD means designing the tests up front on a high level as part of the planning phase - alongside the architectural design. These tests may change during development as more information becomes available, but they have been carefully considered and offer a good guide as to what the code should actually do. To me that makes perfect sense.

share|improve this answer
6  
+1 "designing the tests up front on a high level as part of the planning phase - alongside the architectural design" Sounds a lot more reasonable to me as well. –  Steven Jeuris Aug 4 '11 at 12:45
10  
@Aaronaught Agile doesn't mean no planning, it means just in time planning. –  Adam Jaskiewicz Aug 4 '11 at 13:54
16  
@Adam Jaskiewicz: I love the "no upfront planning" thing. C'mon, planning is upfront by definition. If you don't plan beforehand but during the event you are not planning at all; you are improvising. ;-) –  CesarGon Aug 4 '11 at 16:48
20  
@Adam - "Do people really jump straight into coding on the first day of an iteration?" erm yep. That's "Agile" man. Last place I worked (and got fired from for not being "Agile") they did an entire 3 month release cycle without ever planning a single line of code or doing a single page of documentation. And yes the code was terrible and yes the software was slow, clunky and buggy. The day I joined I was told proudly by the manager that they were "The most Agile shop in London". They sure were. –  user23157 Aug 4 '11 at 18:20
5  
Can add another problem: as long as it passes the test, it mush be good. Never mind that the test itself may well be flawed and thus causes false negatives and/or false positives. And of course requiring "100% test coverage" and anything that has such is by definition perfect, causing useless tests that don't actually test anything but are written solely to achieve that 100% coverage, code that's undocumented because your coverage counter counts comments as uncovered code, etc. etc. –  jwenting Aug 9 '11 at 6:22

This (Clojure author) Rich Hickey interview contains the following. I feel 100 % sympathetic:

Life is short and there are only a finite number of hours in a day. So, we have to make choices about how we spend our time. If we spend it writing tests, that is time we are not spending doing something else. Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design.

Another similar statement from Donald Knuth in Coders at Work book interview, copy-pasted from here:

Seibel: Speaking of practical work, in the middle of working on The Art of Computer Programming you took what turned into a ten-year break to write your typesetting system TeX. I understand you wrote the first version of TeX completely away from the computer.

Knuth: When I wrote TeX originally in 1977 and ’78, of course I didn’t have literate programming but I did have structured programming. I wrote it in a big notebook in longhand, in pencil. Six months later, after I had gone through the whole project, I started typing into the computer. And did the debugging in March of ’78 while I had started writing the program in October of ’77. The code for that is in the Stanford archives—it’s all in pencil—and of course I would come back and change a subroutine as I learned what it should be. This was a first-generation system, so lots of different architectures were possible and had to be discarded until I’d lived with it for a while and knew what was there. And it was a chicken-and-egg problem—you couldn’t typeset until you had fonts but then you couldn’t have fonts until you could typeset. But structured programming gave me the idea of invariants and knowing how to make black boxes that I could understand. So I had the confidence that the code would work when I finally would debug it. I felt that I would be saving a lot of time if I waited six months before testing anything. I had enough confidence that the code was approximately right.

Seibel: And the time savings would be because you wouldn’t spend time building scaffolding and stubs to test incomplete code?

Knuth: Right.

share|improve this answer
20  
I think you have to look at the type of work you are doing. Knuth and Hickey are talking about the design of new (innovative) applications. If you look at where TDD is popular (broad overgeneralization), then you come to the realization that most of the applications written in a TDD manner already have a well known architecture. –  sebastiangeiger Aug 4 '11 at 11:26
3  
I can see what Rich Hickey means, but I would like to add this: My experience is, that when you are writing the tests, you need to really think about your design and think about how you make your code testable, which, in my experience, results in better design. –  Niklas H Aug 4 '11 at 11:51
3  
Seeing as the OP asks for negative experiences with TDD neither of your example seem relevant to this question as neither shows an example of TDD in action. –  Winston Ewert Aug 4 '11 at 15:05
4  
@Michael Borgwardt: It's relatively easy to add tests to an existing, good design, to get rid of bugs in the implementation. But getting rid of bad design often means a complete rewrite. So the primary objective should be in getting the design right; execution is easier to fix later on. –  Joonas Pulakka Aug 4 '11 at 16:18
4  
Most business applications don't have the benefit of being written and/or maintained by Donald Knuth. The lifetime of an application is usually MUCH longer than its primary development. I wish people had written more tests up front on my current project. Now maintenance is like a minefield of infinite regressions. –  Matt H Aug 5 '11 at 0:49

My negative experience with TDD was my first one. TDD sounded great to me, I'd done QA for years and still had the horrors fresh in my mind. I wanted to squash every bug before it made it into a build. Unfortunately, using TDD doesn't guarantee that you'll write good tests. In fact, my initial predisposition was to write simple tests that generated simple code. Really simple code that contained few abstractions. Really simple tests that were intertwined with the class internals. And once you have a few thousand itty bitty tests in place you sure don't feel like you're moving any faster when you have to change a hundred of them to refactor your code to use very important domain concept X.

The light came on for me - TDD is not a testing skill. It's a design skill. It can only lead you to good, simple, workable code with practice and a constant awareness of the design directions it leads you in. If you're writing tests for the sake of code coverage you're going to create brittle tests. If you're writing tests to help you design your abstractions then it's just a more rigorous way of writing top-down code. You get to see the code from the caller's perspective first, which encourages you to make his life easier, rather than mirroring a class's internals to its outside edge.

I think TDD is useful, but I'm not dogmatic about it. If those "no-value tests" are making maintenance difficult - Delete them! I treat tests the same way as the rest of the code. If it can be refactored away and make things simpler, then do it!

I haven't seen it personally, but I've heard that some places track code coverage and test counts. So if metrics gathering is a side effect of TDD, then I could see that as a negative as well. I will enthusiastically delete 1000 lines of code, and if that obsoletes 20 tests and drops my code coverage %, oh well.

share|improve this answer
5  
You nailed it in paragraph 2. –  Sheldon Warkentin Aug 5 '11 at 12:58
4  
"I haven't seen it personally, but I've heard that some places track code coverage and test counts" I've lived in such an environment and indeed no code ever got thrown out because doing so would cause a test to fail. Until I started debugging the actual tests that is, and found a lot of them having such serious flaws they forced code to produce incorrect results in order for the tests to pass. That's when I coined the question: "who is testing the tests" to which so far I've never had a satisfactory answer from the TDD community. Unit tests for unit tests, anyone? –  jwenting Aug 10 '11 at 7:36
2  
@StevenA.Lowe - I know that now, but 9 years ago it wasn't so clear :) "TDD is not a testing skill" –  Steve Jackson Feb 10 '12 at 17:50

I'm going to go out on a limb here and declare with brutal honesty that it's literally a ritualistic waste of time. (In the majority of situations.)

I bought a book on Unit Testing which also discussed TDD, and while I agree with the benefits of UT, after about a hundred hours of trying out TDD, I gave up on it for a myriad of reasons. I'm kind of cross-posting here, but TDD:

  1. Isn't better documentation than real documentation.
  2. Doesn't catch bugs or regressions.
  3. Doesn't really make my designs better than they end up being if I apply some functional programming and composability concepts.
  4. Is time that could be better spent doing code reviews or polishing documentation and specifications.
  5. Gives managers a false sense of security when they see a list of hundreds of green icons.
  6. Improves productivity in the implementation of algorithms with limited input-output mappings.
  7. Is clumsy in that you may know what you're doing as a result of TDD, but you aren't earning any understanding of why it works so well, why your designs come out the way they do.

Another concern is the debated degree of perfection to which one must do TDD to do it successfully. Some insist that if TDD isn't done persistently by everyone on the team from the beginning of the project, you'll only suffer. Others insist that no one ever does TDD by the book. If these are both true, then it follows that TDD practitioners are suffering, whether they realize it or not.

Of course, if it's being argued that by doing things in a TDD-like manner you'll come to designs that can work easily with TDD, well, there are much quicker ways to achieve that -- namely, by actually studying the concepts of composability. There's plenty of resources out there, a lot of rigorous mathematical theory even (largely in functional programming but also in other fields). Why not spend all your TDD time learning?

Culturally, TDD shows symptoms of being a ritualistic practice. It rides on guilt; it encourages procedure over understanding; it has tons of doctrines and slogans ("fake it 'til you make it" is really quite alarming if you look at it objectively). Wikipedia's definition of the word "ritual" is in fact quite apposite:

In psychology, the term ritual is sometimes used in a technical sense for a repetitive behavior systematically used by a person to neutralize or prevent anxiety; it is a symptom of obsessive–compulsive disorder.

share|improve this answer
5  
@jwenting People do TDD because they don't know what makes a design modular, so they can never take their 35" training wheels off of their 40" bikes. Making a modular design is always a manageable task if you understand the concepts, because each dilemma in your design will be of a manageable size due to the fact that it is, in its conception, in the process of becoming modular. I don't disagree that TDD is effective at forcing modularity, but I disagree that one needs to be forced into creating modular designs. Modular design is a perfectly teachable and learnable skill. –  Rei Miyasaka Aug 10 '11 at 9:03
1  
@Kall In that interview, he says "about 15 to 35% more time" -- not just 15% as you've quoted him. The study also only involves Java/C++/C# (probably C# 2, given the date) -- all languages of the same imperative OOP paradigm. I'm certainly more than 15% and probably more than 35% more productive in a functional language (and in C# 3 even), and I produce much fewer bugs writing stateless, composable code -- the same kinds of bugs that testing ameliorates, because both things solve the exact same types of problems. In other words, sure, 60-90% reduction in bugs, but compared to what? –  Rei Miyasaka Oct 28 '11 at 18:59

My primary negative experience is trying to use TDD to edit another programmer's code who has no tests, or very, very basic integration tests. When I go to add a feature, or fix a problem with said code; I would prefer to write a test first (the TDD way). Unfortunately, the code is tightly coupled, and I cannot test anything without a lot of refactoring.

The refactoring is a great exercise anyway, but it is required to get the code into a testable state. And after this step, I have no checks and balances to see if my changes broke anything; short of running the application and examining every use case.

On the other hand, adding features/fixing bugs to a TDD project becomes very straightforward. By nature, code written with TDD is usually quite decoupled with small pieces to work with.

In any case, TDD is a guideline. Follow it to the point where you find you get maximum effectiveness. Decent test coverage and decoupled code, well written code.

share|improve this answer
2  
That's not a negative experience with TDD, that's a negative experience with crappy code. –  Rei Miyasaka Aug 5 '11 at 20:35

I made the experience that I sometimes rely on my tests too much when it comes to the design of the system. I am basically too low in the nitty-gritty implementation details to take the step back to look at the bigger picture. This often results in an unnecessary complex design. I know, I am supposed to refactor the code but sometimes I have the impression that I could save a lot of time by taking the step back more often.

That being said, if you have a framework like rails where your architectural decisions are very limited these problems are basically non existent.

Another problem is when you trust your tests blindly. Truth is - like any other code - your tests can have bugs too. So be as critical towards your tests as you are towards your implementation.

share|improve this answer
2  
Maybe you should write some tests for your tests! :p...... I Kid, I Kid –  Aren Aug 9 '11 at 17:02

To add, another concern with TDD I noticed is:

TDD causes an inadvertent shift in development team's focus from Quality Code to Testcases and Code Coverage! I personally did not like TDD as it makes me less creative and makes software development a dull mechanical process! Unit testcases are useful when judiciously used but becomes a burden when treated the goal of software development.

I know a guy who is a manager and is technically dull once got obsessed with TDD. It was such a magical thing for him that he believed would bring in magical solutions to all the issues in his poorly architectured software with least maintainable code. Not to say what happened to that project-it failed miserably in his hands, while all his testcases were green. I guess TDD helped him get some kind of statistical information like "99/ 100 of my cases are green" etc and that was the reason for his obsession as he would never be able to evaluate the quality or suggest enhancements in design.

share|improve this answer
2  
Sounds like PHB hell! It reminds me of companies that introduce bonus schemes - what happens of course is that developers instead of focusing on quality code, focus on meeting whatever the bonus requirements are. Inevitably you get crappier code. (There's also an analogy here to the current banking crisis :-) ) –  TrojanName Aug 24 '11 at 11:27

I have yet to come across more than one scenario as a game developer where TDD was worthwhile. And the instance in which it was, was a piece of code which was purely mathematical in nature and needed a solid approach to test a huge number of edge cases simultaneously - a rare need.

Maybe something, someday will change my mind, but amongst XP practices, the idea of refactoring mercilessly, and of code evolving its own form are far more important and lead to the greatest productivity for me, cf. a quote from a paper by James Newkirk:

Simplicity - "What is the simplest thing that could possibly work?"
The message is very clear. Given the requirements for today, design and write your software. Do not try to anticipate the future, let the future unfold. It will often do this in very unpredictable ways making anticipation a cost that is often too expensive to afford."

The concepts of courage and of tightening feedback loops which he mentions are also, to my mind, key to productivity.

share|improve this answer
9  
The trouble is, without unit tests how do you know that your merciless refactoring results in code which does the same thing? In my experience, I've found that depending on the problem it can take me less time to write tests+code than it does to write the code on it's own! The reason mainly boils down to the fact that for some problems I can try out a re-factor and retest automatically much quicker than I could test it manually, which can increase the velocity of iterations quite significantly. –  Mark Booth Aug 4 '11 at 12:21
5  
For games, very often the results can be seen. If they can be seen, and appear good enough, they will be accepted, since a game is intended to be a subjective experience anyway. On the other hand, taking eg. Diablo 2 as an example, the number of errors in combat formulae showed where TDD would have brought huge value and saved them vast amounts of work on patches. For very well-defined problems such as solving equations, and where these cannot be judged by visual outputs at runtime, TDD is a must-have to ensure correctness. But that is a small fraction of the code in most games. –  Nick Wiggill Aug 4 '11 at 12:27
2  
Unit tests make refactoring much easier, in my experience. –  Tom Kerr Aug 4 '11 at 14:34
1  
@Nick The big problem in gaming industry are the deadlines, which are always - "we had to deliver half a year ago". I think the time plays against unit-tests in time constrained environments. In some cases it's not a correct decision, but in most cases shipping without test writing is faster. Depends, really depends... –  Coder Aug 8 '11 at 21:53

My negative TDD experience, limited as it may be, is simply knowing where to start! For instance, I'll try to do something TDD and either have no idea where to begin barring testing trivial things (can I new up a Foo object, can I pass in a Quux to the Baz, and the like. Tests that don't test anything), or if I am trying to implement it in an existing project then I find that I would have to rewrite various classes to be able to use them in TDD. The end result is that I quickly abandon the notion entirely.

It probably doesn't help that often I'm the only person in the entire company who knows what unit testing (TDD or otherwise) is and why it's a good thing.

share|improve this answer
1  
This is where mocking frameworks come in. Instantiate Foo with Mock objects rather than Quux and Baz directly, then you can call the function you want to test and then check that the mocks were called with the functions you expect. Mock objects are the enabling technology that helps decouple units and make them unit testable. This is why singletons are evil, since you often can't just Mock them out. *8') –  Mark Booth Aug 4 '11 at 16:32

As a big fan of TDD I sometimes see these drawbacks

  • Temptation to write too many test for the sake of nearly 100% code coverage. In my opinion it is not necessary to write tests
    • for simple property getters/setters
    • for every case when an exception is thrown
    • that checks the same functionality through different layers. (Example: if you have a unittest to check input validation for every parameter then it is not necessary to repeat all these tests through an integration test, too)
  • Maintenance costs of test code for similar tests, that vary only slightly (created through code-duplication (aka copy-paste-inheritance)). If you already have one it is easy to create a similar one. But if you do not refactor the test code, by eliminating code duplication into helper-methods you might need some time to fix the tests if implementation details of you business-code changes.

  • If you are under time pressure you may be tempted to eliminate broken tests (or comment them out) instead of fixing them. This way you lose the investment in the tests

share|improve this answer

If you use TDD from these "fanatic" articles you will get wrong safety feeling that your software have no errors.

share|improve this answer

TDD zealots.

For me, they are just one of a long line of religious loonies knocking on my door, trying to prove that my way of doing things is irreparably broken and the only path to salvation is Jesus, Kent Back, or Unit Testing.

IMO, their biggest lie is that TDD will lead you to salvation a better algorithm design. See the famous Soduku solver written in TDD: here, here, here, here and here

share|improve this answer

TDD have some benefits:

  • You focus on how to call your code and what to expect first (as you write the test first) instead of focusing on solving the problem and then you glue in a call from the application. The modularity makes it easier to mock and wrap.
  • The tests ensure that your program works the same before and after a refactoring. This does not mean your program is error free, but that it keeps working the same way.

TDD is about long term investment. The effort pays off when you reach maintenance mode of your application, and if the application is not planned to reach that point you may never recover the investment.

I consider the TDD red-green-cycle with the baby steps similar to a checklist for a plane. It is annoying and tedious to check each and every thing in the plane before take-off especially if it is trivially simple (the TDD baby steps) but it has been found that it increases safety. Besides verifying everything works, it essentially resets the plane. In other words, a plane is rebooted before every take-off.

share|improve this answer
2  
Benefit point 2 can be achieved with simple unit tests without a TDD approach as well. Benefit 1 you should be doing anyhow. (Focusing on the API) It's still entirely possible to create a crappy API using TDD, but yes, you are guaranteed that it will work (for the written tests). –  Steven Jeuris Aug 4 '11 at 12:34
2  
The question was not asking about the benefits of TDD. There are already plenty of other questions on that. –  Aaronaught Aug 4 '11 at 13:18
1  
@aaronaught, I am addressing his pain points. –  user1249 Aug 4 '11 at 15:05
1  
@aaronaught, then write some of those. –  user1249 Aug 5 '11 at 0:10

My negative experience about TDD is something I feel with a lot of new and hyped stuff. In fact I enjoy TDD because it ensures the validity of my code, and even more important: I can recognize failing tests, after adding new code or any kind of refactoring.

What annoys me about TDD is the fact that there are a lot of rules or guidelines about it. As it is still quite new, we most of us experience to be beginners at TDD. So what works well for some of us, might not work for others. What I want to say is, that there is no real "wrong or right" way to perform TDD: There is the way that works for me - and my team if I have one.

So as long as you write tests - before or after the production code doesn't really matter IMHO - I am not sure if test driven really means you have to follow all guidelines that are stated right now, since they are not yet proven to be the ideal solution for every day work. If you find a better way in writing tests, you should post it in a blog, discuss it here, or write an article about it. So in ten years or so we might have shared enough experience to be able to tell which rule of TDD can be assumed to be good or not in a certain situation.

share|improve this answer

I have, on more than one occasion, written code that I discarded the next day since it was clumsy. I restarted with TDD and the solution was better. So I have not had too much in the line of negative TDD experience. However, that being said, I have spent time thinking about a problem and coming up with a better solution outside of the TDD space.

share|improve this answer

I've found TDD performs poorly when it comes to emergent systems. I'm a video games developer, and recently used TDD to create an system that uses multiple simple behaviours to create realistic-looking movement for an entity.

For instance, there are behaviours responsible for moving you away from dangerous areas of different types, and ones responsible for moving you towards interesting areas of different types. Amalgamating the output of each behaviour creates a final movement.

The guts of the system were implemented easily, and TDD was useful here for specifying what each subsystem should be responsible for.

However I ran into problems when it came to specifying how the behaviours interact, and more importantly how they interact over time. Often there was no right answer, and although my initial tests were passing, QA could keep finding edge cases where the system didn't work. To find the correct solution I had to iterate thorugh several different behaviours, and if I updated the tests each time to reflect the new behaviours before I checked they worked in-game, I may have ended up throwing out the tests time and time again. So I deleted those tests.

I should have possibly had stronger tests that captured the edge cases QA discovered, but when you have a system like this that sits on top of many physics and gameplay systems, and you're dealing with behaviours over time, it becomes a bit of a nightmare to specify exactly what's happening.

I almost certainly made mistakes in my approach, and like I said for the guts of the system TDD worked brilliantly, and even supported a few optimising refactors.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.