Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

There are a lot of very smart and experienced people, who claim that if you don't write unit tests then your code will be buggy and God will kill all kittens. On other hand, another bunch of smart people will say that unittesting is counterproductive. So if you will write unittest then kittens will be damned too.

Also, I heard very good arguments on both sides for and against integration testings, for and against TDD.

Also, there are lot of popular open source projects. Some of them use some automated testings and some of them not.

So, why different testing methods are so contradicting topic?

Update 1

I wrote couple of examples of anti-unittesting sentiments in comments. However, just to be clear there are A LOT of very successful open source projects which doesn't have unittests. This implies that not everybody thinks that unittests are useful.

Update 2

As soon as we start discussing some particular piece of testing (as example mocking, which came up in the comments), we immediately try to distinguish between being a core value or a mean. However, testing by itself isn't a core value. It's just a way to decrease support cost, which in it's turn, just a way to increase profitability. So, as I see it - unittests are something like 4 connections aways from a profit.

Update 3

It's interesting that amount of people in "balanced" camp is quite small. Most of people have really strong opinion on this topic.

share|improve this question

closed as not constructive by Robert Harvey, Martijn Pieters, Justin Cave, Eric King, Caleb Feb 28 '13 at 5:15

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

4  
Who says unit tests kill kittens? That never struck me as a popular opinion. –  delnan Feb 27 '13 at 20:27
1  
BTW. Apache HTTP Server doesn't have unittests. Are they wrong? –  Victor Ronin Feb 27 '13 at 20:35
2  
Mocking is overused. You have to be very, very good at it to maintain any semblance of productivity, and my impression is that much of it happens because of over-engineering. –  Robert Harvey Feb 27 '13 at 20:38
1  
Interesting study about TDD theruntime.com/blogs/jacob/archive/2008/01/22/… –  Ryathal Feb 27 '13 at 21:24
1  
I think part of the polarisation is that comprehensive unit testing is almost an "all or nothing" commitment (and I'm not referring to 100% coverage, but rather the approach / architecture in general). The decision one way or another will heavily impact your architecture, so it's not to be made lightly. The knowledge required to make it work (well) is non-trivial, requiring much investment of time, and this probably raises the barrier to entry (as well as causing many failed attempts -> more polarisation). –  Daniel B Feb 28 '13 at 8:19
show 9 more comments

7 Answers

up vote 5 down vote accepted

My first IT job was as a manual tester on an application. Within two months, I moved to an automated front-end tester. Now, on my app work, I write Unit tests to test my code. In my short career, I've pretty much been all over the app testing horizon.

The reason I bring that up is because every type of testing has value. On the first app with my first employer, we had an app with 670k lines of code, and only moderate Unit Testing coverage, which pretty much justified that job. That being said, a fair number of bugs we discovered and documented had a Unit Test created for them. Contrast to my current job; a project I am currently working on has < 2k lines of code, yet about 80% unit test coverage. Our testers haven't yet pounded away on it, yet I have a feeling that when they do, my team will still have our share of suprises. It's inevitable.

The problem with any type of testing is, it can never be 100% complete. You will asymptotically close on 100% coverage through any venue; the closer you get to a fully testable application, the closer you are incapable of getting to a fully-testable application. That thought in mind, I've grown to adopt the mindset of, 'what does a particular type of testing do to ensure the quality of my app?'

Unit Testing: Ensures that internal dependencies and complex interactions work as designed.

Automated Front-End Testing: Regressively ensures that recent changes don't break known-good features

Load/Performance Testing: Discovers issues that arise from large numbers of concurrent users.*

Manual Front-End Testing: Discovers issues that arise as a result of emergent behavior.

I have actually found bugs from executing Load Testing. Actually, you'd be suprised what you can find with a well-formed Load/Performance Test.

Answer to original question: both 'camps' make the same argument that some of us have encountered through blood, sweat, and tears: any sort of testing has its limits. However, those two camps you encountered are mutually wrong, because to ensure quality, all testing is necessary testing.

I think the controversy comes from resources. Not everyone has resources to perform all types of testing, yet I think we all know at some level that it would really help if we did. I see the problem of this controversy is less one of technology, and more one of the human element.

No kittens were harmed in the acquisition of this experience.

share|improve this answer
1  
Thanks. Really interesting answer. However, I partially disagree that controversy comes from resources. Yeah... people tend to say, if I have more time I would implemented tests. However, I believe in reality, one camp says something like 'good unittests will decrease time to deliver" (so doing it is benefitial with any amount of resources) and other camp says "unittests won't save as any time" (so doing it is counter productive on any amount of resources). –  Victor Ronin Feb 27 '13 at 22:36
    
@VictorRonin Perhaps I misunderstand, but the argument about disagreeing that controversy comes from resources appears to be affirming it (or, I worded my thought extremely obtusely, which I did.) Where I'm talking about 'actual' available resources for the spending, though, I think your answer deals with 'perceived' available resources for the spending, which is actually way more interesting than my answer! –  Andrew Gray Feb 27 '13 at 22:38
1  
Yeah.. Probably we were talking about a little bit different resources. You are talking from a perspective of manager, who looks at the schedule and says that we can allocate 120 hours of testing. And within this 120 hours people have to choose where to apply them. I am talking from a perspective of a person who wasn't give a quota and have to decide on his own what would bring the most value for his 8 hours work day (he can choose both to do testing or to skip it). –  Victor Ronin Feb 27 '13 at 22:54
add comment

I try to take a balanced approach. On my personal projects, when company policy doesn't mandate tests, I don't test everything, but I will add unit tests the second time I fix a similar bug. Complex areas of my code end up with very good coverage, and the rest of the code is hardly covered at all. Perhaps that makes me a neutral party, if such a thing is possible in such a polarized topic.

I see the differences as simple personality conflicts. Some methods work better for some people. Testers like the methodical structure being added, and consider the time it takes to write and maintain tests as offset by productivity gains. The manual double checkers have good instincts and don't tend to find many additional bugs with unit testing, and therefore consider the time it takes to write and maintain tests as not adding much value, especially since they tend to manually double check anyway.

share|improve this answer
    
First of all, I like to see somebody with a balanced approach. I think I am in this camp. Second, I wonder why it's so polarized. Is this because there is no way to measure end result properly? –  Victor Ronin Feb 27 '13 at 20:55
add comment

I think the basic issue is that unit testing is an often-poorly-understood tradeoff. I write a lot of unit tests, but not for everything. There are many things I don't think should have unit tests.

The basic thing you have to understand is that unit tests are regression tests. If a test fails, it does not mean there is a bug, and if a test passes it does not mean there isn't a bug. If a test that was passing is now failing, it means, simply, that something changed. It's an invitation to look and make sure that the change is correct and the test case was wrong, but it is an invitation only. Additionally I have found that there are often many unit tests which get coded poorly and so they break in unexpected ways and one has to follow these back and find out why. When the answer is, "We tested for the English-language OS-level message and we got a localized message from the OS in German" the correct response may, in fact be, to just get rid of that unit test since we can show it is not testing what it is supposed to test and there may not be an easy way to fix it.

In general, the more carefully spec'd out a portion of the code is, and the tighter the contract, the more important it is to unit test. Such test failures will often show you contract violations rather than bugs. Portions of the code which are not well spec'd out, subject to frequent change, and not part of a public API do not need as many (of in some cases any) unit tests. Manual testing there is ideal because you get real user feedback and automated tests will give you many false positives and false negatives.

In LedgerSMB we aggressively test things like number formatting and parsing, heavily test (but not as heavily) our database routines to varying degrees depending on, of course, how well defined the desired behavior is and how complex the code is. Something like a cash-basis income statement or a 1099 report gets a lot of testing. Something like saving a customer record gets light testing. The user interface layer gets light testing which amounts to "got an HTTP success and did not get an error page from the application" and test coverage there is unlikely to ever be complete.

But let me give the other side, namely LedgerSMB before the unit tests and after the fork. We forked from a codebase (of what was then a successful open source project!) which had no unit tests, and where most of the code cannot be tested because it relies on sloppy scoping. When we started adding unit tests the first ones we added were something simple: rounding numbers (keep in mind this is an accounting and ERP application, where rounding numbers is important). We got significant numbers of failures and we spent a fair bit of time correcting those failures. That single experience taught me that not to trust code which has no unit tests at all.

Unit tests are an important tool but "you must write unit tests for everything" is counterproductive. Similarly a lack of unit tests is an indication that there is a lack of, if you will, contract enforcement, and that behavior of some parts of the codebase might be very ill-defined. In the end, the tradeoff of unit testing a given portion of the code needs to be better understood and articulated, and the tradeoff is really that of rigidity in contract. There are many places where you want to test and many places where testing just gets in the way of making desired changes over development process.

share|improve this answer
add comment

The people who believe in unit testing fall into two camps. The "Having good coverage saved my ass from a bug that would have been a show stopper if that test wasn't there" camp and the "Everyone says Unit Testing is a good thing and so I say it is too." (Usually the second camp is paying lip service and not doing TDD until one day they decide to really do it and inevitably transition to the first camp).

The people who don't believe in unit testing see more detriment (extra time writing and maintaining unit tests) than benefit (possibility of catching bugs early).

What they fail to realize is the freedom TDD gives you to maintain/update/change your code. Knowing that you can verify that the change you made didn't create an unwanted side effect by running your unit test suite allows you to make changes more rapidly. In essence you're paying an up front cost to build the tools that will allow you to go faster as the project grows.

share|improve this answer
1  
Agreed, with two caveats: 1. passing unit tests doesn't actually prove that you didn't introduce a bug, it only verifies the behaviors that you codified in the unit tests; and 2. you may have to change your unit tests anyway as a result of refactoring. –  Robert Harvey Feb 27 '13 at 20:51
1  
You are putting here everybody who don't use unit tests in just one bucket - "fail to realize is the freedom TDD gives you to maintain/update/change your code". Why in such case Apache HTTP Server doesn't have unittests? I tlooks like it had a lot of changes across decade and I am pretty sure a lot of very experienced developer worked on it. –  Victor Ronin Feb 27 '13 at 20:51
    
@RobertHarvey I agree with both of those points...I actually considered those facts but glossed over them in the final answer. –  Mike Brown Feb 27 '13 at 20:54
    
@VictorRonin I'm grossly generalizing here. (The "Everyone says..." camp was actually tongue in cheek...mostly). Other reasons for forgoing unit testing "my code is that damned good", "I don't have the time", "when this project was first conceived, unit testing didn't exist and who the hell is going to write tests for such a complicated codebase?" –  Mike Brown Feb 27 '13 at 20:58
add comment

So, why different testing methods are so contradicting topic?

What's the old joke? "The only people who don't love it are those who haven't tried it."

I don't mean to be snarky, but I don't know of anyone who worked in an environment where there was ubiquitous, small, isolated, speedy, repeatable, maintained automated tests and then argue against their utility in non-throwaway code.

That said, there are many developers who have worked in environments where one or more of those criteria failed. Tests suck when they're not trustworthy. They suck when they take forever. They suck when they're fragile. There are tons of ways to say you're doing unit tests, and not actually do unit testing.

So yeah, many smart people will argue against what they know to be unit testing, which is a huge waste of time. Mostly though, the arguments I get against unit testing are from older developers than even me - "we've made quality software for decades without unit tests". They fail to see the costs needed to achieve that quality, or how much more quality software they could have made by spending less time doing integration and bug hunting.

share|improve this answer
    
"there was ubiquitous, small, isolated, speedy, repeatable, maintained automated tests". I can see two problems with that. All these characteristics depends on type of application, programming language, platforms, frameworks used. The best case scenario (where all of them align) is reasonably rare. Also, even if stars aligner perfectly, the question whether investment will pay off still applies. –  Victor Ronin Feb 27 '13 at 21:56
    
@VictorRonin - No, not really. These characteristics can exist in almost every single application ever written. Maybe not for the entire application, but nobody argues for 100% coverage. –  Telastyn Feb 27 '13 at 21:59
    
There is one more characteristic - time. yes, probably you can create nice unit tests for any application on any platform. However, in some cases it will be very beneficial and in some cases it will be very wasteful. –  Victor Ronin Feb 27 '13 at 22:21
    
@victorRonin - not in my experience. The time saved from making code testable (and thus more maintainable and flexible against changing requirements) and implementing the automated tests (saving debugging, manual testing, regression testing, and smoke testing time) is pretty much always worth it (though I am sure there are exceptions if you look hard enough...). But I've seen good unit testing done in 5 significantly different codebases now - it is not some mythical beast. –  Telastyn Feb 27 '13 at 22:46
    
I have thrown out unit testing for some portions of an application, just not the whole one. I think a lot of the controversy has to do with extremes. At their best, unit tests are contract testing not code testing and that's the secret to using them productively. –  Chris Travers Feb 28 '13 at 1:38
show 1 more comment

To understand where the two wildly different viewpoints come from, you need to have some understanding of history.

The concept of unit testing originated in the Smalltalk community. There are a lot of differences between Smalltalk and the languages that most people build large projects with these days, but the relevant one here is that Smalltalk is dynamically typed. And specifically (so that this doesn't end up turning into a static-vs-dynamic flamefest, which is not the intention here,) it has no compiler capable of verifying type correctness prior to execution. This is the primary problem that unit testing was originally created to solve: catching details like type errors that the compiler would have caught for you, if you'd been able to use one. And that makes a lot of sense, when you don't have a compiler available.

The problem is, a bit later down the line, a bunch of Java folks (who did have a compiler and a strict type system available) heard about the idea. Only they didn't really understand it, most likely because the problem it's designed to solve does not exist in Java. They thought, "oh, look at this! We can test our software's functionality before we use it, and that will help catch bugs!" even though the type of bugs that unit testing was invented to catch don't happen in Java.

Java being very popular, the idea got picked up and spread quickly, and the claims about it continued growing more and more grandiose, until eventually we ended up with ridiculous things like Uncle Bob claiming that, if you write all your code with a proper testing methodology:

think about what would happen if you walked in a room full of people working this way. Pick any random person at any random time. A minute ago, all their code worked.

Let me repeat that: A minute ago all their code worked! And it doesn't matter who you pick, and it doesn't matter when you pick. A minute ago all their code worked!

Just in case you didn't catch the claim there, unit testing is supposed to be a magical thing that will make all your code work!

This is complete nonsense, of course. If all of your tests pass, that means that all of your tests pass, nothing more, nothing less. And specifically, it does not mean that:

  • your tests correctly describe what the code is supposed to be doing in the first place
  • your tests cover every possible error case
  • your tests, being code too, are free from errors themselves. (Follow that one to its logical conclusion and you get stuck in infinite recursion. It's tests all the way down!)

Also, bear in mind that your unit tests only cover the things you thought to write tests for. These are, of course, the things that were on your mind when you were writing them, areas where you anticipated would likely cause problems. These are areas you're paying attention to, which you're likely to get right in the first place. As any experienced developer will tell you, the majority of bugs (not all, of course, but a good majority) come from cases you never anticipated, which is why they turn out buggy. Tests are far less useful there.

There's another serious problem with unit testing, one of those issues that you don't really notice as a problem until your project gets really big and then it's too late, which is that changing requirements requires changing code, which breaks your tests. Sometimes it can break a non-trivial fraction of your tests, like 10% of them. And if you follow Uncle Bob's advice and write "every year... thousands of tests" like he exults about, eventually you have a whole lot of tests, enough that 10% of that whole lot is still a whole lot.

And the thing is, you just intentionally changed something. Unit testing is supposed to help you not accidentally make a change that breaks something, but you just changed something intentionally. And now you have hundreds or thousands (or more) of failing tests screaming at you, and you have to go through each one and individually, manually verify whether or not that test is still valid according to the new requirements. (And for the ones that aren't, you might well have to write new tests to replace them!)

And the interesting thing is, long before Uncle Bob started making his ridiculous claims about unit tests magically making all your code work, long before unit testing spread to Java and from there to the rest of the programming world, long before Java even existed in the first place, the whole concept was already known to be a silly idea by people who actually understood programming!

It is now two decades since it was pointed out that program testing may convincingly demonstrate the presence of bugs, but can never demonstrate their absence. After quoting this well-publicized remark devoutly, the software engineer returns to the order of the day and continues to refine his testing strategies, just like the alchemist of yore, who continued to refine his chrysocosmic purifications.

-- Edsger W. Djikstra. (Written in 1988, so it's now closer to 4.5 decades.)

The reason why there are two wildly different viewpoints out there regarding unit testing is that some people have enough perspective, and enough knowledge of history and/or of the fundamentals of computer science and engineering, to understand these principles and see the basic truth behind all the snake oil.

And some people don't. What they see is that on some small project they've been working on, at some point they happened to catch a nasty bug one time with an automated test. Or they know someone who did. Personal experiences loom large in our memory--that's basic human psychology. That's the thing that they really remember, and so they think it has to be some wonderful, magical thing, logic and reason notwithstanding.

They hold to the concept of unit testing with dogmatic fervor, and like any dogma, its Absolute Truth is necessarily stronger than any fact. Just look at how often TDD proponents pull out the old "if you don't like it/if it doesn't work for you, you must be doing it wrong" argument when confronted with the facts. That's not a logical argument; that's an emotional attack meant to shut down logical discussion. And it works far too well (again, basic human psychology,) which is why people keep using it.

share|improve this answer
    
I read this rant a few months ago and it was just as wrong then as it is now. Nobody, not Uncle Bob, or anyone else, has ever once stated it magically makes your code work. It always is predicated on the fact that your code is in a working state as per the understood requirements. All unit tests really do is verify that behavior is preserved. That's it. It's not difficult. Function a when given argument b produces result c. You made a change somewhere that wasn't intended to change the result of a when given b. Did it still return c? Good. Behavior preserved. –  Anthony Pegram Feb 27 '13 at 23:02
1  
Frankly, I find the confirmation of preserved behavior to be extremely valuable. Oh, and if you ever have a single change that breaks thousands of unit tests or 10% of tests or whatever the unseemly high number is, that suggests something more about the coupled nature of the code or low isolation of the tests. –  Anthony Pegram Feb 27 '13 at 23:03
    
@Anthony: I provided a link to the article that makes the claim, just in case someone wanted to try and say I was taking it out of context. Feel free to read the article yourself. It says exactly what it says: write your tests, and all your code works! And it can't be "predicated on the fact that your code is in a working state as per the understood requirements," and yet at the same time also serve as a demonstration of the same. That's like asking Bob to pick up Alice and then (while he's still holding her) asking Alice to pick up Bob. –  Mason Wheeler Feb 27 '13 at 23:04
    
You are simply make a dramatic leap in logic that was not intended. Out of context, it fits your viewpoint, but it is still a leap. I'm reminded of Inigo Montoya. "I don't think that word means what you think it means." –  Anthony Pegram Feb 27 '13 at 23:07
    
@MasonWheeler: It's interesting to see someone from the camp who are against unittest's and TDD. I agree with Anthony Pegram that most likely code is too coupled and tests aren't well isolated, if some changes breaks 10% of tests. However, I agree with you that passing tests mean only that tests are passing and it's not guarantee that overall productivity will improve because of the tests. –  Victor Ronin Feb 27 '13 at 23:13
show 6 more comments

Unit test is a good alternative

Perhaps unit testing is hard or costly, but it is a good practice that has been widely embraced by our profession for having many advantages. Unit test:

  • Brings the discovery of defects to the developer sooner with less visibility of those defects to others within the organization, let alone customers.
  • Make developers more responsible for testing vs. a loosely defined developer test role.
  • Permits whitebox/glassbox testing in ways only developers can perform.
  • Broadens the base and diversity of people who contribute to the overall test automation used in the product.
  • Supports continuous integration (CI) by permitting newly introduced defects to be automatically detected at the earliest opportunity.

On this last point, I recently saw a case where a driver developer made a change that had positive impact on the driver, but components from other groups had widespread failures. Without a pool of unit tests that ran automatically, the breakage would have been detected much later.

Software Development Thrives on Alternatives and Tools

Agile development is full of alternatives:

  • Iterative/incremental vs. waterfall or spiral software life cycle models.
  • Pair programming vs. inspections/code reviews.
  • Developer gathered stories vs. system engineer driven formal requirements.
  • More developers who unit test vs. more functional / regression testers.

As a developer embracing unit test you move against the tide of applying industrial revolution concepts like division of labor to software and instead embrace the change in which developers have a richer and more complete involvement with crafting their deliverable. This is also true when a developer helps annotate user stories or acceptance criteria in collaboration with customers.

Evolution and Complementary Options

Often, questions on SE Programmers question the value of broadly accepted practices. We are free to develop options by both evolving our tools and introducing new techniques. Source control tools evolved significantly from SCCS to CVS to SVN to Git. Software testing has as well, with test runners, assertion based testing often in conjunction with constant integration servers being one step along the way toward things that will be even more effective.

Other options that complement the benefits from unit test include:

  • Static checkers.
  • Code coverage tools.
  • Code inspections and reviews.
share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.