Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This question already has an answer here:

It seems to be generally assumed (on Stack Overflow at least) that there should always be unit tests, and they should be kept up to date. But I suspect the programmers making these assertions work on different kinds of projects from me - they work on high quality, long-lasting, shipped software where bugs are A Really Bad Thing.

If that's the end of the spectrum where unit testing is most valuable, then what's the other end? Where are unit tests least valuable? In what situations would you not bother? Where would the effort of maintaining tests not be worth the cost?

share|improve this question

marked as duplicate by gnat, Martijn Pieters, MichaelT, Glenn Nelson, MainMa Mar 3 '13 at 13:24

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2  
"[in Big Ball Of Mud kind code] ...unit tests, these do things opposite to how I use them in good code, like breaking at reasonable changes and failing to catch the real mistakes I make. Which is painful but not surprising - what else would one expect from testing units which have bad design to start with? Instead of helping you improve the design, unit tests often work to preserve bad code..." - I wrote about that in another answer –  gnat May 3 '12 at 11:39
4  
@gnat That depends on the direction the testing comes from. Unit tests are especially brittle when they delve into minutiae and when written after implementation, so unit testing becomes a fruitless catch-up exercise and therefore become very expensive to maintain. Targeted testing of legacy code with judicious refactoring can help, but only to a certain degree. OTOH, a green field behavior-driven project tends to result in tests that preserve the requirements instead of the tests, because changed requirements necessitate the removal of older unsuitable tests that no longer match requirements. –  S.Robins May 3 '12 at 12:06
2  
This question asked something similar, and the conclusion was that it was not worth doing it for them, as they would get payed to fix bugs later: programmers.stackexchange.com/q/64592/1249 –  user1249 May 3 '12 at 12:11
show 5 more comments

16 Answers 16

up vote 55 down vote accepted

I will personally not write unit tests for situations where:

  1. The code has no branches is trivial. A getter that returns 0 doesn't need tested, and changes will be covered by tests for its consumers.

  2. The code simply passes through into a stable API. I'll assume that the standard library works properly.

  3. The code needs to interact with other deployed systems; then an integration test is called for.

  4. If the test of success/fail is something that is so difficult to quantify as to not be reliably measurable, such as stenography being not noticeable to humans.

  5. If the test itself is an order of magnitude more difficult to write than the code.

  6. If the code is throw-away or placeholder code. If there's any doubt, test.

share|improve this answer
7  
Overall I agree, +1. But regarding #1: Branchless code doesn't have to be trivial, and even if it currently is due to fortunate circumstances, the intended behaviour may still warrant a test. –  delnan May 3 '12 at 12:32
25  
If the test itself is an order of magnitude more difficult to write than the code. Actually I get that feeling most of the time :P –  Songo May 3 '12 at 12:34
7  
@CarlManaster I think that's the whole point. If it's difficult to write a test case for, it's consequently difficult to understand and will likely create many bugs in your program. The trick is how to avoid methods like these. –  Neil May 3 '12 at 14:41
4  
+1 for integration tests over unit tests. If you are interacting / retrieving data from an external system, then it is actually counter-productive and may introduce bugs to have stand-alone tests that use "mocks." –  noahz May 3 '12 at 15:04
14  
"The code needs to interact with other deployed systems; then an integration test is called for." - that's what mock objects are for –  BlueRaja - Danny Pflughoeft May 3 '12 at 15:41
show 26 more comments

Unless you are going to write code without testing it, you are always going to incur the cost of testing.

The difference between having unit tests and not having them is the difference between the cost of writing the test and the cost of running it compared to the cost of testing by hand.

If the cost of writing a unit test is 2 minutes and the cost of running the unit test is practically 0, but the cost of manually testing the code is 1 minute, then you break even when you have run the test twice.


For many years I was under the misapprehension that I didn't have enough time to write unit tests for my code. When I did write tests, they were bloated, heavy things which only encouraged me to think that I should only ever write unit tests when I knew they were needed.

Recently I've been encouraged to use Test Driven Development and I found it to be a complete revelation. I'm now firmly convinced that I don't have the time not to write unit-tests.

In my experience, by developing with testing in mind you end up with cleaner interfaces, more focussed classes & modules and generally more SOLID, testable code.

Every time I work with legacy code which doesn't have unit tests and I have to manually test something, I keep thinking "this would be so much quicker if this code already had unit tests". Every time I have to try and add unit test functionality to code with high coupling, I keep thinking "this would be so much easier if it had been written in a de-coupled way".

Comparing and contrasting the two experimental stations that I support. One has been around for a while and has a great deal of legacy code, while the other is relatively new.

When adding functionality to the old lab, it is often a case of getting down to the lab and spending many hours working through the implications of the functionality they need and how I can add that functionality without affecting any of the other functionality. The code is simply not set up to allow off-line testing, so pretty much everything has to be developed on-line. If I did try to develop off-line then I would end up with more mock objects than would be reasonable.

In the newer lab, I can usually add functionality by developing it off-line at my desk, mocking out only those things which are immediately required, and then only spending a short time in the lab, ironing out any remaining problems not picked up off-line.


TL;DR version:

Write a test when the cost of writing the test, plus the cost of running it as many times as you need to is likely to be less than the cost of manually testing it as many times as you need to.

Remember though that if you use TDD, the cost of writing tests is likely to come down as you get better at it, and unless the code is absolutely trivial, you probably end up running your tests more often than you expect.

share|improve this answer
1  
Exactly this! I have colleagues who time after time say "I'm done with the ticket, but now I have to write some unit tests." Those unit tests often end up bloated, awkward, and difficult to maintain. –  waxwing May 3 '12 at 21:16
1  
dude, you speak my mind and my experience exactly. Recently I just bite the bullet and wrote mock object for some legacy API for my unit tests because it takes about 10 minute to upload my code to server, deploy and test. –  Alvin May 3 '12 at 23:40
2  
Ok, but your answer is "TDD is great!" and my question was "when wouldn't you write unit tests?" Do you really never do any coding where unit testing is inappropriate? (That is, ignoring situations where you'd like to, but can't.) –  Steve Bennett May 4 '12 at 8:02
2  
@SteveBennett - I answer that the first section, you don't write tests if the cost of manual testing is very low and/or the cost of writing tests is very high. TDD helps shift the economics involved by making writing tests cheaper and running tests routine. Hopefully the summary I've added at the end helps to make that clear. –  Mark Booth May 4 '12 at 8:40
show 7 more comments

Personally, I generally do not write unit tests for:

  • Changes to existing non-unit-tested (legacy) code. While theoretically it's never too late to start unit testing, in practice the gains from adding unit tests to code that was never TDDed are small. The sole advantage is regression prevention; you can see what effect a change makes on areas you may or may not have expected it to affect. However, the unit test only proves that the code still works the way it did when you wrote the test; because the test wasn't written based on the original requirements, the test does not prove the current implementation is "correct". Making unit-tested changes to such a codebase is even more of a nightmare in my experience then simply making the change and manually exercising affected areas of the program; you first have to write a proper unit test for the code in question (which may require refactoring the code to allow for mocking and other TDD-friendly architecture), prove it passes AND that the refactor didn't break anything (and when this is the first unit test ever written for a million-LoC codebase, that can be difficult), then change the test to expect something different and change the code to make the changed test pass.

  • UI Layout and top-level behavior. In some cases it's just not feasible; many IDEs with "designers" declare UI elements as private, and they shouldn't be public conceptually. Making them internal would require putting the unit tests into the same assemblies as production code (bad) and making them protected would require a "test proxy" that exposes what we need to know about the controls (also generally not the way it's done). Even if it was acceptable to make all controls public and thus have access to their type, position, formatting properties, etc, tests that verify the UI is designed the way you expect are extremely brittle. You can usually still unit-test event handlers in codebehinds/viewmodels and then it's trivial to prove that clicking the button fires the handler.

  • Database Persistence. This is, by definition, an integration test. You always mock the repository when dealing with code that passes data to the repository, and you mock the ORM's hooks when developing within the repository itself. While there are "database unit tests" and you can "mock" a database using something like SQLite, I find that these tests just never fit at the unit level.

  • Third-party code. You test YOUR code however you want, and trust that the developers of the third-party tools you're using (including components of the .NET Framework itself like WPF, WCF, EF, etc) did their job right, until proven wrong. A unit test that proves you make the correct WCF call does not require actually crossing the service proxy, and it is unnecessary to test that a particular WCF service call works correctly in a unit test (whether it works or not depends on environment-specific variables that may be perfect in a local testing environment but not at the build-bot, or in the QA, Staging or UAT environments; this is where higher-level integration, AAT and manual testing still have value).

In almost every other case, a test-first methodology has significant value in enforcing requirements and YAGNI, preventing regression, identifying requirements conflicts, etc. Even in cases of "trivial" code, that verification that "yes, this property does still return this value and so you can trust that in other tests" has value to me as a developer.

share|improve this answer
1  
One more case: A routine that returns something random, especially something complex and random. This is very rare in the business world, in the game world plenty of programs will have such routines in them. –  Loren Pechtel May 5 '12 at 0:50
5  
-1. If you don't write tests for existing code, then how can you have any confidence that your code works?? You have 0% code coverage, or no confidence at all that your existing code works. –  CodeART May 5 '12 at 14:28
3  
If I am working in a legacy codebase, it already has 0% code coverage. Am I confident it works? Well, several major in-house applications have been in use through three iterations of the company through M&A, so I'd have to say yes. While adding unit tests to these codebases is a noble goal for regression, the tests only prove the code works the same way it always has, not necessarily that the results are "correct". I stand by my original statement but have edited to clarify the specific situation. –  KeithS May 7 '12 at 15:52
show 5 more comments

You should really have tests for any code that you intend to run on more than 1 occasion.

If you intend to run it once then throw it away, tests may not be useful, though if the domain is complex, I might well write the tests anyway to help me with my design and implementation.

Otherwise, even if the only thing the tests do is confirm that your assumptions are still correct, they are valuable. And any code that lasts more than a single session will need modifications, at which point the tests will pay for themselves 10 times over.

share|improve this answer
2  
I'd appreciate an insight into why somebody down voted this post. –  Bill Michell May 3 '12 at 13:38
11  
Did not down-vote, but I would offer that the idea that all code must have a test is border-line religious, and not always practical. –  noahz May 3 '12 at 15:03
2  
@noahz Yes, I'll buy that. But on the "must, should, would" scale, I selected "should" and not "must"... –  Bill Michell May 3 '12 at 16:34
show 4 more comments

Where are unit tests least valuable?

You should be writing unit tests only for logical code. Definition of logical code is below:

Logical code is any piece of code that has some sort of logic in it, small as it may be. It’s logical code if it has one or more of the following: an IF statement, a loop, switch or case statements, calculations, or any other type of decision-making code.

If that's the end of the spectrum where unit testing is most valuable, then what's the other end?

  • Tightly coupled code can't be easily tested. Sometimes it can't be tested at all. For example ASP.NET web forms are a nightmare and sometimes impossible to test, as opposed to ASP.NET MVC which is built with unit testing in mind.

  • I may choose not to write tests for very small and simple projects. This is perfectly justifiable in a world with limited time and resources. Yes, it's risky, but it's perfectly acceptable when the company you work for has a limited time to market.

What situations would you not bother? Where would the effort of maintaining tests not be worth the cost?

Once unit tests are in place, their maintenance cost becomes very insignificant. Unit tests with high code coverage give you a valuable sense of confidence that system behaves how you expect it to. The same applies to integration tests, but that's not your question.

My answer is based on the following assumptions:

  • You know difference between unit, integration and functional tests.
  • You are proficient with one of the unit-testing frameworks, and it takes you very little effort to write a unit test.
  • You don't feel like you are doing extra work when you are writing unit tests.

Reference: Art of Unit Testing

share|improve this answer
3  
This answer (in light of the Q) would really benefit from an example of code (as in: a function / a class / ...) that isn't "logical". –  Martin Ba May 3 '12 at 11:41
1  
Auto property isn't logical, unless it encapsulates some logic inside, i.e. if/else statement. –  CodeART May 3 '12 at 11:56
show 3 more comments

Kent Beck (the man behind the idea of Unit Testing) says that it's fine not to write Unit Tests when working on an idea (eg. prototype) that you might throw away (if you decide not to work further on). On the contrary, if you write your prototype and decide to go for it, Unit Testing will be helpful for all the reasons covered in the previous answers.

share|improve this answer
show 1 more comment

There is a trade-off between the cost of writing unit tests, and the value they add. Neglecting this risks the possibility of over-testing.

blueberryfields touched on one variable that affects the value of unit tests: stability of the project (or pace of development). I'd like to expand on that.

The value to having unit tests roughly follows the following plot (yay, MS Paint!):

utility vs stability

  1. In the first zone, the code is unstable to the point that unit tests will often need to be rewritten or entirely scrapped. This might be a prototype that you know won't be used. Or it might be a project where you are new to the programming language or framework (and are often needing to redo large amounts of code).

    Unit tests are good for refactoring at the module level (i.e. changing functions). The problem is when code is unstable at the architectural level, where entire modules might be rewritten, removed, or completely redesigned. The unit tests then must be changed/removed/added, which is a burden.

    At this stage, limiting to system level integration tests, or even manual testing is a better approach (which one you choose again being dependent on the cost of writing the test vs. the value you get).

  2. The unit tests add the most value in the second zone, where code becomes stable above the module level, but not yet at the module level. If you have a stable codebase, and you are adding a module that has a high chance of staying, this is when unit testing is a big win.

  3. Unit tests also begin to have less value when the stability of the code under test is very high. Testing a simple wrapper of a standard library function is one example where unit testing is overkill.

    If a piece of code you are writing will likely never need to change, and you are confident you've got it right (maybe it's super simple?), there is less value to testing.

Unit testing becomes beneficial when the utility is greater than the cost, as shown in the below graph in zone II.

utility and cost

This cost line could be at different levels for different people (line 1 vs line 2). If you are new to the test framework used in a project, this increases the level, and makes the net utility less. A higher cost (line 2) shrinks the range that unit testing is appropriate.

The cost line doesn't necessarily need to be flat. Especially when you consider other variables such as project size, and whether a test framework already exists for the project, this might be so.

share|improve this answer
show 3 more comments

Sometimes you don't have time to unit test. That's okay.

External constraints exists and you may not feel the need for tests in a given moment, as you may think you have a firm grasp on the whole system state and know how changes will ripple trough it.

But I still believe you should feel a little guilty about it. So when things start to fall apart, you will know it's because: "oh, yeah, I'm not testing stuff, let's start testing."

Before that moment, the feel of guilt will help you avoid certain architectural choices that would prevent you from being able to automate code tests later.

share|improve this answer
4  
...and btw, I'm up to my neck in guilt right now. –  ZJR May 3 '12 at 16:38
1  
you don't have time to unit test - huge BS. So, no time to unit test, but there is always time to fix bugs. –  BЈовић May 3 '12 at 19:38
show 3 more comments

I have no clue about unnecessary but I'll shoot for when I think Unit Tests are inappropriate in that they add cost without proportional benefit:

If our organization / dev team isn't doing any unit tests, then unless you are able to convince your fellow developers of running, fixing (and also creating on their own) unit tests, the tests you create aren't going to be run and maintained. (This implies you are working on a piece of software together with someone else.)

Not everyone is doing unit tests, they're sometimes shoddily supported by local team culture, and in such environs you first need to get them in place for the majority of devs and probably also for the build-process ("[continuous] integration") before actually starting to write unit tests makes sense.

Given time pressure and delivery dates and "your" relative standing with the team -- the time remaining until writing unit tests makes sense can easily be months or years.

share|improve this answer
1  
+1 for mentioning the maintenance problem. Though I've used unit tests with success as the only dev using them. Only, when I was on a different project for a few months, it took weeks until all the unit tests worked again... –  sleske May 9 '12 at 8:58
add comment

The cost isn't in maintaining tests, but rather in not maintaining working code. If you divorce testing and implementation, you are assuming that what you write will work without the benefit of proof. Tests give you that proof, and the confidence to maintain the code as time passes. Sure you might think that you'll never revisit that app you thought was a throw-away, however the reality is that any time you spend writing software is an investment that your employer doesn't want to treat lightly. Let's face it, software developers don't generally come cheap, and time spent on code to throw away is money simply burned to your employer.

If you are spiking a problem, writing a few lines of code to see how something works, or simply throwing together an example for Programmers.SE, then it's really a case of creating something that you fully intend to throw away once you've satisfied your understanding of a problem. In such cases, unit testing can be a hindrance to the task and can safely be left out. In all other cases however, I'd say that testing is always appropriate, even if the program seems relatively unimportant, and you should always aim to write software to a professional standard with enough test support to reduce the difficulty in maintaining the code in the future.

share|improve this answer
4  
Tests don't prove that code works, they give you proof that they don't fail in the way the tests test for, assuming the tests are correct. –  dan_waterworth May 3 '12 at 10:55
1  
If you don't think my comment invalidates your point then you don't understand my comment. Tests don't prove correctness, but you assert that they do when you say "Tests give you that proof [that what you write will work]". –  dan_waterworth May 3 '12 at 11:22
1  
To clarify, the word proof has a definite meaning and I understand working code to mean code that does what it was intended to do without exception. I would use an interactive proof assistant to create proofs. Tests provide assurance, but they can't provide a proof that your code works unless they do exhaustive model checking, which is hardly ever possible. –  dan_waterworth May 3 '12 at 12:04
3  
The context is software. If I ask for a proof of correctness I don't expect a test suite. –  dan_waterworth May 3 '12 at 12:59
2  
@S.Robins, You're completely missing the point. If I wanted to write provable correct software, I'd use a dependently typed language or other formal methods. "Hand-checks and costly & document heavy Q/A systems" are forms of testing which as I've already said do not prove correctness. –  dan_waterworth May 4 '12 at 10:00
show 7 more comments

Unit tests are nice and all, but in complicated systems with lots of "moving parts", they really don't actually test much.

As we know, Unit tests ensure that given input X for a unit, you get output Y where X is all appropriate input and Y is all appropriate output. However, I find that most code does what the developer thought it should.. but not necessary what it actually should do.
(You might think of it like a gun. A gun has a bunch of precision parts manufactured separately and then assembled. It doesn't matter if each particular part is tests well for functionality if the gun explodes when actually fired. Disclaimer: I know nothing about fire arms (: )

In this cases, we need actual system tests in a model production environment. Since functionality is tested here anyway, the value of unit tests is slightly diminished.

However, unit tests are still valuable in this case because they provide assurance that functionality that you wished to remain unchanged does in fact remain unchanged. This is quite valuable because it gives you confidence for areas of the system need focused testing.

share|improve this answer
2  
Remember that down votes aren't supposed to be an I don't like your answer they are simply a I didn't find your answer useful, so you have to learn not to take it personally. As it is, I'm not sure how this answers the actual question. –  Mark Booth May 4 '12 at 8:50
show 4 more comments

The amount and quality of unit testing to be done in a project is directly proportional to the pace of development and the required stability of the product.

At one end, for a one-off script, trivial demo, or in a quick-paced-startup environment during the very early stages, it's unlikely unit tests will matter or be useful for anything whatsoever. They are almost always, almost certainly, a waste of time and resources.

At the other end, when writing software to control pace makers, or pilot rocket-ships, or when working at maintaining stable, highly complex, software systems (say, a compiler with heavy adoption across the multiverse) multiple layers of unit test, and other coverage, are crucial.

[There are loads of other situation specific factors which each contribute one way or the other, such as the size of your team, your budget, the talent of your developers, your preferred software dev strategy, etc... Generally, and at the highest level of abstraction, I find these two make the largest difference]

share|improve this answer
add comment

In my view, product/code prototypes don't require testing as they are often delivered using a completely different process and for completely different purposes than a finished product.

Imagine this common scenario. You have some sort of technology that your company is famous for, your sales team or a chief exec goes to a client and is able to sell them a different product that is leveraging your technology. They sell it there and then and promise the client they'll see a beta version in a month time. You don't have anything, even a development team.

What happens next is you quickly design a solution based on the points agreed/sold to the client, assemble a team of (often star) engineers and start writing code. It would be ideal of course if your dates didn't slip (they always do) and you managed to write code in three weeks time rather than four. In this case you have an extra week, and if you were able to secure a QA team's time for this week - you will test it. In reality you won't have an extra week and/or your QA team is booked for another two months supporting other product deliveries. Moreover you won't have any design specs, so QA team won't have any idea what they are supposed to be testing. So you end up delivering what you wrote with a bit of your own smoke testing, which isn't really a testing as such, simply because you aren't qualified to test your own code. The biggest factor here, is that the client will see some of the promised functionality very quickly. If your prototype does the job, then you're back to the drawing board, where you design and plan the actual version 1.0 - properly this time.

As far as unit-tests go, they are good to maintain certain contracts (I promise my code will do this, if it doesn't - it's broken). In case of prototyping, you may not care so much for contracts, because often enough your prototypes will not live to become version 1.0, whether because the client has went away, or because you ended up developing your version 1.0 using different technologies/tools. An example here would be a quick prototype website written very quickly on PHP in a month's time by one person, and then you assemble a team to rewrite it in JEE. Even the database will often not survive this change.

share|improve this answer
4  
I've seen too much "protype" and "temporary" code go into production to ever assume that it won't. It's one thing to hope/plan on re-implementing. It's another thing to find the time in the schedule to do it. –  jwernerny May 3 '12 at 14:55
show 1 more comment

You should write unit tests when when the logical comprehension required to maintain or extend or communicate the logic of the application without flaws exceeds (or will likely in the future exceed) the ability of the least knowledgeable member of your team to meet that understanding.

Basically depends how much you can keep in your head at once. On your own you might have more confidence that you can code (in full knowledge of the impact of your changes) until a certain complexity has been reached. Until that point unit tests are a distraction, a hindrance and will slow you down. In a team, the threshold is going to be much lower so you pretty much need unit tests straight away to counteract group-stupidity. If you pick up a testless project from someone else (or you've forgotten about a project from a while ago) you'll want to write unit tests before you change anything because your understanding will be zero.

Note: writing less code and practising YAGNI, keeps down complexity and hence the number of required unit tests.

share|improve this answer
1  
So... you're advocating to only write unit tests to help you learn/protect a system you're unfamiliar with, otherwise don't unit test?!! Maybe I've misunderstood your answer. If not, then this really isn't what unit testing is meant for. In particular, if you're a TDD advocate, you'll appreciate that writing a test first guides your efforts to implement minimal code, while Refactoring helps you to reinforce those efforts. Thus YAGNI ends up both a central tenet and a byproduct of your method of development. If your unit testing is getting complicated, IMHO you're probably doing it "wrong". –  S.Robins May 4 '12 at 9:37
show 4 more comments

In a fast paced smaller organization, I find that developers are so pressured to stay productive and move fast that they often simply don't bother to test their own changes.

We have a general policy that states that if you make a change (whether it is a bug fix or enhancement) the developer should test it him/herself before submitting the code for further QA.

We don't actually write unit tests for most instances because our screens are mostly designed in a way where if a change is made on the screen, the rest of the logic has to be traversed through in order to find out if the change was successful regardless.

I find it imperative that the developer does their own testing, whether it is documented or not, because they are the ones who best know where the potential weak points in the logic may be.

So I would say, even if you don't write unit tests for every little part of the system, common sense testing should be done regardless.

share|improve this answer
add comment

IMO unit tests are inappropriate (not unnecessary!) in any codebase where it would require tons of hours of refactoring or even an outright rewrite in order to introduce unit testing, because it would be a very hard sell to management to allow for that time allotment.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.