Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I understand the value of automated testing and use it wherever the problem is well-specified enough that I can come up with good test cases. I've noticed, though, that some people here and on StackOverflow emphasize testing only a unit, not its dependencies. Here I fail to see the benefit.

Mocking/stubbing to avoid testing dependencies adds complexity to the tests. It adds artificial flexibility/decoupling requirements to your production code to support mocking. (I disagree with anyone who says this promotes good design. Writing extra code, introducing things like dependency injection frameworks, or otherwise adding complexity to your codebase to make things more flexible/pluggable/extensible/decoupled without a real use case is overengineering, not good design.)

Secondly, testing dependencies means that critical low-level code that's used everywhere gets tested with inputs other than those that whoever wrote its tests explicitly thought of. I've found plenty of bugs in low-level functionality by running unit tests on high level functionality without mocking out the low-level functionality it depended on. Ideally these would have been found by the unit tests for the low-level functionality, but missed cases always happen.

What's the other side to this? Is it really important that a unit test doesn't also test its dependencies? If so, why?

Edit: I can understand the value of mocking external dependencies like databases, networks, web services, etc. (Thanks to Anna Lear for motivating me to clarify this.) I was referring to internal dependencies, i.e. other classes, static functions, etc. that don't have any direct external dependencies.

share|improve this question
9  
"overengineering, not good design". You're going to have to provide more evidence than that. Lots of folks would not call it "over engineering", but "best practice". –  S.Lott Apr 6 '11 at 1:09
    
@S.Lott: Of course this is subjective. I just didn't want a bunch of answers ducking the issue and saying mocking is good because making your code mockable promotes good design. Generally, though, I hate dealing with code that's decoupled in ways that have no obvious benefit now or in the foreseeable future. If you don't have multiple implementations now and don't anticipate having them in the foreseeable future, then IMHO you should just hard code it. It's simpler and doesn't bore the client with the details of an object's dependencies. –  dsimcha Apr 6 '11 at 1:12
2  
Generally, though, I hate dealing with code that's coupled in ways that have no obvious rationale except poor design. It's simpler and doesn't bore the client with the flexibility to test things in isolation. –  S.Lott Apr 6 '11 at 1:13
1  
@S.Lott: Clarification: I meant code that goes significantly out of its way to decouple things without any clear use case, especially where it makes the code or its client code significantly more verbose, introduces yet another class/interface, etc. Of course I'm not calling for code to be more tightly coupled than the simplest, most concise design. Also, when you create lines of abstraction prematurely they usually end up in the wrong places. –  dsimcha Apr 6 '11 at 1:18
    
Consider updating your question to clarify whatever distinction you're making. Integrating a sequence of comments is difficult. Please update the question to clarify and focus it. –  S.Lott Apr 6 '11 at 1:26
add comment

10 Answers 10

up vote 62 down vote accepted

It's a matter of definition. A test with dependencies is an integration test, not a unit test. You should also have an integration test suite. The difference is that the integration test suite may be run in a different testing framework and probably not as part of the build because they take longer.

For our product: Our unit tests are run with each build, taking seconds. A subset of our integration tests runs with each check-in, taking 10 minutes. Our full integration suite is run each night, taking 4 hours.

share|improve this answer
10  
+1 for distinguishing between unit / integration tests. –  robjb Apr 6 '11 at 2:02
3  
Don't forget about regression tests to cover discovered and fixed bugs to prevent their reintroduction into a system during future maintenance. –  RBerteig Apr 6 '11 at 7:15
    
I would not insist on the definition even if I agree with it. Some code might have acceptable dependencies like a StringFormatter and it still considered by most a unit test. –  danip Sep 25 '13 at 7:06
    
danip: clear definitions are important, and I would insist on them. But it's also important to realize that those definitions are a target. They're worth aiming for, but you don't always need a bullseye. Unit tests will often have dependencies on lower-level libraries. –  Jeffrey Faust Dec 3 '13 at 13:54
add comment

Testing with all the dependencies in place is still important, but it's more in the realm of integration testing as Jeffrey Faust said.

One of the most important aspect of unit testing is to make your tests trustworthy. If you don't trust that a passing test really means things are good and that a failing test really means a problem in the production code, your tests aren't nearly as useful as they can be.

In order to make your tests trustworthy, you have to do a few things, but I'm going to focus on just one for this answer. You have to make sure they are easy to run, so that all of the developers can easily run them before checking code in. "Easy to run" means that your tests run quickly and there's no extensive configuration or setup needed to make them go. Ideally, anyone should be able to check out the latest version of the code, run the tests right away, and see them pass.

Abstracting away dependencies on other things (file system, database, web services, etc.) enables you to avoid requiring configuration and makes you and other developers less susceptible to situations where you're tempted to say "Oh, tests failed because I don't have the network share set up. Oh well. I'll run them later."

If you want to test what you do with some data, your unit tests for that business logic code shouldn't care about how you get that data. Being able to test the core logic of your application without depending on supporting stuff like databases is awesome. If you're not doing it, you're missing out.

P.S. I should add that it's definitely possible to overengineer in the name of testability. Test-driving your application helps mitigate that. But in any event, poor implementations of an approach don't make the approach less valid. Anything can be misused and overdone if one doesn't keep asking "why am I doing this?" while developing.


As far as internal dependencies are concerned, things get a bit muddy. The way I like to think about it is that I want to protect my class as much as possible from changing for the wrong reasons. If I have a setup like something like this...

public class MyClass 
{
    private SomeClass someClass;
    public MyClass()
    {
        someClass = new SomeClass();
    }

    // use someClass in some way
}

I generally don't care how SomeClass is created. I just want to use it. If SomeClass changes and now requires parameters to the constructor... that's not my problem. I shouldn't have to change MyClass to accommodate that.

Now, that's just touching on the design part. Far as unit tests are concerned, I also want to protect myself from other classes. If I'm testing MyClass, I like knowing for a fact that there are no external dependencies, that SomeClass didn't at some point introduce a database connection or some other external link.

But an even bigger issue is that I also know that some of my methods' results rely on the output from some method on SomeClass. Without mocking/stubbing out SomeClass, I might have no way to vary that input in demand. If I'm lucky, I might be able to compose my environment inside the test in such a way that it'll trigger the right response from SomeClass, but going that way introduces complexity into my tests and makes them brittle.

Rewriting MyClass to accept an instance of SomeClass in the constructor enables me to create a fake instance of SomeClass that returns the value I want (either via a mocking framework or with a manual mock). I don't generally have to introduce an interface in this case. Whether to do so or not is in many ways a personal choice that may be dictated by your language of choice (e.g. interfaces are more likely in C#, but you definitely wouldn't need one in Ruby).

share|improve this answer
    
+1 Great answer, because external dependencies are a case I wasn't thinking of when I wrote the original question. I've clarified my question in response. See the latest edit. –  dsimcha Apr 6 '11 at 1:27
    
@dsimcha I expanded my answer to go into more detail on that. Hope it helps. –  Anna Lear Apr 6 '11 at 1:41
    
Anna, can you suggest a language where "If SomeClass changes and now requires parameters to the constructor... I shouldn't have to change MyClass" is true? Sorry, that just leapt out at me as an unusual requirement. I can see not having to change the unit tests for MyClass, but not having to change MyClass... wow. –  Мסž Apr 6 '11 at 4:34
1  
@moz What I meant was that if the way SomeClass is created, MyClass doesn't have to change if SomeClass is injected into it instead of created internally. Under normal circumstances MyClass shouldn't need to care about setup details for SomeClass. If SomeClass's interface changes, then yeah... MyClass would still have to be modified if any of the methods it uses are affected. –  Anna Lear Apr 6 '11 at 4:37
    
@Anna Lear: Ah. I got confused because it's not until the last para that you move to injection. Which I still struggle with. Ok, new question :) –  Мסž Apr 6 '11 at 5:07
show 3 more comments

Aside from the unit vs integration test issue, consider the following.

Class Widget has dependencies on classes Thingamajig and WhatsIt.

A unit test for Widget fails.

In which class lies the problem?

If you answered "fire up the debugger" or "read through the code until I find it", you understand the importance of testing only unit, not the dependencies.

share|improve this answer
    
@Brook: How about seeing what the results are for all Widget's dependencies? If they all pass, it's a problem with Widget until proven otherwise. –  dsimcha Apr 6 '11 at 3:55
2  
@dsimcha, but now you are adding complexity back in the game to check the intermediate steps. Why not simplify and just do simple unit tests first? Then do your integration test. –  asoundmove Apr 6 '11 at 5:19
1  
@dsimcha, that sounds reasonable until you get into a non-trivial dependency graph. Say you've got a complex object that has 3+ layers deep with dependencies, it turns into an O(N^2) search for the problem, rather than O(1) –  Brook Apr 6 '11 at 11:52
    
@asoundmove: Because then you have to design your codebase to support mocking. This is the whole point of the question. I know mocking has some benefits in the way you describe, but I'm skeptical that they're large enough to justify the artificial requirements they introduce into your design. –  dsimcha Apr 6 '11 at 14:18
    
@dsimcha, I don't really understand what you mean "design your codebase to support mocking". Do you mean following SRP? You dislike interface segregation? It seems to me your issue isn't mocking but rather the SOLID principles in general. –  Brook Apr 6 '11 at 15:03
show 5 more comments

Imagine that programming is like cooking. Then unit testing is the same as making sure your ingredients are fresh, tasty, etc. Whereas integration testing is like making sure your meal is delicious.

Ultimately, making sure that your meal is delicious (or that your system works) is the most important thing, the ultimate goal. But if your ingredients, I mean, units, work, you will have a cheaper way to find out.

Indeed, if you can guarantee that your units/methods work, you are more likely to have a functioning system. I emphasize "more likely", as opposed to "certain". You still need your integration tests, just like you still need someone to taste the meal you cooked and tell you that the end product is good. You will have an easier time getting there with fresh ingredients, that's all.

share|improve this answer
3  
And when your integration tests fail, unit test may zero in on the problem. –  Tim Williscroft Apr 6 '11 at 4:30
3  
To stretch an analogy: When you find that your meal tastes terrible, it may take some investigation to determine that it was because your bread is moldy. The outcome, though, is that you add a unit test to check for moldy bread before cooking, so that that particular problem can't happen again. Then if you have another meal failure later, you've eliminated moldy bread as the cause. –  Kyralessa Nov 3 '11 at 16:24
    
Love this analogy! –  thehowler May 17 at 10:00
add comment

Testing units by completely isolating them will allow to test ALL possible variations of data and situations this unit can be submitted. Because it is isolated from the rest you can ignore variations that do not have a direct effect on the unit to be tested. this in turn will greatly diminish the complexity of the tests.

Testing with various level of dependencies integrated will allow you to test specific scenarios that might not have been tested in the unit tests.

Both are important. Just doing unit test you will invariably more complex subtle error occurring when integrating components. Just doing integration testing means you are testing a system without the confidence that the individual parts of the machine have not been tested. To think you can achieve better complete test just by doing integration testing is almost impossible as the more components you add the number of entry combination grows VERY fast (think factorial) and creating a test with sufficient coverage becomes very quickly impossible.

So in short, the three levels of "unit testing" I typically use in almost all my projects :

  • Unit tests, isolated with mocked or stubbed dependencies to test the hell of a single component. Ideally one would attempt complete coverage.
  • Integration tests to test the more subtle errors. Carefully crafted spot checks that would show limit cases and typical conditions. Complete coverage is often not possible, effort focused on what would make the system fail.
  • Specification testing to inject various real life data in the system, instrument the data (if possible) and observe the output to ensure it conforms to the specification and business rules.

Dependency injection is one very efficient means to achieve this as it allows to isolate components for unit tests very easily without adding complexity to the system. Your business scenario may not warrant the use of injection mechanism but your testing scenario almost will. This for me is sufficient to make then indispensable. You can use them as well to test the different levels of abstraction independently through partial integration testing.

share|improve this answer
add comment

What's the other side to this? Is it really important that a unit test doesn't also test its dependencies? If so, why?

Unit. Means singular.

Testing 2 things means that you have the two things and all the functional dependencies.

If you add a 3rd thing, you increase the functional dependencies in the tests beyond linearly. Interconnections among things grow more quickly than the number of things.

There are n(n-1)/2 potential dependencices among n items being tested.

That's the big reason.

Simplicity has value.

share|improve this answer
add comment

Remember how you first learned to do recursion? My prof said "Assume you have a method that does x" (e.g. solves fibbonacci for any x). "To solve for x you have to call that method for x-1 and x-2". In the same regards, stubbing out dependencies lets you pretend they exist and test that the current unit does what it should do. The assumption is of course that you are testing the dependencies as rigorously.

This is in essence the SRP at work. Focusing on a single responsibility even for your tests, removes the amount of mental juggling you have to do.

share|improve this answer
add comment

(This is a minor answer. Thanks @TimWilliscroft for the hint.)

Faults are easier to localize if:

  • Both the unit-under-test and each of its dependencies are tested independently.
  • Each test fail if and only if there is a fault in the code covered by the test.

This works nicely on paper. However, as illustrated in OP's description (dependencies are buggy), if the dependencies aren't being tested, it would be hard to pinpoint the location of the fault.

share|improve this answer
    
Why wouldn't the dependencies be tested? They're presumably lower level and easier to unit-test. Moreover, with unit tests covering specific code, it's easier to pinpoint the location of the fault. –  David Thornley Apr 6 '11 at 16:02
add comment

A lot of good answers. I'd also add a couple of other points:

Unit testing also allows you to test your code when you dependencies don't exist. E.g. you, or your team, have not written the other layers yet, or maybe you are waiting on an interface delivered by another company.

Unit tests also mean that you don't have to have a full environment on your dev machine (e.g. a database, a web server etc). I would strong recommend that all developers do have such an environment, however cut down it is, in order to repro bugs etc. However, if for some reason it is not possible to mimic the production environment then unit testing at least gives yu some level of confidence in your code before it goes in to a larger test system.

share|improve this answer
add comment

About the design aspects: I believe that even small projects benefit from making the code testable. You don't necessarily have to introduce something like Guice (a simple factory class will often do), but seperating the construction process from the programming logic results in

  • clearly documenting the dependencies of every class via it's interface (very helpful for people who are new on the team)
  • the classes becoming much clearer and easier to maintan (once you put the ugly object graph creation into a seperate class)
  • loose coupling (making changes much easier)
share|improve this answer
1  
Method: Yes, but you have to add extra code and extra classes to do this. Code should be as concise as possible while still being readable. IMHO adding factories and whatnot is overengineering unless you can prove that you need or are very likely to need the flexibility it provides. –  dsimcha Apr 6 '11 at 13:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.