Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything. I'm literally looking at my code and creating examples based on the flow of logic through my public methods. And every time the implementation changes, I have to go and change those tests, again, not really feeling that I'm accomplishing anything useful (be it mid- or long-term). I also do integration tests (including non-happy-paths) and I don't really mind the increased testing times. With those, I feel like I'm actually testing for regressions, because they have caught multiple, while all that unit tests do is show me that the implementation of my public method changed, which I already know.

Unit testing is a vast topic, and I feel like I'm the one not understanding something here. What's the decisive advantage of unit testing vs integration testing (excluding the time overhead)?

share|improve this question
    
My two cents: Don't overuse mocks (googletesting.blogspot.com/2013/05/… –  Juan Mendes Jul 11 at 15:33

4 Answers 4

up vote 8 down vote accepted

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything. I'm literally looking at my code and creating examples based on the flow of logic through my public methods.

This sounds like the method you are testing needs several other class instances (which you have to mock), and calls several methods on its own.

This type of code is indeed difficult to unit-test, for the reasons you outline.

What I have found helpful is to split up such classes into:

  1. Classes with the actual "business logic". These use few or no calls to other classes and are easy to test (value(s) in - value out).
  2. Classes that interface with external systems (files, database, etc.). These wrap the external system and provide a convenient interface for your needs.
  3. Classes that "tie everything together"

Then the classes from 1. are easy to unit-test, because they just accept values and return a result. In more complex cases, these classes may need to perform calls on their own, but they will only call classes from 2. (and not directly call e.g. a database function), and the classes from 2. are easy to mock (because they only expose the parts of the wrapped system that you need).

The classes from 2. and 3. cannot usually be meaningfully unit-tested (because they don't do anything useful on their own, they are just "glue" code). OTOH, these classes tend to be relatively simple (and few), so they should be adequately covered by integration tests.


An example

One class

Say you have a class which retrieves a price from a database, applies some discounts and then updates the database.

If you have this all in one class, you'll need to call DB functions, which are hard to mock. In pseudocode:

1 select price from database
2 perform price calculation, possibly fetching parameters from database
3 update price in database

All three steps will need DB access, so a lot of (complex) mocking, which is likely to break if the code or the DB structure changes.

Split up

You split into three classes: PriceCalculation, PriceRepository, App.

PriceCalculation only does the actual calculation, and gets provided the values it needs. App ties everything together:

App:
fetch price data from PriceRepository
call PriceCalculation with input values
call PriceRepository to update prices

That way:

  • PriceCalculation encapsulates the "business logic". It's easy to test because it does not call anything on its own.
  • PriceRepository can be pseudo-unit-tested by setting up a mock database and testing the read and update calls. It has little logic, hence few codepaths, so you do not need too many of these tests.
  • App cannot be meaningfully unit-tested, because it is glue-code. However, it too is very simple, so integration testing should be enough. If later App gets too complex, you break out more "business-logic" classes.

Finally, it may turn out PriceCalculation must do its own database calls. For example because only PriceCalculation knows which data its needs, so it cannot be fetched in advance by App. Then you can pass it an instance of PriceRepository (or some other repository class), custom-tailored to PriceCalculation's needs. This class will then need to be mocked, but this will be simple, because PriceRepository's interface is simple, e.g. PriceRepository.getPrice(articleNo, contractType). Most importantly, PriceRepository's interface isolates PriceCalculation from the database, so changes to the DB schema or data organisation are unlikely to change its interface, and hence to break the mocks.

share|improve this answer
    
I thought I was alone not seeing the point in unit testing everything, thanks –  enthrops May 22 '13 at 11:27

What's the decisive advantage of unit testing vs integration testing?

That's a false dichotomy.

Unit testing and integration testing serve two similar, but different purposes. The purpose of unit testing is to make sure your code does what it is supposed. In practical terms, the unit tests make sure that the code fulfills the contract outlined by the unit tests. This is evident in the way that unit tests are designed: they specifically state what the code is supposed to do, and assert that the code does that.

Integration tests are different. Integration tests exercise the interaction between software components. You can have software components that pass all of their tests and still fail integration tests because they don't interact properly.

However, if there is a decisive advantage to unit tests, it is this: unit tests are much easier to set up, and require far less time and effort than integration tests. When used properly, unit tests encourage the development of "testable" code, which means the final result is going to be more reliable, easier to understand, and easier to maintain. Testable code has certain characteristics, like a coherent API, repeatable behavior, and it returns results that are easy to assert.

Integration tests are more difficult and more expensive, because you often need elaborate mocking, complex setup, and difficult assertions. At the highest level of system integration, imagine trying to simulate human interaction in a UI. Entire software systems are devoted to that sort of automation. And it is automation that we're after; human testing is not repeatable, and doesn't scale like automated testing does.

Finally, integration testing makes no guarantees about code coverage. How many combinations of code loops, conditions and branches are you testing with your integration tests? Do you really know? There are tools that you can use with unit tests and methods under test that will tell you how much code coverage you have, and what the cyclomatic complexity of your code is. But they only really work well at the method level, where unit tests live.


If your tests are changing every time you refactor, that's a different problem. Unit tests are supposed to be about documenting what your software does, proving that it does that, and then proving that it does that again when you refactor the underlying implementation. If your API changes, or you need your methods to change in accordance with a change in the system design, that's what is supposed to happen. If it's happening a lot, consider writing your tests first, before you write code. This will force you to think about the overall architecture, and allow you to write code with the API already established.

If you're spending a lot of time writing unit tests for trivial code like

public string SomeProperty { get; set; }

then you should reexamine your approach. Unit testing is supposed to test behavior, and there is no behavior in the line of code above. You have, however created a dependency in your code somewhere, since that property is almost certainly going to be referred to elsewhere in your code. Instead of doing that, consider writing methods that accept the needed property as a parameter:

public string SomeMethod(string someProperty);

Now your method doesn't have any dependencies on something outside of itself, and it is now more testable, since it is completely self-contained. Granted, you won't always be able to do this, but it does move your code in the direction of being more testable, and this time you're writing a unit test for actual behavior.

share|improve this answer
1  
I am aware that unit testing and integration testing server different goals, however, I still don't get how unit tests are useful if you stub and mock all public calls that unit tests make. I would understand 'code fulfills the contract outlined by the unit tests', if it wasn't for stubs and mocks; my unit tests are literally reflections of logic inside of methods that I'm testing. You (I) aren't really testing anything, just looking at your code, and 'converting' it to tests. Regarding difficulty automating and code coverage, I'm currently doing Rails, and these both are well taken care of. –  enthrops May 17 '13 at 3:29
2  
If your tests are just reflections of the method logic, you're doing it wrong. Your unit tests should essentially hand the method a value, accept a return value, and make an assertion about what that return value should be. No logic is required to do that. –  Robert Harvey May 17 '13 at 3:35
1  
Makes sense, but still have to stub all the other public calls (db, some 'globals', like current user status, etc) and end up with testing code following method logic. –  enthrops May 17 '13 at 3:38
    
db is integration testing. Globals hinder unit testing; if you need globals, set up a mock object or a context object, and pass that to the method. But it's better to not rely on global state at all. –  Robert Harvey May 17 '13 at 3:49
1  
So I guess unit tests are for mostly isolated stuff that kind of confirms to 'set of inputs -> set of expected results'? –  enthrops May 17 '13 at 4:33

The unit tests with mocks are to make sure the implementation of the class is correct. You mock the public interfaces of the dependencies of the code that you are testing. This way you have control over everything external to the class and are sure that a failing test is due to something internal to the class and not in one of the other objects.

You are also testing the behavior of the class under test not the implementation. If you refactor the code (creating new internal methods, etc) the unit tests should not fail. But if you are changing what the public method does then absolutely the tests should fail because you have changed the behavior.

It also sounds like you are writing the tests after you have written the code, try instead writing the tests firsts. Try outlining the behavior that the class should have and then write the minimum amount of code to make the tests pass.

Both unit testing and integration testing are useful for ensuring the quality of your code. The unit tests examine each component in isolation. And the integration tests make sure that all the components interact properly. I want to have both types in my test suite.

Unit tests have helped me in my development as I can focus on one piece of the application at a time. Mocking the components that I haven't made yet. They also a great for regression, as they document any bugs in the logic that I have found (even in the unit tests).

UPDATE

Creating a test that only makes sure that methods are called has value in that you are making sure that the methods actually do get called. Particularly if you are writing your tests first, you have a check list of methods that need to happen. Since this code is pretty much procedural, you don't have much to check other than that the methods get called. You are protecting the code for change in the future. When you need to call one method before the other. Or that a method always gets called even if the initial method throws an exception.

The test for this method may never change or may change only when you are changing the methods. Why is this a bad thing? It helps reinforce using the tests. If you have to fix a test after changing the code, you will get in the habit of changing the tests with the code.

share|improve this answer
    
And if a method doesn't call anything private, then there's no point in unit testing it, correct? –  enthrops May 17 '13 at 1:49
    
If the method is private, you don't test it explicitly, it should be tested via the public interface. All public methods should be tested, to ensure that the behavior is correct. –  Schleis May 17 '13 at 12:45
    
No, I mean if a public method does not call anything private, does it make sense to test that public method? –  enthrops May 17 '13 at 17:19
    
Yes. The method does something doesn't it? So it should be tested. From a testing stand point I don't know if it is using anything private. I just know that if I provide input A, I should get output B. –  Schleis May 17 '13 at 17:23
    
Oh yes, the method does something and that something is calls to other public methods (and only that). So the way you would 'properly' test that is stub the calls with some return values and then set up message expectations. What EXACTLY are you testing in this case? That the right calls are made? Well, you wrote that method and you can look at it and see exactly what it does. I think unit testing is more appropriate for isolated methods that are supposed to be used like 'input -> output', so you can set up a bunch of examples and then do regression testing when you refactor. –  enthrops May 17 '13 at 18:02

Though the op has already marked an answer, I am just adding my 2 cents here.

What's the decisive advantage of unit testing vs integration testing (excluding the time overhead)?

And also in response to

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything.

There is a helpful but not exactly what OP has asked:

Unit Tests work but there are still bugs?

from my little experience on testing suites, I understand that Unit Tests are always to test the most basic method level functionality of a class. In my opinion every method either public, private or internal deserves to have a dedicated unit tests. Even in my recent experience I had a public method that was calling other small private method. So, there were two approaches:

  1. don't create a unit test(s) for private method.
  2. create a Unit Test(s) for a private method.

If You think logically, the point of having a private method is: main public method is getting too large or messy. In order to solve this You refactor wisely and create out small chunks of code that deserve to be separate private methods which in turn make Your main public method less bulky. You refactor by keeping in mind that this private method might be reused later on. There can be cases in which there are no other public method depending upon that private method, but who knows about future.


Considering the case when private method is reused by many other public methods.

So, If I had chosen approach 1: I would have duplicated Unit tests and they would have been complicated, as You had number of Unit Tests for each branch of public method as well as private method.

If I had chosen approach 2: the code written for unit tests would be relatively less, and it would be much easier to test.


Considering the case when private method is not reused There is no point of writing a seperate unit test for that method.

As for as Integration Tests are concerned, they tend to be exhaustive and more of high level. They will tell You that given a input, all of Your classes should come to this final conclusion.To understand more about usefulness of integration testing please see the mentioned link.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.