How to Avoid Fragile Unit Tests?

Question

We have written close to 3,000 tests -- data has been hard coded, very little reuse of code. This methodology has began to bite us in the ass. As the system changes we find ourselves spending more time fixing broken tests. We have unit, integration and functional tests.

What I'm looking for is a definitive way to write manageable and maintainable tests.

Frameworks

• FakeItEasy
• nUnit
• AutoFixture

Answer 1

Don't think of them as "broken unit tests", because they are not.

They are specifications, which your program no longer supports.

Don't think of it as "fixing the tests", but as "defining new requirements".

The tests should specify your application first, not the other way around.

You can't say you have a working implementation until you know it works. You can't say it works until you test it.

A few other notes that might guide you:

1. The tests and the classes under test should be short and simple. Each test should only check for a cohesive piece of functionality. That is, it doesn't care about things which other tests already check.
2. The tests, and your objects, should be loosely coupled, in a way that if you change an object, you're only changing its dependency graph downwards, and other objects which use that object are not affected by it.
3. You might be creating and testing the wrong stuff. Are your objects built for easy interfacing, or easy implementation? If it's the latter case, you're going to find yourself changing a lot of code that uses the old implementation's interface.
4. In the best case, strictly adhere to the Single Responsibility principle. In the worse case, adhere to the Interface Segregation principle. See SOLID Principles.

Answer 2

What you describe may actually not be such a bad thing, but a pointer to deeper problems your tests discover

As the system changes we find ourselves spending more time fixing broken tests. We have unit, integration and functional tests.

If you could change your code, and your tests would not break, that would be suspicious to me. The difference between a legitimate change and a bug is only the fact that it is requested, an what is requested is (TDD assumed) defined by your tests.

data has been hard coded.

Hard coded data in tests is imho a good thing. Tests work as falsifications, not as proofs. If there is too much calculation, your tests may be tautologies. For example:

assert sum([1,2,3]) == 6
assert sum([1,2,3]) == 1 + 2 + 3
assert sum([1,2,3]) == reduce(operator.add, [1,2,3])

The higher the abstraction, the closer you get to the algorithm, and by that, closer to comparing the acutal implementation to itself.

very little reuse of code

The best reuse of code in tests is imho 'Checks', as in jUnits assertThat, because they keep the tests simple. Besides that, if the tests can be refactored to share code, the actual code tested likely can be too, thus reducing the tests to the ones testing the refactored base.

Answer 3

Seems to me like your unit testing works like a charm. It is a good thing that it is so fragile to changes, since that's sort of the whole point. Small changes in code break tests so that you can eliminate the possibility of error throughout your program.

However, keep in mind that you only truly need to be testing for conditions which would make your method fail or give unexpected results. This would keep your unit testing more prone to "break" if there is a genuine problem rather than trivial things.

Though it seems to me that you're heavily redesigning the program. In such instances, do whatever you need to and remove the old tests and replace them with new ones afterwards. Repairing unit tests is only worthwhile if you're not fixing due to radical changes in your program. Otherwise you may find that you're dedicating too much time towards rewriting tests to be applicable in your newly written section of program code.

Answer 4

I am sure others will have a lot more input, but in my experience, these are some important things that will help you:

1. Use a test object factory to build input data structures, so you don't need to duplicate that logic. Perhaps look into a helper library such as AutoFixture to cut down on the code needed for test setup.
2. For each test class, centralize creation of the SUT, so it will be easy to change when things get refactored.
3. Remember, that test code is just as important as production code. It should also be refactored, if you find that you are repeating yourself, if the code feels unmantainable, etc, etc.

Answer 5

Handle tests like you do it with source code.

Version control, checkpoint releases, issue tracking, "feature ownership", planning and efforts estimation etc etc. Been there done that - I think this is the most efficient way to deal with problems like you describe.

Answer 6

I've had this problem as well. My improved approach has been as follows:

1. Don't write unit tests unless they're the only good way to test something.

   I am fully prepared to admit that unit tests have the lowest cost of diagnosis and time-to-fix. This makes them a valuable tool. The problem is, with the obvious your-mileage-may-vary, that unit tests are often too petty to merit the cost of maintaining the code mass. I wrote an example at the bottom, have a look.

2. Use assertions wherever they are equivalent to the unit test for that component. Assertions have the nice property that they are always verified throughout any debug build. So instead of testing the "Employee" class constraints in a separate unit of tests, you're effectively testing the Employee class through every test case in the system. Assertions also have the nice property that they do not increase the code mass as much as unit tests (which eventually require scaffolding/mocking/whatever).

   Before someone kills me: production builds should not crash on assertions. Instead, they should log at the "Error" level.

   As a caution to someone who hasn't thought about it yet, do not assert anything on user or network input. It is a huge mistake™.

   In my latest code bases, I've been judiciously removing unit tests wherever I see an obvious opportunity for assertions. This has significantly lowered the cost of maintenance overall and has made me a much happier person.

3. Prefer system/integration tests, implementing them for all of your primary flows and user experiences. Corner-cases probably do not need to be here. A system test verifies behavior at the user-end by running all components. Because of that, a system test is necessarily slower, so write the ones that matter (no more, no less) and you will catch the most important problems. System tests have very low maintenance overhead.

   It's key to remember that, since you're using assertions, each system test will run a couple hundred "unit tests" at the same time. You're also rather assured that the most important ones get run multiple times.

4. Write strong APIs that can be tested functionally. Functional tests are awkward and (let's face it) kind of meaningless if your API makes it too hard to verify functioning components on their own. Good API design a) makes testing steps straightforward and b) begets clear and valuable assertions.

   Functional testing is the hardest thing to get right, especially when you have components communicating one-to-many or (even worse, oh god) many-to-many across process barriers. The more inputs and outputs attached to a single component, the harder functional testing is, because you have to isolate one of them to really test its functionality.

---

On the issue of "don't write unit tests," I will present an example:

TEST(exception_thrown_on_null)
{
    InternalDataStructureType sink;
    ASSERT_THROWS(sink.consumeFrom(NULL), std::logic_error);
    try {
        sink.consumeFrom(NULL);
    } catch (const std::logic_error& e) {
        ASSERT(e.what() == "You must not pass NULL as a parameter!");
    }
}

The writer of this test has added seven lines that do not contribute at all to the verification of the final product. The user should never see this happening, either because a) nobody should ever be passing NULL there (so write an assertion, then) or b) the NULL case should cause some different behavior. If the case is (b), write a test that actually verifies that behavior.

My philosophy has become that we should not test implementation artifacts. We should only test anything that can be considered an actual output. Otherwise, there's no way to avoid writing twice the basic mass of code between the unit tests (which force a particular implementation) and the implementation itself.

It is important to note, here, that there are good candidates for unit tests. In fact, there are even several situations where a unit test is the only adequate means by which to verify something and in which it is of high value to write and maintain those tests. Of the top of my head, this list includes nontrivial algorithms, exposed data containers in an API, and highly-optimized code that appears "complicated" (a.k.a. "the next guy will probably screw it up.").

My specific advice to you, then: Start deleting unit tests judiciously as they break, asking yourself the question, "is this an output, or am I wasting code?" You will probably succeed in reducing the number of things that are wasting your time.

Answer 7

You should definitely have a look at Gerard Meszaros's XUnit test patterns. It has a great section with many recipes to reuse your test code and avoid duplication.

If your tests are brittle, it could also be that you don't resort enough to test doubles. Especially, if you recreate whole graphs of objects at the beginning of each unit test, the Arrange sections in your tests might become oversized and you might often find yourself in situations where you have to rewrite the Arrange sections in a considerable number of tests just because one of your most commonly used classes has changed. Mocks and stubs can help you here by cutting down on the number of objects you have to rehydrate to have a relevant test context.

Taking the unimportant details out of your test setups via mocks and stubs and applying test patterns to reuse code should reduce their fragility significantly.