Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A well-tested codebase has a number of examples, but testing certain aspects of the system results in a codebase that is resistant to some types of change.

An example is testing for specific output--e.g., text or HTML. Tests are often (naively?) written to expect a particular block of text as output for some input parameters, or to search for specific sections in a block.

Changing the behavior of the code, to meet new requirements or because usability testing has resulted in change to the interface, requires changing the tests as well--perhaps even tests that are not specifically unit tests for the code being changed.

  • How do you manage the work of finding and rewriting these tests? What if you can't just "run 'em all and let the framework sort them out"?

  • What other sorts of code-under-test result in habitually fragile tests?

share|improve this question
How is this significantly different from…? – AShelly Sep 21 '10 at 14:39
That question mistakenly asked about refactoring--unit tests should be invariant under refactoring. – Alex Feinman Sep 21 '10 at 15:08

I know the TDD folks will hate this answer, but a large part of it for me is to choose carefully where to test something.

If I go too crazy with unit tests in the lower tiers then no meaningful change can be made without altering the unit tests. If the interface is never exposed and not intended to be reused outside the app then this is just needless overhead to what might have been a quick change otherwise.

Conversely if what you are trying to change is exposed or re-used every one of those tests you are going to have to change is evidence of something you might be breaking elsewhere.

In some projects this may amount to designing your tests from the acceptance tier down rather than from the unit tests up. and having fewer unit tests and more integration style tests.

It does not mean that you cannot still identify a single feature and code until that feature meets its acceptance criteria. It simply means that in some cases you do not end up measuring the acceptance criteria with unit tests.

share|improve this answer
I think you meant to write "outside the module", not "outside the app". – SamB Sep 21 '10 at 19:24
SamB, it depends. If the interface is an internal to a few places withing one app, but not public I would consider testing at a higher level if I thought the interface is likely to be volatile. – Bill Sep 21 '10 at 19:36

I just completed a major overhaul of my SIP stack, rewriting the entire TCP transport. (This was a near refactor, on a rather grand scale, relative to most refactorings.)

In brief, there's a TIdSipTcpTransport, subclass of TIdSipTransport. All TIdSipTransports share a common test suite. Internal to TIdSipTcpTransport were a number of classes - a map containing connection/initiating-message pairs, threaded TCP clients, a threaded TCP server, and so on.

Here's what I did:

  • Deleted the classes I was going to replace.
  • Deleted the test suites for those classes.
  • Left the test suite specific to TIdSipTcpTransport (and there was still the test suite common to all TIdSipTransports).
  • Ran the TIdSipTransport/TIdSipTcpTransport tests, to make sure they all failed.
  • Commented out all but one TIdSipTransport/TIdSipTcpTransport test.
  • If I needed to add a class, I'd add it write tests to build up enough functionality that the sole uncommented test passed.
  • Lather, rinse, repeat.

I thus knew what I still needed to do, in the form of the commented-out tests (*), and knew that the new code was working as expected, thanks to the new tests I wrote.

(*) Really, you don't need to comment them out. Just don't run them; 100 failing tests isn't very encouraging. Also, in my particular setup compiling fewer tests means a faster test-write-refactor loop.

share|improve this answer
I've done this too some months ago and it worked quite well for me. However I couldn't absolutely apply this method when pairing with a colleague in the ground-up redesign of our domain model module (which in turn triggered the redesign of all the other modules in the project). – Marco Ciambrone Jun 22 '11 at 9:04

When tests are fragile, I find its usually because I'm testing the wrong thing. Take for example, HTML output. If you check the actual HTML output your test will be fragile. But you aren't interested in the actual output, you are interested in whether it conveys the information that it should. Unfortunately, doing that requires making assertions about the contents of user's brains and so can't be done automatically.

You can:

  • Generate the HTML as a smoke test to make sure it actually runs
  • Use a template system, so you can test the template processor and data sent to the template, without actually testing the exact template itself.

The same sort of things happens with SQL. If you assert the actual SQL your classes attempt to make you are going to be in trouble. You really want to assert the results. Hence I use a SQLITE memory database during my unit tests to make sure that my SQL actually does what its supposed to.

share|improve this answer
It might also help to use structural HTML. – SamB Sep 21 '10 at 19:23
@SamB certainly that would help, but I don't think it'll solve the problem completely – Winston Ewert Sep 21 '10 at 20:23
of course not, nothing can :-) – SamB Sep 21 '10 at 21:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.