Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Generally when writing automated unit tests (eg JUnit, Karma) I aim to:

  • cover all the boundary conditions
  • get a high level of coverage.

I heard someone say:

coverage and boundary conditions aren't enough for a unit test, you need to write them so they will break if the code changes.

This sounds good to me in theory - but I'm not sure how to apply it.

My question is: Should I write automated unit tests that fail when the code changes? If so, how.

share|improve this question
32  
Automated tests aren't meant to break when code changes. They're meant to ensure that when you change code that your code still works to spec, and if it doesn't THEN fail. – Pieter B Jan 9 at 11:18
3  
There is a reason for a code to fail. That is when a procedure or business logic changes. Unit tests cannot predict domain changes, so if you do that, then the tests should fail and they should be rewritten. If you, however, only refactor code to make it faster, better (, stronger?), then all your unit tests should work even after you are done with the changes. – David Packer Jan 9 at 11:22
1  
I can see writing unit tests that run if and only if code changes. Even a basic Makefile can do that. But the unit test should only "fail" if the changed code actually fails the unit test. You can always have the unit tests emit a message to say when it actually runs. – technosaurus Jan 9 at 21:13
3  
That edit completely changes the meaning of the question, by the way. I see three answers, and they've all been invalidated. – Blacklight Shining Jan 10 at 12:41
2  
@BlacklightShining: There are actually six answers; three, including mine, have been deleted. This question should have been allowed to die with dignity instead of edited. – Blrfl Jan 10 at 13:20
up vote 28 down vote accepted

Your aim should be not to write unit tests that fail when the code changes, but unit tests that fail when the behaviour changes. Here, behaviour means anything that an external caller of the method wants it to do, like returning the right response to a question or saving the right thing to a databse. How it achieves that is its own internal implementation, not its behaviour.

By testing behaviour rather than implementation, you can refactor code to make improvements, and instantly verify whether you've accidentally changed the way it behaves by running your tests.

In reality, it's not possible to perfectly achieve this goal. If you have a method:

int add(int x, int y) {
    return x + y;
}

You can write as many unit tests as you want for it, but it's extremely unlikely any of them will fail if you modify it to:

int add(int x, int y) {
    if(x==10731 && y == -405571) {
        return 0;
    }
    return x + y;
}

However, you can take some sensible steps to get as close to full behavioural coverage as is practical:

  • As you said, think about boundary conditions and corner cases. These are the places where you're most likely to see accidental behaviour change.
  • Think of line-by-line coverage as "necessary, but not sufficient". Imagine trying to be as lazy as you possibly can while still getting full line-by-line coverage, and you'll see how easy it is to write an inadequate set of tests that follow this
  • Think about the behaviour your method is supposed to provide, and the branches it can follow. Ideally you should test that for every route through its implementation, it provides all the behaviour that's expected.
  • When you've written a set of tests, ask "Are there any implementations of my method which are at least as simple as the existing one, which would pass all of my tests, but which have the wrong behaviour?" This is a good rule of thumb to see if your behavioural coverage is good enough.

    As shown in the example add method above, you can never really fully defend against modifications to your method that add extra stuff to it, making it less simple. But much more likely, bugs are going to sneak in through modifying or removing parts, without adding to the complexity (because, why would you refactor something in a way that adds to its complexity?). So by adding the "at least as simple" condition, you get something practically achievable.

share|improve this answer
3  
I will note that your example bug would be caught if you ensure 100% branch coverage when running your tests. Such tests might not need to be written by hand. Even an automated tool like (C#) Pex (e.g., paste this code into Pex4Fun) would catch this type of bug, though I'm unsure what sort of tools exist for Java. – Brian Jan 9 at 16:25
    
@Brian Agreed. There's also things like property-based tests that can make special cases at least more likely to be caught. Nothing's foolproof though, and I didn't want to muddy the answer with increasingly advanced ways to plug an increasingly small hole – Ben Aaronson Jan 9 at 17:48
    
This is a really helpful constructive answer. I've updated the quote - would you consider updating your question? – hawkeye Jan 10 at 11:02
    
@Brian: And let's not forget branch-aware fuzzing, AFL or the LLVM's libFuzzer are examples of libraries that observe the code behavior at run-time and try to generate input that triggers new code paths. – Matthieu M. Jan 10 at 12:23
    
Fuzzing is interesting - I'm told the tool that specifically addresses this problem is a mutation test tool: mutation-testing.org – hawkeye Jan 10 at 12:27

The answers given - that this is a wrongheaded thing to do, and violates every principle of good testing - are correct.

But this is programming. There are always some sensible rationales for even the strangest request, and it can be productive to consider them.

Say you have some critical kernel code, or some high-security part of your SSH library, which absolutely, under no circumstances, should ever change its behavior, not even for some tiny subset of the infinite values it takes, all of which you could never test against.

And say you have a company set up such that there is a QA department responsible for writing unit tests against code provided by the Dev deparment, to ensure it meets business requirements.

In this situation, it makes sense to say "the critical code paths should only be modified by Dev when QA has been notified with signoff by upper management." Any change without signoff and notification is non-deliberate, and may be malicious.

In that case, a test which uses introspection to get a hash of the contents of each method, and compare that to a stored hash, could be legitimate. You can't serialize Method(), but you could subclass Method to be serializable. Perhaps a better approach is to get a hash of the file itself. Or just compare the file to a secure backup of the file.

But probably the best approach if the requirement is to fail on any modification, is to check the versioning system's logs and see if any changes were downloaded for the relevant files. If yes, then fail.

share|improve this answer
    
"QA department responsible for writing unit tests" Please, no. – Celos Jan 15 at 8:03
    
@Celos - Coders should write low-level unit tests as they code. But another layer of higher level tests written by people who are not the people writing the code exposes a plethora of spec issues that you simply can't ever through coder testing. All too many places leave spec-compliance-testing to some combo of [arbitrary coder decision; manual QA tests; wishful thinking]. Like art and UI design, automated testing's a separate concern and specialization from coding, and most coders who think that they can do these things just fine, simply haven't a clue. – Dewi Morgan Jan 16 at 0:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.