Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

If I have some code that has 80% test coverage (all tests pass), is it fair to say that it's of higher quality than code with no test coverage?

Or is it fair to say it's more maintainable?

share|improve this question
2  
100% coverage does not mean that it has been well tested. But 0% means that it has not been tested at all. –  mouviciel Oct 26 '10 at 9:21
    
Technically no. Practically, yes. Experience has taught lots of software engineers and testers that when code coverage reaches around 80%, the types of defects for which unit testing is adequate start to level off. It's the pareto principle. Basically once you get to the point where you're covering 80% of the code, regardless of the quality of your tests, you've probably tested the 20% of the code that causes most of the potential issues fairly thoroughly. This is not an absolute, but rather a conventional wisdom. You have to be more thorough if lives depend on your testing. –  Joe Rounceville Apr 23 at 17:54
    
@JoeRounceville I'm not sure... I can achieve high test coverage while testing nothing truly useful. Coverage merely tells you how much of the code is being touched by the test suite, not whether the tests are meaningful. –  Andres F. Apr 23 at 20:31
    
@AndresF. That's why I said "technically no, practically yes". People aren't idiots (generally). They don't (generally) test only no-brainer cases. So, based on experience, many shops stop somewhere around 80% coverage, making the (fairly safe) assumption that their people aren't morons. –  Joe Rounceville Apr 29 at 18:11
add comment

7 Answers 7

up vote 14 down vote accepted

In a strict sense, it is not fair to make any claims until the quality of the test suite is established. Passing 100% of the tests isn't meaningful if most of the tests are trivial or repetitive with each other.

The question is: In the history of the project, did any of those tests uncover bugs? The goal of a test is to find bugs. And if they didn't, they failed as tests. Instead of improving code quality, they might only be giving you a false sense of security.

To improve you test designs, you can use (1) whitebox techniques, (2) blackbox techniques, and (3) mutation testing.

(1) Here are some good whitebox techniques to apply to your test designs. A whitebox test is constructed with specific source code in mind. One important aspect of whitebox testing is code coverage:

  • Is every function called? [Functional coverage]
  • Is every statement executed? [Statement coverage-- Both functional coverage and statement coverage are very basic, but better than nothing]
  • For every decision (like if or while), do you have a test that forces it to be true, and other that forces it to be false? [Decision coverage]
  • For every condition that is a conjunction (uses &&) or disjunction (uses ||), does each subexpression have a test where it is true/false? [Condition coverage]
  • Loop coverage: Do you have a test that forces 0 iterations, 1 iteration, 2 iterations?
  • Is each break from a loop covered?

(2) Blackbox techniques are used when the requirements are available, but the code itself is not. These can lead to high-quality tests:

  • Do your blackbox tests cover multiple testing goals? You'll want your tests to be "fat": Not only do they test feature X, but they also test Y and Z. The interaction of different features is a great way to find bugs.
  • The only case you don't want "fat" tests is when you are testing an error condition. For example, testing for invalid user input. If you tried to achieve multiple invalid input testing goals (for example, an invalid zip code and an invalid street address) it's likely that one case is masking the other.
  • Consider the input types and form an "equivalence class" for the types of inputs. For example, if your code tests to see if a triangle is equilateral, the test that uses a triangle with sides (1, 1, 1) will probably find the same kinds of errors that the test data (2, 2, 2) and (3, 3, 3) will find. It's better to spend your time thinking of other classes of input. For example, if your program handles taxes, you'll want a test for each tax bracket. [This is called equivalence partitioning.]
  • Special cases are often associated with defects. Your test data should also have boundary values, such as those on, above, or below the edges of an equivalence task. For example, in testing a sorting algorithm, you'll want to test with an empty array, a single element array, an array with two elements, and then a very large array. You should consider boundary cases not just for input, but for output as well. [This is call boundary-value analysis.]
  • Another technique is "Error guessing." Do you have the feeling if you try some special combination that you can get your program to break? Then just try it! Remember: Your goal is to find bugs, not to confirm that the program is valid. Some people have the knack for error guessing.

(3) Finally, suppose you already have lots of nice tests for whitebox coverage, and applied blackbox techniques. What else can you do? It's time to Test your Tests. One technique you can use is Mutation Testing.

Under mutation testing, you make a modification to (a copy of) your program, in the hopes of creating a bug. A mutation might be:

Change a reference of one variable to another variable; Insert the abs() function; Change less-than to greater-than; Delete a statement; Replace a variable with a constant; Delete an overriding method; Delete a reference to a super method; Change argument order

Create several dozen mutants, in various places in your program [the program will still need to compile in order to test]. If your tests do not find these bugs, then you now need to write a test that can find the bug in the mutated version of your program. Once a test finds the bug, you have killed the mutant and can try another.


Addendum: I forgot to mention this effect: Bugs tend to cluster. What that means is that the more bugs you find in one module, the higher the probability that you'll find more bugs. So, if you have a test that fails (which is to say, the test is successful, since the goal is to find bugs), not only should you fix the bug, but you should also write more tests for the module, using the techniques above.

So long as you are finding bugs at a steady rate, testing efforts must continue. Only when there is a decline in the rate of new bugs found should you have confidence that you've made good testing efforts for that phase of development.

share|improve this answer
add comment

I would call it more-refactorable. Refactoring gets extremely easy if code is covered with lot of tests.

It would be fair to call it more maintainable.

share|improve this answer
add comment

By one definition it's more maintainable, as any breaking change is more likely to be caught by the tests.

However, the fact that code passes the unit tests doesn't mean it's intrinsically of higher quality. The code might still be badly formatted with irrelevant comments and inappropriate data structures, but it can still pass the tests.

I know which code I'd prefer to maintain and extend.

share|improve this answer
add comment

I would agree about the maintainable part. Michael Feathers recently posted a video of an excellent talk of his called "The deep synergy between testability and good design" in which he discusses this topic. In the talk he says that the relationship is one way, that is, code that is well designed is testable, but testable code is not necessarily well designed.

It's worth noting that the video streaming is not great in the video, so it might be worth downloading if you want to watch in full.

share|improve this answer
add comment

Code with absolutely no tests can be extremely high quality, readable, beautiful and efficient (or total junk), so no, it's not fair to say that code with 80 % test coverage is of higher quality than code with no test coverage.

It could be fair to say that code 80 % covered with good tests is probably of acceptable quality, and probably relatively maintainable. But it guarantees little, really.

share|improve this answer
add comment

I have been asking myself this question for some time now in relation to "condition coverage". So how about this page from atollic.com "Why code coverage analysis?"

More technically, code coverage analysis finds areas in your program that is not covered by your test cases, enabling you to create additional tests that cover otherwise untested parts of your program. It is thus important to understand that code coverage helps you understand the quality of your test procedures, not the quality of the code itself.

This seems to be quite relevant here. If you have a test case set that manages to attain a certain level of (code or otherwise) coverage, then you are quite likely invoking the code under test with a rather exhaustive set of input values! This will not tell you much about the code under test (unless the code blows up or generates detectable faults) but gives you confidence in your test case set.

In an interesting Necker Cube change-of-view, the test code is now being tested by the code under test!

share|improve this answer
add comment

There are many ways to guarantee that a program does what you intend, and to ensure that modifications will carry no unintended effects.

Testing is one. Avoiding mutation of data is another one. So is a type system. Or formal verification.

So, while I agree that testing is generally a good thing, a given percent of testing might not mean much. I would rather rely on something written in Haskell with no tests than on a well tested PHP library

share|improve this answer
    
is this only your opinion or you can back it up somehow? –  gnat Apr 23 at 17:38
2  
Testing is not a way to guarantee a program does what you intend. –  Andres F. Apr 23 at 20:29
    
Then I am left to wonder what testing is –  Andrea Apr 24 at 6:22
    
@gnat this is of course my opinion. Still, it says what is says. I took Haskell as an example of language whose compiler is very strict and gives many guarantees about the well-formedness of input, the types, the side effects, the mutation of data. I took PHP as an example of a language whose interpreter is very lenient and which does not even have a specification. Even in the absence of tests, the presence of all the guarantees from the types and effects system usually yields a decent degree of reliability. To compensate that with tests, one would need to have a very comprehensive suite –  Andrea Apr 24 at 13:51
    
I was maybe a little rushed when I wrote - I was on a phone - but I still think there is a point. I do not want to bash on PHP, but I think that saying that in comparison Haskell gives a much greater degree of reliability is an objective statement –  Andrea Apr 24 at 13:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.