Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are starting a push for code coverage here at my work, and it has got me to thinking.... How much code coverage is enough?

When do you get to the point of diminishing returns on code coverage? What is the sweet spot between good coverage and not enough? Does it vary by the type of project your are making (ie WPF, WCF, Mobile, ASP.NET) (These are C# classes we are writing.)

share|improve this question
    
There is really no good answer to this; "How Much Unit Test Coverage Do You Need?" on the Artima Developer forums has some helpful advice. –  RN01 Jan 26 '12 at 1:03

11 Answers 11

up vote 14 down vote accepted

We aim for at least 70%. On things that are more easily testable (functional data structures, for example), we aim for 90% and most individuals aim for as near to 100% as possible. On WPF-related things and other frameworks that are very difficult to test, we get much lower coverage (barely 70%).

share|improve this answer
    
Is WPF inherently hard to test, or have you not yet spent the effort working out the best strategy to get better coverage? –  JBRWilkinson Oct 19 '10 at 21:30
    
A lot of it stems from the fact that WPF input is hard to fake. Our testing is as unit-y or API-y as we can get it, and the inability to easily fake the layer that sits "on top" of WPF (input, at least) makes it difficult to test. It isn't a huge problem, as the non-GUI parts of the API are easy to test, but it's just that last stretch that goes from our model (or view model) to WPF that is challenging. –  Noah Richards Oct 20 '10 at 2:03
1  
Yeah, WPF is a bitch to test. And I could live with that if there were compile time checking for the bindings in the views. So at least the build would break if you change a property that the view binds to. But it doesn't. That has led us to use GUI automation in our acceptance tests, which is also a bitch to write. But at least it gives us confidence that the system works. –  Pete Jan 30 '11 at 8:52
    
I like the number (70%). As my teams get higher, I tend to start finding tests for coverage than for value. On the WPF comment, we are just in the beginning days. Meaning, we aren't building/structuring WPF code to be easily testable. Mock models help. When designing code, design it to be testable. And, yes, at this point there are limited examples so you'll have think about it. No different than where most developers were as TDD was introduced to them for the first time just less industry experience. –  Jim Rush Jan 30 '11 at 15:25
    
to be more specific WPF is a bitch to unit test, if you need greater coverage for some reason the easiest way to up coverage of WPF classes is with coded UI tests/integration tests –  jk. Jan 26 '12 at 8:17

I'm of the opinion that code coverage alone is a poor metric. It's easy to produce tons of useless tests that cover the code, but don't adequately check the output, or don't test edge cases, for example. Covering code just means it doesn't throw an exception, not that it's right. You need quality tests- the quantity isn't that important.

share|improve this answer
6  
Code Coverage should be one of the outputs of your automated tests that are performed in your automated build system. Tests that don't check the output are hardly worth having. Below-threshold coverage indicates new features with no/insufficient tests. It shouldn't be something to beat people up with - but certainly it can flag up untested code. –  JBRWilkinson Oct 19 '10 at 21:32
3  
I second JBRWilkinson here, although not an indicator of good code, code coverage can be an indicator of a lack of tests. Your unit tests can also deliver other metrics by the way, like performance measures, so that you are not suddenly surprised when a new release comes crashing down the server under the unexpected workload. –  Matthieu M. Jan 30 '11 at 16:52
4  
I think this is a false choice. High-quality tests that only touch a small amount of code are poor overall measures of quality, as are "tests" that touch a large amount of code but don't really check results. Put another way, imagine a quality assurance process for cars that was very thorough at testing the front driver's-side wheel, but no other part of the car. That would be bad in the same way as a QA process that was just a guy eyeballing the whole car and saying "yeah, looks good." –  Noah Richards Jan 31 '11 at 21:05

"Enough" is when you can make changes to your code with confidence that you're not breaking anything. On some projects, that might be 10%, on others, it might be 95%.

It's almost never as high as 100%. However, sometimes trying to get 100% code coverage can be a great way to remove cruft from the code base. Don't forget that there's two ways to increase code coverage - write more tests or take out code. If code isn't covered because it's hard to test, there's a good chance you can simplify or refactor to make it easier to test. If it's too obscure to bother to test, there's usually a good chance that nothing else in the code is using it.

share|improve this answer
17  
"Test until fear turns to boredom" –  Brad Mace Oct 5 '10 at 19:05
2  
Upvote for the comment about removing code. I use code coverage metrics for that all the time (in VS, where it highlights the lines that aren't covered). –  Noah Richards Jan 31 '11 at 21:09
    
Great quote @bemace –  jayraynet May 18 '11 at 19:57

Code coverage approaches 100% asymptotically. Consequently, that last 5% is probably more effort than it is worth, as you begin to achieve vanishingly small returns for the effort expended.

share|improve this answer

Coverage is a metric to keep an eye on, but it shouldn't be the ultimate goal. I've seen (and admittedly written!) plenty of high coverage code - 100% coverage (TDD, of course), yet:

  • bugs still come up
  • design can still be poor
  • you can really kill yourself shooting for some arbitrary coverage target - pick your battles :p

There's a "The Way of Testivus" entry that I think is appropriate to reference here :)

share|improve this answer

Only 20% of most code will run 80% of the time. A code coverage analysis is not very useful unless paired with a call graph to determine what needs to be tested the most. That tells you where your edge cases are likely to be. You may come up with 100 tests just for those edge cases, which constitute less than 5% of the actual code.

So, make sure to cover 100% of the 20% that defines critical paths, and at least 50% of the rest (according to the call graph). This should get you (roughly) 70% - 75% total coverage, but it varies.

Don't burn up time trying to get over 70% total coverage while leaving critical edge cases without checks.

share|improve this answer
    
Don't Code Coverage tools generate a Call Graph by definition? –  JBRWilkinson Jan 30 '11 at 20:19

When you start considering changes that would affect run time performance, security, flexibility or maintainability to allow more code coverage it is time to end the quest for more code coverage.

I have projects where that point is 0% because coverage is impossible to calculate without harming the design and other projects where that is as high as 92%.

Code coverage metrics are only useful in pointing out were you might have missed some tests. They tell you nothing about the quality of your tests.

share|improve this answer

Space critical software requires 100% statement coverage.

At first it makes no sense. Everybody knows that a full test coverage doesn't mean that the code is fully tested and that it is not that difficult to get 100% coverage without actually testing the application.

Nevertheless, 100% coverage is a lower limit: although 100% coverage is not a proof of a bug-free software, it is certain that with a lesser coverage the code is not fully tested and this is simply unacceptable for space critical software.

share|improve this answer

Use coverage as a guide to indicate areas not tested. Rather than having a mandate for coverage it is wiser to understand the reason for code not covered. Recording a reason for the shortfall is good discipline that allows the risks to be balanced.

Sometimes the reason is less than desirable 'e.g. ran out of time' but might be OK for an early release. It is better to flag areas to return to for a boost in coverage later.

I work on critical flight software where 100% statement coverage is considered suitable for non-critical systems. For the more critical systems we check branch/decision coverage and use a technique call MC/DC which is sometimes not stringent enough.

We also have to ensure that we have also covered the object code.

It is a balance between risk, in our case very high, against value/cost. An informed choice is needed based upon the risk of missing a bug.

share|improve this answer

I think it depends on the part of the application you are testing. E.g. for business logic or any component involving complex data transformations, I would aim at 90% (as high as possible) coverage. I have often found small but dangerous bugs by just testing as much of the code as possible. I would rather find such bugs during testing than letting them occur at a customer's site one year later. Also, a benefit of a high code coverage is that it prevents people from changing working code too easily, since the tests have to be adapted correspondingly.

On the other hand, I think there are components for which code coverage is less suited. For example, when testing a GUI it is very time consuming to write a test that covers all the code that gets executed when clicking on a button in order to dispatch the event to the right components. I think in this case it is much more effective to use the traditional approach of performing a manual test in which you just click on the button and observe the behaviour of the program (does the right dialog window open up? does the right tool get selected?).

share|improve this answer

I don't have that high opinion about using code coverage as a measure for knowing when your test suite has enough coverage.

The main reason why is because if you have a process where you first write some code, then some tests, and then look at the code coverage to discover where you have missed a test, then it is you process that needs improving. If you do true TDD, then you have a code coverage 100% out of the box (admittedly, there are some trivialities I don't test for). But if you look at the code coverage to find out what to test, then you will likely write the wrong tests.

So the only thing you can conclude from the code coverage is that if it is too low, you don't have enough tests. But if it is high, there is no guarantee that you have all the right tests.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.