|
We are starting a push for code coverage here at my work, and it has got me to thinking.... How much code coverage is enough? When do you get to the point of diminishing returns on code coverage? What is the sweet spot between good coverage and not enough? Does it vary by the type of project your are making (ie WPF, WCF, Mobile, ASP.NET) (These are C# classes we are writing.) |
|||
|
|
We aim for at least 70%. On things that are more easily testable (functional data structures, for example), we aim for 90% and most individuals aim for as near to 100% as possible. On WPF-related things and other frameworks that are very difficult to test, we get much lower coverage (barely 70%). |
|||||||||||||
|
|
I'm of the opinion that code coverage alone is a poor metric. It's easy to produce tons of useless tests that cover the code, but don't adequately check the output, or don't test edge cases, for example. Covering code just means it doesn't throw an exception, not that it's right. You need quality tests- the quantity isn't that important. |
|||||||||||||
|
|
"Enough" is when you can make changes to your code with confidence that you're not breaking anything. On some projects, that might be 10%, on others, it might be 95%. It's almost never as high as 100%. However, sometimes trying to get 100% code coverage can be a great way to remove cruft from the code base. Don't forget that there's two ways to increase code coverage - write more tests or take out code. If code isn't covered because it's hard to test, there's a good chance you can simplify or refactor to make it easier to test. If it's too obscure to bother to test, there's usually a good chance that nothing else in the code is using it. |
|||||||||||
|
|
Code coverage approaches 100% asymptotically. Consequently, that last 5% is probably more effort than it is worth, as you begin to achieve vanishingly small returns for the effort expended. |
|||
|
|
|
Coverage is a metric to keep an eye on, but it shouldn't be the ultimate goal. I've seen (and admittedly written!) plenty of high coverage code - 100% coverage (TDD, of course), yet:
There's a "The Way of Testivus" entry that I think is appropriate to reference here :) |
|||
|
|
|
When you start considering changes that would affect run time performance, security, flexibility or maintainability to allow more code coverage it is time to end the quest for more code coverage. I have projects where that point is 0% because coverage is impossible to calculate without harming the design and other projects where that is as high as 92%. Code coverage metrics are only useful in pointing out were you might have missed some tests. They tell you nothing about the quality of your tests. |
|||
|
|
|
Only 20% of most code will run 80% of the time. A code coverage analysis is not very useful unless paired with a call graph to determine what needs to be tested the most. That tells you where your edge cases are likely to be. You may come up with 100 tests just for those edge cases, which constitute less than 5% of the actual code. So, make sure to cover 100% of the 20% that defines critical paths, and at least 50% of the rest (according to the call graph). This should get you (roughly) 70% - 75% total coverage, but it varies. Don't burn up time trying to get over 70% total coverage while leaving critical edge cases without checks. |
|||
|
|
|
Use coverage as a guide to indicate areas not tested. Rather than having a mandate for coverage it is wiser to understand the reason for code not covered. Recording a reason for the shortfall is good discipline that allows the risks to be balanced. Sometimes the reason is less than desirable 'e.g. ran out of time' but might be OK for an early release. It is better to flag areas to return to for a boost in coverage later. I work on critical flight software where 100% statement coverage is considered suitable for non-critical systems. For the more critical systems we check branch/decision coverage and use a technique call MC/DC which is sometimes not stringent enough. We also have to ensure that we have also covered the object code. It is a balance between risk, in our case very high, against value/cost. An informed choice is needed based upon the risk of missing a bug. |
|||
|
|
|
I don't have that high opinion about using code coverage as a measure for knowing when your test suite has enough coverage. The main reason why is because if you have a process where you first write some code, then some tests, and then look at the code coverage to discover where you have missed a test, then it is you process that needs improving. If you do true TDD, then you have a code coverage 100% out of the box (admittedly, there are some trivialities I don't test for). But if you look at the code coverage to find out what to test, then you will likely write the wrong tests. So the only thing you can conclude from the code coverage is that if it is too low, you don't have enough tests. But if it is high, there is no guarantee that you have all the right tests. |
|||
|
|
|
I think it depends on the part of the application you are testing. E.g. for business logic or any component involving complex data transformations, I would aim at 90% (as high as possible) coverage. I have often found small but dangerous bugs by just testing as much of the code as possible. I would rather find such bugs during testing than letting them occur at a customer's site one year later. Also, a benefit of a high code coverage is that it prevents people from changing working code too easily, since the tests have to be adapted correspondingly. On the other hand, I think there are components for which code coverage is less suited. For example, when testing a GUI it is very time consuming to write a test that covers all the code that gets executed when clicking on a button in order to dispatch the event to the right components. I think in this case it is much more effective to use the traditional approach of performing a manual test in which you just click on the button and observe the behaviour of the program (does the right dialog window open up? does the right tool get selected?). |
|||
|
|
|
Space critical software requires 100% statement coverage. At first it makes no sense. Everybody knows that a full test coverage doesn't mean that the code is fully tested and that it is not that difficult to get 100% coverage without actually testing the application. Nevertheless, 100% coverage is a lower limit: although 100% coverage is not a proof of a bug-free software, it is certain that with a lesser coverage the code is not fully tested and this is simply unacceptable for space critical software. |
|||
|
|
