Any success stories continuously using commercial static analysis tools for C++?

Question

I can't decide whether an offer of a commercial static analysis tool is worth spending the resources. We tried the tool on several million lines of our C++ code and it found something like 50 real issues.

We wanted to find how those issues might have affected users. We grabbed a year old version of stable branch and analyzed it and then looked into the defects base - none of the issues found in the old code has caused any of the problems users reported.

Not only those defects don't seem to manifest themselves to users but using the tool requires nontrivial effort - address false positives, analyze every warning the tool produces. Also the tools doesn't find all defects, it of course only finds some defects.

So once again

• a noticeable effort
• very low number of issues found
• issues found don't seem to affect customers

The tool is licensed at 7K euro for a team of five per year. Currently it looks like it's a lot of money and effort and no return except now our code will maybe have less defects that likely don't manifest themselves. It feels like we could have spent that effort on addressing issues that hurt right now.

The supplier claims that using their tool in the development process helps drastically improve code quality. I currently can't get any explanation of what this drastic improvement can be - all the facts I considered are listed above.

I'd like to hear stories that go like this: before we licensed X we were miserable, because (what specifically was wrong), now using X at all times we have (what's much better now than before), so we're pretty happy - money were spent well and negative experience with enough details would also be helpful.

Has anyone had really positive experience continuously using commercial static analysis tools? Is there a drastic improvement in source code quality that directly affects customers? Was the result worth the license fee and effort?

Answer 1

I've used several different tools on embedded projects at different companies. One of the big gotcha's on some of these projects is that the code is expected to stay up and running for months if not years at a time. Little things like a tiny memory leak can go undetected in testing (which takes weeks), but tend to manifest themselves post launch.

In that context, here are my comments.

PC-Lint

One project I worked on involved adding features to an existing product. The product had been shipping for several years and was (and is) widely used. As part of our work, we decided to run PC-Lint on the C source code. It turned up some scary source code [if (a = b = f(c))..., and those really are =, not ==] along with some other SPM (Silly Programmer Mistakes). In the end, PC-Lint helped us to resolve some long standing "features" that everyone (including our customers) knew about, but no one had been able to find. It didn't really tell us what was wrong, but it gave us good areas to look in because of the high number of issues it found.

Coverity

A little while later, we deployed Coverity for C++ analysis on start-up project. Coverity was run as part of our nightly builds, the results were published for developers, and developers were tasked with making sure their issues were resolved. Yes, there were some false positives, but we did find many SPMs in the code. The typical ones were ignoring of error returns, not checking for NULL pointers, and not properly handling allocated memory (use after delete, missing delete when the method exited and loosing the pointer to the memory, etc.)

The interesting side to this use was we also applied Coverity to some vendor provided source code. It turned up issues in their code they did not even know they had! The fall-out of that was that the vendor started using Coverity also. [BTW, we had run Coverity on their tool because we were seeing some "strangeness" in it's behavior.]

Klocwork

On yet another project at yet another company, Klocwork was put in place.

On one C++ project it was put into use on an existing code base. This is probably the worst way to use the tool, but still worth while. [It is much better to build in quality from the start than to try to tack it on at the end.] The biggest problem that was found was that Klocwork identified many, many potential issues. Sort through them to determine what needed to be addressed became a large task. Even so, it did find somethings that were serious and needed to be resolved.

On two other projects, it has been put in at the front end. One of these projects uses the Eclipse integration. The integration invokes Klocwork on a file when it is opened or saved. If issues are found, it marks the lines in the code view of Eclipse. This gets the developer to immediately address the issue instead of just deferring it. Even I, with my super human programming skills (:)), find myself making some SPMs from time to time.

Summary

The success and easy of using any of the static analysis tools really comes down in how you use it. If you try to use it as a last minute quality gate, you are going to be very disappointed in the amount of time it takes to find the few issues that might be there. If you use it as part of your continual development process (especially integrated with CI), then you will be much happier.

I like to compare static analysis tools to a trip to the dentist. Personally, I don't like to have to be bothered, but I have learned it is much better to have periodic check-ups and cleanings than to have to endure a root canal. Integrating it as part of the day-to-day development is like the checkups. Trying to put it in is like a root canal.

Answer 2

It can have a huge benefit - if your code is pants. If your code is good then it's not surprising you don't get much benefit from it.

Our parent company over in the states ran such a tool on their code, oo dear. It (Coverity) found a significant number of defects that the static analysis detected. So for them, it was worth it. The fact that they had run this became a bit of a selling point for the latest version (though I guess 'buy this version, we fixed a load of bugs we shouldn't have put in there' makes sense to marketing and sales)

I find that the free OSS tools are just as good for my code (ie they find the occasional whoopsie) and we've put them in the build server. I use cppcheck so even if it pops up no bugs found, it was free. Its also really easy to run against your code.

Answer 3

maybe have less defects that likely don't manifest itself

Yeah, but when they do, those are the hardest bugs to solve, because you look at the code and say: nah, these lines are unlinkely to cause a bug.

John Carmack used a handful of those during the development of RAGE (the upcoming game of id software). He has an overall positive reaction. You can watch his keynote on QuakeCon 2011 where he gives his thoughts on the matter (as well as many anecdotes, I found it very entertaining).

Regarding the analyzer's false positives, I share his view: if the analyzer thought it was bad, you should change that code even if it's a false positive. The next human who reads it would have a similarly hard time figuring out what's going on.

Edit: I just remembered that I came across an article about static analysis ROI. I'm yet to read it, but it does seem informative. The author is the scientific adviser of Viva64, the vendor of PVS-Studio (and I guess many parts of it were written by himself).

Answer 4

I understand the money issue (don't we all ?) and it is true that this may seem extremely extensive. I guess it is then a matter of measuring the impact of the potential bugs found...

I am not sure of your experimental protocol though: static analysis is cheap in the sense that it can run 24/24 7/7 without any supplementary cost (apart the fact that a machine is up and running, I guess). This means you can run it on each and every commit, and thus detect defect even before they hit QA.

Instead of diving head-first for this expensive tool though, why don't you try your hand with cheaper alternatives. There are several versions free of charge (I head good things about VC++ Analysis module, Clang has a built-in static analyzer, etc...). Since most static analysis tools are blind in some way or another, having several around means you're covering a greater scope; and if one is really obnoxious because of its false positive rate, you can always ditch it.

Once you've acquired experience with cheaper tools, you may make an enlightened decision about whether or not paying a royal fee for a supposedly better tool, and even evaluate it in comparison to those already integrated.

Answer 5

Static analysis will be helpful only with day-to-day usage. In this case you can find program errors and copy-paste mistakes quickly and fix it immediately. Rules for using of static analysis:

1. Integrate it with continuous integration systems (CruiseControl, Hudson, TFS).
2. Integrate it with version control systems (SVN, Git, Mercurial).
3. If tool support feature like "Incremental Analysis" (analyse few files after compilation) then you can use it.

Also you must remember, if you have a important code base then static analysis is a way to improve code quality.

Answer 6

PC-Lint here.

Low cost, fast, flexible and easy to configure. Prevents many trivial errors and guards much of our coding standard. Output is configured as part of a normal build process - there is basically no difference if warning comes from compiler or Lint. It's critical to use it from the beginning of the development, otherwise there are zillions of false alarms if you try to run just before deadline trying to "fix" quality issues.