I can't decide whether an offer of a commercial static analysis tool is worth spending the resources. We tried the tool on several million lines of our C++ code and it found something like 50 real issues.
We wanted to find how those issues might have affected users. We grabbed a year old version of stable branch and analyzed it and then looked into the defects base - none of the issues found in the old code has caused any of the problems users reported.
Not only those defects don't seem to manifest themselves to users but using the tool requires nontrivial effort - address false positives, analyze every warning the tool produces. Also the tools doesn't find all defects, it of course only finds some defects.
So once again
- a noticeable effort
- very low number of issues found
- issues found don't seem to affect customers
The tool is licensed at 7K euro for a team of five per year. Currently it looks like it's a lot of money and effort and no return except now our code will maybe have less defects that likely don't manifest themselves. It feels like we could have spent that effort on addressing issues that hurt right now.
The supplier claims that using their tool in the development process helps drastically improve code quality. I currently can't get any explanation of what this drastic improvement can be - all the facts I considered are listed above.
I'd like to hear stories that go like this: before we licensed X we were miserable, because (what specifically was wrong), now using X at all times we have (what's much better now than before), so we're pretty happy - money were spent well and negative experience with enough details would also be helpful.
Has anyone had really positive experience continuously using commercial static analysis tools? Is there a drastic improvement in source code quality that directly affects customers? Was the result worth the license fee and effort?