It's pretty easy to track when we fix security vulnerabilities in existing code. But to make sure the whole team is staying on their toes about writing secure code, I'd like to also track how well we are preventing and avoiding writing new security vulnerabilities. What is the right way to measure that?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
You can never be 100% safe about security vulnerabilities, but a good approach is to check how much of your code got reviewed by a peer, because then both developers have to look over it and look for security problems. We're using Crucible (a tool by Atlassian) to accomplish that and every commit is reviewed.
Depending on your programming language there are also some tools which use algorithms to search for typical vulnerabilities in your code or even a crawler which checks your program or website in a blackbox approach.
Again, this is not to give 100% security, but it's one way to at least try to measure the improvements.
There is no answer, unfortunately. As noted in the comments, "How do you know if your elephant repellent has worn off?"
That said, if you want to report a somewhat informative metric, invest in a static code analysis tool (Fortify, Veracode, Checkmarx) and see how many issues it turns up. Such tools require tuning in order to find only the issues you are interested in.
A static analysis tool will not catch security defects in logic or design.
You can start by going down the OWASP ASVS checklist to verify that you're covering all your bases when it comes to security efforts. To comply with ASVS's requirements you will need some sort of active auditing of the code to look for security issues.
There are several approaches you can take:
Examples of static analysis tools include HP Fortify, Coverity, IBM AppScan. Be warned that there is a lot of snake oil vending here, because these tools are heavily oversold (both in price and in ability). In code I've worked on we ended up developing our own tool with a bunch of regexes to check the codebase during each build for things like improperly quoted variables inserted into SQL statements, because it was cheaper and more effective. Tools that are more general-purpose, like Sonar, can be repurposed to the task, so consider those as well.
There are also dynamic analysis tool which scan a running application, like Zed Attack Proxy and sqlmap. These tools are what an actual hacker would use to scan your application for vulnerabilities, so it can pay off to use them to check your own work.
You're trying to test a negative - that is, you're testing for something being absent. The problem with this is that it's like testing for the absence of the Loch Ness Monster - the loch is deep, you can't check all of it, and you certainly can't check it all at once. You can never be sure whether it's not there, or if you just missed it.
Your best bet is to flip your test and look for positives. Get some vulnerability-scan software and check your application for holes, both by checking the code for bad things like injection-vulnerable unescaped variables, and by scanning the application as it runs to look for vulnerabilities to known exploits. If your budget can stretch far enough, hire some penetration testers - or recruit some to work in-house - and get them to poke around your software to look for a way in. The aim of all this is to turn your negative "looking for something that's not there" into a positive "we found this vulnerability; fix it". If they don't find anything... well, you can never be completely sure, but at least you have a list of things that you've checked, so that when someone inevitably finds a zero-day exploit in something you're using, you can at least show (to your boss, your customer, or whoever else tries to point fingers) that you were diligent and did all that could be expected to make sure the application was secure.