Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

It's pretty easy to track when we fix security vulnerabilities in existing code. But to make sure the whole team is staying on their toes about writing secure code, I'd like to also track how well we are preventing and avoiding writing new security vulnerabilities. What is the right way to measure that?

share|improve this question
Why the downvote? – Mark Rushakoff Apr 27 '13 at 6:10
How do you know if your elephant repellent has worn off? Ultimately you're trying to measure and/or prove a negative - how well protected you are from Bad Things. – Dan Pichelman Apr 27 '13 at 12:57
@DanPichelman - it certainly feels that way. I was hoping there was some kind of non-obvious, pragmatic practice for this. – Mark Rushakoff Apr 27 '13 at 17:02
Nope, @DanPichelman is right on the money. One pragmatic way to prevent vulnerabilities is to hire people who know what they're doing and do their development with care and discipline. – Blrfl Apr 28 '13 at 15:40
Security issues are quality issues. If you say there's no way to double-check work performed on security then you are saying there is no way to double-check work performed on quality. If that were true QA would be pointless. – Joeri Sebrechts Apr 29 '13 at 7:50

You can never be 100% safe about security vulnerabilities, but a good approach is to check how much of your code got reviewed by a peer, because then both developers have to look over it and look for security problems. We're using Crucible (a tool by Atlassian) to accomplish that and every commit is reviewed.

Depending on your programming language there are also some tools which use algorithms to search for typical vulnerabilities in your code or even a crawler which checks your program or website in a blackbox approach.

Again, this is not to give 100% security, but it's one way to at least try to measure the improvements.

share|improve this answer

There is no answer, unfortunately. As noted in the comments, "How do you know if your elephant repellent has worn off?"

That said, if you want to report a somewhat informative metric, invest in a static code analysis tool (Fortify, Veracode, Checkmarx) and see how many issues it turns up. Such tools require tuning in order to find only the issues you are interested in.

A static analysis tool will not catch security defects in logic or design.

share|improve this answer

You can start by going down the OWASP ASVS checklist to verify that you're covering all your bases when it comes to security efforts. To comply with ASVS's requirements you will need some sort of active auditing of the code to look for security issues.

There are several approaches you can take:

  1. Dedicated code reviews for security (all code is reviewed)
  2. Static analysis via automated tools, custom-made or standard
  3. Dynamic analysis of the running application.

Examples of static analysis tools include HP Fortify, Coverity, IBM AppScan. Be warned that there is a lot of snake oil vending here, because these tools are heavily oversold (both in price and in ability). In code I've worked on we ended up developing our own tool with a bunch of regexes to check the codebase during each build for things like improperly quoted variables inserted into SQL statements, because it was cheaper and more effective. Tools that are more general-purpose, like Sonar, can be repurposed to the task, so consider those as well.

There are also dynamic analysis tool which scan a running application, like Zed Attack Proxy and sqlmap. These tools are what an actual hacker would use to scan your application for vulnerabilities, so it can pay off to use them to check your own work.

share|improve this answer

You're trying to test a negative - that is, you're testing for something being absent. The problem with this is that it's like testing for the absence of the Loch Ness Monster - the loch is deep, you can't check all of it, and you certainly can't check it all at once. You can never be sure whether it's not there, or if you just missed it.

Your best bet is to flip your test and look for positives. Get some vulnerability-scan software and check your application for holes, both by checking the code for bad things like injection-vulnerable unescaped variables, and by scanning the application as it runs to look for vulnerabilities to known exploits. If your budget can stretch far enough, hire some penetration testers - or recruit some to work in-house - and get them to poke around your software to look for a way in. The aim of all this is to turn your negative "looking for something that's not there" into a positive "we found this vulnerability; fix it". If they don't find anything... well, you can never be completely sure, but at least you have a list of things that you've checked, so that when someone inevitably finds a zero-day exploit in something you're using, you can at least show (to your boss, your customer, or whoever else tries to point fingers) that you were diligent and did all that could be expected to make sure the application was secure.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.