Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What are useful metrics to capture for source code?

How can metrics, like for example (Executable?) Lines of Code or Cyclomatic Complexity help with quality assurance or how are they beneficial in general for the software development process?

share|improve this question

closed as primarily opinion-based by MichaelT, GlenH7, Kilian Foth, World Engineer Oct 29 '13 at 13:15

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

33  
The only valid measure is WTF / sec. :) –  terminus Dec 18 '10 at 10:51
2  
@terminus osnews.com/story/19266/WTFs_m :-) –  thbusch Jan 12 '11 at 14:35
show 1 more comment

17 Answers

What confuses me about code metrics is that it isn't done more. Most companies report on the efficiency of their employees, suppliers, and systems in place, but nobody seems to want to report on code. I will definitely agree with answers that state that more lines of code is a liability but what your code does is more important.

Lines Of Code: As 'Ive mentioned this is a vital measurement and should be taken the most seriously, but on each level. Functions, classes, files and interfaces can indicate do-everything code that is hard to maintain and costly in the long term. It's infinitely hard to compare the total lines of code versus what a system does. It could be something that does many things and in that case there will be many lines of code!

Complexity: This measurement is good to do on code bases you haven't worked on, and can give you a good indication of where problem areas lie. As a useful anecdote I measured complexity on one of my own code bases, and the highest complexity area was the one that I was spending the most time when I needed to change it. Working towards reducing the complexity resulted in a massive reduction in maintenance time. If management had these measurements at hand they could plan refactoring iterations or redesigns of specific areas of a system.

Code duplication: This is a very important measurement as far as I'm concerned. Code duplication is a very bad sign and could point to either deep problems in low levels of a system's design or developers that are copy pasting, causing massive problems in the long term and systems that are unmaintainable.

Dependency Graphs Finding bad dependencies and circular dependencies are an important measurement in code. This almost always points to an incorrect high level design that needs revising. Sometimes one dependency can suck in a whole lot of unnecessary other ones, because someone is using addNumber inside an e-mail library to do their finance calculations. Everyone is shocked when the e-mail library is changed and finance breaks. If everything is dependent on one thing it can also point to do-everything libraries that are hard to maintain and badly designed.

A good measurement will always tell you that every feature of a system has a small footprint. Less dependencies, less complexities, less duplication. This points to loose coupling and high cohesion.

share|improve this answer
add comment

Take a look on Jeff's posts on the subject:

A Visit from the Metrics Maid

Software Engineering: Dead?

There is an old, but good, post from Joel too, closely related to software metrics, and I strongly recommend its reading: The Econ 101 Management Method

The key point, for me, is this, quoting Jeff: "Responsible use of the metrics is just as important as collecting them in the first place."

share|improve this answer
show 1 more comment

There are many situations at my work where I use code metrics:

While writing code

The biggest and perhaps most important use in my daily job is in Checkstyle, a tool for java developers which continually checks the metrics (among other things) of my code against a set of rules we've defined and flags places where my code does not comply to those rules. As I develop code, it tells me in real time if my methods become to long, to complex or to coupled allowing me to step back and think about refactoring it to something better.

Developers are completely free to break all the rules since they will never apply to all situations. The "rules" are there to stimulate thought and say "Hey, is this the best way to do this?"

During QA/Code Reviews

The first thing I generally do when I perform a code review is to check the code coverage of the code I am reviewing in conjunction with a code coverage tool which highlights which lines of code have been covered. This gives me a general idea of how thorough the test code is. I don't really care if the coverage is 20% or 100% so long as the important code is well tested. Thus the percent covered is somewhat meaningless, but 0% sure stands out like a sore thumb as something I want to look carefully at.

I also check which metrics agreed by the team have been 'broken', if any, to see if I agree with the developer that it was OK or if I can suggest ways to improve it. Having these development metrics agreed upon in our team for writing new code has made big in-roads into improving our code. We write a lot less monolithic methods and are much better at the single responsibility principle now.

Trending improvements to legacy code We have a lot of legacy code around that we'd like to improve. The metrics at any point in time are fairly useless, but what's important to us is that over time code coverage goes up and things like complexity and coupling go down. Therefore, our metrics are plugged into our Continuous Integration server allowing us to look over time to ensure we are on the right track.

Getting to grips with a new code base About the only time I ever use lines of source code metric is when looking at a code base I'm not familiar with. It allows me to quickly gauge the rough size of the project compared to others I've worked with. Using other metrics I can get a further rough idea of the quality of the project too.

The key things are to use metrics as starting points for trending, discussions or ways forward and not to religiously manage them to exact figures. But I do strongly believe that they can help you improve the code you right when used properly.

share|improve this answer
show 1 more comment

An anecdote to show why KLOC counts are useless (and even harmful) to gauge performance.

Years ago I worked on a large project (70+ people in our company, another 30+ at our customer) which used KLOC counts as the sole measure of performance of teams and individuals.

For our Y2K effort (tells you how long ago it was :) ) we did a large cleanup of the section of the code my team was responsible for. We ended up for the release writing about 30.000 lines of code, not a bad 3 months of work for 5 people. We also ended up scrapping another 70.000 lines of code, a very good job for 3 months of work, especially combined with the new code.

End result for the quarter: -40.000 lines of code. During the performance review following the quarter we got an official reprimand from the company for failing to meet our productivity requirements of 20.000 lines of code produced per quarter (after all, the tools had shows we'd produced -40.000 lines of code), which would have resulted in all of us being listed as underperforming and bypassed for promotions, training, pay increase, etc. etc. had not the project manager and QA team intervened and gotten the reprimand overturned and replaced by a commendation.

A few months later (such things take time) we were told the company was reviewing their productivity standards and had hired a team of experts to create a new system based on function point analysis.

share|improve this answer
2  
Your answer doesn't show that KLOC are useless, it shows how to not use them. –  Neil N Nov 17 '11 at 18:19
1  
it shows that relying on them as a measure of productivity is shortsighted, relying on them as the only measure is idiotic. In other projects using KLOC as a measure of productivity and even quality we easily inflated the numbers by creating coding standards that caused loads of lines (C++ bracing practices, extra empty lines with just a short comment everywhere, splitting the conditions in an if statement over 3 lines, etc.). –  jwenting Nov 21 '11 at 6:28
show 3 more comments

Metrics are only useful if you know what to do with the answers you get. In essence a software metric is like a thermometer. The fact that you measure something at 98.6° F doesn't mean anything until you know what the normal temperature is. The above temperature is good for body temperature but really bad for ice cream.

Common metrics that can be useful are:

  • Bugs discovered/week
  • Bugs resolved/week
  • # Requirements defined/release
  • # Requirements implemented/release

The first two measure trends. Are you finding bugs faster than you can fix them? Two possible outcomes: maybe we need more resources fixing bugs, maybe we need to stop implementing new features until we catch up. The second two provide a picture of how close you are to being done. Agile teams call it a "burn down" chart.

Cyclomatic Complexity is an interesting metric. At it's base concept it's the number of unique execution paths in a function/method. In a unit-test heavy environment this corresponds to the number of tests needed to verify every execution path. Nevertheless, just because you have a method that has a Cyclomatic Complexity of 96 doesn't mean it is necessarily buggy code--or that you have to write 96 tests to provide reasonable confidence. It's not uncommon for generated code (through WPF or parser generators) to create something this complex. It can provide a rough idea of the level of effort needed to debug a method.

Bottom Line

Every measurement you take needs to have the following defined or it is useless:

  • An understanding of what "normal" is. This can be adjusted over the life of the project.
  • A threshold outside of "normal" where you need to take some sort of action.
  • A plan for dealing with the code when the threshold is exceeded.

The metrics you take may vary widely from project to project. You may have a few metrics that you use accross projects, but the definition of "normal" will be different. For example, if one project discovered an average of 5 bugs/week and the new project is discovering 10 bugs/week it doesn't necessarily mean something is wrong. It just might be the testing team is more meticulous this time around. Also, the definition of "normal" may change over the life of the project.

The metric is just a thermometer, what you do with it is up to you.

share|improve this answer
show 2 more comments

I'm surprised no-one's mentioned Statement/Decision Coverage of Unit tests (percentage of code exercised by unit tests) yet.

Code coverage is useful in that you know what percentage of the application doesn't fail catastrophrically; with the rest of its usefulness depends on the quality of the unit tests.

share|improve this answer
show 2 more comments

I often work on a giant C++ package, and when looking for problematic code worth refactoring the Cyclomatic Complexity or horrible FanIn/FanOut are usually pretty good red flags to look for. Fixing problems there will usually lead to improvements in the whole codebase.

Of course these numbers can only serve as a hint on what would be worth looking at. Making this some hard threshold after which to fail a build or to refuse a commit would be ridiculous.

share|improve this answer
add comment

Source code is a liability, not an asset. With that in mind, measuring lines of code is analogous to tracking dollars spent while building a house. It needs to be done if you want to stay under budget, but you wouldn't necessarily think that spending $1000 a day is better than spending $50 a day; you'd want to know how much of the house got built for that money. It's the same with lines of code in a software project.

In short, there are no useful metrics for source code because measuring source code by itself isn't useful.

share|improve this answer
add comment

Q: What are useful metrics to capture for source code?

For business:

A: Number of man-hours

For coder's supervisor:

A: Doesn't matter. Let's do everything today

For coder's self-esteem:

A: Number of SLOC (Source Lines of code)

For coder's mother:

A: Eat more of these soft French rolls and drink tea

continued in comments below...

share|improve this answer
add comment

These are not very useful absolute metrics in terms of progress, but can be used to give a general idea of the state of the code.

Notably Cyclomatic Complexity I have found to be useful in terms of visualizing how modularized a given code base is. You generally want a low complexity as this means that the number of sources per module is low and there are many modules.

share|improve this answer
add comment

Remember: All code can be reduced by at least 1 instruction. All code has at least 1 bug. Therefore, all code can be reduced to a single instruction which does not work. Hope that helps!

share|improve this answer
show 2 more comments

Source code metrics for quality assurance aim at two objectives:

  • writing code with less bugs inside
  • writing code for easy maintenance

Both lead to writing code as simple as possible. This means:

  • short units of code (functions, methods)
  • few elements in each unit (arguments, local variables, statements, paths)
  • and many other criteria more or less complex (see Software metric in Wikipedia).
share|improve this answer
add comment

The smaller the commits the better, usually. This is about SCM tools, not code per-se, but it's a very measurable metric. The smaller the commit the easier it is to see each change as an atomic unit; the easier it is to revert specific changes and pin-point when things broke.

As long as no commit breaks the build...

share|improve this answer
add comment

To the best of my knowledge, the number of bugs found directly correlates with lines of code (probably churn), modulo language, programmer, and domain.

I don't know of any other straightforward and practical metric well-correlated with bugs.

One thing I'd like to do is start running the numbers for different projects I'm on - Test Coverage :: kLOC, and then discuss "perceived quality" to see if there is a correlation.

share|improve this answer
2  
@Thor:l yep yep. shocker, huh? –  Paul Nathan Dec 18 '10 at 7:13
1  
@hlovdal: 2-3 errors per KSLOC is already a very low figure. The lowest figures I know from aerospace and security domains are of the order of 0.1 errors per KSLOC. Typical figures seem to be 20 to 50 errors per KSLOC. For reference, Google for Andy German's paper titled "Software Static Code Analysis - Lessons Learnt". –  Schedler Jan 12 '11 at 11:03
show 4 more comments

"Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs."- Bill Gates

share|improve this answer
2  
Please don't update non-answers. –  Eric Wilson Jan 12 '11 at 14:04
1  
While an amusing anecdote, this answer does little to contribute to the answer of this question. –  Chris Knight Feb 28 '11 at 10:31
6  
@Chris This answer got a lot of up-votes (or "updates" as FarmBoy wants to call it) because many developers believe that software metrics are useless. If you disagree or feel that you have a better response to the question, then post your own answer. Commenting like you've done here isn't productive; you've contributed nothing yourself. –  chrisaycock Feb 28 '11 at 15:30
4  
My downvote and comment are intended to discourage answers which lack depth and don't directly address the OP's question. This could be a much better answer if you went into more detail about why you believe software metrics to be useless with regards to software development and quality assurance and focused on more than just LOC. –  Chris Knight Feb 28 '11 at 23:27
add comment

Won't this "source code metrics" crap EVER die?

Raw source lines of code (SLOC) is the oldest, easiest, most basic metric there is.

Halstead originally proposed a whole bunch of metrics. Lots of people were having lots of fun writing measurement programs until some spoilsport did the obvious study, and demonstrated that each and every single Halstead metric was strongly directly correlated with SLOC.

At that point, Halstead's metrics were abandoned, because SLOC is always easier to measure.

share|improve this answer
4  
Interesting study, though their study only looked at programs generally between 50 and 100 lines of code. With such a small well-defined problem to solve, the end result doesn't seem so surprising. –  Chris Knight Feb 28 '11 at 23:48
show 3 more comments

Since source code is simply a combination of sequence, selection, and repetition. If I were to describe the most optimal piece of software that we could ever reasonably expect to produce it would be as follows. Software with nearly 100% testing code coverage using the least amount of lines of code necessary to do the job and yet flexible enough to withstand changes.

share|improve this answer
1  
100% coverage is only 100% if it covers all paths, not just all lines. In any realistic piece of software 100% path coverage is a bad goal to set, because it will be very expensive to reach, and still will only tell you that your code behaves as designed, not that the design itself is sound. You could have gaping security holes, and have 100% path coverage. –  Joeri Sebrechts Jan 12 '11 at 10:58
show 2 more comments

Not the answer you're looking for? Browse other questions tagged or ask your own question.