Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm wondering if someone has done some experiments correlating code metrics (SLOC, Cyclomatic Complexity, etc) with bug density in Object Oriented applications.

I'm not looking for experiments that only prove or disprove a correlation, but on both. I'm not trying to find a silver bullet as I believe that the bug density of a project might correlate to one or more metrics for a given project or team and the correlation can change during the lifetime of the project / team.

My goal is to

  1. Measure all interesting metrics for 2-3 months (we already have quite a few from sonar).
  2. Find one metric that correlates with the number of new bugs.
  3. Do a root-cause analysis to check why this happens (e.g. Do we lack a certain design skill?).
  4. Improve the skill and measure change for a couple of itereations.
  5. Rinse and repeat from 2.

If you don't have any experience on this, but remember seeing a paper / blog on this subject, I would appreciate if you can share it.


So far I've found the following links with some information about this subject

share|improve this question
1  
If you want to avoid closure, you should rephase your question. Stack Exchange sites are not search engines and users are not personal research assistants. Instead of asking for links to papers, the emphasis should be on asking what types of metrics have been correlated with defects and defect density. –  Thomas Owens May 17 '12 at 11:21
    
I'm sorry that the question came across as a request to become my personal search assistant, it's definitely not what I wanted to do, but finding these type of papers is not something very common. I've changed the title so other people don't have the same impression. –  Augusto May 17 '12 at 12:33

3 Answers 3

up vote 8 down vote accepted

Whenever I hear of attempts to associate some type of code-based metric with software defects, the first thing that I think of is McCabe's cyclomatic complexity. Various studies have found that there is a correlation between a high cyclomatic complexity and number of defects. However, other studies that looked at modules with a similar size (in terms of lines of code) found that there might not be a correlation.

To me, both number of lines in a module and cyclomatic complexity might serve as good indicators of possible defects, or perhaps a greater likelihood that defects will be injected if modifications are made to a module. A module (especially at the class or method level) with high cyclomatic complexity is harder to understand since there are a large number of independent paths through the code. A module (again, especially at the class or method level) with a large number of lines is also hard to understand since the increase in lines means more things are happening. There are many static analysis tools that support computing both source lines of code against specified rules and cyclomatic complexity, it seems like capturing them would be grabbing the low hanging fruit.

The Halstead complexity measures might also be interesting. Unfortunately, their validity appears to be somewhat debated, so I wouldn't necessary rely on them. One of Halstead's measures is an estimate of defects based on effort or volume (a relationship between program length in terms of total operators and operands and program vocabulary in terms of distinct operators and operators).

There is also a group of metrics known as the CK Metrics. The first definition of this metrics suite appears to be in a paper titled A Metrics Suite for Object Oriented Design by Chidamber and Kemerer. They define Weighted Methods Per Class, Depth of Inheritance Tree, Number of Children, Coupling Between Object Classes, Response for a Class, and Lack of Cohesion in Methods. Their paper provides the computational methods as well as a description of how to analyze each one.

In terms of academic literature that analyze these metrics, you might be interested in Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects, authored by Ramanath Subramanyam and M.S. Krishna. They analyzed three of the six CK metrics (weighted methods per class, coupling between object classed, and depth of inheritance tree). Glancing through the paper, it appears that they found these are potentially valid metrics, but must be interpreted with caution as "improving" one could lead to other changes that also lead to a greater probability of defects.

Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults, authored by Yuming Zhou and Hareton Leung, also examine the CK metrics. Their approach was to determine if they can predict defects based on these metrics. They found that many of the CK metrics, except for depth of inheritance tree and number of children) had some level of statistical significance in predicting areas where defects could be located.

If you have an IEEE membership, I would recommend searching in the IEEE Transactions on Software Engineering for more academic publications and IEEE Software for some more real-world and applied reports. The ACM might also have relevant publications in their digital library.

share|improve this answer
    
Thanks for you answer Thomas. I'll check the metrics and papers you mentioned. –  Augusto May 21 '12 at 10:15
    
The Halstead metrics have all been shown to be strongly correlated with raw SLOC (number of source lines of code). At that point, anything that is correlated with any Halstead metric became known to be correlated with raw SLOC, and it is easier to measure SLOC than any of the Halstead metrics. –  John R. Strohm Jul 30 '12 at 14:00
    
@JohnR.Strohm I don't agree that it's easier to count SLOC than compute the Halstead metrics, when you are using tools to do the computation. Assuming that the Halstead metrics are valid (which is actually debated, but nothing matters for an invalid metric), knowing the amount of time required to develop the code or the projected number of defects in the system is a more useful value than knowing the amount of lines. I can build schedules with time data, quality plans with defect data, or allocate enough time to a code review with difficulty. It's harder to use raw SLOC for those things. –  Thomas Owens Jul 30 '12 at 14:13
    
@JohnR.Strohm I'm sure that a Halstead metric computation program takes a little longer to execute than a SLOC counting program. But assuming valid output becomes valid input into a decision making, I'd rather have meaningful time, effort, and defect data than a raw SLOC count. The added value of the more complex metric is often worth the additional computation time, again assuming valid inputs and valid outputs of computation. –  Thomas Owens Jul 30 '12 at 14:15
    
@ThomasOwens, the question of whether software effort, and hence cost and schedule, may be estimated directly from estimates of raw SLOC has been done to death. After considerable research on real project data, the question was resolved, in the affirmative. See "Software Engineering Economics", by Barry Boehm, 1981. –  John R. Strohm Jul 31 '12 at 1:21

I have discussed possible correlations in one of my blog posts:

Correllation between Cyclomatic Complexity and Bugs density: Is this the real Issue?

The answer is no. Keeping the size constant, studies show no correlation between CC and defect density. However, there are other two interesting correlations to study:

The first one is: Does CC strongly correlate with the duration of detecting and fixing defects? In other words, if CC is lower, would we spend less time debug and fix defects?

The second one is: Does CC strongly correlate with the Fault Feedback Ratio (FFR, the average number of defects which results from applying one change or fixing one defect)?

It needs more investigation to see if anyone has ever studied this correlation empirically. But, my gut feeling and the feedback I get from the teams I work with is that there is strong positive correlation between cyclomatic complexity on one side and the duration of detecting and fixing a defect or the change impact on another side.

This is a good experiment to do. Keep alert for the results!

share|improve this answer

In the book Code Complete, p.457, Steve McConnell says that "control-flow complexity is important because it has been correlated with low reliability and and frequent errors". He then mentions a few references that support that correlation, including McCabe himself (who is credited with having developed the cyclomatic complexity metric). Most of these pre-date the widespread use of object-oriented languages, but as this metric applies to methods within those languages, the references might be what you are looking for.

Those references are:

  • McCabe, Tom. 1976. "A Complexity Measure." IEEE Transactions on Software Engineering, SE-2, no. 4 (December): 308-20
  • Shen, Vincent Y., et al. 1985. "Identifying Error-Prone Software--An Empirical Study." IEEE Transactions on Software Engineering SE-11, no.4 (April): 317-24.
  • Ward, William T. 1989. "Software Defect Prevention Using McCabe's Complexity Metric." Hewlett-Packard Journal, April, 64-68.

From my own experience, McCabe's metric, as it can be calculated by a program on many code sections, is useful in finding methods and functions that are overcomplicated and that have a high probability of containing errors. Although I have not calculated, the distribution of errors within high-cyclomatic complexity functions versus low-cyclomatic complexity functions, investigating those functions has allowed me to discover overlooked programming errors.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.