21

I'm wondering if someone has done some experiments correlating code metrics (SLOC, Cyclomatic Complexity, etc) with bug density in Object Oriented applications.

I'm not looking for experiments that only prove or disprove a correlation, but on both. I'm not trying to find a silver bullet as I believe that the bug density of a project might correlate to one or more metrics for a given project or team and the correlation can change during the lifetime of the project / team.

My goal is to

  1. Measure all interesting metrics for 2-3 months (we already have quite a few from sonar).
  2. Find one metric that correlates with the number of new bugs.
  3. Do a root-cause analysis to check why this happens (e.g. Do we lack a certain design skill?).
  4. Improve the skill and measure change for a couple of itereations.
  5. Rinse and repeat from 2.

If you don't have any experience on this, but remember seeing a paper / blog on this subject, I would appreciate if you can share it.


So far I've found the following links with some information about this subject

3
  • 2
    If you want to avoid closure, you should rephase your question. Stack Exchange sites are not search engines and users are not personal research assistants. Instead of asking for links to papers, the emphasis should be on asking what types of metrics have been correlated with defects and defect density.
    – Thomas Owens
    May 17, 2012 at 11:21
  • 2
    I'm sorry that the question came across as a request to become my personal search assistant, it's definitely not what I wanted to do, but finding these type of papers is not something very common. I've changed the title so other people don't have the same impression.
    – Augusto
    May 17, 2012 at 12:33
  • As a pointer, I remember that Clean Code - A Handbook of Software and CraftsmanShip by Robert Martin mentions several studies as a basis for the arguments presented. A bit old, though.
    – Pac0
    Dec 16, 2020 at 12:45

4 Answers 4

13

Whenever I hear of attempts to associate some type of code-based metric with software defects, the first thing that I think of is McCabe's cyclomatic complexity. Various studies have found that there is a correlation between a high cyclomatic complexity and number of defects. However, other studies that looked at modules with a similar size (in terms of lines of code) found that there might not be a correlation.

To me, both number of lines in a module and cyclomatic complexity might serve as good indicators of possible defects, or perhaps a greater likelihood that defects will be injected if modifications are made to a module. A module (especially at the class or method level) with high cyclomatic complexity is harder to understand since there are a large number of independent paths through the code. A module (again, especially at the class or method level) with a large number of lines is also hard to understand since the increase in lines means more things are happening. There are many static analysis tools that support computing both source lines of code against specified rules and cyclomatic complexity, it seems like capturing them would be grabbing the low hanging fruit.

The Halstead complexity measures might also be interesting. Unfortunately, their validity appears to be somewhat debated, so I wouldn't necessary rely on them. One of Halstead's measures is an estimate of defects based on effort or volume (a relationship between program length in terms of total operators and operands and program vocabulary in terms of distinct operators and operators).

There is also a group of metrics known as the CK Metrics. The first definition of this metrics suite appears to be in a paper titled A Metrics Suite for Object Oriented Design by Chidamber and Kemerer. They define Weighted Methods Per Class, Depth of Inheritance Tree, Number of Children, Coupling Between Object Classes, Response for a Class, and Lack of Cohesion in Methods. Their paper provides the computational methods as well as a description of how to analyze each one.

In terms of academic literature that analyze these metrics, you might be interested in Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects, authored by Ramanath Subramanyam and M.S. Krishna. They analyzed three of the six CK metrics (weighted methods per class, coupling between object classed, and depth of inheritance tree). Glancing through the paper, it appears that they found these are potentially valid metrics, but must be interpreted with caution as "improving" one could lead to other changes that also lead to a greater probability of defects.

Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults, authored by Yuming Zhou and Hareton Leung, also examine the CK metrics. Their approach was to determine if they can predict defects based on these metrics. They found that many of the CK metrics, except for depth of inheritance tree and number of children) had some level of statistical significance in predicting areas where defects could be located.

If you have an IEEE membership, I would recommend searching in the IEEE Transactions on Software Engineering for more academic publications and IEEE Software for some more real-world and applied reports. The ACM might also have relevant publications in their digital library.

6
  • The Halstead metrics have all been shown to be strongly correlated with raw SLOC (number of source lines of code). At that point, anything that is correlated with any Halstead metric became known to be correlated with raw SLOC, and it is easier to measure SLOC than any of the Halstead metrics. Jul 30, 2012 at 14:00
  • @JohnR.Strohm I don't agree that it's easier to count SLOC than compute the Halstead metrics, when you are using tools to do the computation. Assuming that the Halstead metrics are valid (which is actually debated, but nothing matters for an invalid metric), knowing the amount of time required to develop the code or the projected number of defects in the system is a more useful value than knowing the amount of lines. I can build schedules with time data, quality plans with defect data, or allocate enough time to a code review with difficulty. It's harder to use raw SLOC for those things.
    – Thomas Owens
    Jul 30, 2012 at 14:13
  • @JohnR.Strohm I'm sure that a Halstead metric computation program takes a little longer to execute than a SLOC counting program. But assuming valid output becomes valid input into a decision making, I'd rather have meaningful time, effort, and defect data than a raw SLOC count. The added value of the more complex metric is often worth the additional computation time, again assuming valid inputs and valid outputs of computation.
    – Thomas Owens
    Jul 30, 2012 at 14:15
  • @ThomasOwens, the question of whether software effort, and hence cost and schedule, may be estimated directly from estimates of raw SLOC has been done to death. After considerable research on real project data, the question was resolved, in the affirmative. See "Software Engineering Economics", by Barry Boehm, 1981. Jul 31, 2012 at 1:21
  • @ThomasOwens: Further, one must recognize that the Halstead metrics are inherently retrospective. You cannot measure the Halstead metrics of software you haven't written yet. On the other hand, it IS possible to estimate raw SLOC for a given task, and, given detailed enough specifications and a little experience, it is relatively easy to come pretty close on the estimate. Further, it is VERY easy to compare estimates to actuals, to fine-tune one's estimating heuristics, and calibrate one's cost estimator. General Dynamics/Fort Worth did a lot of work on this in the early 1980s. Jul 31, 2012 at 1:25
7

I have discussed possible correlations in one of my blog posts:

Correllation between Cyclomatic Complexity and Bugs density: Is this the real Issue?

The answer is no. Keeping the size constant, studies show no correlation between CC and defect density. However, there are other two interesting correlations to study:

The first one is: Does CC strongly correlate with the duration of detecting and fixing defects? In other words, if CC is lower, would we spend less time debug and fix defects?

The second one is: Does CC strongly correlate with the Fault Feedback Ratio (FFR, the average number of defects which results from applying one change or fixing one defect)?

It needs more investigation to see if anyone has ever studied this correlation empirically. But, my gut feeling and the feedback I get from the teams I work with is that there is strong positive correlation between cyclomatic complexity on one side and the duration of detecting and fixing a defect or the change impact on another side.

This is a good experiment to do. Keep alert for the results!

1
  • Not worthy of a downvote, but that should be "some studies show no correlation", because other studies do show a correlation. Jul 22, 2015 at 3:11
3

In the book Code Complete, p.457, Steve McConnell says that "control-flow complexity is important because it has been correlated with low reliability and and frequent errors". He then mentions a few references that support that correlation, including McCabe himself (who is credited with having developed the cyclomatic complexity metric). Most of these pre-date the widespread use of object-oriented languages, but as this metric applies to methods within those languages, the references might be what you are looking for.

Those references are:

  • McCabe, Tom. 1976. "A Complexity Measure." IEEE Transactions on Software Engineering, SE-2, no. 4 (December): 308-20
  • Shen, Vincent Y., et al. 1985. "Identifying Error-Prone Software--An Empirical Study." IEEE Transactions on Software Engineering SE-11, no.4 (April): 317-24.
  • Ward, William T. 1989. "Software Defect Prevention Using McCabe's Complexity Metric." Hewlett-Packard Journal, April, 64-68.

From my own experience, McCabe's metric, as it can be calculated by a program on many code sections, is useful in finding methods and functions that are overcomplicated and that have a high probability of containing errors. Although I have not calculated, the distribution of errors within high-cyclomatic complexity functions versus low-cyclomatic complexity functions, investigating those functions has allowed me to discover overlooked programming errors.

1

However the question was asked 8 years ago I would like to add some more information.

I asked this question to myself 2 years ago when I was working on software complexity analysis for our project. I was asked to provide practical guidance how to improve metrics in our codebase. I doubted that some of them did not matter. But then I decided to prove it with data.

I collected metrics with 1-2 month periods for 3 years timeline for every class. Then I collected bug-fixes for the classes. End of the day I had a data set containing:

  • Time period
  • Values of metrics
  • Number of bugs for the time period

Then I ran linear regression with different parameters and played around with statistics.

It turned out that some of the metrics really had a correlation with future bug counts. Some of them did not have any dependency at all.

Then I took several other projects (some of them were open source form internet with good bug-tracker) and did the same experiments again. I got different image.

My conclusion was that the metrics depend on various factors and you need to measure the correlation yourself for the project.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.