Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 26 down vote favorite
9

Does anyone know if there is some kind of tool to put a number on technical debt of a code base, as a kind of code metric? If not, is anyone aware of an algorithm or set of heuristics for it?

If neither of those things exists so far, I'd be interested in ideas for how to get started with such a thing. That is, how can I quantify the technical debt incurred by a method, a class, a namespace, an assembly, etc.

I'm most interested in analyzing and assessing a C# code base, but please feel free to chime in for other languages as well, particularly if the concepts are language transcendent.

share|improve this question
9  
Technical debt comes from decisions, not code. It accrues because of bad management choices. It's not clear that "method, a class, a namespace, an assembly" contain technical debt by themselves. They represent a liability when there's a better choice available. – S.Lott Feb 20 '12 at 19:19
2  
I would argue (in the context of the metaphor of debt) that managers may be the debt-holders, but the code artifacts represent the debt valuation and could be quantified. That is, I agree that managers may make a decision like "forget unit testing because we don't have time" and thus incur technical debt. But, I certainly think you can put a number to individual code elements as a heuristic. Think of it this way - if management makes a series of horrible decisions for the future, but no code has been written, is there any debt at that moment? – Erik Dietrich Feb 20 '12 at 19:25
3  
"is there any debt at that moment?" Debt does need to accumulate, you're right. But it's not the code; it's the volume of "work" done that needs to be undone. Specifications, designs, code, DBA-work, all of it has to be reworked. Measuring debt from software artifacts (like source lines of code) is similar to predicting development cost. – S.Lott Feb 20 '12 at 19:42
6  
Measuring technical debt is hard, plus it confuses managers. However, I can tell you a good way to fight technical debt: cheap, nice and working prototypes, particularly if the code base revolves around GUI. As Joel suggested here: joelonsoftware.com/articles/fog0000000332.html, spend a bit of time each day cleaning things up. The change must be positive improvements, not "OMG, our technical debt = pentablobs and it is rising exponentially at a rate of ... the sky is falling." Just spend a bit of time each day on kaizen in a way that does not break things that work. Make friends. – Job Feb 20 '12 at 20:22
2  
Taking decisions about technical debt based on an automatic score seems like a sure way to increase technical debt! – Andrea Oct 2 '12 at 13:36
show 2 more comments

8 Answers

up vote 21 down vote accepted

Technical debt is just an abstract idea that, somewhere along the lines of designing, building, testing, and maintaining a system, certain decisions were made such that the product has become more difficult to test and maintain. Having more technical debt means that it will become more difficult to continue to develop a system - you either need to cope with the technical debt and allocate more and more time for what would otherwise be simple tasks, or you need to invest resources (time and money) into reducing technical debt by refactoring the code, improving the tests, and so on.

There are a number of metrics that might give you some indication as to the quality of the code:

  • Code coverage. There are various tools that tell you what percentage of your functions, statements, and lines are covered by unit tests. You can also map system and acceptance tests back to requirements to determine the percentage of requirements covered by a system-level test. The appropriate coverage depends on the nature of the application.
  • Coupling and cohesion. Code that exhibits low coupling and high cohesion is typically easier to read, understand, and test. There are code analysis tools that can report the amount of coupling and cohesion in a given system.
  • Cyclomatic complexity is the number of unique paths through an application. It's typically counted at the method/function level. Cyclomatic complexity is related to the understandability and testability of a module. Not only do higher cyclomatic complexity values indicate that someone will have more trouble following the code, but the cyclomatic complexity also indicates the number of test cases required to achieve coverage.
  • The various Halstead complexity measures provide insight into the readability of the code. These count the operators and operands to determine volume, difficulty, and effort. Often, these can indicate how difficult it will be for someone to pick up the code and understand it, often in instances such as a code review or a new developer to the code base.
  • Amount of duplicate code. Duplicated code can indicate potential for refactoring to methods. Having duplicate code means that there are more lines for a bug to be introduced, and a higher likelihood that the same defects exist in multiple places. If the same business logic exists in multiple places, it becomes harder to update the system to account for changes.

Often, static analysis tools will be able to alert you of potential problems. Of course, just because a tool indicates a problem doesn't mean there is a problem - it takes human judgement to determine if something could be problematic down the road. These metrics just give you warnings that it might be time to look at a system or module more closely.

However, these attributes focus on the code. They don't readily indicate any technical debt in your system architecture or design that might relate to various quality attributes.

share|improve this answer
I currently use NDepend (ndepend.com), CodeRush and VS code metrics to keep an eye on the metrics you mention (with the exception of the Halstead measures, which I'll look into further). I was thinking I might use some amalgamation of these metrics to attempt to put some kind of number on a given code element that would roughly indicate, at a glance, how costly it was to ongoing development. – Erik Dietrich Feb 20 '12 at 20:04
@ErikDietrich You might be able to, but I probably wouldn't quantify that value. Perhaps an "executive summary" style report on what your metric tools tell you, with respect to changes over time, would be more appropriate. – Thomas Owens♦ Feb 20 '12 at 20:17
Another simple metric I'd add to the list is number of TODO/HACK/WTF? comments in a codebase... – MaR Feb 21 '12 at 12:31
@Mar That assumes that you properly use these and aren't gaming them for your advantage. Want some extra time to clean up the code base, just add these comments where they aren't appropriate. Don't care about the codebase, just remove them from where they should be. Comments can lie, code can't. – Thomas Owens♦ Feb 21 '12 at 12:49
1  
@Thomas Owens: agreed, but almost any metric alone can be cheated. If used right and honestly, "TODO metric" provides cheap overview what code is actually missing or should be changed (=invisible debt for code-only based metrics). – MaR Feb 21 '12 at 14:19
show 2 more comments
up vote 9 down vote

Sonar has a technical debt heuristic as well as several other features useful to a software project.

It also supports a pretty wide range of languages.

share|improve this answer
Cool, thanks! I have and use NDepend for my C# work, but I also do a bit of Java work and am interested in metrics there as well. At the very least, this gives me functionality for Java, and it may turn out to be a nice complement to NDepend. – Erik Dietrich Feb 20 '12 at 20:09
Awesome, we use Sonar where I work and it does some really nice things that give you insight into the state of your codebase. – Robert Greiner Feb 20 '12 at 20:10
+1 for sonar, just looked at the site seems like a good tool to have in the utility belt. – Mike Brown Feb 20 '12 at 21:35
@ErikDietrich, FYI Sonar has a C# plugin too. – Péter Török Feb 20 '12 at 23:06
up vote 4 down vote

I think the question is how much would it cost to "buy back" your technical debt--that is, how much work is it to fix it? Well, it's up to the team to figure that out.

During sprint planning, I ask the team to estimate the complexity of fixing technical debt items in the same way they would estimate the complexity of a user story. At that point, it's a negotiating game between the team an the product owner to determine which technical debt is high enough priority to be done in the current sprint (displacing actual user stories) and what can wait.

If you're not doing scrum, I'd stick to my premise--technical debt should be measured by the cost of the remedy.

share|improve this answer
So, in the context of story points, is it fair to say that you could add a few points to each story if there was a high degree of technical debt represented by the affected areas of the code? That is, if story X involves adding to code element Y, which is just awful, you tack on a few points to the story specifically because of the nature of Y? And that number of points is the same as or related to the number of points to perform the fix that you mentioned estimating? – Erik Dietrich Feb 20 '12 at 19:33
@Erik Dietrich - Well, the TD is definitely adding complexity to the solution. The difficulty may be that fixing TD piecemeal may be more costly than a wholesale solution. Thus it might be that you have 3 stories that would be rated at 5 each if the the TD was eliminated, but are 8 each with the debt in place--so that adds up to 9 points of TD. The task to fix the TD as a whole (irrespective of stories) may actually be an 8. So you can argue that the wholesale solution costs less (8) than the piecemeal(9). This would be part of the negotiation – Matthew Flynn Feb 20 '12 at 19:42
That makes sense. And, certainly, what I'm looking to get to is making a (somewhat) objective case to say something like "in one year, we can develop X new features if we just keep plowing ahead, but X+Y new features if we pay off some of this technical debt". – Erik Dietrich Feb 20 '12 at 20:07
up vote 4 down vote

I hate to use an analogy from finance but it seems really appropriate. When you're pricing something (assets of any kind), it can have both intrinsic and extrinsic value. In this case, the existing code has intrinsic value which would be a quantity corresponding to the relative quality of said code and it would also have extrinsic value (value from what could be done to the code) and those quantities would be additive. The intrinsic value can be broken down into credits and debits (good vs. bad) using whatever methodology you're using to score the code (+5 for comments/readability, -10 for code coverage, etc.)

I certainly don't know of any tools that quantify this today and I think you'd have an entirely new discussion on your hands if you argue the merits of different "debt valuation" strategies but I agree with Matthew -- the debt is the cumulative cost of getting the code as good as you can possibly get it, using whatever method you use to cost out the man-hours it takes to get there.

Something else to consider is that there is certainly a measure of cost-effectiveness whereby as one gets closer to "perfection", the value of an hour spent on the code base is more than likely decreasing exponentially so there is probably an additional optimization problem to maximize utility of the work done.

share|improve this answer
Yes, the concept of diminishing marginal returns is certainly something that I'd want to address in coming up with and refining the metric. So, not just "here's my objective argument for refactoring this class from a business perspective" but also "here's my rationale for not bothering at this point." – Erik Dietrich Feb 20 '12 at 20:15
up vote 2 down vote

There's a pretty strong platform out there called CAST to look for technical debt in big applications. We used it on a project where we took over a big enhancement to a legacy system. It doesn't tell you what was in people's heads who wrote the code, but it examines code and finds code and architecture flaws, then quantifies to technical debt if you want to. The real use in looking at this, though, is not the $ amount but the list of problems already in the code. This tells you about a portion of the technical debt you have (so I do disagree with some of the answers up above). There is some technical debt that's purely design-based and that's very subjective - like pornography - you know it when you see it and know the context. I would argue whether that's really "technical" debt. There is some technical debt that's purely in the implementation and I believe that's worth measuring and tracking.

share|improve this answer
I shared this question on twitter, and someone responded talking about CAST. I'm not really clear on what all it does after checking out their website. Is there a freebie or demo version of it to take for a test drive, by any chance? – Erik Dietrich Feb 24 '12 at 23:51
up vote 1 down vote

I'd say the standard code metrics can be used as a high-level relative view of technical indebtedness. VS Ultimate includes a Code Analyzer that will give you a "Maintainability Index" based on Cyclomatic Complexity, Coupling, LoC, and Depth of Inheritance. You can dive down into any trouble spots and see details (down to the function level). I just ran it on my project and the lowest scores we got were 69 on our Data package (configuring and initializing EF) and our Test Suite. Everything else was 90 or above. There are other tools that will give you more metrics like those discussed in Uncle Bob's PPP

share|improve this answer
So, say you had something not in the test suite or data package that was scoring below 90. Do you have a numerical threshold there where you say, "alright, that's not good enough and we're going to refactor"? Or, do you use this information to make the case to management or some stakeholder that a refactoring is necessary? That is, do managers/stakeholders care about Microsoft's maintainability index, or do you present that information in some other way? Or, do you just not present it and quietly fix the issue on your own? – Erik Dietrich Feb 20 '12 at 20:23
I love that question. My answer will always be that writing the best code you can is not something you ask permission to do. I use what Uncle Bob calls the "boyscout rule" (always leave the code in better condition than when you arrived) and I call opportunistic refactoring. The idea is that when you have to modify existing code, take the time to a)cover it in unit tests b) refactor it to be cleaner. Michael Feathers Working Effectively with Legacy Code provides some guidance on doing this. – Mike Brown Feb 20 '12 at 20:49
@Mike-That will get you fired in many development environments where tight control of all code changes are tracked and monitored. Especially if your seemingly innocent improvement that nobody told you to correct ended up breaking something that once worked. – Dunk Feb 20 '12 at 21:16
Note I didn't say dive in and willy-nilly clean up the code. I said to clean up the code that you are already working in. I've also worked with highly regulated code (get assigned a work item, have to provide a list of changes being made to address the work item for approval, perform approved changes). 9/10 times explaining the refactoring in the change request would result in approval. – Mike Brown Feb 20 '12 at 21:33
up vote 1 down vote

In between developers a fairly reliable measure of technical debt seem to be WTFs/minute.

Issue with this "metric" is that it is typically rather difficult to communicate "outside".

Metric that worked for me in communicating technical debt to "outsiders" was amount of testing and bug fixing effort (especially for fixing regression bugs) needed for successful delivery.

A word of caution: although this approach is quite powerful, one would better double-check with good old WTFs/minute before resorting to it. Thing is, it is quite cumbersome: to get the data, one has to carefully track time and accurately log it per appropriate categories.

  • it is so much easier to state 3 weeks total spent on implementing feature A than
     
    I spent 14 hours on draft implementation of feature A then 29 hours on smoke testing it then 11 hours on implementing fixes for regressions I discovered, then 18 hours testing the QA-ready feature implementation. After that, QA guys spent 17 hours on testing the initial candidate release. After that I spent 13 hours analyzing bugs submitted by QA for the initial candidate release and 3 hours implementing the fixes. After that, I spent 11 hours on smoke testing the changes I made to initial candidate release. After that...

Anyway data about testing and bug fixing effort has been quite easy to communicate in my experience.

For recent release, we spent about 90% time on testing and fixing regression bugs. For next release, suggest to allocate some effort on getting this value down to 60-70%.

Another word of caution. Data like 90% above could be interpreted not only as an indication of technical debt, but also (surprise surprise) as indication of one being not quite proficient in programming / particular technology. "You just make too much bugs in your code".

If there is a risk of data being misinterpreted that way, it helps to have an additional, reference data on something less WTF prone to compare against.

  • Say if there are two similar components / applications maintained by same developer(s), first releasing at "waste rate" about 50% and second at 80-90, this makes a pretty strong case in favor of second being subject of technical debt.

If there are dedicated testers in the project, they could also contribute to more objective evaluation of the data. As I mentioned in another answer,

With testers, you get someone to backup your understanding of design issues. When there are only developers complaining about code quality, this often sounds like subjective WTFs from behind the closed door.
 
But when this is echoed by QA guy saying something like component A had 100 regression bugs for 10 new features, as opposed to component B which had 10 regression bugs per 20 new features, communication suddenly turns into whole another game.

share|improve this answer
1  
I like this answer a lot. With a dedicated QA department, the ratio of regression defects to new defects is very straightforward to calculate and could definitely tell you a lot about technical debt. – Erik Dietrich Feb 22 '12 at 15:55
up vote 0 down vote

I wouldn't think of technical debt as dollars where you need a fancy model to quantify it. I would think of it as favors. If someone does you a favor and you are likely to forget, you write it down. When you take a short cut, write it down. This helps you remember, and more impotent forces you to acknowledge it. No fancy tool is needed. Notepad or Ecxel can do the trick.

share|improve this answer
2  
From a realpolitik perspective, I would argue that the people most willing to incur long term badness for short term results are probably also the people least likely to document their decisions. So, I agree with your idea in theory, but I think that serial "favor requesters" would be the least likely to keep track of the balance of favors. – Erik Dietrich Feb 20 '12 at 20:20
@ErikDietrich - I agree. And the worse serial offenders don't even know they're adding to their debt. (Similar to the worst credit card offenders crushing their credit ratings.) But the starting point assumes the desire to quantify, and it's hard for the non-writer to quantify it. You don't know where the poop is unless it's your dog, or you happened to step in it. – MathAttack Feb 21 '12 at 1:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.