Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have been told that the average number of bugs/defects per line of code is "constant" for different programming languages. 10 KLOC of Ruby would have the same number of bugs as 10 KLOC of c++. The argument is usually used to promote the use of expressive languages (think python/ruby over c++/assembly) since the number of lines to describe the same functionality would be smaller.

Does anybody know where this claim comes from? Does higher-level languages lead to fewer bugs?

share|improve this question
8  
Seems unreasonable considering that some languages encourage a style that packs more statements into a single line than others do. –  Caleb Feb 2 '13 at 16:08
9  
Bugs / LOC is a very wrong metric for everything. It depends on the language, but it depends far more on the programmer, writing it. So taking the average for the language makes no sense, since the large fluctuations are in the other variable. This is just IMO, ofc. –  K.Steff Feb 2 '13 at 16:39
1  
I thought bugs/loc are at best measurable by a specific programmer/developer and not for languages. Can you provide a source for your statement? –  mri Feb 2 '13 at 17:35
3  
I can tell you that the number of bugs/line I write in Perl will be much greater than the number I write in C. A friend of mine is a Perl wizard, and for him the bugs/line is much greater in C than in Perl. Hard to see how this metric could possibly be useful. –  Caleb Feb 2 '13 at 20:31
1  
Do you really think that {1≥⍴⍵:⍵⋄e←⍵[?⍴⍵]⋄ (∇(⍵<e)/⍵) , ((⍵=e)/⍵) , ∇(⍵>e)/⍵} is as likely to contain an error as int pivot = arr.Count / 2;? –  svick Feb 7 '13 at 20:03

8 Answers 8

Contrary to intuition, the number of errors per 1000 lines of does seem to be relatively constant, reguardless of the specific language involved. Steve McConnell, author of Code Complete and Software Estimation: Demystifying the Black Art goes over this area in some detail.

I don't have my copies readily to hand - they're sitting on my bookshelf at work - but a quick Google found a relevant quote:

Industry Average: "about 15 - 50 errors per 1000 lines of delivered code."
(Steve) further says this is usually representative of code that has some level of structured programming behind it, but probably includes a mix of coding techniques.

Quoted from Code Complete, found here: http://mayerdan.com/ruby/2012/11/11/bugs-per-line-of-code-ratio/

If memory serves correctly, Steve goes into a thorough discussion of this, showing that the figures are constant across languages (C, C++, Java, Assembly and so on) and despite difficulties (such as defining what "line of code" means).

Most importantly he has lots of citations for his sources - he's not offering unsubstantiated opinions, but has the references to back them up.

It seems to boil down to this: The average number of defects per kloc seems to be more a property of the fact that developers are fallible humans than of the peculiar advantages or disadvantages of a particular language or platform.

(Aside: If you don't already have Code Complete, go buy yourself a copy and read it thoroughly - it's well worth the investment. )

Update: There's another factor at play with some of the answers here - large scale statistics are useful for making general predictions but not specific ones. Consider, population mortality tables can predict about how many people will be killed in traffic accidents this year but can't tell you which people will die. Similarly, industry statistics that show a relatively constant number of defects per kloc can't be used to predict how well - or how poorly - a particular developer will perform or what will happen on a given project.

share|improve this answer
1  
Don't have a copy of Software Estimation, but in Code Complete McConnel cites Capers Jones "Program Quality and Programmer Productivity" 1977 report as the source of a table of errors per LOC per project size. The point McConnel tries to make is that errors increase dramatically as the project's size increases, and notes that the data are only a "snapsot of the industry" and that "the numbers may bear little resemblance to those for the projects you’ve worked on". I don't really see anything in there that has anything to do with this question. –  Roc Martí Feb 3 '13 at 0:09
    
Which edition of Code Complete do you have @RocMartí? I know that the second edition was a major update. Will have to dig it out and see what it says when I get to work Monday. –  Bevan Feb 3 '13 at 3:01
    
I think your edit (Update:) is the core of the problem. Or, as Mark Twain said, there's three kinds of lies: Lies, Damn Lies, and Statistics. –  GalacticCowboy Mar 26 '13 at 17:59

The claim is - at best - naive.

SLOC isn't exactly a reliable metric for anything useful, except perhaps comparing the size of two or more projects. Furthermore there are two distinct types of SLOC, physical LOC and logical LOC, and those might differ significantly. Consider this example, from Wikipedia:

for (i = 0; i < 100; i += 1) printf("hello"); 

Here we have one physical LOC, but two logical ones (for and printf statements). But we could of course write the example as:

for (i = 0; i < 100; i += 1) 
  printf("hello"); 

Which would give us two physical and two logical LOCs. I think it's clear that any "bug per loc" measurement that would depend on physical LOCs would be tainted by programming style, thus our measurement would be largely useless.

If, on the other hand, we went with logical LOCs then our measurement would heavily depend on the language's syntactic idiosyncrasies. Although the resulting metric might be a bit useful when comparing projects written in the same language, it would be fairly useless for projects written in different languages.

One possible source for the claim is Les Hatton's Software failures-follies and fallacies:

We can conclude that programming language choice is at best weakly related to reliability.

Later on, the paper mentions similar defect densities for C and C++:

In a recent study comparing two similar systems of similar size, (around 50,000 lines each), one in C and one in object-designed C++, the resulting defect densities were shown to be around the same at 2.4 and 2.9 per 1000 lines respectively.

This, however, doesn't mean that "bug per LOC" is constant across programming languages, or that it would be significant if it was.

share|improve this answer
    
If you assume that bugs/statement is constant then there is a difference for languages. The C example commonly has bugs in the for() and the printf() args. If you had to code the printf functionality out in full you would have proportionally more bugs and if you had a higher level language with a single printRepeat() call there would be less opportunities to get it wrong. –  Martin Beckett Feb 2 '13 at 18:36
1  
Summary: bugs per statement/function point is constant, low level languages have more code written by the fallible programmer, high level languages you type less - therefore less bugs. Although totally incorrect design type bugs are probably the same! –  Martin Beckett Feb 2 '13 at 18:37
2  
Let alone that what constitutes "one bug" is highly subjective, and that bugs differ wildly in severity, impact, and importance. –  tdammers Feb 2 '13 at 21:35
    
@tdammers And that importance can be negative. We have a handful of bugs that the client is used to/expects/wants, so we can't fix them... –  Izkata Feb 3 '13 at 17:19
    
@Izkata: depends on your definition of a bug... –  tdammers Feb 3 '13 at 19:35

I don't find Bugs per LOC to be constant at all for a given language. Bugs per LOC seem like a metric some Managers use to determine the quality of developers when it comes to review time.

Now outside of that, some languages are more prone to errors or defects than others. Usually, but not always this is a lower level language than a higher one. For example coding in C versus C# (or Java.) I say usually because the reality of it and the crux of the answer you are looking for comes down to quality of developer and the coding practices in place. I have seen very good C developers with much higher code quality and lower defect counts than average Java/C# developers. This is one item that separates a senior developer from a junior one. Not how many LOC they write in a given time frame, but quality of the code the write regardless of language, LOC or time frame.

The only thing answer I can give that might relate is that the more LOC there are the more likely-hood of there being a defect and the more defects that exists.

share|improve this answer
    
My question is about average number of defects per line of code independent of language. –  Kristian Feb 2 '13 at 17:52
4  
@Kristian there is no such number. It changes per person relative to the job and expertise of the developer and language they are coding in. I don't think there is a universal average. –  Akira71 Feb 2 '13 at 19:47

Bugs Per Line Of Code

Bugs / LOC is only relative to an individual. For businesses that implement bug tracking tools that link with their source code repository. It's possible for a manager to organize issues by developer, sorted by past issues and code changes.

Bugs Are Relative To Your Job

A senior software developer, who is highly experienced, highly skilled, very smart and able to take on independent jobs is far more likely to have way more bugs logged in a tracking system, then a junior developer with little experience.

How is that possible?

Senior developers are often engaged in higher risk development tasks. Refactoring of code and building new systems as an example. Junior developers are often assigned to fix known issues that aren't worth the time of a senior developer.

Therefore, by task assignment a junior isn't introducing bugs but fixing them, and a senior developer is allowed the risk of introducing them, because the benefit of what they are trying to archive is more important then the minor issues that are raised completing those tasks.

Language Syntax Is Important

The argument that a language introduces less bugs, because it can achieve more in fewer lines of code is a complete myth. Highly structured languages like C++/C#/Java force the developer to clearly express in writing what the desired instruction should be, where as languages like Python/PHP are very unstructured. Those languages allow for written expressions that not only will confuse a developer, but also the language parser.

The Compiler Reduces Bugs

How many bugs in Python/PHP have made it out into production servers, because there was no compiler to warn the developer that something was incorrect. When you measure bugs per LOC is that before or after a compiler has processed the source code?

Idiots Per Language

People program software at all different levels, from beginner to advance and there are programming languages for each of those. While developers will argue until their blue in the face about which language is better. There is no questioning the fact, that some languages attract more idiots then others. Some people learn Java, because they simply want to earn more money (not for any reason based upon the features of the language).

So if you would like to select a programming language, with the hopes of seeing fewer bugs, then select the language that has a smaller population of idiots in it. You'll be less likely to hire them later when the project has grown.

share|improve this answer
2  
Re compiler reducing bugs: Both Python and PHP technically have compilers, they just don't do the same checking that statically typed languages do. I also don't agree that such checking has a significant effect on the ending bug count because virtually all errors that can be caught by a compiler are caught with minimal testing. –  Winston Ewert Feb 2 '13 at 21:53

This observation is very old, and comes from a very venerable source, namely Fred Brooks in his book "The Mythical Man Month". He was a top manager at IBM, and managed many programming projects including the milions-of-lines operating system OS/360. In fact he reported that the number of bugs in a program is not proportional to the length of code, but quadratic! According to his research, the number of bugs was proportional to the length of the program to the power 1.5. In other words, a program that is ten times longer has 30 times more bugs. And he reported that this held over all programming languages, and levels of programming languages.

share|improve this answer

FWIW, in my experience

  1. There are two kinds of bugs: a) where the program does not meet expectations, and b) where the program cannot meet any reasonable expectations, because it crashes/hangs/won't compile.

  2. Regardless of language, bugs of type (b) are caused by redundancy in data/class structure, where changing something in one part of the data structure puts the structure in an inconsistent/broken state until one or more corresponding changes are made in other parts. Contributing to this is redundancy of source code, where an edit to one line of code makes the code incorrect until one or more changes are made in other parts. These two types of redundancy are closely related, of course, and since programmers are not super-persons they get distracted, forget things, and make mistakes, thereby putting in bugs.

These things (again, in my experience) are not really a function of the language, but of the skill/maturity of the programmer. Programs that are much less bug-prone also tend to be much smaller, in terms of LOC, for a given set of functionality.

I've seen systems where some people write programs, while others write directories, and the former tend to "just work" compared to the latter.

share|improve this answer

I would expect that a key factor in coding errors relates to what I call the "semantic gap" between a particular type of solution definition and the code to solve it - where these are close reformulation errors would be more apparent, where the code is very different, many errors could be expected. The paradigm of certain languages closely match certain problem domains - spreadsheets are very appropriate to everyday business calculations, resulting in both very little "code" and the "code" being very close to the problem domain. The expected code is both very concise (little KLOC) and few errors. Conversely using assembler would require many KLOC and is likely to produce immense number of errors.

share|improve this answer

Instead of talking about lines of code - which are indeed a useless metric - I'd like to address this part of your question:

Does higher-level languages lead to fewer bugs?

This is different than bugs/LOC, because higher-level languages do more with less code. Implementing some feature requirement might take 500 lines of LISP vs 15000 lines of x86 assembly.

So, even if bugs/LOC is constant between all languages, the higher-level language will still yield fewer bugs.

share|improve this answer
    
Lines of code a "useless metric"? No, it is a rough approximation of program complexity. It can be useful because it is easy to measure and is also closely related to development time. –  dan1111 Apr 21 '13 at 7:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.