Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Recently I attended a lecture given by Greg Wilson (Chief Scientist of Software Carpentry). From the abstract:

The idea that claims about software development practices should be based on evidence is still foreign to software developers, but this is finally starting to change: any academic who claims that a particular tool or practice makes software development faster, cheaper, or more reliable is now expected to back up that claim with some sort of empirical study.

Overall, the lecture was very informative and left me thinking quite deeply about my approach to development. In particular, I now find myself looking for citations to back up a lot of statements. Previously, I had slipped into the habit of simply repeating offered truths, with perhaps a mental note to go check up on it later.

Putting it bluntly, I was being gullible.

Here's an example taken from the lecture:

"If more than 25% of the code needs refactoring, it's quicker to rewrite it".

Sounds plausible, but is it true? Where's the study backing this up? Is it true for all languages? And so on.

OK, it's quite possible to take this to an extreme and not believe anything by anyone unless you have derived it yourself from first principles. That way lies madness (or maybe mathematics ;-) ). But, if someone comes up to you with a statement along the lines of "Hey, by doing this in [pick language of moment] we'll be able to boost productivity by [pick multiple of 10]%" are you inclined to just accept it, or are you going to ask for proven evidence?

If it's the latter (and I hope it is) then

  1. where would you go to find this evidence?
  2. how stringent would you be?

In short, if someone offers you an unverified statement, will you respond with "citation needed"?

share|improve this question

closed as not a real question by Mark Trapp, David Thornley, Gaurav, John Straka, bigown Dec 14 '10 at 20:23

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

In how many fields, outside science, do people demand empirical evidence? In my observation, not nearly as many as I'd like. – David Thornley Dec 13 '10 at 18:32
How about some comments on the close votes? "Too localized" and "Not a real question" isn't really self-explanatory in this context. – Inaimathi Dec 13 '10 at 18:36
Yes, I too would like to know the reasoning behind the close votes. – Robert Harvey Dec 13 '10 at 18:48
@Robert Thanks for the edit. Much less inflammatory on reflection. – Gary Rowe Dec 13 '10 at 18:52
Great question. I saw Prof. Wilson speak at CUSEC last year and was also greatly influenced by what he had to say. The best part was when a student challenged him to cite his claim that claims should be cited, and he did without missing a beat. – Matthew Read Dec 13 '10 at 19:03
up vote 3 down vote accepted

The problem with these sort of statements is that even if you had empirical evidence supporting the claim it would be very difficult to determine if the study that lead to the evidence applied to your current situation.

Almost everything in the profession has a caveat, or several so every improvement in one place has the likelihood of being a disservice someplace else.

The folks down in the trenches know the difference through experience and generally do not have the funding/time/resources to try to prove it through a scientific study.

The folks that try to prove it through a scientific study obviously have resources to dedicate to such studies and are therefore highly likely to be selling you something so I would say that you should be even more stringently applying your own personal experience to anything that claims to be backed up by empirical research.

If someone told you "If more than 2% of the code needs refactoring, it's quicker to rewrite it" you would know that to be false as much as if someone told you "Only if more than 98% of the code needs refactoring, it's quicker to rewrite it". Where the actual number is depends on what you are doing and how far from ideal the current code is.

The idea that after a certain point it is easier to do a nuclear refactor is obviously true in many cases, but the actual percentage is more of an example that you need to consider through the lens of your (team's) own experience and current situation.

share|improve this answer
+1 for a thorough analysis of the example statement. Do you really think that all scientific research has a commercial angle that has to be exploited, though? (Forgive my naivety but I'm genuinely curious about this) – Gary Rowe Dec 13 '10 at 20:36
@Gary : All things being perfect no, but it is very difficult if not impossible to determine the bias of a study from the outside. Doubly so when there are no agreed upon metrics that cover the entire situation – Bill Dec 13 '10 at 21:32
If someone offers you an unverified statement regarding software development practices, do you respond with “citation needed”?

No, I post it here and see if it gets any upvotes. Social proof is better than no proof!

share|improve this answer
You might get somewhere with this model, but I shudder at the thought of a paper citing programmers.SE as a source. – Inaimathi Dec 13 '10 at 19:00
@Inaimathi: it's at least as reliable as wikipedia, if not more so! – Steven A. Lowe Dec 13 '10 at 19:50
+1 for seeking proof - always good advice. How many upvotes before you believe it? ;-) – Gary Rowe Dec 13 '10 at 20:29
@Gary: on SO, ten. on this site, twenty. on meta, a hundred - unless it involves unicorns and waffles, in which case it must be true – Steven A. Lowe Dec 13 '10 at 21:37
Love the unicorn reference - must get me one of them – Gary Rowe Dec 13 '10 at 22:10

Many developers base their moment to moment decisions on experience in the trenches working with code and the customers which that code serves.

When a class or method has become so fragmented by bug fixes and customer change requests that it has become unmaintainable, a developer will sometimes make the decision to rewrite it rather than refactor, under the theory that he will save time and effort over the long term, because the resulting code will be of higher quality.

This kind of experience intelligence is what HR departments call "human capital." It is one of the things that makes experienced developers valuable, and one of the reasons that good companies do things to try and preserve the longevity of their people.

It doesn't seem fair or even practical to ask experienced developers to come up with a study and empirical data as proof that their techniques are valid. Experience doesn't work that way. To the contrary, experience is something of a "felt sense." In the refactoring world, we often call it "smell."

Ultimately, a statement like "If more than 25% of the code needs refactoring, it's quicker to rewrite it" cannot be proven to work under all circumstances, so the statement [citation-needed] is really a way to inform the dogmatic programmer who seeks to force his views on others that it's not always "His Way or the Highway."

share|improve this answer
Ahh, good old human capital and human resources, those wonderful twin phrases promoting the ongoing dehumanization of workers everywhere... – Aaronaught Dec 13 '10 at 19:04
@Aaronaught: The execution of a concept can sometimes fall short of the lofty vocabulary used to describe it. Which is why skeptical people sometimes ask for proof. – Robert Harvey Dec 13 '10 at 19:11
Wouldn't it be a good thing for the execution of those particular concepts to fall short? – Aaronaught Dec 13 '10 at 19:18
+1 for a good use of the "citation needed" defence against a dogmatic programmer - very useful – Gary Rowe Dec 13 '10 at 20:27

I think with anything you never know until you try it. Even with proof to back up a statement, it's always possible to bend facts to the benefit of your point. That being said you shouldn't try every new thing that hits the interwebs. Make your best judgment. Remember, if something is sounds too good to be true it probably is. Always ask yourself why do you need to adopt something? What do you have to gain? Does it make sense from a business prospective? Never blinded except something on faith.

share|improve this answer
+1 for asking "why do you need to adopt something". Sometimes stepping back from the leading edge is a good thing. – Gary Rowe Dec 13 '10 at 20:32
I find that too often developers get caught up in the next best thing without analyzing it and understanding how it could both benefit and determent them. I have seen situations where organizations adopt something like Asp.Net MVC over Asp.Net Webforms but don't adopt TDD. – Carlosfocker Dec 13 '10 at 22:18

The example from the lecture is a heuristic, a rule of thumb and nothing more. That should be implicitly obvious.

Heuristics are like anything else: they are subject to a certain context and dependent on any number of unstated assumptions, and their usefulness can be very non-deterministic. As much arbitrary judgement goes into finding them useful as goes into formulating them in the first place.

Does that means that they are without worth? I wouldn't say so at all.

Heuristics are one of the approaches that we can take toward addressing NP-complete problems, and in many respects, software engineering is itself an NP-complete problem.

share|improve this answer
Good points. =) – Pablo Dec 13 '10 at 22:45

It depends. :) When someone's statement contradicts repeated, reflected-on, and personally verified experience, then yes, I'd want to see some sort of reference of a study. On the other hand, if someone echoes an idea you've seen and lived many times, there's not much reaction provoked (doesn't mean that the idea is infallible though).

As an example, the book "Code Complete" cites scores of studies in each chapter to make its points, sometimes about seemingly small matters (like indentation and spacing, or variable name length). I recall some (younger) developers whom I introduced the book to thinking that that level of detail and evidence was silly. But a few months later with more production coding experience and after a few code reviews, some of those same developers had the honesty to admit that yes, even the number of spaces in indentation does matter. Good comments matter. Encapsulation matters. etc. etc.

On the other end, when a vendor claims some new IDE is 50% more productive, my first reaction is bull$%^&!

share|improve this answer
It depends, but even the code complete examples depend on your situation. For Example: If you have a parsing formatter and your diff tool ignores whitespace then the indentation argument becomes fairly trivial – Bill Dec 13 '10 at 18:43
+1 for the Code Complete example (also true for Rapid Development by the same author Steve McConnell). – Gary Rowe Dec 13 '10 at 20:47
@Bill It's interesting that as technology improves the problems that required proof earlier disappear. I wonder if this is an effect that reduces our need to ask for proof? – Gary Rowe Dec 13 '10 at 20:49
@gary I don't think this (in the general sense) is a case of improvement as much as variation. The code complete examples are definitely referring to a specific toolset at a specific point in time so they are both, but the "when to refactor" type has a lot more to do with the situation than with the tech. Think of a safety critical software in a vehicle vs a data processing solution. The testing requirements are going to be a lot higher on the safety system, so the refactor point is always going to be a lot higher before there is a net gain. – Bill Dec 13 '10 at 22:22

Isn't that something that depends on a whole lot of intangible variables (variables that have no way of being scientifically measured)?

In my opinion, they are talking about an empirical method for measuring emotions. Something that not even Spock could accomplish. =)

share|improve this answer
+1 for an interesting take on it. Putting the (intentionally) woolly example aside, if someone said to you that Rails is a better framework than SpringMVC how would you go about determining it's validity? – Gary Rowe Dec 13 '10 at 20:40
As Benjamin Graham states in his classic book about investing ("Security Analysis"), that (investment) tools were to be measured in two different, but lonely aspects: The Qualitative (intangible, feelings), and the Quantitative (real numbers, math, computation, logic). – Pablo Dec 13 '10 at 22:32
The Quantitative is what you are measuring through a scientific method. The Qualitative is what you measure through you own instinct. There is no possible to judge emotion versus emotion without having emotions about the analysis. – Pablo Dec 13 '10 at 22:34
To say the least, when we talk about opinions and differences in them, we just cannot agree on a reasonable, scientific, and tangible method to measure the abstract. – Pablo Dec 13 '10 at 22:36
My answer is that we can only measure technical aspects, not abstract thoughts/opinions. Also, I don't mean to sound like an ass. Those posts were not meant as some kind of foolish response. – Pablo Dec 13 '10 at 22:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.