Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Software development today isn't evidence-based. That is, we don't really choose technologies and practices based on the outcome of research studies the way (we hope!) they do in, say, medicine.

As an example, there was a study published in early 2011 in the Journal of the American Medical Association that suggested removing lymph nodes of breast cancer patients does not improve survival rates, which had an immediate impact on surgical practice.

Imagine you found yourself reading a paper about a software engineering study which suggested that a software practice or technology that you currently use did not actually have the benefit you thought it did: i.e., this study was in contradiction with your personal experience. Assuming that the paper was published in a reputable venue that you trusted, what properties would the study need to have for you to actually change your practices based on the result?

share|improve this question

closed as primarily opinion-based by gnat, jwenting, Kilian Foth, mattnz, Martin Wickman May 27 '14 at 13:28

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

My practices while on payroll do depend on my manager's manager's manager's manager understanding software. – Job Jun 20 '11 at 2:47
You mean like "The Mythical Man-Month" or any of a range of other, similar, research-based publications? I hope all developers and their managers have read them (but I realise that's not correct). But then, handwashing in medicine is still not universally practiced despite the literature being quite solidly behind the idea. Maybe we all have unfulfilled aspirations? – Мסž Jun 20 '11 at 3:01

the study would have to be repeatable and free of bias - which is nearly impossible when dealing with developers.

for example, to determine if language X is better than language Y for a specific kind of problem in an objective manner, one would have to have the development team implement the system from frozen specs (a rarity in itself) using language X, then erase their memories and implement the system again in language Y, to account for bias introduced in already solving the problems with language X and knowing more about the domain and solution after the first implementation

for things outside of pure development, it is somewhat easier, as a large enough study could show correlation - but the typical reaction to such things is likely to be that the effort of learning how to do something differently (e.g. wideband delphi estimation) exceeds the benefits of the improvements, and/or the circumstances of the study (e.g. a large team of similarly-skilled peers) do not apply to your situation

share|improve this answer
+1: I was a research chemist before I was a programmer, and my doctoral adviser was a stickler for treating data as sacrosanct, being conservative about conclusions, and not treating journal articles even from famous researchers as holy writ. However, in my experience, few programmers, technical managers, or even doctors approach things that way. If there's ever a, I'll post some stories there, like how rats react differently to purified, white drugs than to brown sludge that's been sitting on a shelf for ten years. – Bob Murphy Jun 20 '11 at 3:29

I don't need a study. I just need someone to say "I tried X and it was good because...", along with some example code. If there is no code, I'm usually not interested.

share|improve this answer
"I tried not surfing the net while I was working" and it was excellent. No code, sorry. – Мסž Jun 20 '11 at 3:53

Painting isn't evidence-based either, but you can measure software production. I remember reading a software engineering textbooks, and I'm not 100% sure, but I believe there are quantitative studies on software deliveries. I believe that is one of the reasons that the industry is moving from waterfall to Agile because Agile approaches tend to offer more predictable results. I'm in the camp that if you don't have evidence to believe in something, then there is no reason to believe it. I will change my practice as long there is scientific data that a software development method delivers high-quality-software that meet expectations on time.

share|improve this answer
I think agile improves predictability of low-risk projects, thus increasing the profitability. I.e. the code quality and developer productivity may not be increased, but the profit rate (revenues / money invested) can be increased because you avoid working on features the customer won't need. – Giorgio May 12 '14 at 7:07

After picking myself up off the floor from laughing at the comparison of the medical and software industries, then I read the "Imagine...." paragraph, and have one question for you - what color is the sky on your planet? Mines a nice shade of blue......

The medical industry stopped (as a rule and because of rules) chasing silver bullets a long time ago. There's nothing in the laws of any country I know that makes it illegal to promise solutions to a programmers every ill. Hence no study can be taken seriously. (Yep, call me a cynic, I won't disagree)

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.