Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Everyone seems to implicitly assume that the free market of ideas will eventually converge on the "right" solutions in software development. We don't assume that in medicine - we recognise that scientific experiments are needed there - so why should we assume it in software development?

I am not arguing for regulation of programmers. It is far too early to even talk about that. Before healthcare could be effectively regulated, there was a need for scientific experiments to establish which treatments worked and which didn't.

Software engineering doesn't even have this scientific evidence base to back up touted methodologies such as Scrum or Agile, or programming paradigms like functional programming or MDA. As

(a) large software projects are responsible for many government project failures (with the UK government being a really good example)

(b) Agile and Lean are being used outside of software development, including in the public sector [of course, Lean originated outside of software development]

this is increasingly politically relevant. Government project failures may be influenced by a failure to use a best practice, or even by using something that is considered by some people to be a best practice, but which actually makes things worse, or just costs money without really helping very much.

The question is, why is this scientific evidence base (for all intents and purposes) nonexistent?

There is a large open source community from which research participants could be drawn. My fear is that the closed-source and in-house software developers would treat with suspicion any research based on this community, fearing (perhaps rightly) that the results would not translate over. And companies that develop closed-source and in-house software would probably not be willing for their developers to participate in any scientific studies. For one thing, it would probably take time away from getting work done; for another, the results could be embarrassing to the company or to senior managers.

share|improve this question
3  
    
It looks like people are not getting the message. One of the answers to that question cited "case studies" (aka anecdotes). Another answer flat out stated "I don't know of any studies, but here is my personal experience". This may all give you warm fuzzies, but it is not scientific. –  Robin Green Mar 17 '12 at 13:33
    
@TomSquires I think it's similar and related, but not identical. That question that you linked to is explicitly about methodologies. This question is about empirical data in software engineering in general. –  Thomas Owens Mar 17 '12 at 14:07

3 Answers 3

There are decent libraries of research into software development projects. Just take a look at the IEEE Computer Science Digital Library and the ACM Digital Library for two examples. More specifically, the IEEE Transactions on Software Engineering and ACM Transactions on Software Engineering and Methodology present academic and industrial research into various aspects of software development, ranging from tools through methodologies and paradigms. In addition, IEEE Software is more of a magazine format that presents relevant topics and information in a easier to digest format for professionals "in the trenches", so to speak. So there is research out there.

However, one of the biggest problems is variation in the software industry. Projects depend on so much - domain, knowledge and skills of the people involved, process methodology, the choices made at various phases of the project (from the technology to use to the system architecture and design - there are often many good choices). It becomes very difficult to look at projects and generalize information in such a way that it is both scientifically valid and useful across the majority of projects.

Your comments about cost and embarrassment are also probably true, at least in some organizations. Organizations pay their engineers to product shippable products. From what I've seen, the majority of the research involving commercial products involves three things: a description of the technique before some change, a description of some technique after a change, and the impact of this change. Going back to embarrassment, some companies aren't going to publish reports that say their productivity decreased or their defects increased and analyze why this happened. Instead, they are going to tend to show off techniques that improved the organization and discuss why. However, in science, the failures are just as important as the successes, especially when something that didn't work on one project can be improved or adopted and work extremely well on another project.

share|improve this answer
1  
People are very different too, and yet randomised controlled trials can be used to account for differences between people in medical and psychological studies. In principle, differences between teams should not be an insurmountable obstacle. –  Robin Green Mar 17 '12 at 13:27
1  
@RobinGreen That's true, and some studies do take a psychological or sociological approach. However, there are many variables in a project, ranging from people to tools. By the time you start reducing the variability, you end up with data that's interesting academically, but all you did was describe one set of projects with one group of people using one set of tools and processes. It becomes harder for me to determine if it's applicable to my people using my tools and processes. And to compound it, companies might be adverse to experimenting with projects that they are relying on to make money –  Thomas Owens Mar 17 '12 at 14:05

Wouldn't that be great if there were. This would probably come from the universities, but they're battling the problem of not producing programmers who know what they're doing when they graduate. Do we see them as teaching theory and doing research or doing technical job training. If you can't deomonstrate graduates get jobs, enrollment/money drops. The money for research has to come from somewhere. You'd think the less technical business managers would be interested in funding the research, so they could use it to base decisions. Applying scientific research in business is just scratching the surface.

The medical profession is a good analogy. When someone graduates from medical school (theory and research), they're not ready to be practicing doctors. They do internships and a residency for several years and possibly go into specializations. This happens at the hospitals. Companies that hire jr. devs need to understand they are still "practicing" programming and require mentorship and training. This could free up the CS departments to devote more time to researching the latest and greatest.

A lot of programmers are in jobs where they don't get to consider the hyped technologies whether there's any data to support them or not. Unlike switching a patient's perscription, rebuilding legacy applications may not be cost effective. How much time does a doctor invest in learning about the new drug compared to what it would take a programmer to learn a new platform or language?

What do we expect the research to tell us? If you fully understand a technology, apply it properly in the right situation, it will work the best. We're in a specialized industry. Yes, your product is as good as you've "hyped" it, but I can't hire enough devs that know how to use it and most projects don't have enough time for learning new things.

share|improve this answer
    
"What do we expect the research to tell us?" - This is exactly the problem with the unscientific mindset. Assuming that you already know the answer. Well maybe you don't. –  Robin Green Mar 17 '12 at 13:31
    
I think you vastly underestimate the amount of time, effort and money goes into your medical analogy. I'm pretty sure its easier for me to convince programmers to rebuild a legacy app than it is for me to convince doctors to change many of their practices. –  Fomite Mar 17 '12 at 23:55

In general, I'd say that it's pretty much impossible to draw any meaningful real-world scientific conclusions because every (non-trivial) software project is unique and software development is, by its very nature, non-deterministic. So I don't think scientific studies would give much useful information.

share|improve this answer
    
That sounds plausible at first glance. However, suppose the hypothesis is that "Scrum, correctly applied, results in fewer cancelled projects than Waterfall". Are you really saying that it's impossible to get useful reliable data on this question, and we have to go with our gut instincts? –  Robin Green Mar 18 '12 at 7:17
    
@Robin You don't need any studies to show that agile (iterative, incremental, collaboration) is vastly superior to waterfall (phased, sequential). That already established and not something we need to dwell on. –  Martin Wickman Mar 21 '12 at 11:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.