Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The Problem - Code Reuse

I was thinking about this question on software delivery, and I kept coming back to the issue of repeatability and / or reproducibility. They matter, because if you don’t repeat a project then it becomes more difficult to improve the process you used to build the project. Engineering involves constantly improving the processes involved with design and construction in order to produce higher quality projects.

Software can rely heavily upon reuse due to its digital form. Instead of rewriting a module, we just call it again or copy it to the other system. Some examples are authentication / login or perhaps a logging function. There are many well known examples for those categories, and conventional wisdom is to reuse what exists instead of rolling your own.

Comparisons to Other Disciplines

Construction

In contrast, construction of physical systems (buildings, bridges) is nowhere near as reusable. It’s true that the blueprint of a house can be reused many times to build the same copy of the house, but the construction must be performed each time. Cut & paste doesn’t work like that in the analog world. Bridge blueprints are less reusable that houses because site conditions will vary.

Master builders are experts recognized for having designed and / or built tens, hundreds, or thousands of things in their area. For example, Frank Lloyd Wright, a world renowned architect and designer designed more than 1,000 structures and completed 532 works. Contrast that with Anders Hejlsberg who has designed “just” five languages (Turbo Pascal; Delphi; J++; C#; Typescript). In many ways, it’s an unfair comparison because the domains are different. But at a broad level, the quantifiable production from two very intelligent people is vastly different.

Martial Arts

Martial artists will say that mastery of a move comes only from thousands of repetitions. After a good portion of those repetitions have been put in, many martial artists are surprised at how a previously perceived to be complex kata or form has become simple. Instructors of those students will also notice how the motion becomes more fluid and purposeful as well as having an economy of motion. Likewise, experienced martial artists are able to pick up more complex katas more quickly than less experienced students. Experience from repetition has given them a framework or process that allows them to learn more quickly.

Woodworking

Woodworkers experience a similar transformation. Hobbyist woodworkers always refer back to their first project that required a lot of drawers. If they complete the project, they gain a new appreciation for the efficiencies that assembly lines produce. There are other benefits such as a better understanding of how to lay out the drawers parts on the sheet stock in order to maximize use of the wood. Compared to hobbyists, professional woodworkers are able to more quickly design, start, and construct items that they have made many times before. They also gain an ability to see inherent issues within someone else's design having made that mistake in their work.

Does software reuse prevent software developers from becoming more proficient?

In many ways, software design and construction is always new. We don’t repeat past works, because if we can reuse a module, library, or system then we do. We’ll preferentially extend an existing system before rewriting the entire thing from scratch. But repetition is what allows us to find efficiency in the design and the construction. Anyone who has practiced a sport or physical activity will tell you that repetition is the key to becoming a good practitioner.

My question: Does software’s ability to be reused prevent the necessary process improvement and efficiency that comes from repeating a project?

share|improve this question
    
If you've written a piece of code, you've essentially solved a problem. If you are good at it, this piece solves a CLASS of problems. If you are really good, it is extensible to a metaclass of problems. And then you lose interest: there is no need to perfect one bicycle if there are unsolved design problems lying around. The thrill of problem solving comes from shining new stuff, not from polishing old problems into perfection. –  Deer Hunter Jul 14 '13 at 19:12
2  
good software projects "shift" a lot of repeatability into QA. When I was a tester in 1,5 year long project, we run test cycles at weekly "checkpoint" releases, about 70 times total through the project. That was... quite repeatable, softly speaking (not much things change in a week). Testing nightly builds has been, naturally, even more repeatable - about 500 times through the project (few entertaining showstopper bugs were too rare to make a difference). Now, tell me a construction company that has built 500 bridges - all with the same team –  gnat Jul 16 '13 at 23:23
    
@gnat - that's an excellent insight and a perspective I hadn't pondered yet. Other aspects of the SDLC become much more efficient due to that repetition. –  GlenH7 Jul 16 '13 at 23:58
1  
@GlenH7 expanded it into the answer, mostly to include pictures of bridges :) –  gnat Jul 17 '13 at 0:58
    
How many problems did Frank Lloyd Wright have to solve in his 1000 structures versus Anders Hejsberg in defining his mere 5 languages? Wright got to make decisions by decree, Anders had to justify decisions to many people just as smart and knowledgeable as him. I'll bet that Anders had to solve many, many more issues. So your throwing numbers in the mix is merely on what you are choosing to count and not any REAL quantifiable comparable numbers. So I like the question, I just don't like the reasoning/examples inspiring the question. SW efficiency has improved tremendously over the years. –  Dunk Jul 22 '13 at 14:46
show 1 more comment

7 Answers

up vote 9 down vote accepted
+50

The ability to reuse software does not prevent process improvement.

If you think about the processes that go into building software - developing requirements, designing the system, implementing the system, deploying the system, managing requirements, managing configurations, verifying and validating work product, tracking changes, and a number of others (see the CMMI process areas for one possible breakdown of key activities in software development) - these are repeated on every project regardless of how much reuse you have. In addition, each has some kind of quantitative and qualitative measures that can be used to determine how good that particular process or activity is, and as a result, how good the development process as a whole is.

In one end of the extreme, we can assume a robust software product line. At the other, you can assume greenfield development. There's still a need to perform all of these processes, to varying degrees, although they may happen at different rates or perhaps even in different sequences. For example, in a high amount of reuse, a greater percentage of the allocated time may be spent on integration and verification/validation activities at a system level (requirements V&V, integration tests, system tests, acceptance tests). With new development efforts, a greater percentage of the time may be required at design and implementation. As long as you perform a process at least once during the course of a project, you can measure it (quantitatively and qualitatively). Once you make adjustments, and see how those adjustments impact some measure of either the process area or the overall capability to deliver software, and then refine the process for other projects.

They key to process improvement is to have some kind of logical breakdown of your activities and processes, determine how to measure them (preferably consistently), and how to understand those measurements to make process changes toward some end. It's not about repeating the project, but about consistency in how you repeat the process.

share|improve this answer
    
depends on what is actually being reused, it might even fall in CMMI Acquisition, i.e. not development work. –  imel96 Jul 17 '13 at 1:15
    
But CMMI hasn't succeeded in any meaningful way. None of the "killer applications" of the 21st century were built by people concerned with the CMMI maturity matrix. Some brilliant people had an idea and implemented it, and then hired more brilliant people to increase the scale of the solution. Contrariwise, projects that probably did at least pay lip service to standards like CMMI have failed miserably, e.g. the US Department of Defense's attempt to build a new payroll application. –  kevin cline Jul 17 '13 at 18:57
1  
@kevincline It doesn't matter that CMMI has or has not succeeded. Sitting in the aerospace/defense industry, I see CMMI in my organization and the companies that we work with, that we are subcontractors to, and that we subcontract. However, my point is that in order to have process improvement, you need to identify your processes. CMMI is a single tool to do just that. There are others out there, and you can define your own. Once you have processes, you can define them, measure them, and improve them. –  Thomas Owens Jul 17 '13 at 21:00
    
@Kevin:"Killer Applications" are by their nature "outside the mainstream". So it would be no surprise that most of the new and innovative work was created by experimentation and hacking rather than via some disciplined process. Although, "killer application" is up to one's definition. Is an application that becomes a fad really a "killer application" or is the DoD program that allows Jet Fighters to safely fly and prevent them from shooting their allies down more of a "killer application". Fads frequently require no skill/innovation at all (e.g. pet rock, hula-hoop)...... –  Dunk Jul 22 '13 at 14:56
    
...including many really popular "fad" application programs. Whereas, large DoD type projects almost always require a tremendous amount of skill and process. Also, your view of CMMI's failure probably says more about your experience (or lack thereof) in industries that use CMMI than CMMI itself. CMMI is not perfect, and probably not even good, but at a minimum it gets companies to at least attempt to write down and follow a process and even try to improve upon it. If that is all CMMI accomplishes then it is a success. –  Dunk Jul 22 '13 at 14:57
add comment

I think idea that other engineering disciplines don't make use of reuse is wrong. Even when designing buildings/machines you still have components that are used by many other projects. For example, do you design you own screws? Engines? Doors or windows? Of course not. Those are often designed by different people who then use them in different product. And they are quite often standardised, which promotes even more reuse.

I think the problem likes in complexity. You simply cannot compare complexity of even most complex buildings to complex software. It is generally accepted idea, that software complexity is what makes it hard to approach from engineering side. The moment you have process in place, that allows you to create software of acceptable quality, you find you the complexity of software you need to create jumps in order of magnitude. So the process cannot be used. So if we had to repeat some part of software multiple times, until we are satisfied with the result, we would never finish that software.

That is why clean code is promoted. Ability to change past code based on new experiences can be said to be form of design reuse. So instead of creating different software multiple times, we refactor and refine single piece of software by reusing new experiences and design on old problems. All while trying to make software do the same thing.

share|improve this answer
    
It's not that other disciplines don't reuse design, the difference is the amount of reuse. All the objects you mentioned have to be physically constructed for each instantiation. I can't just copy & paste a door, for example. The repetition that comes from construction leads to identifying efficiencies and improvements that are not obvious at the onset. Build a set of kitchen cabinets and you'll have discovered new things between the 1st and the last. You have a point with the overall complexity, as the virtual nature of software allows us to pile complexity on unknowingly. –  GlenH7 Jul 15 '13 at 11:46
1  
@GlenH7 The thing is, software development is not building. Its designing. With building stuff, you are doing same thing over and over again. But with design, you always have different goals and problems. You should not compare it to construction of building, but to creation of it's blueprint. –  Euphoric Jul 15 '13 at 12:51
2  
I'm not sure I fully agree with your point on software development. SW development is both design and construction. The construction should provide a feedback loop to the design. In both analog and digital realms, good architects "get their hands dirty" and build in order to complete the feedback loop. Even if we focus on design alone, I think repetition of the design identifies efficiencies that leads to better design. SW doesn't repeat itself like other fields do. Each bridge requires modification from a general approach tailoring it to the site it's going in to. –  GlenH7 Jul 15 '13 at 14:26
    
SW dev is not so complex compared to the design the architect would draw up. Its just that we think its hard because we don't treat software as a proper engineering discipline, and because we keep reinventing things. If only you knew what went into design of other stuff you'd see that most software should be trivial, but we make it hard for ourselves :( –  gbjbaanb Jul 17 '13 at 12:43
    
To compare to the bridge - you're right, bridges are a solved problem. You want a new bridge, dust off the old designs and make a few tweaks and you have a new bridge (I exaggerate the simplicity here of course). So why isn't a web service similarly constructed in software? That's why software isn't engineering IMHO, we treat it more like a craft (or art) where every project is custom work. –  gbjbaanb Jul 17 '13 at 12:45
add comment

Does software’s ability to be reused prevent the necessary process improvement and efficiency that comes from repeating a project?

I have worked as a systems and software engineer in the same large project for the past 17 years, incidentally (thinking of the Airbus A380 reference in your first link) in the aircraft industry, though my responsibilities lie in the military aircraft sector. Stories like that are basically pure fiction, and actually really funny to watch when you have insider insight.

But for your brief and concise question: From my experience, I would say both yes and no.

Let me first say that I am all for software recycling in all forms (well, maybe not all...). The advantages of reusing just about anything, from cut-and-paste code snippets and algorithms, to whole code modules and function libraries, is on the whole far better than to always start from the beginning again (to push it a little).

The downside is, as you point out (or at least infer), that when you add functionality by simply putting together a given set of components (and, yes, I am simplifying this to the extreme), you do not really evolve as a programmer, engineer or whatever.

Just looking at the software engineers around me at work, I know from long experience that a majority of them do not know, and worse - have no interest in learning, anything about the product we are constructing other than the bare minimum they need to produce the document or the piece of code that they are assigned to do.

I am reeling a bit off topic here, but my point is that when the programmers do not need to learn what the code they are constructing will really be used for, and do not need to learn the inner workings of the system since they can just reuse already written and tested components, then most of them just will not bother to do so.

Granted, this is also due to other circumstances, such as that the product we are constructing is incredibly complex, and it would be impossible for one person learn about all of it (and I am just talking about one of the computers in the aircraft - the most complex one of them, but still).

If our software engineers did not have the option to re-use as much code, I am convinced that they would become better at their profession generally, and much greater assets to the project specifically.

Oh, and you may have noticed that I talk about them a lot here. I am of course also included among these software engineers. The exception being that I seem to be a lot more inquisitive and eager to learn new things then the others :-) When faced with a new task, I always take it upon myself to learn as much about it that I can, both in the form of facts and by studying source code (yes, I actually enjoy that too).

Ah - dang, side-tracked again... My excuse is that I have not slept for 32 hours, so my focusing ability is a bit... what was I saying?

If anyone is still reading, my conclusion is that:

Yes, too much reuse of software makes for less knowledgeable software engineers, which makes them markedly less efficient when they actually need to know how the stuff works. Problem analysis is a good example, or even just being able to tell if a suggested design solution is viable. And of course, process improvement is also more difficult to achieve when you do not really know what you are doing :-)

and No, reusing software with care, potentially give you a lot of spare time to consider and plan process improvements.

share|improve this answer
    
Isn't the fact that most sw developers can get by without knowing the inner workings of the system a strong indicator of extensive reuse? I also find it funny how government project stories get out that sound simply dreadful but if you had any knowledge about government work then you would understand how clueless the author is. The $1500 hammers etc... All become understandable when you recognize that government processes required 10 people to review and get competitive quotes before purchase OR it was simply not moving funds between accounting buckets. –  Dunk Jul 22 '13 at 15:21
add comment

As pointed out in the accepted answer in another Programmers question, analogies with construction are to be taken with care:

a recommended reading for this is The Software Construction Analogy is Broken

software is often likened to construction. Unfortunately this analogy is flawed and lessons learned from the construction industry are suspect...

What I observed is, good software projects "shift" a lot of repeatability into quality assurance.

When I was a tester in 1,5 year long project, we run test cycles at weekly "checkpoint" releases, about 70 times total through the project. That was... quite repeatable, softly speaking (not much things change in a week). Testing nightly builds has been, naturally, even more repeatable - about 500 times through the project (few entertaining showstopper bugs were too rare to make a difference).

Now, following that "suspect" analogy, tell me a construction company that has built 500 bridges - all with the same team.

  • Following it further, tell me a company that has been using mostly the very same bricks and iron at each of their new bridges (here, I refer to the fact that releases we tested had mostly the same bits of code day by day, week by week - "not much things change").
    http://upload.wikimedia.org/wikipedia/commons/thumb/0/0c/GoldenGateBridge-001.jpg/220px-GoldenGateBridge-001.jpg http://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Ponte25Abril1.jpg/220px-Ponte25Abril1.jpg

Master builders are experts recognized for having designed and / or built tens, hundreds, or thousands of things in their area.

Fine, following up your explanation of repeatability quoted above, I can say so what? Back then, our little, not very special group of testers has verified, see above ("about 500") hundreds of things in our area.

As for project developers, they have literally built ("nightly builds") - see, the word is the same, and the meaning is right in this context - hundreds things in their area.

If one wants to continue that "suspect" analogy further up to "thousands of things", these amounts are, again, nothing spectacular in software development when one looks at the right things.

  • As an example, yet another of my past projects (again, nothing spectacular, rather regular one), this time in dev role, has been going on for more than 5 years (8 major releases, several dozens minor ones). There were similar weekly checkpoints (5x52~=250 of them), similar nightly releases (5x365~=1800 of them) and similarly, same dev / QA teams working on these. Day by day, week by week, month by month, mostly repetitive stuff (not much change between two nightly builds) - as promised, in the range of thousands times (1800).

Longer lived projects like Windows or Java, or AutoCAD can span 10, 20, 30 years, that easily accounts for repeating as many "thousands" nightly builds and nightly tests as it gets.


Concept of repeatability shift to QA becomes even more prominent with Continuous integration...

the practice... of merging all developer working copies with a shared mainline several times a day... CI can be seen as an intensification of practices of periodic integration..

In addition to automated unit tests, organisations using CI typically use a build server to implement continuous processes of applying quality control in general — small pieces of effort, applied frequently. In addition to running the unit and integration tests, such processes run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code and facilitate manual QA processes. This continuous application of quality control aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. This is very similar to the original idea of integrating more frequently to make integration easier, only applied to QA processes...

Repeatability? it's right there, as much of it as one can think of.

With frequent / continuous QA, things that go wrong quickly get back to developers who are forced to repeat attempts at doing it right until failing tests successfully pass. In a sense, that cycle of repeating until it passes resembles code cata,

an exercise in programming which helps a programmer hone their skills through practice and repetition...

share|improve this answer
1  
Excellent points, and I think the escapes that were then rolled back into the automated test suite capture some of the experience that I'm alluding to. Regarding the "same team" claims, I defer back to Wright's experience. With over 500 buildings constructed, he was a common element for all of those. But the point is made, and I agree with some of the premise. –  GlenH7 Jul 17 '13 at 2:13
    
@GlenH7 yeah impact of repetition has been truly profound and it went far beyond test suites. Knowledge accumulated everywhere where repetition happened - you know, everything tends to settle at optimum after 20...30...50 times doing it. Checkpoint / nightly preparations, bugs submissions (and the whole "bug life" at all), dev / QA / mgmt / sysadmins communication, documenting stuff etc etc. I only focused on repeatability, because diving into matters of knowledge accumulation that naturally followed would have a firehose effect on presenting my point –  gnat Jul 17 '13 at 2:35
add comment

Software is different than most other disciplines, so the economics of where we best spend our time is often different.

In construction, you spend a certain amount of time and money on a blueprint (and software is far more like producing a blueprint than like building a building), then, roughly speaking, a whole lot more on actually building it one or more times. So it is worth it to put quite a lot of work into getting the blueprint right. More specifically to your question - it's worth repeating the effort of doing it over from scratch to make the end product a little better.

In software, when you have the blueprint, it is much cheaper to build the product than it was to make the blueprint. At least most of the time - if the software will be embedded in a pacemaker you are much closer to the situation of a bridge builder in some ways. But in general, reusing software might save 90% of the cost of your biggest budget item, vs. saving 90% of a much smaller budget item for building a bridge. So, re-use wins a lot more often.

As far as productivity - when you build, say, a bridge, you face really significant real world constraints. Imagine if architects were paid large sums of money to design bridges for massive multiplayer online games, where construction costs were near 0 and limitation significant less than the real world. They would design bridges that are freakishly complex by real-world bridge standards. The blueprint phase might take a little longer.

Also, there are a limited number of bridges to build, and, since design is a smallish part of the cost, you can pay for the best, and a few of the best can do most of the design. There are hundreds of thousands of software developers, and basically all of them have a giant backlog of things they would do if they had time. You aren't going to find one guy who does a massive portion of all that - it's kind of surprising that there are people who sort of come close, really.

Your real point seems to be that we may be losing something by re-using instead of trying to repeat and improve things. I think you have a point. The problem is that though it would most likely be more globally efficient to rewrite some of the foundational stuff and try to improve it, whoever takes that on gets all the risk and probably not that much of the reward. (There is also a huge practical problem of dependency hell, which probably takes some of the win out of rewriting things, but not to the point that it wouldn't be worthwhile, at least looking at the global picture. Copyright and patents also may force a proposed re-engineering effort bite off quite a bit of rewrite work to redo smaller pieces of existing code).

In terms of learning from repetition - in all disciplines this happens less in design than in construction, because there is less repetition, so less chance to learn, and perhaps less benefit. Also, the process of design probably just isn't that repeatable. It's a bit like having a process for writing a novel. A good process can almost certainly help, and software is generally far more collaborative than a novel, but repeating a process when the goal is to invent something new is problematic. Even novelists learn from the past, very much so, but a repeatable process is a secondary factor for creative endeavors. And if any part of software development is really truly repeatable, why isn't a computer doing it?

share|improve this answer
1  
Very good points surrounding the risk to reward aspects of reuse and rewrites. –  GlenH7 Jul 17 '13 at 2:16
add comment

Some of what you say is true: e.g. libraries solve functions not solved by high level languages which solve problems not solved by assembly which solve problems not solved by machine code. When you call System.out.println() in java you are losing sight of how a CPU outputs to a device.

So yes, you're losing something. What you gain is the ability to focus on unsolved problems. Now it may be that you need to immerse yourself in some other aspects of technology ( e.g. how networks function ) to solve a problem. But you don't need to become an expert in reading machine language when all you want to do is build a web page.

The same way, bridge builders are solving a slightly different problem every time ( its a different river ). They don't worry about how to create steel beams of a certain tensile strength, or how to machine bolts to a certain tolerance. They leave that to specialists who have solved that problem.

Look closely, and you'll see our entire society and infrastructure are built on 99% reuse and only 1% true progress. Most new things are just old things with a little extra something added or removed. It's the accumulation of human knowledge. You can write code in a high level language with decent libraries because someone figured out all of the mind-blowingly complicated stuff required to get to this point. It allows you to solve new and interesting problems.

To tie this all together and respond to comments: You don't have to solve problems that have already been solved to develop proficiency. Further, much of what you do will be reinventing the wheel. So in short, the answer is no - you don't need to re-implement the functions of libraries to become proficient. There is plenty of opportunity, some of it rote, some of it creative to hone your craft.

share|improve this answer
    
how toes this answer the question asked: "Does the ability to reuse software prevent process enhancements that come from repeating a project?" –  gnat Jul 17 '13 at 3:46
1  
I think you touch upon some potentially valid points, but I don't see them tying together to answer the question. And I don't agree with your 99 : 1 ratio for reuse. I think that grossly overestimates how much reuse occurs. Even within software dev, re-use rates are nowhere near that high. –  GlenH7 Jul 17 '13 at 13:50
add comment

It's all about resources. Years ago, if you developed software projects for large mainframe computers they might be around for 15 years or so with largely a static development environment. The FORTRAN program written to calculate payroll or the COBOL program was perfected over decades because it was constantly being used. There were resources to see how this could be improved. We don't have that sort of slow environment any longer to fine tune and polish the skills for a specific project. But we do take the skills and adapt them to the next project resources permitting. But in the end if becomes a choice of money spent on the new project to do the job, or to do the new job with a large amount of gold-plating. Gold-plating a project means to improve it to the nth degree and add a ton of bells and whistles even if the user didn't specifically ask for it.

The best we can do, is look at the overall design of a new project and see how it can be improved based on the past experience of the team. But it takes a real experience software architect to have a vision about what is actually considered improving the design to improve skills vs simply casing the latest buzzword in development such as Agile, OOP, etc.

share|improve this answer
2  
I understand some of the points you're trying to make, but they are based in presumption and unfamiliarity. I used to develop for mainframes, and I can assure you that the rate of development was just as fast as on open systems. The process was different, but the pace was the same. Your answer would be stronger by focusing on the transferable skills component, and expounding upon potential efficiencies gained that way. –  GlenH7 Jul 18 '13 at 11:14
    
You need to look at the history, there were not new technologies coming out every year or so on mainframes systems for CDC 6600 Kronos OS, for example. It was basically static for 15 years. Now things move much more quickly and there isn't the time to have the deep of knowledge gained over 15 years. There are no Flash programmers with 15 years of experience just doing Flash, for example. After re-reading my post, I stand by my original post. –  Edward Jul 18 '13 at 18:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.