In the Aggregate: How Will We Maintain Legacy Systems? [closed]

Question

NEW YORK - With a blast that made skyscrapers tremble, an 83-year-old steam pipe sent a powerful message that the miles of tubes, wires and iron beneath New York and other U.S. cities are getting older and could become dangerously unstable.

July 2007 Story About a Burst Steam Pipe in Manhattan

---

We've heard about software rot and technical debt.

And we've heard from the likes of:

• "Uncle Bob" Martin - Who warned us about "the consequences of making a mess".
• Michael C. Feathers - Who gave us guidance for 'Working Effectively With Legacy Code'.

So certainly the software engineering community is aware of these issues.

---

But I feel like our aggregate society does not appreciate how these issues can plague working systems and applications.

As Steve McConnell notes:

...Unlike financial debt, technical debt is much less visible, and so people have an easier time ignoring it.

If this is true, and I believe that it is, then I fear that governments and businesses may defer regular maintenance and fortification against hackers until it is too late. [Much like NYC and the steam pipes.]

---

My Question:

• Is there a way that we can avoid the software equivalent of NYC and the steam pipes?

Answer 1

A key problem relating to the maintenance of legacy systems is the lack of people who a) are up to speed on those systems and b) willing to continue maintaining them.

I recently asked a question along similar lines regarding whether younger programmers were interested in mainframes at all. The consensus leaned towards no.

Maintaining legacy systems is seen as career suicide. In a lot of companies where inertia rules, there is little investment in training staff to remain on top of those systems, leading to single points of failure on the personnel side. A lot of people I know who work on similar systems are looking for routes out because they see no long term future in the systems and they see only career detriment.

In some industries, datamaintenance regulations may be a key factor ensuring that legacy systems are reasonably watched over. This is particularly the issue in the financial industry I think. Those regulations - as far as I am aware - are generally time limited.

However, I think in practice, what will happen is:

There will come a point on the graph where the cost of cutting over to a more modern system which allows you to route around the disadvantages of a legacy system will fall to below the cost of maintaining the legacy system because it is cheaper.

IBM are selling a lot of mainframes at the moment, and they are working very hard to ensure that their large machines don't shut out swathes of younger professionals. But I don't think it's enough at this stage. They have some USPs in terms of carbon footprint and actual processing power.

However, for every one person who will buy an IBM mainframe because you can run Linux on it, it costs less in electricity, and is highly efficient, you will have several more who will choose a server farm because they are more familiar with that world and so are many more programmers.

Ultimately a lot depends on the industry involved. I work in a particular industry which has been very dependent on mainframes for years and years and they are still widely in use. But hosted solutions are becoming more popular which allows for the consolidation of skills in larger companies - this removes some of the issues faced by individual companies in terms of points of failure - and additionally, some suppliers are very strongly looking at non-mainframe based solutions for the problems inherent in that industry.

So I guess in summary what I am saying is that by and large, there will be a move towards migration from legacy systems as soon as an economic point of maintenance versus migration falls in favour of migration. It will, however, be largely invisible to a lot of people because it's not new and funky and doesn't have a very public face in the way that a next-big-thing does. That migration may be to service providers or to newer technology (especially where the service providers are the ones directly affected).

My experience - particularly in networks - is that there is a move to remove reliance on legacy systems.

Answer 2

Most businesses are already ignorant of technical debt, and don't even realize things are bad until it literally collapses around them and sends them into bankruptcy (if it ever gets to that point). I've actually seen that happen, and it wasn't pretty; what made it worse was the fact I repeatedly tried to make the business owners aware of the mounting technical debt and that it would have to be fixed, and every single time it was rejected due to an unwillingness to spend the required time and resources to fix it. The end result was after 10+ years the system finally failed catastrophically (after I was gone) and they couldn't recover, and went out of business, because they were too stupid to realize during those ten years that spending a bit of money to fix the problem would have been better than ignoring it entirely. I could rant for hours about the absurd stupidity of that company, and what pained me the most was it could have been avoided entirely if the owners weren't just out to make a quick buck by cutting everything else entirely.

It's insanely difficult trying to tell a business if their systems are badly written and need some heavy refactoring (if not a total rewrite which is normally the case because it is that bad). Most of the time they'll just shoot you down even if you warn them that it's hard to make changes, or add new features (the right way, I mean, not just layering more crap onto the pile), or even consider you a detriment to the business because you see problems with the system in its current state.

I've honestly come to the conclusion it's a losing battle, and one that isn't worth fighting. The people smart enough to know about technical debt don't need to be told twice about it and are aware of the dangers from the start, and everybody else won't listen to any kind of reason or warning until it's too late anyways. The best (and of course, most unrealistic) option would be to let natural selection kick in and let the ignorant people go extinct, leaving only the intelligent ones. I don't know of any more down-to-earth way of handling it, because everything I have personally tried in the past has either been ignored completely, reduced my value in the eyes of the company (for "complaining") or even resulted in my termination because I was "too focused" on fixing "what isn't broken", when in reality it was broken and nobody else had the proper state of mind to see it was broken.

Answer 3

The miles of tubes, wires and iron beneath New York and other U.S. cities are getting older and could become dangerously unstable.

For the anecdote, the same argument was made in Paris in the 16-17th century. So many holes and tunnels had been dug underneath it (in addition to the natural holes due to the geology of the area) that an occasional building would crumble.

By the time entire city blocks were collapsing into the ground, directives had been given to fill the unneeded holes and tunnels with gravel and bones (they also had over-crowded graveyard problems). The city survived that way until concrete was invented.

My point here is that many organizations tend to wait for the last minute to do any software maintenance, but coders (like civil engineers) as a lot get the job done, fast and well.

We survived the Y2k bug. The Y2036 bug will force many an organization to upgrade its hardware and software. The world could end in 2012. But computer scientists are not sociologists or literary critics.

Oh, and as the saying goes in the meanwhile: write code as if the next maintainer is a vicious psychopath who knows where you live.

Answer 4

This is already a big problem. And it shows no signs of changing.

In the 60s and 70s large institutions of all kinds went from doing accounting on paper to doing accounting on computing systems. Overwhelmingly they chose COBOL. Most are still using updated versions of those COBOL systems. See http://cis.hfcc.edu/faq/cobol for some statistics on this

Every so often we get random reminders of this, such as when Arnold Schwarzenegger discovered a couple of years ago that he couldn't simply cut the pay of 200,000 state workers without 6 months of development first (see http://www.infoworld.com/d/developer-world/californias-cobol-conundrum-067 for verification).

Given the risks of switching, it is very hard to justify changing these systems. Ever. That has been reality for my lifetime. But I see no reason for that fact to change in my children's lifetime. Or their children's lifetime either.

I have friends who have maintained code that is older than they are. I have a friend who returned to a company 30 years after she worked there the first time, to find that her programs were still running, unchanged, in a language that she didn't even remember!

Let me finish with a true story of what both can happen.

In the 1970s, a company was founded to provide an online market for traders. The PDP-11 was a good price/performance fit for them, so they chose that. They were pushing the performance boundaries of a machine, so they wrote their system in highly optimized PDP-11 assembly. A few years later the PDP-11 stopped being sold. However business was great, the machines lasted, and second hand replacements were easy to come by. They kept their platform. Some years after that replacements became harder to come by. A major project was made to replace the trading platform. It failed. They tried again. And failed again. A major cause of the failure was that nobody knew how they trading system worked, and nobody could read PDP-11 assembly any more. Then salvation hit. Someone created a PDP-11 assembler that ran on Linux.

Thus it was that in 2000, trades coming into a billion dollar/year business went to Linux machines, over an ethernet-decnet bridge, to emulated PDP-11 machines that executed the trade on a software system which was written in highly optimized PDP-11 assembly. For speed.

I have had no connection to said company in the last decade. So I can't tell you whether they are still running on emulated PDP-11s. I know that decimalization was squeezing their margins painfully, so they had yet another effort to migrate. (I only learned the story because I interviewed several people who had been laid off from there, and I asked what had happened.) However given previous failures, I'd give it better than even odds that they have not successfully replaced that system.

Answer 5

Sounds like a very genuine concern from a user's view. For the rot to be delayed, or better, avoided, the software in question needs to be free of its shackles. Its publishers should have set the source code free, or be in the business of maintaining and upgrading the sources until its last user stops using it. Otherwise, there is very good chance that they could go out of business due to similar rots in the business world, thus, leaving the software fully open to the rots.

That is to say, the length of term of software copyrights and restricting licences should be very short, at the end of which, the software and its code base enters the public domain and stays there. Thus, making it possible for the user to keep upgrading the sources, or, hire someone to do it, thereby, delaying and/or avoiding the rots.

Or, if you are a bit open to the idea of free software, you could write your programs under a free license such as the AGPL or the GPL or other free software licenses. From what I have seen, when a software's sources attract no more interest in the developers to improve it for any reason, the source base gets cannibalised and takes on new life. Packages in Debian operating system tend to follow this life cycle, as far as I have seen.

Answer 6

I forget, what do we consider legacy code these days ? Last year's code, last decade's code or last century's code ?

Money drives the conversation around the maintenance of legacy systems. Technical debt takes its form as an increased cost to change the system.

I've worked with poorly designed and intelligently designed systems. What is interesting is that the costs to maintain them don't vary by that much. The biggest issues are incorrect architectures for current use, which usually show up in scaling issues or when a major change is desired. You don't easily convert a major area of code from single threaded to multi-threaded.

The most significant problems I experience are the development languages used. Older systems are written in languages that are less popular today so you have to either train more or hire more seasoned (expensive) and rare resources. In either case, because of the smaller pool, you also struggle to find skilled individuals which tends to lead as many problems as solutions.

As for the promised rewrite, most systems that have had huge investments don't justify a rewrite. It is amazing how long software can be kept working and enhanced. Hardware changes (some systems my company supports use specialized hardware) tend to be our biggest problem. The ability to enhance is often only limited by the mechanisms to integrate legacy code with new features.

Answer 7

Having supported a variety of government and private industry applications I would say that most companies and at least the US Government is well aware of the dangers of letting code rot and not staying on top of the latest security trends.

We regularly have to get our software certified for various susceptibilities and most government electronic systems, even old ones, get regular updates to keep them secure.

Of course there are exceptions, and hackers are always on the move, but on the whole I think people are pretty aware you can't just throw something out there and never touch it again.

Answer 8

Warning: This is going to be a bit free-form...

I think there are 2 ways to look at your concern.

If you think of it, some space shuttles and satellites have been running the same code that originally launched them. On the other hand, some have been designed to be updated even if they are (very) remote.

What matters is the environment. Obviously, as long as you don't modify the environment, your code won't become obsolete. In this instance, code rot doesn't really exist: the code itself (or the produced binary) cannot rot. It might break if you just start attacking it from a completely different angle. It's not that it's rotting, it's that it's not adapting to its environment. Think of it as an evolutionary problem.

But our environment changes. And somehow, what is the key to your problem is also the solution. Our environment changes so fast, that nowadays we wouldn't expect a software solution to not evolve over time. We overlook software projects that have not been updated in the past year, and will moan about product and customer support that doesn't produce a clear roadmap. And even when this works out well - you get a clear roadmap, good support, regular updates... - there's always the chance now that a challenger will surface, with exponential growth. We often make the mistake of thinking that the big established companies will always dominate, exactly because they dominate. However, the same way the dominant element in a herd gets older, the super-massive software/hardware/whatever vendor gets older. Or just a bit lazy. And a challenger comes in and turns things around even faster than the established dominant might have done it 5 or 10 years before. Or the dominant will just take a good beating, barely surviving while we see a disruption in the market (economically speaking, with impacts on different fields), and then things will go on. Maybe that looks imperfect, but in itself it's an organic process.

So, from the perspective of the user, I guess the issue is not that big. Code rot won't happen from the user's perspective, as he'll get to use an alternative (possibly with a seamless transition/migration... hopefully).

Now assuming we're not seeing things from the point of view of the user, or that we are talking about a system that is immune - for reasons unknown, governmental development, spac travel, etc... - to competition and is really supposed to be built to live/survive for a very long time, we need to look at the texts you referenced. And probably some more literature on dependable systems and fault-tolerant system. Though we probably want to push further. We don't just want fault-tolerance, we want evolutionary systems.

The problem with evolution, is that it introduces changes, and changes introduce points of failures. Let's look at these now and at what we can do to address them.

We can still rely on the infrastructure/architecture/emgineering metaphor while we do so (after all, we all ourselves software engineers, though there's arguably no such thing as software engineering... for now). There's a reason while the tube system is still active (with some glitches), while Big Ben still works (with some glitches) or the Eiffel Tower is still standing. It's because we do with vital (or not so vital) elements of an infrastructure what we should be doing with software just as well: continuous inspection. These entities were not necessarily designed to last this long, but have benefited from permanent oversight and timely improvements and repairs when it was needed. Call that your hotfixes if you will.

On the other hand, some designs were meant to last, and run durably with no interruption, even knowing that continuous inspection won't be possible. In this case we turn towards good design and formal models. Elements of dependability (Availability, Reliability, Safety, Integrity, Maintainability) can be quantified - for a given environment. Stats do the rest to plan for the rest and the future. Which brings the question: is it even possible for us to build systems that will be evolutionary, in the real sense?

Answer 9

Jeff Langer in Clean Code asks a similar question ... without references to steam pipes : )

What if there were four simple rules that you could follow that would help you create good designs as you worked? What if by following these rules you gained insights into the structure and design of your code, making it easier to apply principles such as SRP and DIP? What if these four rules facilitated the emergence of good designs?

Many of us feel that Kent Beck’s four rules of Simple Design are of significant help in creating well-designed software.

According to Kent (in Extreme Programming Explained), a design is “simple” if it follows these rules:

• Runs all the tests
• Contains no duplication
• Expresses the intent of the programmer
• Minimizes the number of classes and methods

In order to run all the tests ... we need tests to run and that is one huge indicator of technical debt. For example, if there are 10,000 test cases in a system like Mercury Quality Center and none of those tests are automated, that is one clear indicator of the technical debt that has been built up.

And that is where Feathers and his book "Working Effectively with Legacy Code" comes in.