How to keep a big and complex software product maintainable over the years?

Question

I have been working as a software developer for many years now. It has been my experience that projects get more complex and unmaintainable as more developers get involved in the development of the product.

It seems that software at a certain stage of development has the tendency to get "hackier" and "hackier" especially when none of the team members that defined the architecture work at the company any more.

I find it frustrating that a developer who has to change something has a hard time getting the big picture of the architecture. Therefore, there is a tendency to fix problems or make changes in a way that works against the original architecture. The result is code that gets more and more complex and even harder to understand.

Is there any helpful advice on how to keep source code really maintainable over the years?

Answer 1

The only real solution to avoid code rot is to code well!

How to code well is another question. It's hard enough even if you're an excellent programmer working alone. In a heterogeneous team, it becomes much harder still. In outsourced (sub)projects... just pray.

The usual good practices may help:

1. Keep it simple.
2. Keep it simple. This applies especially to the architecture, the "big picture". If developers are having hard time to get the big picture, the are going to code against it. So make the architecture simple so that all the developers get it. If the architecture has to be less than simple, then the developers must be trained to understand that architecture. If they don't internalize it, then they shouldn't code in it.
3. Aim for low coupling and high cohesion. Make sure everyone in the team understands this idea. In a project consisting of loosely coupled, cohesive parts, if some of the parts becomes unmaintainable mess, you can simply unplug and rewrite that part. It's harder or near impossible if the coupling is tight.
4. Be consistent. Which standards to follow matters little, but please do follow some standards. In a team, everyone should follow the same standards of course. On the other hand, it's easy to become too attached with standards and forget the rest: please do understand that while standards are useful, they are only a small part of making good code. Don't make a big number of it.
5. Code reviews may be useful to get a team to work consistently.
6. Make sure that all tools - IDEs, compilers, version control, build systems, documentation generators, libraries, computers, chairs, overall environment etc. etc. - are well maintained so that developers don't have to waste their time with secondary issues such as fighting project file version conflicts, Windows updates, noise and whatever banal but irritating stuff. Having to repeatedly waste considerable time with such uninteresting stuff lowers the morale, which at least won't improve code quality. In a large team, there could be one or more guys whose main job is to maintain the developer tools.
7. When making technological decisions, think what it would take to switch the technology; which decisions are irreversible and which are not. Evaluate the irreversible decisions extra carefully. For example, if you decide to write the project in Java, that's a pretty much irreversible decision. If you decide to use some self-boiled binary format for data files, that's also fairly irreversible decision (once the code is out in the wild and you have to keep supporting that format). But colors of the GUI can easily be adjusted, features initially left out can be added later on, so stress less about such issues.

Answer 2

Unit tests are your friend. Implementing them forces low coupling. It also means that the "hacky" parts of the program can easily be identified and refactored. It also means that any changes can be tested quickly to ensure they don't break existing functionality. This should encourage your developers to modify existing methods rather than duplicating code for fear of breaking things.

Unit tests also work as an extra bit of documentation for your code, outlining what each part should do. With extensive unit tests your programmers shouldn't need to know the whole architecture of your program to make changes and use exiting classes/methods.

As a nice side effect, unit tests will also hopefully reduce your bug count.

Answer 3

My answer is going to be short and sweet...

Everybody here is quick to mention code rot and I completely understand and agree with this, but it still misses the bigger picture and the bigger issue at hand here. Code rot doesn't just happen. Further Unit tests are mentioned which are good but don't really address the problem, one can have good unit test coverage and relatively bug free code however still have rotted code and design.

You mentioned that the developer working on a project has difficulty implementing a feature and misses the bigger picture of the overall architecture, and thus implements a hack into the system. Where is the technical leadership to enforce and influence the design? Where are the code reviews in this process?

You are not actually suffereing from code rot, but you are sufferring from Team Rot. The fact is that it shouldn't matter if the original creators of the software are no longer on the team. If the technical lead of the existing team fully and truly understands the underlying design and is any good at the role of being a tech lead, then this would be a non-issue.

Answer 4

There are several things we can do:

Give one person overall responsibility for architecture. When choosing that person, ensure they have the vision and skill to develop and maintain an architecture, and that they have the influence and authority to help other developers follow the architecture. That person should be a seasoned developer who is trusted by management and who is respected by their peers.

Create a culture where all developers take ownership of the architecture. All developers need to be involved in the process of developing and maintaing architectural integrity.

Develop an enviromnet where architectural decisions are easily communicated. Encourage people to talk about design and architecture - not just in the context of the current project, but in general, too.

The best coding practices make architecture easier to see from code - take time to refactor, to comment code, to develop unit tests etc. Things like naming conventions and clean coding practices can help a lot in communicating architecture, so as a team you need to take time out to develop and follow your own standards.

Ensure that all necessary documentation is clear, concise, up-to-date and accessible. Make both high- and low-level architecture digarams public (pinning them to the wall can help) and publically maintainable.

Finally (as a natural perfectionist) I need to recognise that architectural integrity is a worth aspiration, but that there can be more important things - like building a team that can work well together and actually ship a working product.

Answer 5

The way I go about this issue is to snip it at the root:

My explanation will be using terms from the Microsoft/.Net, but will be applicable to any platform/toolbox:

1. Use standards for naming, coding, checkins, bug flow, process flow - basically anything.
2. Dont be afraid to say goodbye to team members who doesnt adhere to standards. Some developers simply cannot work within a defined set of standards and will become 5th column enemies on the battlefield to keep the code-base clean
3. Dont be afraid to allocate lower skilled team members to testing for long periods of time.
4. Use every tool in your arsenal to avoid checking in rotting code: this involves dedicated tools, as well as pre-written unit tests that test the build files, project files, directory structure etc.
5. In a team of about 5-8 members, have your best guy do refactoring almost constantly - cleaning up the mess the others leave behind. Even if you find the best specialists in the field, you will still have a mess - its unavoidable, but can be constrained by the constant refactoring.
6. Do write unit tests and maintain them - do NOT depend on the unit tests to keep the project clean, they do not.
7. Discuss everything. Don't be afraid to spend hours to discuss things in the team, this will disseminate the information and will remove one of the root causes for bad code: confusion on technologies, goals, standards etc.
8. Be VERY careful with consultants writing code: their code will, almost by definition, be the real shitty stuff.
9. Do reviews preferably as the process step before checkin. Dont be afraid to rollback commits.
10. Never use open/close principle unless in the last stage before release: it simply leads to rotting code being left to smell.
11. Whenever a problem is encounted, take the time to understand it to the fullest before implementing a solution - most code rot comes form implementing solution to problems not fully understood.
12. Use the right technologies, these will often come in sets and be fresh: Its better to depend on a beta version of a framework you are ensured support from in the future, than to depend on extremely stable, but obsolete frameworks, that are unsupported.
13. Hire the best people.
14. Lay off the rest - you are not running a coffee shop.
15. If management is not the best architects and they interfere in the decision making process - find another job.

Answer 6

Clean rotted code by refactoring, while writing unit tests. Pay down (this) design debt on all the code you touch, whenever you:

• Develop a new feature
• Fix a problem

Greatly speed your test-first development cycle by:

• Refactoring to convert code modules to a scripting language
• Use fast, cloud-based test machines

Refactor code to use low coupling (of highly internally-cohesive units) by:

• Simpler, (more) functions (routines)
• Modules
• Objects (and classes or prototypes)
• Pure functions (without side effects)
• Preferring delegation, over inheritance
• Layers (with API's)
• Collections of small, one-purpose programs which can operate together

Organic growth is good; big up-front design is bad.

Have a leader who is knowledgeable about the current design. If not, read the project's code until you are knowledgeable.

Read refactoring books.

Answer 7

I work on a code-base for a product that has been in continuous development since 1999, so as you can imagine it is pretty complex by now. The biggest source of hackiness in our codebase if from the numerous times we've had to port it ASP->ASP.NET, ADO->ADO.NET, Postbacks->AJAX, switching UI libraries, coding standards, etc.

All in all we've done a reasonable job of keeping the code base maintainable. The main things we have done that contributed to that are:

1) Constant refactoring - If you have to touch a piece of code that is hacky or hard to understand, you are expected to take the extra time to clean it up and are given the leeway in the schedule to do so. Unit tests make this a lot less scary because you can test against regressions more easily.

2) Keep a neat development environment - Be vigilant about deleting code that is no longer used, don't leave backup copies/working copies/experimental code exist in the project directory.

3) Consistent Coding Standards for the life of the Project - Let's face it, our views on coding standards evolve over time. I suggest sticking with the coding standard you started with for the life of a project unless you have time to go back and retrofit all the code to comply with the new standard. It's great that you are over hungarian notation now, but apply that lesson to new projects and don't just switch mid-stream on that new project.

Answer 8

Simple answer: you can't.

That's why you should aim for writing small and simple software. It's not easy.

That's only possible if you think long enough about your seemingly complex problem to define it in as a simple and concise manner as possible.

Solution to problems that truly are big and complex can often still be solved by building upon small and simple modules.

In other words, as others pointed out, simplicity and loose coupling are the key ingredients.

If that's not possible or feasible, you are probably doing research (complex problems with no known simple solutions, or no known solution at all). Don't expect research to directly produce maintainable products, that's not what research is for.

Answer 9

Since you've tagged Project Management, I've tried to add some non-code points :)

Plan for turnover - assume that the entire development team will have disappeared by the time it hits its maintenance phase - no developer worth their salt wants to be stuck maintaining his / her system forever. Start preparing handover materials as soon as you have time.

Consistency / Uniformity cannot be stressed enough. This will discourage a culture of 'go it alone' and encourage new devs to ask, if they are in doubt.

Keep it mainstream - technologies used, design patterns and standards - because a new developer to the team (at any level) will have more chance of getting up and running quickly.

Documentation - especially architecture - why decisions were made, and coding standards. Also keep references / notes / roadmaps into documenting the business domain - you would be amazed how difficult it is for corporate business to explain what it is they do to a developer with no domain experience.

Lay down the rules clearly - not just for your current development team, but think about future maintenance devs. If this means putting a hyperlink to relevant design and coding standard documentation on every page, so be it.

Ensure that the architecture and especially code layers are clearly demarcated and separated - this will potentially allow for the replacement of layers of code as new technologies come along, e.g. replace a Web Forms UI with an HTML 5 jQuery UI etc, which may buy a year or so of added longevity.

Answer 10

One principle that has not been mentioned but that I find important is the open / closed principle (see e.g. here).

You should not modify code that has been developed and tested: any such piece of code is sealed. Instead, extend existing classes by means of sub-classes, or use them writing wrappers, decorator classes or using whatever pattern you find suitable. But do not change working code.

Just my 2 cents.

Answer 11

In addition to the other answers I'd recommend layers. Not too many but enough to separate different types of code.

We use an internal API model for most apps. There is an internal API that connects to the database. Then a UI layer. Different people can work on each level without disrupting or breaking other parts of the applications.

Another approach is to get everyone read comp.risks and thedailywtf.com so they learn the consequences of bad design and bad programming, and they will dread seeing their own code posted on thedailywtf

Answer 12

One property of highly maitainable code nobody mentioned yet is function purity.

Purity means that functions should return the same result for the same arguments, i.e. they should not depend on side effects of other functions. Additionally, it is useful if they do not have side effects themselves.

This property is easier to witness than coupling/cohesion properties, you don't have to go out of your way to achieve it and I personally consider it more valuable.

When your function is pure, its type is a very good documentation by itself. In addition, writing and reading documentation in terms of arguments/return value is much easier than one mentioning some global state (possibly accessed by other threads O_O).

As an example of using purity extensively to help maintainability you can see GHC. It is a large project about 20 years old where large refactorings are being done and new major features are still being introduced.

Last, I don't like the "Keep it simple" point too much. You can't keep your program simple when you are modelling complex things. Try making a simple compiler and your generated code will likely end up dead slow. Sure, you can (and should) make individual functions simple, but the whole program will not be simple as a result.

Answer 13

I've come to call this the "Winchester Mystery House Effect". Like the house, it started simply enough but over the years many different workers added on so many odd features without an overall plan that nobody really understands it anymore. Why does this staircase go to nowhere and why does that door only open one way? Who knows?

The way to limit this effect is to begin with a good design that's made where it's flexible enough to handle expansion. Several suggestions have been offered already on this.

But, often you'll take a job where the damage has already been done and it's too late for a good design without performing an expensive and potentially risky redesign and rewrite. In those situations, it's best to try to find ways to limit the chaos while embracing it to some degree. It may annoy your design sensibilities that everything has to go through a huge, ugly, singleton 'manager' class or the data access layer is tightly coupled to the UI but learn to deal with it. Code defensively within that framework and try to expect the unexpected when 'ghosts' of programmers past appear.

Answer 14

Be a Scout. Always leave the code cleaner than you found it.

Fix the broken windows. All those comments "change in version 2.0" when you're on version 3.0

When there are major hacks, design a better solution as a team and do it. If you can't fix the hack as a team, they you don't understand the system well enough. "Ask an adult for help." The oldest people around might have seen this before. Try drawing or extracting a diagram of the system. Try drawing or extracting the use cases that are particularly hacky as interaction diagrams. This doesn't fix it but at least you can see it.

What assumptions are no longer true that pushed the design in a particular direction? There might be a small refactoring hiding behind some of that mess.

If you explain how the system works (even just one use case) and find yourself having to apologize of a subsystem over and over again, its the problem. What behavoid would make the rest of the system simpler, (No matter how hard it looks to implement compared to what is there) The classic subsystem to rewrite is one that pollutes every other subsystem with it's operating semantics and implementation. "Oh, you have to groz the values before you feed them into the the froo subsystem, then you un-groz them again as you get output from the froo. Maybe all values should be groz'ed when read from the user & storage, and the rest of the system is wrong ? This gets more exciting when there are two or more different grozifications.

Spend a week as a team removing warnings so that real problems are visible.

Reformat all the code to the coding standard.

Ensure your Version control system is tied to your bug tracker. This means future changes are nice and accountable, and you can work out WHY.

Do some archeology. Find the original design documents and review them. They might be on that old PC in the corner of the office, in the abandoned office space or in the filing cabinet nobody ever opens.

Republish the design documents on a wiki. This helps institutionalize knowledge.

Write checklist-like procedures for releases and builds. This stops people having to think so they can concentrate on solving problems. Automate builds wherever possible.

Try continuous integration. The sooner you get a failed build , the less time the project can spend off the rails.

If your team lead does not do these things, well that's bad for the company.

Try to ensure all new code gets proper unit tests with measured coverage. So the problem can't get much worse.

Try to unit test some of the old bits that are not unit tested. This helps cut back the fear of change.

Automate your integration and regression test if you can. At least have a checklist. Pilots are smart and get paid lots and they use checklists. They also screw up pretty rarely.

Answer 15

Since many of these answers seem to focus on biggish teams, even from the outset, I am going to put my view as part of a 2-man dev team (3 if you include the designer) for a startup.

Obviously, simple designs and solutions are best, but when you have the guy that literally pays your salary breathing down your neck, you don't necessarily have time to think about the most elegant, simple and maintainable solution. With that in mind, my first big point is:

Documentation Not comments, code should be mostly self documenting, but things like design documents, class hierarchies and dependencies, architectural paradigms, etc. Anything that helps a new, or even existing, programmer to understand the code base. Also, documenting those odd pseudo-libraries that pop-up eventually, like "add this class to an element for this functionality" can help, since it also prevents people from re-writing functionality.

However, even if you do have a severe time limit, I find that another good thing to keep in mind is:

Avoid Hacks and Quick Fixes. Unless the quick fix is the actual fix, it is always better to figure out the underlying problem to something, then fix that. Unless you literally have a "get this working in the next 2 minutes, or you're fired" scenario, doing the fix now is a better idea because you aren't going to fix the code later, you're just going to move onto the next task you have.

And my personal favorite tip is more of a quote, though I cannot remember the source:

"Code as if the person that comes after you is a homicidal psychopath that knows where you live"

Answer 16

Read then re-read Code Complete by Steve McConnell. It's like a bible of good software writing, from initial project design down to a single line of code and everything in between. What I like most about it is that it is backed up by decades of solid data, it's not just the next best coding style.

Answer 17

I just like to throw my two cents.

The start of the project is very important. For me, the most important part is to choose a right language and developing tool

For example, JScript is a loose language, too many buggy while project go big. So, I never see a really big JScript project.

C#, on other hand, strong typed. And you can cut it to section(areas) much easy to go big and go well.

Answer 18

Code refactoring and unit testing are perfectly fine. But since this long running project is running in to Hacks, this means that he management is not putting its foot down to clean the rot. The team is required to introduce hacks because somebody is not allocating sufficient resources to train people and analyze the problem/request. Maintaining and long running project is as much a responsibility of project manager as an individual developer. People dont introduce hacks because they like it, they are forced by circumstances

Answer 19

My idea would be to setup a wiki to determine and document the elusive big picture, as well as smaller observations (and problems). Then, instead of writing code without knowing such contexts, encourage developers to do research and put results on wiki for others to see.

Answer 20

Don't use dynamic languages if you want your project to be maintainable in the long run. Type information is an important part of documentation. Don't sweep it under the rug.