Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Scenario

Currently, I am apart of a health care project whose main requirement is to capture data with unknown attributes using user generated forms by health care providers. The second requirement is that data integrity is key and that the application will be used for 40+ years. We are currently migrating the client's data from the past 40 years from various sources (Paper, Excel, Access, etc...) to the database. Future requirements are:

  • Workflow management of forms
  • Schedule management of forms
  • Security/Role based management
  • Reporting engine
  • Mobile/Tablet support

Situation

Only 6 months in, the current (contracted) architect/senior programmer has taken the "fast" approach and has designed a poor system. The database is not normalized, the code is coupled, the tiers have no dedicated purpose and data is starting to go missing since he has designed some beans to perform "deletes" on the database. The code base is extremely bloated and there are jobs just to synchronize data since the database is not normalized. His approach has been to rely on backup jobs to restore missing data and doesn't seem to believe in re-factoring.

Having presented my findings to the PM, the architect will be removed when his contract ends. I have been given the task to re-architect this application. My team consists of me and one junior programmer. We have no other resources. We have been granted a 6-month requirement freeze in which we can focus on re-building this system.

I suggested using a CMS system like Drupal, but for policy reasons at the client's organization, the system must be built from scratch.

This is the first time that I will be designing a system with a 40+ lifespan. I have only worked on projects with 3-5 year lifespans, so this situation is very new, yet exciting.

Questions

  • What design considerations will make the system more "future proof"?
  • What questions should be asked to the client/PM to make the system more "future proof"?
share|improve this question
57  
Future proofing is one thing, but I believe that a client to ask for software that is expected to have a lifespan that is 10x-20x longer than the current history of mobile/tablet computing or 5x-8x longer than the current history of the language in use is... unreasonably optimistic about the stability of a given model of computing. –  MichaelT Oct 28 '13 at 15:17
30  
designing to be 40+ year 'future proof' sounds like an exercise in futility. –  whatsisname Oct 28 '13 at 15:21
10  
To fullfill the requirement of a database being usefull for the next 40 years I would put it all on paper. Paper has proven itself, whereas digital storage mostly has proven how to lose lots of data fast. (but ofcourse preserve all data which should be destroyed) –  Pieter B Oct 28 '13 at 15:47
34  
They're giving two contract developers 6 months to build this system? Collecting years of legacy data AND anticipating new requirements decades into the future? If you're not already walking away from the project, start running. This is way bigger than two people can handle in anything close to the time frame alotted. The client has completely unreasonable expectations and is unwilling to commit proper resources to the project. –  Sean McSomething Oct 28 '13 at 22:57
11  
6 months for 2 people to architect and implement an application that needs to last 40+ years? Doesn't matter how good you are, that sounds like a setup for failure. If you can't convince your organization how unreasonable that is, then I would suggest you start looking for other employment ASAP. –  dsw88 Nov 4 '13 at 17:30

11 Answers 11

up vote 122 down vote accepted

Data is King

I think its a bit unreasonable to expect a web application circa 2013 to be still up and runnable in 2053. Technologies are going to change. Platforms are going to come and go. HTML may be a quaint memory by then. But your data will still be around.

So data is your primary focus. As long as your data is still there, people will be able to adapt to whatever new technologies come about. Make sure your data schemes are well thought out, and well suitable for expansion. Take your time spec'ing them out.

Regarding the actual applications, your company is probably correct here in having a 'build from scratch' directive. I maintain a couple 10+ year old web apps, and I'm very glad they are not locked into the prevailing CMS systems of 2003. They use home grown, very simple frameworks. I think for something like this you are better off with a very basic framework that you create specficially for the needs of the project.

But the reality is, over 40 years, the company will (hopefully) be making quite a few front-end and back end services to adapt to evolving platforms. So given that, I'd target a 5-10 year lifetime for individual user-facing applications.

share|improve this answer
12  
"we won't possibly be using this code in 40 years!" is why there was a Y2K bug to begin with. Expecting your code to be replaced is just bad practice. –  DougM Oct 28 '13 at 16:03
63  
The Y2K 'bug' was a data problem - 2 digits instead of 4 were stored. Which is why I suggest focusing on data. –  GrandmasterB Oct 28 '13 at 16:10
21  
Good point. Keeping this in mind, if someone really expects their data (and possibly database as well) to be in use 40+ years from now, it might be best to design the database with as few vendor-specific features as possible. Whoever has to untangle/rewrite all of your code that relies on special Oracle/MS-SQL/whatever functionality 20 years from now will not be happy with you. ;) –  FrustratedWithFormsDesigner Oct 28 '13 at 17:27
4  
This is solid advice. There are plenty of Cobol programs still running that were originally written 20-30 years ago. Although the technology moves on, if your data and object model is solid, and the data remains of interest, your code will remain in use in some form or other. –  Bobble Nov 2 '13 at 10:34
6  
Since someone brought up the Y2K: be mindful of the UNIX Y2K (en.wikipedia.org/wiki/Year_2038_problem). –  MrFox Nov 4 '13 at 19:51

We produce software that has been in use by paying customers for over 20 years. The codebase has outlasted several generations of source control tools. Our software hits all your bullet points except for the tablet thing.

Some of the concerns include ESIGN and UETA. Our lawyers believe that we need to keep electronic records readable for a minimum of 10 years. For documents that are retained whole, you should look into PDF/A.

For your database, don't worry too much about normalization. Instead you should be concerned about logging everything and having audit tables that track changes/deletes in data. When upgrading versions, plan on testing new versions in parallel for enough time to ensure that you've got your data migrated. This testing of new versions also includes new operating systems - we've had some very unpleasant surprises over the years. Preserve installation media and license keys in the event that a rollback needs doing. Test backups. If you are going to serialize objects to store in the database, do so as XML instead of the serialization supplied by your development framework.

For your staffing, long term code bases need long term memory. Ideally you'd want people around who have been around a long time. If that is institutionally impossible, then you need to document everything in something like a wiki. And my advice is a wiki that can tie in with your bug tracking system.

For your codebase, make sure you have comments in your code. Migrating from one version control system to another will almost always lose your check-in comments. I'm a fan of naming unit tests after spec and bug numbers. That way if the unit test Test_Bug_1235 breaks, then you know what and where to track down what it is supposed to be testing. It isn't as "sexy" as naming your tests Check_File_Save_Networked_Drives but that sort of test is hard to backtrack to specifications, requirements or bugs unlike Test_requirement_54321_case_2.

share|improve this answer
    
Thank you for sharing your experiences. I definitely need some of those done as the current architect hasn't commented any of his code nor provided us with any documentation. It's been a logistical nightmare, but I hope to change that. The PDF/A is something that I will definitely investigate as it is something that our client will need - especially for auditing. Thanks again for your advice! –  Pete Oct 28 '13 at 21:30
4  
This is a comprehensive and well thought answer. You make some good points about the importance of auditing, both for changes to the data and data quality, but also for legal reasons behind tracking who is viewing what data, See HIPAA. If your software is to be used in the United States then you will have certain security constraints and requirements that are required under this law. –  maple_shaft Oct 28 '13 at 23:09
3  
…at least SVN to git is possible with full commit history. –  feeela Mar 26 at 20:20
    
Not just SVN to Git, but most non-stone-age systems, though old ones like CVS often need manual adjusting with reposurgeon. Most exporters emit a git-fast-export stream, too, which is simple enough that it can be imported by non-Git VCSes as well. –  grawity Jun 21 at 19:56
1  
I suggest you don't name Tests only after Bug Tracking & Specification numbers, as: a) bug numbers tend to start over from 0 (since nobody seems to like >5-digit numbers & bug tracking software gets exchanged; b) specifications tend to get lost (ugly but it happens often enough); c) Sexy names often make it clear enough. Though, having a reference to spec/bug id is always a good idea. –  bernstein Jun 23 at 15:51

Rather than trying to figure out how this application is going to still be in operation 20 years from now, I think you should spend your six months fixing the problems you found that the original architect caused, put in place a sensible, robust architecture, and move forward from there.

Partial de-normalization of a database is not necessarily entirely unexpected in a medical setting. Some parts of medical databases have characteristics which make them a good fit for the EAV (Entity/Attribute/Value) model.

share|improve this answer
2  
@user2708395 Be careful with EAV design as it might not be the most performant or easy to query. It also might not be a good choice for reporting. –  maple_shaft Oct 28 '13 at 16:36
    
@maple_shaft That's what I've read as well. I'm going to be very cautious with it as I heard some horror stories where people overuse it. Looking at using some reporting database to make it easier to query since the client only generates reports once a month. –  Pete Oct 28 '13 at 16:39
3  
@maple_shaft: Usually data is extracted from the EAV schema/database to a separate reporting schema/database. –  FrustratedWithFormsDesigner Oct 28 '13 at 21:55
    
@FrustratedWithFormsDesigner That is an excellent point. The flexibility in your schema that EAV provides is unmatched but it certainly isn't a silver bullet for all persistence. –  maple_shaft Oct 28 '13 at 23:11
    
While I'll grant that EAVs can be used, you'd be surprised at the relationships you can find. That said, the extra attributes that often show up for these kinds of industries (healthcare, customer relations, etc) do have to go somewhere... Just make sure to back it with an attribute-key table, to get a canonical list of attributes. –  Clockwork-Muse Oct 29 '13 at 9:19

Answer from a front-end perspective:

Don't listen to everyone saying it can't be done, because an experimental San Francisco State University web service I co-wrote in 1996 finally went to Internet heaven a couple years ago, and never needed a single browser compatibility fix in that time; that's almost half of your 40-year goal. And this JavaScript-based front-end I made in 1998 for a Stanford Research Institute project was replaced with something flashier a few years later, but there's no reason the original UI couldn't still be running today with minor compatibility fixes.

The trick is to ensure your app uses only widely-supported W3C/ECMA standards and has a clean design under your control. While a lot of web apps written to the trendy 90s-era technology won't work well or at all today, 90s-era web apps written to major standards still do. They may look passé, but they work.

The goal here isn't to write a web app that will go up on their server and remain there for 40 years without anyone touching it again. It's to build a foundation that can still be in use decades down the line, which can grow to support new features without having to be rebuilt from scratch.

First of all, you have to code to official standards and only to official standards. No JavaScript features not part of a ratified ECMAScript standard; ES5.1 is the current version and is generally supported so that's safe to target. Likewise, current versions of HTML5, CSS, and Unicode are good. No experimental JavaScript, CSS3 or HTML features (the ones with vendor-prefixes or without 100% agreement between browsers). And no browser-specific compatibility hacks. You can start using a new feature once it's in the standard and everyone supports it without prefixes.

ES5 support would mean dropping IE8 or earlier, which I suggest anyway since it requires browser-specific hacks which will be useless in a couple of years. I'd suggest ES5's Strict Mode for the best chance at longevity, which actually sets your baseline browser compatibility at IE10 and recent versions of everyone else. Those browsers also have native support for many of HTML5's form validation and placeholder features, which will be useful for a very long time.

New editions of ECMAScript maintain compatibility with older versions, so it'll be much easier to adopt upcoming features if your code is written according to current standards. For instance, classes defined using the upcoming class syntax will be fully interchangeable with classes defined with the current constructor.prototype syntax. So in five years a developer can rewrite classes into the ES6 format on a file-by-file basis without breaking anything -- assuming, of course, that you also have good unit tests.

Second, avoid trendy JavaScript app frameworks, especially if they change the way you code your app. Backbone was all the rage, then SproutCore and Ember were, and now Angular's the framework everyone loves to promote. They may be useful, but they also have something in common: they often break apps and require code changes when new versions come out and their longevity is questionable. I recently updated an Angular 1.1 app to 1.2, and quite a bit had to be rewritten. Likewise, going from Backbone 2 to 3 requires a lot of HTML changes. Standards are slow-moving for a reason, but these frameworks move fast and things breaking periodically are the cost.

Plus, new official standards often leave old frameworks obsolete, and when that happens those frameworks either mutate (with breaking changes) or are left behind. You know what's going to happen to all the world's competing promise libraries once ECMAScript 6 is ratified and all browsers support its standardized Promise class? They'll become obsolete and their developers will stop updating them. If you picked the right framework your code might adapt well enough, and if you guessed poorly you'll be looking at a major refactoring.

So if you're thinking of adopting a third-party library or framework, ask yourself how hard it'll be to remove in the future. If it's a framework like Angular that can't ever be removed without rebuilding your app from scratch, that's a good sign it can't be used in a 40-year architecture. If it's a third-party calendar widget that you abstracted with some custom middleware, replacing it would take a few hours.

Third, give it a good, clean app structure. Even if you aren't using an app framework you can still take advantage of developer tools, build scripts, and a good clean design. I'm personally a fan of Closure Toolkit's dependency management because it's lightweight and its overhead is completely removed upon building your app. LessCSS and SCSS are also great tools for organizing your stylesheets and building out standards-based CSS stylesheets for release.

You can also organize your own code into single-use classes with an MVC structure. That will make it much easier to come back several years into the future and know what you were thinking when you wrote something, and to replace only those parts that need it.

You should also follow the W3C's advice and keep presentational information completely out of your HTML. (That includes cheats like giving elements presentational class names, like "big-green-text" and "two-columns-wide".) If your HTML is semantic and CSS is presentational, it'll be much easier to maintain and adapt it to new platforms in the future. It'll also easier to add support for specialized browsers for blind or disabled people.

Fourth, automate your tests and make sure you have near-full coverage. Write unit tests for every class, whether server-side or in JavaScript. On the front end, make sure each class performs according to its spec in every supported browser. Automate these tests from your build bot for every commit. This is important for both longevity and reliability, since you can catch bugs early even when current browsers obscure them. Both Jasmine and Google Closure's JSUnit based testing frameworks are good.

You'll also want to run full UI functional tests, which Selenium/WebDriver are good at. Basically, you write a program that steps through your UI and uses it as if a person were testing it. Wire those up to the build bot as well.

Lastly, as others have mentioned your data is king. Do think through your data storage model and make sure it's built to last. Make sure that your data schema is solid, and make sure that's tested thoroughly as well on every commit. And make sure your server architecture is scalable. This is even more important than anything you do on the front end.

share|improve this answer
1  
The good advice on 'JS frameworks' applies to backend frameworks as well. See Uncle Bob Martin's advice. –  Brian Low Jun 22 at 6:12
    
Frankly, I'd be cautious of JS entirely given the context. I can imagine HTML being around in 40 years; I wouldn't rely on whatever converter is being used then to necessarily support JS the way you want (and consider that your JS is possibly doing the wrong thing since the preferred output device may be unimaginably different). –  Eamon Nerbonne Jun 23 at 7:28

After I've read the question and the other (very well thought out) answers, I've thought that I'll leave my two cents as well. Note: I don't have to maintain any really old application/software at all. What I use as reference is my own work-in-progress hobby web app that grabs some open government data, parses and displays it in different formats. The app started as a project where I wasn't alone and where the gov just announced that it will offer this data somewhen, somehow. So it was clear that lots of things will change over time. And they did. What I learned from it:

  • Split things up in mini-applications. Each fully able to fulfill its task on its own. This makes switching out a single piece very fast, very secure and very easy. And when you have to go back in, it's not really hard to get around why things happen and how you they happen. If you or someone else will have to change something later on, it's easier to replace a single part than a whole set of things.
  • Get a solid constant middleware/-layer that is used for communication between the different parts. In this case I used JSON, but XML, ini or similar standards would be fine as well. It's easy to repeat and can be transformed into nearly anything. All are proven standards that will survive quite some time. Even if the underlying data and/or storage model will change. Each of the apps can use their own data storage for its specific task. This makes the amount of scanned data for a task smaller, therefore easier to handle and maintain and is easier exchangeable.
  • Don't worry about programming language decisions. Those will change over time. Just make sure that you use the language that you're comfortable with or that fits a task best.
  • Make sure that your data storage is "horizontally scaleable" and that it's easy to plug in additional storage modules.
  • Get a common point (in my case it's URIs) where the mini-apps are called and/or exchange data.

Summed up: The thing I'd care most about is the separation of concerns and exchange-ability of parts that are assigned for tasks. You simply know that in 40 years (even in 5 or 10) hardware, interfaces, storage, etc. will change a lot. And later developers will have to react on those changes and exchange parts of your application, be it the data container or parts of the User Interface.

share|improve this answer
1  
Very good advice! Thanks. I definitely agree with the separation of tasks and creating the mini-apps. The current build everything is coupled making it hard to integrate new features and requirements. I'm hoping to go with a RESTful interface and use JSON. Not to complain, but when I first joined, the offshore architect wouldn't let me use JSON. So I just told him I was passing "strings" and left out the part that these strings were in JSON format. :) –  Pete Nov 5 '13 at 16:05

Leaving aside the issues of the unreasonable expectations of your client and focusing on the design issues, I wouldn't go as far as 40 years, but the problem you seem to have, of long-term evolvability, is precisely what REST was created for. By that I really mean REST as an architecture style, not the buzzword driven development that's so commonly associated with the term these days.

To some extent, people get REST wrong because I failed to include enough detail on media type design within my dissertation. That’s because I ran out of time, not because I thought it was any less important than the other aspects of REST. Likewise, I suspect a lot of people get it wrong because they read only the Wikipedia entry on the subject, which is not based on authoritative sources.

However, I think most people just make the mistake that it should be simple to design simple things. In reality, the effort required to design something is inversely proportional to the simplicity of the result. As architectural styles go, REST is very simple.

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution.

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven#comment-724

You mentioned you intend to use a RESTful interface. That comment by itself suggests you should do some serious research into that and try to understand what REST really is about. You're probably associating it merely with the mapping of HTTP methods to CRUD operations that most people think REST is, but it has nothing to do with that.

Think of REST as a formalization of the architecture of the web itself. In one way or the other, there are many parts of the web written a decade ago or more that are still available and can be used with a client made today, so we got something right in that department. It will be a lot of work, I guarantee you, because doing REST right is hard, but the long-term benefits are worth it.

share|improve this answer
    
This is very helpful! Thank you. I was doing some more research into REST and I can see that massive benefits of it and how it can be extended beyond the HTTP methods. It's pretty powerful stuff and I'm pretty excited to be working with it. Thanks for the link as well! I just wish I had more time! –  Pete Nov 12 '13 at 15:39

To make things as "future-proof" as possible, plan for change. That is, try your hardest not to optimize for anything other than the ability to easily change. So no normalization, no strict validation, and loose coupling galore.

  • Use major open-source technologies. For data, closed-source systems are a major source of risk as one can't plan on which companies will go under or change strategies, taking with it all access to the data. Also, minor open-source projects without a vibrant community are also more likely to lose support.

  • Use a schema-less NoSQL database. The sorts of unstructured data being used is almost straight out of the textbook for a document store like MongoDB. Traditional relational databases and their normalization is good when you know how your data is going to be structured, but that's really a fiction, especially here. The costs of changing schema in a RDBS just get larger and larger as the system expands. Know that whatever structure is chosen now is going to end up changing.

  • Decouple the system heavily using widely accepted standards. Breaking up all data access and mutation into web services is one step towards this. Combining that with message queues for submitting changes and such will help individual parts of the system change languages or technologies over time.

share|improve this answer
    
Unfortunately, using a schemaless database doesn't mean that restructuring and reorganizing data has zero cost. –  Alex D Jun 28 at 11:57

OK, so I'm going to say some things here which are probably going to be pretty unpopular, but stick with me here.

As this is your first project where the data and/or the application is supposed to last for 20+ years, and you are the one leading the project, you need to take a step back and think about what the odds of this project succeeding are. Because they are basically next to zero.

You need to spend a huge amount of time focusing on the database design and getting that right. For this project to succeed you need to bring a data architect into the project, and sooner rather that later. Without someone who is experienced in database design, and who is well practiced in looking forward to how the data can be used in the future the odds of the data still being useful after 5 years much less 40 years is slim to none.

Expecting two people (one of whom has the title of jr. dev) to build something from scratch which is expected to last 40 years, is probably not going to succeed. There should be a team of people many of who are experienced in working with large projects like this who are working on the data design, the API design and the application design. Something like this isn't a 2 person project.

Wanting to tie the project to something like Drupal shows why the project needs people who are used to working on these sorts of projects. You don't want to tie the application to anything that may go out of style within just a few years. If you did, finding someone to work on the system in 5-10 years could get very difficult very quickly.

I would take this advice to management and explain to them that you need to get more senior people on the project. Otherwise the project is doomed to fail, and you'll probably end up being the one that takes the blame for it.

share|improve this answer

The application need not survive 40 years without any evolution. But, because it would or should be built from scratch, it could still be 'functioning'.

What is the most key thing is the 'data architecture' that allows for some stability and governance as well as extensible.

We've designed data architecture and taxonomy that could almost survive the end of humanity but still be extensible. You've got find a true DATA TAXONOMY / DATA ARCHITECTURE person to do this for you.

share|improve this answer
    
I think that was the failure of this project from the beginning is that it was started without a proper data architect. This is definitely very sound advice. –  Pete Jun 23 at 13:46
    
Time to call and hire me :) Doing a Data Governance & Taxonomy thing for some companies as we speak :) –  Alex S Aug 20 at 6:50

The key here is to focus on the database (as several above have said). This needs to be coherent and completely describe the operation. It needs to grow with the operation as it changes. If it is not easy to change then it will get out of date and that is the kiss of death. The rest is relatively less important.

I disagree with those above who suggest normalisation is not important, though there are always cases where the current systems are not up to the job. In these cases denormalise but ensure the database handles the extra writes/ changes as part of an atomic transaction. That way you can effectively ignore the problem till you can fix it.

The company I worked for before retirement is running a system that was written from scratch and has grown almost continuously for 25 years, and covers virtually all aspects of a medium retailer. Aspects of this system that I feel were important are:

  • Integration is vital.
  • The database needs to be as correct and understandable to both IT and other staff, so an almost paranoid emphasis on naming is required. We have tables of defined mnemonics which are then combined to make up table and field names and all “codes” were likewise named as constants and stored in a EAV table structure.
  • We encapsulated business logic into database triggers. This is painful at first and requires additional work to transmit error messages to clients and to allow triggers to be changed flexibly without locking the whole table on some systems, but this quickly becomes a massive time saver, and forces the database to be much more correct than otherwise.
  • Assume you will keep at least reference tables (ideally all but the fastest and least important transactions) for the life of the system, even when “deleted” so your references are correct.
  • Because of the above ensure any unique identifiers and transaction numbers are sized for the long term. (I originally jokingly suggested that they needed to last until I retired).
share|improve this answer

I would suggest using event sourcing and command and query responsibility segregation. This is mainly because the only thing that you can be sure about is the events that already appeared. New types of events might come. So even if you put heavy thoughts on a model, it is sure that this will get outdated. Migrating all data with every release might not be feasible. So the best is to have a model that fits your current needs and which is calculated from already recorded events every time you need it and then to pass out events that are computed from the current state of that model.

Also have testing in mind. If the application is in use in ten years from now, testers have to make sure that it is still doing what its supposed to do. So make integration testing your application as easy as possible.

share|improve this answer

protected by Telastyn Jun 23 at 3:01

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.