Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This question already has an answer here:

We have a legacy system to bring up-to-date because:

  1. It uses an unpopular (amoung our users) non-sql database (Btrieve)
  2. Provides only a text interface
  3. Is written in Turbo Pascal (but compiled in free pascal) which is hard to recruit/retain developers to maintain it

To give you an idea of the scale, the code base is 15MB over 756 files.

All the studies I've looked in to such as those listed here suggest it is vastly more expensive to rewrite than to adapt what you have.

If I thought we could encapsulate the legacy stuff behind a web service and then design new client(s), that would be fine. But I'm stuck with this legacy database which 90% of those 756 files reference.

We even attempted replacing the low-level DB code with code to convert to SQL, and while it worked fine, it was too slow because the code is still accessing records one at a time generating far too many SQL statements than it would need. Plus this approach does not allow any improvements in the DB design.

When faced with such an upheaval as replacing the DB that is referenced so heavily in code and the other requirements listed, can it still pay to reuse or are we better off treating the new version as a whole new product and just listing the current systems abilities as requirements (bearing in mind no handy requirements doc exists for the current system).

I know it's too hard to give a definitive answer on this, so just any advice would be appreciated especially if you've experienced a similar situation.

share|improve this question

marked as duplicate by MichaelT, Martijn Pieters, GlenH7, BЈовић, Dan Pichelman Feb 28 at 22:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

3  
@SK-logic Many would argue the opposite... –  maple_shaft Aug 6 '12 at 12:24
2  
@SK-logic Specifically the non-sql (not no-sql) is Btrieve, I don't know if it's more replaceable with a no-sql db, that an sql one but don't get the wrong idea, it's not a "no-sql" db as it stands, it's a record based ISAM db. –  weston Aug 6 '12 at 12:38
2  
The SQL vs NoSQL database question really depends on what you are storing in it. So what sort of data are you storing? And why don't the users like the database? –  Sign Aug 6 '12 at 12:54
1  
@SK-logic it's partially a sales thing. It's hard to sell anything that is not MS SQL server to our clients these days. Clients usually have the DB server already, and have the skills to manage it in house. They also like to report off the DB, using standard SQL reporting tools. –  weston Aug 6 '12 at 12:55
6  
For those of you who would like to have a discussion about SQL vs. NoSQL in regards to this question, I created a room... SQL vs. NoSQL Flame-o-rama 2012 Please divert all comments regarding this to the chat room I created. I will be purging all comments after this point. –  maple_shaft Aug 6 '12 at 13:24
show 9 more comments

5 Answers

up vote 3 down vote accepted

Its true that re-writing is the most radical approach and should be avoided whenever possible.

Try to create an intermediate level between your application and the DB. It needn't be a separate project necessarily, but all DB requests should go through it. If you manage to do this first step, it will be easy to change the DB any time, you'll just need to re-write one single module of your project.

Once this is done, try to divide your project into small logical pieces and re-write ("translate") them one after another (if you really think it can't be avoided). If you have unit-tests, they would be a great help.

share|improve this answer
    
I like the idea, but I think introducing an intermdiate layer would mean modifiying the 90% of 756 files/classes that access the db. A large proportion of the files (150 or so) probably do nothing but manage db tables, so I guess these files would sit in the legacy db implementation module. –  weston Aug 6 '12 at 10:50
    
Since you've determined to re-write the project, and you've alreadt faced the difficulty of modifying a lot of files to access the DB correctly, I suggest you do this once again and forget about it forever. You'll have to change those files anyway –  superM Aug 6 '12 at 10:53
1  
Thanks, I think this is a sensible way forward. It's refactoring the project to how it should have been done in the first place WRT to persistence, then it's a simpler step to change DB, and later replace the business logic if required from a recriutment point of view. –  weston Aug 6 '12 at 11:12
    
@weston, you're welcome )) –  superM Aug 6 '12 at 11:15
    
If you want to replace the database, you must do this anyway. –  user1249 Aug 6 '12 at 13:03
add comment

Before making any decisions about what approach that you would like to take, I always feel it is important for the development team to always consider the option of a complete rewrite as a possibility, even though many think it is nearly always a bad choice.

Take some time to write up some high level milestones that must be achieved, and make some very high level architectural decisions about what core technologies will comprise the new software. At this point make a rough estimate of the time you would feel is needed to complete such a project. It is good no matter what choice you make to at least have a good idea what a rewrite effort will cost you in terms of time and money.

The other very important cost of a rewrite that many neglect to consider can be the exorbitant cost of deployment in both money, time, and man hours. Users get comfortable with using older flawed systems, they memorize the quirks and bugs, the workarounds become part of their routine, they memorize the hotkeys to quickly perform their favorite tasks, they become surprisingly productive over time.

I have seen rewrites that by all accounts should have been a major success, the development came in under time and under budget, the major issues no longer exist, the newer system was easier and cheaper to find talent to maintain, the learning curve was smaller... the only major problem that had been neglected was user adoption and Agile based user involvement. When they were told that their text based mainframe system had been replaced with a GUI based desktop client, they were unfamiliar with it and productivity dropped. User input only came from a single individual who no longer performed these job functions so important key combinations, lack of keyboard support, hot keys, poor tab ordering, and the need for a lot of point and click mouse interaction caused productivity to plummet. Sure they would have gotten comfortable and productive eventually, and without the major bugs and improved performance they would have been vastly more productive with the new GUI based system once the kinks were worked out.

What unfortunately happened is that there was open revolt against the new system, and despite management buy-in, they could not force the users on the ground to adopt it. The project failed at deployment. The hidden costs and risks of deploying and training for a rewrite are important to consider when comparing to the increasing cost of maintaining the legacy system.

Another option to consider is to look into the technologies that are causing the most grief, most expensive to license, hardest to find talent for, inherently buggy, etc. A good start might be to rewrite small pieces of the application in such a way as to decouple the core components. Once this is done it is easier to swap and replace components as time becomes available to rewrite them, and this greatly reduces the risk of refactoring in general.

If I were you I would start with replacing just the database. You can use one that is more popular and that is more easily maintainable by the talent pool. As you replace the schema you can slowly start seperating database access specific logic into seperate modules, making it easier to decouple the business logic from the database implementation and database access code. It can be justified in improved performance, decreased cost of maintainability, and quicker turnaround.

share|improve this answer
2  
At one place I worked, we were actively maintaining two systems because of what you called "user revolt". They liked the newer, GUI-based web application for navigation and viewing, but preferred the legacy mainframe application for data entry. They felt hampered by the browser based one, and when I saw how fast they were on the old application I understood why. –  Michael K Aug 6 '12 at 12:18
    
That's an excellent point. The system is 20 years old, and think it really could be an issue amoungst users. We'll definitely have to factor that in. It might be that we provide a text interface as well for some operations (thinking shop floor tasks) to save retraining/avoid revolution. As long as it's just another thin client that won't bother me or the users management. –  weston Aug 6 '12 at 12:52
1  
@MichaelK could you create and provide a faster data entry over the web? –  user1249 Aug 6 '12 at 12:59
    
@ThorbjørnRavnAndersen I believe so. The root problem isn't limitations on web technology, but inadequate GUI design. With careful thought to what screens were required and the use of asynchronous input, a lot can be done to speed up data entry. However, it requires that active users be involved in the design. As maple_shaft showed, as soon as the users feel you've left them hanging, the project cannot be as successful as it could be with their full support. –  Michael K Aug 6 '12 at 15:01
    
@ThorbjørnRavnAndersen About web tech - WebSockets may prove to be a good solution for this type of problem, once it matures somewhat. –  Michael K Aug 6 '12 at 15:03
show 2 more comments

Some random thoughts:

Big-bang rewrites are prone to big-bang problems, which is why many people suggest an incremental approach to rewriting an application. Ideally, then, you need to Divide and conquer.

If you're going for an incremental approach, the challenges are:

  • How you divide the existing application into upgradable components
  • How you upgrade each component
  • How you integrate new components with the old ones

Look for logical divisions that already exist in the application. Are there separate pieces of hardware? Different technologies? Layers? Does it serve different classes of business user? Or perform different groups of business function?

The better organised your original code, the easier your task will be. As a result, breaking up an application in this way will almost certainly involve some amount of refactoring of the original application. For example, pulling out database calls into a single set of modules / classes. Be aware that you're not trying to improve your old application - your purpose is just to modularise it. Fixing other problems should be easier in the new technology (after all, that is presumably one of the reasons you're migrating in the first place).

The smaller the chunks you're dealing with the easier your task will be. So take things a step at a time. Even if you only manage a few hundred lines of code, at least that is a start.

If you can automate the upgrade process in any way, then do so. For example, you may be able to buy / make a tool to convert program code or database schemas.

Write unit tests for both old and new code.

share|improve this answer
add comment

Your suggested approach - encapsulate the legacy stuff behind a webservice is a good practice to follow. it will ensure that business logic will stay same as it was, and minimize the development time for better looking application.

Consider some No-Sql databases currently available in the market.

Currently, we are using Raven DB for the project needs. Although, it is quit odd to start getting into it. However, it seems quit a good match for a pure DDD approach.

Here is a post explaining strong sides of Raven DB among the No-Sql databases - Why Raven DB?

share|improve this answer
    
I think we would do that if it wasn't for the requirement to also replace the legacy database. –  weston Aug 6 '12 at 10:40
    
@weston, ok you may consider No-sql alternative –  Yusubov Aug 6 '12 at 10:53
1  
I'll look in to that thanks –  weston Aug 6 '12 at 11:07
add comment

Two issues.

One, the tiger team issue. Usually these developments fail as the old system would be the requirement for the new one, while the old one also changes, it's a Zeno's paradox. Uncle Bob wrote an article about this, but it's also part of several of his talks and perhaps some of his books as well.

Second, the UI issue.

It's not just that people are used to the UI. That's the tip of the iceberg.

For the users, the system IS the UI. That's why it was called the "user interface" at the first place. I don't even really understand how could they hate the underlying database system, as they have nothing to deal with it? Isn't it lying "under" enough?

If the user interface, the behaviour, the data, and the performance stays the same, you could change every bit of the system, the users wouldn't take a notice. You could convert it from Borland Pascal to Turbo C and vice-versa, you could change the server OS from FreeBSD to Solaris, from Windows to Linux, from Intel to PPC, whatever.

And this is good and bad for you. As long, as you keep the UI, you can change the system. If you change the interface, make sure the logic stays the same behind for a while.

(Your only problem will be, that the users don't have a problem with the underlying system: they have a problem with its user interface. Every single user-initiated change request is UI-based, this is what SVN / CVS / git logs show of every system, because that's the language the user speaks.)

It's like refactoring: you either change the code structure, but don't modify behaviour, or you change the behaviour, but don't change code structure. And have automated tests for that.

I don't know how you can test it, usually we used screenscrapers for reaaallly old applications. I've even seen banking frontend which literally typed the results of a form submission into a terminal to the old code. Not joking.

Change one step at a time. Have a hybrid system for a while. Change UI, retain system, change system, retain UI, change UI, retain system, change system, retain UI...

It's good that you have the source. Even if it's Pascal, it makes your life a few billion magnitudes easier.

For one refactor, I collected all input the system got in a half-an hour intensive testing period, and collected all the database changes it was to do. You can read about it here

Perhaps you should collect real system usage, by the use of keyloggers and terminal capture like kibitz or script.

Oh, and pray that your tests cover everything.

share|improve this answer
    
Thanks for the answer. The IT maintenance staff of the client don't like the db. This particular DB (pervasive/btrieve) requires that all machines that run the UI have pervasive client software installed. It is also not popular due to its per seat licencing. So no, the client's end users don't access the DB, but unfortunately that doesn't stop it being a royal pain in the arse for the client. –  weston Aug 7 '12 at 13:32
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.