Using ESB for database synchronisation / replication

Question

We're starting to look at implementing an ESB / Microservices architecture. I (think) I know about the concepts, but there's one thing I don't seem to be able to get a good idea about: data replication / synchronisation.

Creating an event for each and every table (maybe even multiple (create,update,delete)) seems like overkill to me, if it's just to synchronize data. Wouldn't an ETL / SQL Replication solution be much easier, in cases where no business logic will be exectute, as it's just to update the local cache/db of the server?

What strategies would you advise?

Simple example, we have an application that manages all product data, we want to build an API (Web Service) that is going to serve a Mobile App that will display that data.

There are several options:

1. API directly accesses the database
2. API has a local database that's being kept up to date using ESB messages
3. API has a local database that's being kept up to date using some replication tool
4. API has a local database that's updated once a day using a batch operation.

In my opinion the reason to use messages would be to further uncouple the systems as the database structure behind it can be changed without affecting the system in that scenario. For all other means and purposes 3/4 seem much let complex, which in my opionion meets the KISS principle.

What would you advice? Where can I get some sort of flowchart/decision tree example on what alternative to use when?

Answer 1

You need to look at what the root problem is.

• Are you seeking data redundancy?

• Are you seeking minimal data access times?

• Are you seeking sharing data across separate environments?

• Are you seeking to minimize security vulnerabilities with access to the data?

Once you decide what the highest priority is, then you can work on finding the best solution.

For data redundancy and access times the most likely solution is using SQL replication. The goal of replication (which those products are very good at) is data redundancy & minimizing access times through slave database servers. This option allows all points to have access to nearly the same data, you just need to monitor replication lag to ensure it stays within business requirements.

For separate environment concerns, I believe either an ESB or automated batch operation is preferable. This enables additional manipulation to occur before/during the operations to ensure any variances between servers can be resolved at data import time.

To minimize security vulnerabilities I would recommend a non-automated batch operation that has an appropriate level of security checks to ensure data validity. By not automating this operation it allows a human to confirm that there are no outstanding issues that may cause data corruption.

For any decision, big or small, you need to do a cost-benefit analysis before implementing any changes. You need to take into consideration:

1. Development time
2. Development cost
3. Future use of the solution
4. Expected performance
5. Complexity of the solution
6. Maintainability of the solution

Answer 2

First of all, be aware that ESB and Microservices can mean different things depending on context. Also, it seems that people usually consider Microservices within a pull-architecture: there are Restful APIs and each service calls other service's APIs to get data as needed. ESB, on the other hand is usually associated with a push-architecture: systems put data in queues where other systems get that data delivered later.

The default use of APIs is to provide an application's functionality to other systems and services and to hide that application's backend data representation from all those other services. So if you'd build your product data app according to those principles than you would build an API inside that app and thus use the same data access functionality (SQL, ORM, etc.) as the rest of the app. If the mobile app needs just the basic product management features which your app provides, it would directly use this API and you'd not have any further separation and not not sync any data.

If, on the other hand, your mobile app needs the data in a more refined for or with some additional functionality, or with higher performance than the original app can deliver (and you can't fix this with a simple caching reverse-proxy) or your old app is old technology and hard to extend, only then you'll get a benefit for putting your new API in a separate application. And only then you need to find a way to sync data between both.

Now, assuming the latter is the case, your means of syncing depend on many things:

• Does data only go one way or both ways?
• If both ways, does the existing application need a way to validate and reject invalid data?
• How often does data change? and how quick do changes need to be reflected?
• What ways of exporting and importing data does the existing application provide? this is of special importance when that app is hard to extend!
• How likely is the data format of the old app to change? In a new Microservice-based architecture you add APIs to encapsulate app data whose format changes often. In an old legacy app, maybe nothing will change any more until the app dies some day. (Also, beware that we often expect this to happen, but as the proverb says: "The condemned live longer.")

In any case, the first question you should answer is: does your new API just provide a new form of access to existing data and functionality or does it provide you functionality and needs a different form to store the data? Use this and other criteria to decide if the API should be part of the existing app or a new application.