I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data with theirs, either by sending data from their systems into ours, or by getting data from our system into theirs. I don't believe we are talking about 'big data.' Our entire database is less than 20 gigs.
The individual transfers from our customers will be very small as well - on the order of 10s of MBs, if not smaller.
A major goal for this project is to minimize the amount of work needed to be done by the customer, who tends to not be very technical. In the past, this problem has been "solved" by having our customers send us an xml file that is transformed by xlst before being saved to the database, or through a WCF application for exporting data. These are not seen as good solutions.
My first thought was to create a REST interface with very good monitoring tools and libraries for our clients.
Another option that is being considered are big tools like IBM datastage or Pentaho (or similar).
My personal problem is that when I look at these tools, I don't see how they can help. It looks like they are aimed at setting up large scale data transfers that happen on a regular schedule with a given format. Adding a new import/export using those tools requires a lot of setup both for us and the customer, which we are trying to avoid. They are also not really useful for real-time data updates. Finally, they seem like they are trying to kill an ant with a flamethrower - they are heavyweights for a light weight problem.
In contrast, with a REST setup, data can be transferred whenever required, and Setup only needs to happen in a very small area.
Am I missing something when it comes to these ETL solutions? Am I not being fair to them?