Extract, Transform, Load - process in a database

learn more… | top users | synonyms

6
votes
3answers
2k views

How to do data synchronisation between two applications?

Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database. Application A is used (...
5
votes
2answers
222 views

Disagreement Concerning Data Integration (I may not understand enterprise ETL tools)

I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data ...
4
votes
2answers
742 views

How can large data handling be a special requirement for job?

I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to ...
4
votes
2answers
302 views

Is ReST useful in Read/Write Operations that involve over 100 Gig

I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time)...
4
votes
1answer
2k views

Designing a Content-Based ETL Process with .NET and SFDC

As my firm makes the transition to using SFDC as our main operational system, we've spun together a couple of SFDC portals where we can post customer-specific documents to be viewed at will. As such, ...
3
votes
2answers
271 views

Data warehouse architecture for mutating schema

I am setting up an ETL process and small data warehouse for querying the data in a few different dimensions. One issue is that the schema for the objects can mutate over time - mainly that some fields ...
3
votes
1answer
421 views

What is the right way to process inconsistent data files?

I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-...
3
votes
1answer
137 views

Enterprise Wide Keys [closed]

I have for a long time been working on an ODS as well as Data Warehouse. Both are integrating a wide variety of data sources from stove pipe applications. One of the uses of the ODS is to provide ...
2
votes
3answers
393 views

ETL Software Research Question

Where I work, we use an in-house ETL solution that's homegrown and has been around for 5-10 years. I'm still new to my data analysis job, but I was wondering about the ETL tools that are out there. ...
2
votes
1answer
220 views

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average ...
2
votes
2answers
310 views

Using ESB for database synchronisation / replication

We're starting to look at implementing an ESB / Microservices architecture. I (think) I know about the concepts, but there's one thing I don't seem to be able to get a good idea about: data ...
1
vote
1answer
636 views

Difference between ESB and ETL

When should a ESB vs ETL tool be used? I have worked on ESB projects using Tibco Businessworks quite a few years ago. The message bus that we built used to consume messages from a source system , ...
1
vote
1answer
114 views

Production or Custom Test Data for Unit Testing?

I've recently had a little disagreement with fellow developers. We're transforming various ontologies from the original source format (Pica+, RDF, etc) into our data format and have several converters ...
0
votes
1answer
217 views

Coordinating a complicated data migration process

A project I'm involved in has suffered a change in scope, and before I set about trying to cook up some homegrown solution, I'm wondering if there is something out there -- some framework, for example ...
0
votes
0answers
49 views

Fault-tolerant, distributed, event dispatcher architecture?

I'm building an event handler/dispatcher for my monolith, aiming to slowly replace big parts of it by delegating jobs to other systems via events. For example, sending emails when a customer buys ...
-1
votes
0answers
41 views

Help building an ETL with SSIS

I am planing an ETL with SSIS and need some help. I have the following basics: I have a dataprovider who stores data from different datasets. Let us say Survey 2012, Survey 2013 and so on. I get a ...