Extract, Transform, Load - process in a database

learn more… | top users | synonyms

1
vote
0answers
32 views

SSIS developer using local data needs to call web services efficiently and interested in patterns [closed]

I have some experience writing ETL programs using Microsoft SQL Services Integration Services (SSIS) Packages and SQL Server Stored Procedures. Once, I needed to call a web service as part of the ...
0
votes
0answers
19 views

Sequence diagram with different scheduling time and events

I have to refactor a system and I want to draw some Sequence Diagram. The point is that I have different scheduling times and real time events and I don't know exactly how to do a clean Sequence ...
4
votes
2answers
79 views

Is ReST useful in Read/Write Operations that involve over 100 Gig

I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a ...
0
votes
1answer
108 views

Coordinating a complicated data migration process

A project I'm involved in has suffered a change in scope, and before I set about trying to cook up some homegrown solution, I'm wondering if there is something out there -- some framework, for example ...
3
votes
1answer
180 views

What is the right way to process inconsistent data files?

I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from ...
3
votes
2answers
192 views

Disagreement Concerning Data Integration (I may not understand enterprise ETL tools)

I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data ...
1
vote
1answer
105 views

Production or Custom Test Data for Unit Testing?

I've recently had a little disagreement with fellow developers. We're transforming various ontologies from the original source format (Pica+, RDF, etc) into our data format and have several converters ...
2
votes
2answers
206 views

Data warehouse architecture for mutating schema

I am setting up an ETL process and small data warehouse for querying the data in a few different dimensions. One issue is that the schema for the objects can mutate over time - mainly that some fields ...
3
votes
1answer
129 views

Enterprise Wide Keys [closed]

I have for a long time been working on an ODS as well as Data Warehouse. Both are integrating a wide variety of data sources from stove pipe applications. One of the uses of the ODS is to provide ...
4
votes
1answer
1k views

Designing a Content-Based ETL Process with .NET and SFDC

As my firm makes the transition to using SFDC as our main operational system, we've spun together a couple of SFDC portals where we can post customer-specific documents to be viewed at will. As such, ...
6
votes
3answers
1k views

How to do data synchronisation between two applications?

Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database. Application A is used ...
4
votes
2answers
569 views

How can large data handling be a special requirement for job?

I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to ...
2
votes
3answers
358 views

ETL Software Research Question

Where I work, we use an in-house ETL solution that's homegrown and has been around for 5-10 years. I'm still new to my data analysis job, but I was wondering about the ETL tools that are out there. ...