Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 6 down vote favorite
5

I frequently encounter a situation where I need to write a program which reads data-objects from a file or system, does a transformation on the data, convert the data to another datamodel and then push the data in another file, or system. Do you know of a object-model which facilitates these type of programs in such a way that it is pluggable and scalable?

share|improve this question
It sounds like you are talking about a pipe and filter architecture? – Macneil Nov 27 '10 at 18:36

3 Answers

up vote 0 down vote

Consider Gstreamer.

Though it is mainly for multimedia domain, and i don't know what is yours; however, here is what you can gain from the overall architecture -

  1. Any kind of data is processed by various elements arranged in a sequence called pipeline

  2. Data exchange between elements is done at interface called pad which are on sides of both the elements. The mechanism is either PUSH based or PULL based.

  3. elements are inherited from variety of capabilities which cab be plugged arbitrarily. They can negotiate (in run time) the common denominator for all.

  4. Application can give command to individual or all elements through a message bus and properties

There are many more stuff - i am not trying the exhaustive about what gstreamer does but only suggesting that this could be directly useful to you or at least can inspire you a lot.

Some people believe that gstreamer is derived out of DirectX. That may be true or false but doesn't bother me. I am not comparing.

share|improve this answer
up vote 1 down vote

Read Enterprise Application Integration Patterns.

There is a website related to it here that contains a wealth of inter-application processing design patterns for you to browse. In particular, I'd draw your attention to these patterns:

Messaging How can I integrate multiple applications so that they work together and can exchange information?

Message Channel How does one application communicate with another using messaging?

Message Router How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions?

I've got a copy of the book myself and it's packed with detail about inter-application and inter-system design patterns.

share|improve this answer
Thanks Gary, I have the book myself :) I do appreciate this book for SOA based design, but I'm looking for a little bit more lowlevel, and need a detailed Object-design. – Edward Smit Nov 28 '10 at 8:18
up vote 2 down vote

here's a link to an article on an alleged Extract/Transform/Load (ETL) design pattern but it sounds like what you want is a framework

share|improve this answer
Thanks for the link, but I'm more looking for a GoF like design pattern, inlcuding Objects, relations etc. – Edward Smit Nov 27 '10 at 15:21
@Edward: if you find one be sure to edit your question and publish it; ETL is not a GoF design pattern. There are ETL frameworks however. good luck! – Steven A. Lowe Nov 27 '10 at 15:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.