Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In which ways could object-orientation (OO) make my data analysis more efficient and let me reuse more of my code? The data analysis can be broken up into

  1. get data (from db or csv or similar)
  2. transform data (filter, group/pivot, ...)
  3. display/plot (graph timeseries, create tables, etc.)

I mostly use Python and its Pandas and Matplotlib packages for this besides some DB connectivity (SQL). Almost all of my code is a functional/procedural mix. While I have started to create a data object for a certain collection of time series, I wonder if there are OO design patterns/approaches for other parts of the process that might increase efficiency?

share|improve this question
You might want to look into ETL (Extract, Transform, Load) design patterns and DAO (Data Access Object) design patterns. –  FrustratedWithFormsDesigner Oct 29 '12 at 18:27
Hm, will do. Thank you! :D –  Konsta Oct 29 '12 at 18:41
add comment

closed as not a real question by gnat, Robert Harvey, Walter, GlenH7, Yusubov Oct 29 '12 at 22:59

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer

Since this is a pure ETL task (as @FrustratedWithFormsDesigner noted), OO can't really add much value. Such tasks are best suited to procedural processing.

What could help you here, if you can do it and if you have huge data sizes, is parallel processing. This makes a great difference. There are existing tools for ETL processing (they don't use OO). I suggest you use one as they have built-in modules that could help with parallel processing and simplify the task of preparing robust applications without much or any code. Also, the proper design of your database to use Star Schema can boost query performance of the end solution.

share|improve this answer
OOD could be used for ETL, for a class hierarchy for different types of transformers (Numeric, Text, Regex, etc...) all implementing a common interface and with methods so that a chain of transformers can be created where one transformer will call the next. Another approach would be to have a Transformer class for each entity being transformed, such as CustomerTransfomer, CustomerAddressTransformer, etc... Whether or not this is a good approach really depends on the data. Some forms of data are really not suited to this approach at all. –  FrustratedWithFormsDesigner Oct 29 '12 at 19:08
Part of the data analysis actually is sped up by parallel processing (PP). But for the other part often small data sets are retrieved, transformed, merged, transformed more and displayed. PP is not appropriate; computation takes ~5min. In contrast, getting an overview of the data set (columns and values) and transforming is the effort I'd like to improve. Creating small functions for each step vs copy&pasting earlier scripts currently seems to be an improvement. Objects holding client/product info and display parameters thereof might be helpful. So, is there anything else for the non-PP part? –  Konsta Oct 29 '12 at 21:05
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.