Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am currently working on a small project in which I work with some "static data", which is extracted from one source. The data might change from time to time (up to once a week). I don't need all of the information the data includes in my project so to allow easier and a more logical access to it I wrote some small tool that converts the extracted data into my desired format.

The conversion is mostly dropping data I don't need, and splitting some strings into arrays. No intensive calculations or anything. But, I drop more than 50% of the original data because their is no need for it.

Now since it is static data would it make more sense to just store that converted data and do all further operations from this data or would it make more sense to convert from the raw data on the fly?

Since the project is in JavaScript speed of course is important, but if converting on the fly has significant advantages it might make more sense to go with that approach.

It is static data from a computer game that defines several units and other objects within the game. They are stored in JSON therefore I can easily use them in JavaScript. But I don't need a lot of the information, just some parts of it to build a kind of "encyclopedia", now since the data is already present in JSON I was wondering if I should just use it or convert it before usage.

share|improve this question
    
Don't think this is answerable if you don't tell us exactly what the conversions / operations are. –  Yannis Rizos Sep 20 '12 at 10:55
    
Most of the conversion is dropping data I don't need in my actual project, there is no hard math or anything involved. Some string splitting to convert some fields into arrays. What might be worth mentioning is that I not even use 50% of the original data, so a lot gets dropped. –  clentfort Sep 20 '12 at 10:59
    
Something smells wrong in your data query, if you drop more than 50% of data. –  Yusubov Sep 20 '12 at 11:03
    
It is static data from a computer game that defines several units and other objects within the game. They are stored in JSON therefore I can easily use them in JavaScript. But I don't need a lot of the information, just some parts of it to build a kind of "encyclopedia", now since the data is already present in JSON I was wondering if I should just it or convert it. –  clentfort Sep 20 '12 at 11:06

3 Answers 3

up vote 4 down vote accepted

Generally speaking, duplication is a greater evil than bad performance. The answer will typically depend on whether the performance is good enough with the on-the-fly conversion. You will have to figure that part out yourself. Stick to the current implementation until you have proof that it's too slow; do not optimise it because you guess that it will be too slow.

An alternative approach would be to take your on-the-fly code, and create a pre-processor out of it, which outputs the cut-down data set. You can then run this code as part of your build process. The benefit is that your core data set is not (really) duplicated, and you get the performance benefits. Also, I realise that this code is in JavaScript, but there are ways to get JS to run on your development machine during the "build".

share|improve this answer

I would write a Data Conversion Process that Runs on a timer. (Once a Day or once a week.) Once the data it conversion is Complete, generate a JSON file from it drop it on you Web server, overwriting your previous JSON file .

share|improve this answer

It sounds like you are leaning towards better performance, then you would rather store the data and make sure that you only update the stored version if any changes were made. Of course this approach can pose it's own challenges such as version control and keeping the data in sync if so desired.

If data integrity is of absolute importance you should rather convert on-the-fly and compromise a bit on performance.

Sit down and think what your application is supposed to do and what is more important to the people that will be using the software. Speed or data integrity?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.