Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

For the sake of an exercice I want to aggregate price listings with prices coming from a series of web sites in a structured form (XML, JSON...). If I have control on both sides how can I go about making price updates the most efficient possible?

Edit: To clarify, I'm looking for a more efficient approach then having a script or application pull in price lists in their entirety from all sources for updates.

share|improve this question
If you just want a database to store and retrieve information, use MongoDB, its simple and supports JSON in storage and search. But be sure to convert XML to JSON to store it in MongoDB (you can store XML too but can't search it if stored as text) –  Ubermensch Feb 20 '12 at 5:15
Re: "I'm looking for a more efficient approach then having a script or application pull in price lists in their entirety from all sources for updates." I am curious why you assume this is not efficient. Perhaps some constraints are not stated in the question? –  codingoutloud Feb 21 '12 at 23:34
@codingoutloud Well, pulling all the lists in, parsing them and then recomposing a central list seems like a lot of effort if only one price has changed in a given lapse of time. I'm thinking through the possible scenarios and how they could be best addressed. –  James Poulson Feb 28 '12 at 11:00
@JamesPoulson the choice of what to put in the feeds is yours. If you only need to deltas, the files will be small (assuming little volatility - though your description left that unstated). If you prefer to repub the whole price list, they'll be larger. (I used a similar approach on an enterprise app with favorable effects - for example, when troubleshooting it is nice to be able to look at the data feed in a web browser that already knows how to format it for human viewing.) You could forgo the XML for more fashionable JSON, but would be more work. –  codingoutloud Mar 5 '12 at 18:36

2 Answers 2

Gathering Data:

One approach is to ask all of the sites to publish an RSS feed with prices in a specific format. Could use ATOM format. Part of ATOM RSS feed is that it specifies metadata inside that says, essentially, "don't check back more frequently than hourly" (or whatever) which may help handle knowing how often to poll each without having to manage that data centrally (assuming price volatility may vary by source). This is basically how an RSS aggregator works (like Google Reader).

Could also look into making the data formats OData (see www.odata.org).

You could alternatively have all the sites call a central API, but that will be harder to manage and likely more complex for the partner sites - and more error prone.

Maintaining Data:

A database! Simple pricing tables may suffice: shred the data as it comes in and store in form that allows easy aggregation. Consider keeping all history (might have other value in trend analysis or otherwise). Recalc aggregates on a schedule or on demand. Consider how to handle "corrections" to prices (maybe a source site can ping the central site with a "correction" hint do the RSS grab can be scheduled sooner than otherwise to pick up the corrections).

You did not mention volume. Very large data sets may require more consideration, but this set up ought to get you pretty far.

share|improve this answer

If you have control over both sides, then I'd have a web service that all client sites post their price changes to. If price changes aren't discrete (that is, if nobody just updates prices, they re-publish whole items/catalogs/whatever) then you'll have to parse whatever updates are offered to get the pricing information. Or you could invert the structure, and have each client site offer some kind of "get latest prices" service that your central server could poll on a schedule. Worst case, you can use a "screen scraping" approach and crawl the site, extracting prices whenever you find a product/service you want to track.

It sounds like you've only got a few distinct things to keep track of (site, "item" [whatever's got a price] and price), so almost any data store would work. A simple relational DB (MySQL or Derby), a key-value store (like MongoDB or Cassandra), or even a spreadsheet. I think it really depends on what you want to do with the prices once you have them.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.