Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For the sake of an exercice I want to aggregate price listings with prices coming from a series of web sites in a structured form (XML, JSON...). If I have control on both sides how can I go about making price updates the most efficient possible?

Edit: To clarify, I'm looking for a more efficient approach then having a script or application pull in price lists in their entirety from all sources for updates.

share|improve this question

closed as too broad by Scant Roger, Ixrec, MichaelT, Bart van Ingen Schenau, Dan Pichelman Dec 28 '15 at 19:41

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

If you just want a database to store and retrieve information, use MongoDB, its simple and supports JSON in storage and search. But be sure to convert XML to JSON to store it in MongoDB (you can store XML too but can't search it if stored as text) – Ubermensch Feb 20 '12 at 5:15
Re: "I'm looking for a more efficient approach then having a script or application pull in price lists in their entirety from all sources for updates." I am curious why you assume this is not efficient. Perhaps some constraints are not stated in the question? – codingoutloud Feb 21 '12 at 23:34
@codingoutloud Well, pulling all the lists in, parsing them and then recomposing a central list seems like a lot of effort if only one price has changed in a given lapse of time. I'm thinking through the possible scenarios and how they could be best addressed. – James Poulson Feb 28 '12 at 11:00
@JamesPoulson the choice of what to put in the feeds is yours. If you only need to deltas, the files will be small (assuming little volatility - though your description left that unstated). If you prefer to repub the whole price list, they'll be larger. (I used a similar approach on an enterprise app with favorable effects - for example, when troubleshooting it is nice to be able to look at the data feed in a web browser that already knows how to format it for human viewing.) You could forgo the XML for more fashionable JSON, but would be more work. – codingoutloud Mar 5 '12 at 18:36

Gathering Data:

One approach is to ask all of the sites to publish an RSS feed with prices in a specific format. Could use ATOM format. Part of ATOM RSS feed is that it specifies metadata inside that says, essentially, "don't check back more frequently than hourly" (or whatever) which may help handle knowing how often to poll each without having to manage that data centrally (assuming price volatility may vary by source). This is basically how an RSS aggregator works (like Google Reader).

Could also look into making the data formats OData (see

You could alternatively have all the sites call a central API, but that will be harder to manage and likely more complex for the partner sites - and more error prone.

Maintaining Data:

A database! Simple pricing tables may suffice: shred the data as it comes in and store in form that allows easy aggregation. Consider keeping all history (might have other value in trend analysis or otherwise). Recalc aggregates on a schedule or on demand. Consider how to handle "corrections" to prices (maybe a source site can ping the central site with a "correction" hint do the RSS grab can be scheduled sooner than otherwise to pick up the corrections).

You did not mention volume. Very large data sets may require more consideration, but this set up ought to get you pretty far.

share|improve this answer

If you have control over both sides, then I'd have a web service that all client sites post their price changes to. If price changes aren't discrete (that is, if nobody just updates prices, they re-publish whole items/catalogs/whatever) then you'll have to parse whatever updates are offered to get the pricing information. Or you could invert the structure, and have each client site offer some kind of "get latest prices" service that your central server could poll on a schedule. Worst case, you can use a "screen scraping" approach and crawl the site, extracting prices whenever you find a product/service you want to track.

It sounds like you've only got a few distinct things to keep track of (site, "item" [whatever's got a price] and price), so almost any data store would work. A simple relational DB (MySQL or Derby), a key-value store (like MongoDB or Cassandra), or even a spreadsheet. I think it really depends on what you want to do with the prices once you have them.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.