Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm creating a website that is going to aggregate women shoes from across the web into one website. Kinda like does with all their products but mine is only going to be shoes. Which makes my job a lot easier but I'm still lost as to how I go about getting all the data.

I have access to 3 APIs so far. Which provide over 15000 line items so far but there are 1000's of sites out there and millions of products.

What is the best method to aggregate all this data? Is it even possible?

Or should I rely on a social aspect as well, like polyvore where users can upload any products that aren't on the website?

Also, do I have to resort to screen scraping?

And is this possible as a one man show? Polyvore has a team of 30 programmers, do I need a team to do this properly?

share|improve this question

closed as not a real question by Martijn Pieters, Karl Bielefeldt, Kilian Foth, thorsten müller, Bart van Ingen Schenau Apr 16 '13 at 13:31

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 1 down vote accepted

From a technical perspective, this is all possible to one degree or another. Your best option is periodic ingestion of another site's collection via APIs that they provide which return machine-readable data. Failing that, scraping could also work, though it is inherently brittle and might need to be tweaked often for every site you scrape from. Also, you are limited to pages that are discoverable through web crawls: anything hidden behind a search form will be inaccessible to you.

Once you have the raw data, you'll need to normalize it into the core items necessary to display a pair of shoes and allow people to shop for them. This is essentially a problem of mapping data in a variety of formats into yours. All the information is probably there, but it will likely be organized differently on each source. There's no rocket science here, but all this data mapping (not to mention building your own site) may make you wish you had a team of 30 programmers. :)

From a business perspective, I think you are on somewhat shaky ground. First of all, what you are doing may or may not be legal: you will likely either make copies of their text and images, which may be copyrighted, or you will be deep linking into their site, which is itself controversial. Second, if you homogenize the shopping experience, you suppress an individual store's unique branding and shopping features and put it on equal footing with its competitors, which it may not like (there goes its hard-won SEO and expensive engineering). To help with legality and safety, respect robots.txt files (if you scrape) and make it easy for stores to opt out (or for you to opt them out). That being said, Google Shopper has been doing this for years (though they recently switched to a model where stores pay to be included in search results--no more controversy).

I think your best bet is to establish business and technical relationships with each store you support. This will give you clean APIs to use and you will be perceived as a friend, not a nuisance (or worse).

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.