Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am working with data feeds from affiliate sites. The basic idea is to provide an interface where the user can paste a link to an XML datafeed (these are huge btw, around 60 mb) that would then be streamed, parsed into small chunks, and mined for the required data which would then be stored in the database.

The problem is that different affiliate sites have different Schemas for their XML's. It is a little hard mapping the elements in an XML to your database attributes when you don't actually know which element contains what.

My Solution: Use XPath to traverse through the first set of parent and it's descendent's, fetch the elements as well as the data and and ask the user to map this data to the attributes in the database by selecting from a set of radio buttons that represent the attributes from the database. This will be done just once for each new Feed, once the system know's what's what it will automatically upload the data from the XML to the database.

Does this sound viable? Is there a better solution? I realize this leaves an uncomfortable opening for human error..


share|improve this question

migrated from Aug 5 '12 at 13:32

This question came from our site for professional and enthusiast programmers.

up vote 1 down vote accepted

Does this sound viable?

It is completely viable. XML data can be fetched by cURL through PHP and parsed as XML data.

For example, YouTube video data can be fetched as XML by:

1) cURL the XML data. YouTube follows the link format of:

2) XML Parser Create the object

3) XML Parse Into Struct the fetched content. You may need to write some custom functions to traverse the data to extract what you want. I usually write custom generic functions such as "getXMLvalueByAttribute", "getXMLvalueByTag", and "getTagStringData" to easily do this.

4) XML Parser Free the object once you are done storing it.

You did not specify exactly what kind of XML data, so I gave you the YouTube example above.

Is there a better solution?

I would definately check if the service/website has an API available. This normally will save you time and energy if one is available.

share|improve this answer
Hi, I forgot about this thread. Thanks for the reply. I have the system up and running, it is still in beta but things are looking good. – SPI Dec 13 '12 at 8:35
Glad to hear you got it working. If my answer works, don't forget to accept answer. ;) – Talvi Watia Dec 19 '12 at 22:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.