Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a large XML file (about 75,000 lines) where I have to build a catalogue (houses) from. Building the lists works fine, but now I have a problem.

The catalogue should have a detailed presentation page of one house. One house (<item id="123">) has about 800-1200 lines of data, based on the house type.

Which is the best way, as referring to making the script faster and saving lines of code, to read these data and present them?

Some houses, for example, have a sauna, and when these data are in the XML file, the presentation page should contain a section sauna.

I tried before to read the whole XML content with a recursive function into arrays and with a lot of foreaches (maximum depth of children is three) with SimpleXML, but it was really ugly slow, and the recursion did not work at all, because my computer could not handle so much input.

Is there any other way to build this data except to query every variable with if?

share|improve this question
3  
What about moving this data into a database, either relational or (maybe better for some types of query NoSQL)? –  thorsten müller Apr 17 '13 at 12:04
1  
Hint: you can almost always remove the word "dynamically" from such questions. It adds nothing (try it: read the text, but skip every "dynamically"). –  Joachim Sauer Apr 17 '13 at 12:45
    
Tearing through XML is one of the (few) areas that PHP excels. I've personally found the DOMDocument class to be more handy than simplexml. –  CamelBlues Apr 17 '13 at 14:21
    
@JoachimSauer, don't worry, I dynamically edited the question to fix that. –  dan1111 Apr 18 '13 at 4:33
    
Thank you about the suggestions! English is not my native language, but now I know to avoid dynamically :) –  Owl Apr 18 '13 at 11:02

2 Answers 2

up vote 8 down vote accepted

XML is an inefficient method of storing large amounts of data. It uses a lot of disk space (look at an XML file and note what a large portion of it is taken up by the syntax and structure definition), and it is slow and memory-intensive to access. The whole tree (or at least a large portion of it) must be parsed just to get a single element, and XML parsers often use several times the size of the file in memory space to do this.

If you need to do something performance sensitive (such as load information onto a web page), 75,000 lines of XML just isn't going to be fast.

If performance matters, you should really move the information into a relational database, as suggested by thorsten müller. Then your task will become trivial. Even if you have no choice but to receive the data in XML, have your program perform a one-time load of that XML file into the database whenever it is updated, and then use the database the rest of the way. Besides being faster, the database will also be a lot easier to work with.

If you choose to stay with XML, you can get some help with your algorithm, but more information is needed. I suggest posting the portion of the code that you describe on Code Review.

share|improve this answer
    
Thank you very much! I learned a lot –  Owl Apr 18 '13 at 11:00
    
There's no need to parse the whole tree at once when reading XML. Large XML files can be handled using event based parsing. –  COME FROM Apr 18 '13 at 12:02
    
@COMEFROM, yes, that is why I added the qualification (or a large portion of it). Since the XML is stored in a text file and doesn't have random access, you have to go through a lot of the file, on average, just to get a single element. –  dan1111 Apr 18 '13 at 12:09
    
I have a question here, I am working with HotelsPro API which contains an XML file of 96000+ hotels around the world. I want to save them all in mysql database without facing execution time error. Is that possible anyhow? Please let me know. –  William Francis Gomes Nov 2 at 10:13
    
@WilliamFrancisGomes, try it yourself, and if you run into specific problems, then post a question about it on the appropriate site (probably Stackoverflow, not here). –  dan1111 Nov 3 at 12:24

Maybe you're running into the limitation that SimpleXML tries to read the entire document in one step, and your document is large or complex. An alternative is to use a stream- or event-based parser which reads bits at a time, sets the state of the parser based on what was read, and lets you register handlers that will react to specific events. Here is a explanation of the idea with a simple example: http://sax.sourceforge.net/event.html

I'm not very familiar with PHP, but it looks like there is something like SAX for you to try: http://php.net/manual/en/book.xml.php. From the introduction:

This extension lets you create XML parsers and then define handlers for different XML events.

Searching for sax in php on Google also shows some promising leads.

share|improve this answer
    
Thank you too, that sounds intresting! –  Owl Apr 18 '13 at 11:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.