Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I currently have a php script set up to do some xml parsing. Sometimes the script is set as an on page include and other times it is accessed via an ajax call. The problem is the load time for this particular page is very long. I started to think that the php I had written to find what I need in the XML was written poorly and my script is very resource intense. After much research and testing the problem is indeed not my scripting (well perhaps you could consider it a problem with my scripting), but it looks like it takes a long time to load the particular xml sources.

My code is like such:

$source_recent = 'my xml feed'; 
$source_additional = 'the other feed I need'; 

$xmlstr_recent = file_get_contents($source_recent);
$feed_recent = new SimpleXMLElement($xmlstr_recent);

$xmlstr_additional = file_get_contents($source_additional);
$feed_additional = new SimpleXMLElement($xmlstr_additional);

In all my testing, the above code is what takes the time, not the additional processing I do below.

Is there anyway around this or am I at the mercy of the load time of the xml URL's?

One crazy thought I had to get around it is to load the xml contents into a db every so often, then just query the db for what I need.

Thoughts? Ideas?

share|improve this question
1  
How large is the file you are processing? –  Jordan Mar 1 '12 at 22:33
    
recent is 277kb and additional is 4kb. Each file is processed when the script is called. I need the info from both at the same time. –  absentx Mar 2 '12 at 1:30
    
Doesn't PHP have a profiler? –  user1249 Mar 2 '12 at 7:39
1  
Do you need all the data in the XML file? Could you create pages and read a page at-a-time, could you remove items not really used by the application? –  Emmad Kareem Mar 2 '12 at 7:43
    
Close voters: Having a little bit of code in the question doesn't mean it automatically belongs to Stack Overflow. This is a question on optimization and it's on topic here. –  Yannis Rizos Mar 3 '12 at 9:57
add comment

4 Answers

up vote 5 down vote accepted

I suggest you look into caching. Chances are the feeds don't change much, and if they do, maybe you can afford getting the changes into your application a little bit later.

Basic caching would go something like this:

  • Do we have the XML data in the cache?
  • If we do, just use the cached data.
  • If we don't, load and parse the XML file, and store the resulting DOM tree in the cache, then use the parsed data.

This would at least reduce your average response time; when the cache expires, one response will take longer, but the rest in between would completely skip the parsing step.

If you don't want any response to take longer, then you need to do the parsing asynchronously. Such a system requires three components: your existing web application, a daemon or cron job, and some kind of shared data store - a plain file in an easy-to-parse format, a memory cache such as memcached, or a database. The daemon process / cron job downloads and parses the XML files at regular intervals (say, every minute, or whatever makes sense) and updates the shared data store. If the data store update itself takes too long, consider using two data stores that you can swap atomically (e.g. using file renames or changing a symlink). The web application then never downloads or parses the XML itself, it simply queries the shared data store. Since the data there has been parsed already, the overhead is gone.

share|improve this answer
    
Thanks for this answer. Ultimately I went with having the xml parsed to a file on a set schedule, then simply get the results I need from that file. A billion times faster, thanks for the direction. –  absentx Mar 6 '12 at 19:02
add comment

Do you know what is actually taking the most time? Is it disk I/O reading the file, is it memory allocation, is it creating the DOM tree? Sometimes it's faster to do stream parsing rather than reading the whole file into memory and building a DOM tree, especially if the data you want is near the beginning of the file.

share|improve this answer
add comment

I also had problems with parsing large XML files. I tried Expresso XML Parser. It can parse files up to 35GB and it's really fast. It's really easy to use. You set up parsing rules on a website and test your file online and then use their client code to access your parsing rules from java or javascript.

They have a free developer version at www.sxml.com.au

share|improve this answer
add comment

Is the problem that it takes a long time to parse the XML, or is the problem that your code sits around waiting for e.g. HTTP requests to return?

It is possible to create nonblocking sockets in PHP, though things might get a tad complicated.

If you have expat installed, you can also set up event handlers to react to elements as they are parsed, which complements an asynchronous read very nicely.

Perhaps you can delegate this task to another process/service developed in a language that supports asynchronous operations. It won't guarantee a speedup (you still have to wait for both HTTP requests to return), but at least you can start working on data as they come in rather than wait until you've received all the data before starting to process.

Note that this is distinct from the second half of tdammers's answer. This answer focuses on keeping the retrieval and processing of the XML at "request time" (necessary when the data in the XML files change frequently and/or the data has to be "real time"), whereas tdammers's answer focuses on keeping the data in a cache (offers much better performance when the data don't change that often and/or if "stale" data are acceptable).

share|improve this answer
1  
the issue completely was the time for the HTTP request to return! The data being a little slower (aka not as fresh) is okay in this situation so tdammers approach is what I went with. I appreciate this angle though if I need an approach in the future where the fresh xml data is imperative. –  absentx Mar 6 '12 at 19:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.