Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I currently code in PHP. Recently I've been working on a project using PHP and Symfony that:

  1. reads large XML files (lots of DOM parsing/reading),
  2. converts large XML files to large arrays,
  3. merges 2 large arrays (lots of array sorting),
  4. takes the 2 large arrays and turns them into a large CSV file.

I finished it in PHP but now it is kind of memory intensive and requires about 8-15 seconds to run. So now I have the following options and need help choosing one:

  1. Try rewriting/refactoring it using better methods in PHP
  2. Choose a different programming language (I have been wanting to learn one, possibly another language processes these things a lot faster?)
  3. Do 1 or 2 and additionally set up something to be constantly reading xml files and write them to MongoDB documents to serve clients from the database instead of scrapping the data.

I am inclined to do 2 or 3 (using a different language), since I am sure there is another language that handles these kind of tasks much faster (e.g Python, C etc.). It's just that I am not sure which.

share|improve this question
7  
define large XML? what is the size? –  Pangea Jan 15 '12 at 17:19
2  
"merges 2 large arrays (lots of array sorting)" - that is lots of O(n log n) operations (at least). Most likely your problem does not lie with the language or even a library, but with the algorithm. You have to get the complexity right & tight. –  Job Jan 15 '12 at 22:55
    
You might want to mention the OS this needs to run on. –  GrandmasterB Jan 16 '12 at 5:09
add comment

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

closed as off-topic by MichaelT, GlenH7, Dan Pichelman, World Engineer Nov 19 '13 at 14:59

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions about what language, technology, or project one should take up next are off topic on Programmers, as they can only attract subjective opinions for answers. There are too many individual factors behind the question to create answers that will have lasting value. You may be able to get help in The Whiteboard, our chat room." – GlenH7, Dan Pichelman, World Engineer
If this question can be reworded to fit the rules in the help center, please edit the question.

5 Answers

up vote 16 down vote accepted

Before choosing a different language, first make sure the language is the bottleneck. So did you actually measure the time for the 4 steps? Do you know that, for example, most waiting time for step 1 is spend in your language interpreter (and not caused by disk IO)? If the latter is the case, then chosing a language like C++ (or even assembler) may not bring you any relevant speed increasement.

XML processing is often dependent on the speed of the XML parser, so that may be the next thing to look at, something that is only partially language dependent. libxml is very fast, it is a C library, but also available as a PHP extension. Building large arrays may be speeded up by a language change, sorting, however, probably not (or not much), if you are using the builtin sort function of PHP (which is a library function coded most probably in C or C++ itself). And writing data into a file is mostly dominated by disk I/O - again something where another language won't help.

So first make sure where the real problem is, otherwise you will be astonished how wrong one can be when making assumptions in optimizing without measuring.

share|improve this answer
    
Yeah Ill take another look at the code tomorrow before giving up on php. Although currently its not even that slow (7s, 2/300 mb ram) but it wont work in a high use environment (way too many servers needed). So I guess I should script something to process and store the stuff for easy retrieval later. I am currently using php's build in DomDocument function (not sure what that uses...). –  NoviceCoding Jan 15 '12 at 23:28
    
@NoviceCoding: if you want more help from this forum, I suggest you work out where in your 4 steps the most running time is spended, provide some more details about the specific step and edit your question above accordingly. Then, I am sure, you will get better answers. –  Doc Brown Jan 16 '12 at 7:09
add comment

The language I would recommend for working with XML is XSLT which is designed for this purpose. It is ideally suited to merging XML files and producing a CSV file as output.

XSLT stands for XSL Transformations, and is a W3C (World Wide Web Consortium recommendation since 1999. XML is also a W3C standard, and thus both XML and XSLT are related standards. In fact XSLT is in fact XML, so if you know XML, which presumably you do if you are working with XML documents, then you already understand the structure of an XSLT program.

Firefox, IE and Google Chrome all implement XSLT processors. Further Microsoft, Apple have implemented XSLT processors and opens source processors such as the toolkit of Gnome exist as well.

Since there have been so many implementations of XSLT processors, the offerings from major vendors are highly optimized an IMHO you are unlikely to find an alternative implementation f0r working with XML files that would outperform on speed.

According to Wikipedia:

Increasingly, however, XSLT processors use optimization techniques found in functional programming languages and database query languages, such as static rewriting of an expression tree (e.g., to move calculations out of loops), and lazy pipelined evaluation to reduce the memory footprint of intermediate results (and allow "early exit" when the processor can evaluate an expression such as following-sibling::*1 without a complete evaluation of all subexpressions). Many processors also use tree representations that are significantly more efficient (in both space and time) than general-purpose DOM implementations.

You may want to see these other questions as well: what is the reason for using xslt? & XSLT for Beginners

share|improve this answer
    
@Marvin, thanks. Yes and I edited the response to fix the typo. –  JonnyBoats Jan 15 '12 at 16:50
    
+1 for XSLT... I once worked on an XML processing application in Symfony and XSLT quickly became part of my toolkit –  CamelBlues Jan 16 '12 at 3:01
add comment

Surprised that no one mentioned this, but instead of reading/parsing the XML data into a DOMDocument, you can use SAX processing. This means that while the XML data is read and parsed, there are functions that you add that will be executed. For example, when the starting tag of an element named "root" happens, it will execute a function called "root_start" or whatever you want.

You can use the SAX processing method by using the XML Parser library. They include a few examples.

You can then skip over storing the XML data as a DOMDocument, and store the data as those large arrays you mention. Then continue to merge/sort as you did before. This should decrease both the memory consumption and the time taken for the task.

Make sure you're using a good sorting/merging algorithm as well. You might want to do some sorting or merging while you're processing the XML document...

share|improve this answer
    
+1 for switching from DOM to SAX. Simply use the php XMLReader library instead of the DOM library. –  James Anderson Jan 16 '12 at 3:31
add comment

If you can learn and use tool other than PHP then MS LinQtoXML is good option, works great for me to load XML from files, Serialize, Query XML like SQL, Add, Remove, SetValues and merge the collections.

Please check this for further detail by Microsoft LINQ to XML Overview

and this LINQ to XML vs. Other XML Technologies

Here is simple example to Generate CSV file from XML msdn.microsoft.com/en-us/library/bb387094.aspx

you can find .Net Mono framework and code editor to play with this from here www.mono-project.com and editor monodevelop from here monodevelop.com/ if you are not using windows platform.

Hope this will help.

share|improve this answer
3  
Can you explain why this is a good option for this project? A one line answer doesn't tell us anything. –  user8 Jan 15 '12 at 19:45
1  
Mark, Muhammad's answer tells us he uses LinQtoXML and it works for him. That's a useful data point. –  Jim In Texas Jan 15 '12 at 20:03
1  
It's not much of a data point; it's a Microsoft tool and we don't know the platform being used, and we don't know exactly wtf using LinQtoXML entails. Is it a new programming language? –  omouse Jan 16 '12 at 3:01
    
LINQ to XML loads the entire XML into memory, which is not a good solution for large files, and the question is about large files. Programming in C#, I don't use LINQ to XML for big files, I use XmlReader instead. –  Konrad Morawski Jan 16 '12 at 10:03
add comment

If you aim at improving the runtime of your program, i think a compiled language ( c, c++ ..) would perform better than an interpreted one (PHP, python) for doing redundant tasks. On googling for an XML parser in C i found http://www.codeproject.com/KB/XML/CXMLFile.aspx. Hope this helps.

share|improve this answer
4  
Can you explain why this is a good option for this project? A one line answer or a quick Google search doesn't tell us anything. –  user8 Jan 15 '12 at 19:45
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.