Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have thousands of .xml files from size 1MB-45MB (no DTDs). I need to parse and further manipulate these XML files before generating separate .xml files with the results of my regex.

What the fastest open-source XML parsing library for C++? Aside from the parsing library, what other approaches can I use to speed up XML parsing?

share|improve this question
    
This would really depend on what kind of data you want to extract. If you can, you should aim for a streaming approach, that also allows you to quickly skip data you're not interested in (it's one thing to seek the matching closing tag and another one to build up the whole data structure in the memory while doing so). –  back2dos Oct 27 '11 at 15:26
1  
How "fast" do you need it? What have you tried that is too slow? –  Angelo Oct 31 '11 at 14:29

2 Answers 2

Is the raw speed of an XML parser or RegEx engine going to make as much of a difference as a faster processor running code that is easy for you to understand & maintain?

From the example you've given I can envisage very big files being processed. Even so would you not be wiser to try and find a way of breaking up the files and processing them in parallel to increase throughput and leverage multiple cores rather than focusing on relatively small differences in library speed.

share|improve this answer
    
Thanks for the suggestion. Is there a specific XML library suited for this task, which threads the XML parsing by default? –  A T Oct 27 '11 at 16:14
    
You'll be better off calling the parser in parallel with different files than by trying to parse a single file using multiple threads. –  TMN Oct 27 '11 at 20:01
    
Okay, well which of the available free, open-source XML libraries is the fastest at parsing? –  A T Oct 28 '11 at 4:24

RapidXml is an attempt to create the fastest XML parser possible, while retaining useability, portability and reasonable W3C compatibility. It is an in-situ parser written in modern C++, with parsing speed approaching that of strlen function executed on the same data.

http://rapidxml.sourceforge.net/

Maybe you could start comparing Expat with this one? Expat is known to be really efficient on speed. RapidXml is used as backend of some boost libraries, mostly Boost.PropertyTree.

Also, I think maybe a more xml-scheme-specific approach could (maybe) be more efficient because of knowledge of the structure of the code. Such claim is just a supposition but if you're interested, CodeSynthesis provide a C++ code generators that takes an xsd file as input. The resulting parsing code might be more helpful... if you take time to define your format in an xsd. There are other similar tools available but it's the one I'm using for my last project. Those tools are mostly based on xerces but you can generate code that is independent. No idea about the performance impact.

share|improve this answer
    
Rapidxml is not suitable for stream processing large files as it is a DOM parser. It needs to load the entire file in memory. Expat or libxml2 will work for OP's scenario and those are what he should benchmark for his data. –  Amit Naidu Jun 8 '13 at 21:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.