Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I've always found XML somewhat cumbersome to process. I'm not talking about implementing an XML parser: I'm talking about using an existing stream-based parser, like a SAX parser, which processes the XML node by node.

Yes, it's really easy to learn the various APIs for these parsers, but whenever I look at code that processes XML I always find it to be somewhat convoluted. The essential problem seems to be that an XML document is logically separated into individual nodes, and yet the data types and attributes are often separated from the actual data, sometimes by multiple levels of nesting. Therefore, when processing any particular node individually, a lot of extra state needs to be maintained to determine where we are and what we need to do next.

For example, given a snippet from a typical XML document:

<book>
  <title>Blah blah</title>
  <author>Blah blah</author>
  <price>15 USD</price>
</book>

...How would I determine when I've encountered a text node containing a book title? Suppose we have a simple XML parser which acts like an iterator, giving us the next node in the XML document everytime we call XMLParser.getNextNode(). I inevitably find myself writing code like the following:

boolean insideBookNode = false;
boolean insideTitleNode = false;

while (!XMLParser.finished())
{
    ....
    XMLNode n = XMLParser.getNextNode();

    if (n.type() == XMLTextNode)
    {
        if (insideBookNode && insideTitleNode)
        {
            // We have a book title, so do something with it
        }
    }
    else
    {
        if (n.type() == XMLStartTag)
        {
            if (n.name().equals("book")) insideBookNode = true
            else if (n.name().equals("title")) insideTitleNode = true;
        }
        else if (n.type() == XMLEndTag)
        {
            if (n.name().equals("book")) insideBookNode = false;
            else if (n.name().equals("title")) insideTitleNode = false;
        }
    }
}

Basically, the XML processing quickly turns into a huge, state-machine driven loop, with lots of state variables used to indicate parent nodes we've found earlier. Otherwise, a stack object needs to be maintained to keep track of all the nested tags. This quickly becomes error-prone and difficult to maintain.

Again, the problem seems to be that the data we're interested in is not directly associated with an individual node. Sure, it could be, if we wrote the XML like:

<book title="Blah blah" author="blah blah" price="15 USD" />

...but this is rarely how XML is used in reality. Mostly we have text nodes as children of parent nodes, and we need to keep track of the parent nodes in order to determine what a text node refers to.

So...am I doing something wrong? Is there a better way? At what point does using an XML stream-based parser become too cumbersome, so that a fully-fledged DOM parser becomes necessary? I'd like to hear from other programmers what sort of idioms they use when processing XML with stream-based parsers. Must stream-based XML parsing always turn into a huge state machine?

share|improve this question
1  
if your using a .net language, you should look at linq to xml aka XLinq. –  Muad'Dib Apr 8 '11 at 14:59

6 Answers 6

up vote 8 down vote accepted

To me, the question is the other way round. At what point does an XML Document become so cumbersome, that you have to start using SAX instead of DOM?

I would only use SAX for a very large, indeterminately-sized stream of data; or if the behaviour the XML is intended to invoke is really event-driven, and therefore SAX-like.

The example you give looks very DOM-like to me.

  1. Load the XML
  2. Extract the title node(s) and "do something with them".

EDIT: I'd also use SAX for streams that may be malformed, but where I want make a best-guess at getting the data out.

share|improve this answer
2  
I think this is a good point. If you're parsing documents that are too big for DOM then you need to consider whether you're parsing documents that are too big for XML –  Dean Harding Apr 8 '11 at 15:15
1  
+1: Given the option, I'd always go with DOM. Unfortunately, it seems our design requirements always include "ability to handle any size document" and "must be performant", which pretty much rule out DOM-based solutions. –  TMN Apr 8 '11 at 15:29
3  
@TMN, in an ideal world that requirements would rule out XML in the first place. –  SK-logic Apr 8 '11 at 15:49
1  
@TMN, that sounds like one of those phantom requirements: "Of course all of our documents are only about 100KB, and the biggest we've seen is 1MB, but you never know what the future holds, so we should keep our options open and build for infinitely big documents" –  Paul Butcher Apr 8 '11 at 15:56
    
@Paul Butcher, you never know. I mean, a dump of Wikipedia is like 30GB of XML. –  Channel72 Apr 8 '11 at 16:10

Must stream-based XML parsing always turn into a huge state machine?

Usually it does, yes.

For me to point to use a fully-fledged DOM parser is when I would need to mimic parts of the file hierarchy in-memory, for example to be able to resolve cross-references within the document.

share|improve this answer
    
+1: Start with DOM. Avoid SAX. –  S.Lott Apr 8 '11 at 15:19

Parsing in general is simply driving a state machine, and XML parsing is no different. Stream-based parsing is always a hassle, I always wind up building a stack of some sort to keep track of ancestor nodes, and defining a lot of events and some kind of event dispatcher that checks a tag or path registry and fires off an event if one matches. The core code is fairly tight, but I wind up with a huge wad of event handlers that mostly consist of assigning the value of the following text node to a field in a structure somewhere. It can get pretty hairy if you need to mix business logic in there too.

I would always use DOM unless size or performance issues dictated otherwise.

share|improve this answer

I don't work with XML too much, bit in my opinion, probably one of the best ways of parsing XML with a library is using XPath.

Instead of traversing the tree to find some specific node, you give a path to it. In the case of your example (in pseudocode), it would be something like:

books = parent.xpath("/book") // This would give you all the book nodes
for-each book in books
    title = book.xpath("/title/text()")
    author = book.xpath("/author/text()")
    price = book.xpath("/price/text()")

    // Do things with the data

XPath is much more powerful than that, you can search using conditions (both on values and attributes), select a specific node in a list, move levels through the tree. I recommend you look for info on how to use it, it is implemented in a lot of parsing libraries (I use it the .Net Framework version and lxml for Python)

share|improve this answer

Not completely language agnostic, but I typically deserialize the XML into objects rather than even think about parsing. Only time to worry about parsing strategies per se is if you have a speed issue.

share|improve this answer
    
That falls under parsing. Unless the XML in question is the output of object serialization and you have a ready-built deserialization library. But then this question doesn't appear. –  delnan Apr 8 '11 at 15:03
    
Many languages/stacks do have ready built deserialization libraries. –  Wyatt Barnett Apr 8 '11 at 16:54
    
Yeah, so what? My points still hold - not all XML files in the wild come in such a format, and if you have one that does, you don't ask this question as you just use that deserialization library and don't parse anything on your own, from streams or otherwise. –  delnan Apr 8 '11 at 18:26

It becomes much less cumbersome if you can use XPath. And in .Net land LINQ to XML abstracts a lot of the less glamorous stuff too. (Edit - these require a DOM approach of course)

Fundamentally, if you are taking a stream based approach (so you can't use nicer abstractions that require a DOM) I think it will always be pretty cumbersome and I'm not sure there is any way around this.

share|improve this answer
    
If you're using XPath, you're using DOM (unless you're using it with a home-grown XPath evaluator). –  TMN Apr 8 '11 at 15:25
    
yes, hence my comment about the abstractions requiring DOM... but I'll clarify, thanks! –  Steve Haigh Apr 8 '11 at 15:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.