Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm struggling with a client-server project where I have Java apps out on the Internet that store data to a backend server. The format of this data is well-defined, but the project is constantly evolving, so the definition keeps changing! To cope with the change I defined a simple REST interface on the server that offers only key-value storage. Clients can store or retrieve a chunk of data by referencing a unique key. This is nice because I don't have to modify the server interface (or the backend database) when the data format changes. To the server, it's just a bunch of opaque blobs.

Of course, the issue then becomes, "What goes inside the blob?" For that I wrote an XML Schema that defines the content of a blob. At first it was great, since the Schema gives a bunch of nice things "for free": A formal yet human-readable spec of the file format, automatic validation of its contents, marshalling/unmarshalling to a stream, and auto-generated Java classes for programmatic access to the data.

But then change happened! The Schema had to be altered, and naturally I ran into forward- and backward-compatibility issues. To deal with the constantly changing Schema, I came up with a solution that embeds a version number into the XML namespace, and I apply a series of XSL Stylesheets to "upgrade" any given blob to the latest version. For example, I'm now on version 1.3 of my Schema, so when I unmarshal a blob, I run it through a 1.0-to-1.1 XSLT, then a 1.1-to-1.2 XSLT, and finally a 1.2-to-1.3 XSLT. This works, but it's not sustainable because the chain keeps getting longer, which lowers performance and sucks up memory, plus I have to keep writing new Stylesheets, which takes time and isn't fun.

Now here's the funny thing... In addition to the Java clients, the project also has iOS apps as clients, and iOS has none of the nice enterprise-y features associated with XML Schemas. There's no validation of the stream, no auto-generation of Objective-C classes, etc., just a low-level event-driven XML parser. But ironically I'm finding this so much easier! For example, if the XML gets a new element, I just add a new if clause. If an element goes away, I remove its clause. Basically, I do a "best effort" at interpreting the XML stream, silently ignoring any unrecognized elements. I don't need to think about what version the file format is or whether it's valid. Plus this is much faster because there's no XSLT chaining, and it saves a lot of my time because I don't have to write any XSLT code.

So far this approach has worked out great, and I've not missed having an XML Schema on the iOS side. I'm now wondering if a Schema, despite its nice feature set, is totally the wrong technology for a file format that often changes. I'm thinking about ditching my XML Schema altogether and using the same "best effort" low-level approach in Java that I'm doing in iOS.

So is my negative assessment of XML Schemas correct? Or is there something I've missed? Perhaps I need to rethink the server interface? Or maybe I shouldn't have been using XML in the first place? I'm open to all suggestions. Thanks for reading!

share|improve this question
add comment

1 Answer

up vote 6 down vote accepted

I think you are really asking a broader question, "is having a strict definition of a file format a good thing for a rapidly evolving project".

To answer your immediate question, though: yes, they are. The XML schema gives you a strict definition of the format, answer a lot of questions about validity, provides great documentation, and allows you to confidently know that a specific version of the document has a specific form.

They are not the be-all and end-all, though: they define structure, not semantics, so you can still change the "meaning" of a tag between version of the document without having to change the schema. That causes just as much trouble.

To answer the question I think you are asking: yes, the XML schema is a good thing.

It is forcing you to address a painful fact, which is that your data exchange is constantly changing versions, and that means you have to adapt your system to account for that.

If you only had the IOS model, where you take a "rough guess" at what this version means, you open the door to all sorts of trouble in the long run. For example, it becomes trivial for someone to assume that "element foo being present means version 1.2, so tag bar means ...".

That is great, right until version 2.0 adds back the foo element, with a different meaning, and tag bar isn't even there. Welcome to "inconsistent behaviour" city.

If you use XML without schemas, or JSON, or something else that doesn't impose that cost on you then only a tiny bit of the problem goes away. You still have to deal with all four versions of the input, but you have less tools to help you out.

You should, in my opinion, generally prefer to make the pain of changes proportional to their real, long term cost. Changing the data exchange format has a high long term cost - you have to deal with compatibility, with data upgrades, and that sort of thing.

If that costs little you will be tempted to do it a lot, and then will pay the maintenance cost tomorrow. If it costs more now, you might think harder - can you do more than one thing in this change? Can you get away without it? Can you do it smarter?

In short, I think your real problem is that your file format changes often, not that you used (or didn't use) XML schemas.

(Also, are your users really happy if the server or client randomly drops content that is represented in the newer version? I would be surprised if that remains true forever - another of those long term costs you need to recognise somewhere...)

share|improve this answer
    
Thanks for the insight! You're right that the "best effort" approach might seem like a good thing now but could come back to bite me as the project matures. So how about this: I keep the XML Schema, and when it changes, I perform a bulk upgrade of all blobs on the server. Clients would no longer try to upgrade the blob; they'd just tell the users they need a newer version of the client. I'd still need to write an XSLT, but the "XSLT chaining" problem would go away. What do you think? –  vocaro Feb 11 '12 at 6:29
    
Ultimately, whatever you decide to do is going to be a trade-off of some sort. I would work to reduce the number of versions you had by encouraging adoption, but keep in mind that the XSLT chain is a very testable and solid technical solution, and can be reasonably trusted without forcing frequent upgrades on your users. Ultimately, though, if you work on solving the "data exchange format changes frequently" issue, you are working on the right problem. –  Daniel Pittman Feb 13 '12 at 2:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.