Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

my map-like app loads data from google map API but it also loads some extra data - in the form of XML - from my server.

When the app loads it really needs only small amount of data to cover the currently visible area but there is a chance of 2% (aka conversion rate) that the user will interact with the map and more data will be needed.

My problem is that I can not really estimate the probability that user will need certain part of the map data, therefore I can not estimate the optimal size of the data packet. Should I sent just the minimal chunk of data that user needs right now or should I send him data for all the nearby screens, or even more?

In theory, what is the trade of between sending one big chunk of data - lots of bytes (lot bandwidth) but few server requests - or sending small pieces of data whenever needed - small but frequent server requests?

The XML chunks will be cached but the bigger chunks I generate the higher the chance I will have to update it every time something changes in the data (that will happen quite frequently).

I can not quantify the sizes and probabilities yet so please treat this question as highly theoretical. I would like to hear your opinions.

share|improve this question
    
Sounds like a case for A/B testing. I know the question is about exploring the theoretical side, so I am not claiming this is an answer, but in reality I think you need to keep all your options open until you get some hard data. –  Steve Haigh Apr 14 '11 at 9:33
add comment

2 Answers

up vote 3 down vote accepted

For server performance it's somewhat dependant on whether you are bottlenecked by CPU or bandwidth, but for lack of better data you can presume the request overhead to be equivalent to 10 kB data.

For the user it's slightly different, anyone with a reasonable speed connection will mostly just want your application to be responsive, for that reason the big packet solution may be preferable.

One more thing, if size matters an XML data format is probably not the right choice.

Edit: Personally I usually end up building a custom quasi-binary data-format for such stuff, but I probably have a tendency to overdo the bit-squeezing a bit because it's fun.

In any case, the things that make the biggest impact on data sizes also happen to be the simplest. JSON is a good base, but the classical JSON style is quite XML-ish. The biggest overhead found in JSON is usually key-value datatypes, like so:

data={
    foo:356682,
    bar:"something",
    foobar:true
}

With the same structure repeated for a load of datapoint, moving from explicit keys to implicit keys is the first step, instead of the above one would use something like:

data=[
    356682,
    "something",
    true
]

The data now has to be in the same explicit order to make unambiguous data extraction possible, but the size advantage is obvious.

Further options include using internally delimited strings rather than arrays of strings, base 64 encoding numbers and in general being creative about choosing the shortest of the many ways any piece of data can be represented.

data="BXFK|something|1"

As for zipping, it ain't magic, especially if using an algorithm that is fast enough to run live on a busy server. Normal deflate doesn't get the broader picture, it will notice that {foo:, ,bar:" and ",foobar: are common patterns and reserve relatively short identifiers for these, but it doesn't get around repeating these identifiers again and again. Further, the more overhead one removes, the better the zip algorithm will handle redundancy in actual data. Don't be surprised if you have made something that seems pretty compact and a simple zip then cut the size in half.

share|improve this answer
    
Thanks for the advice. What would you recommend instead of XML? JSON? Isn't the XML zipped on transfer (like any other text file)? (+1) –  daniel.sedlacek Apr 14 '11 at 12:59
    
I'd let the client make that call -- both get you to the same place, both compress reasonably well, arguable which one is more trouble to deal with on the far end. You could very easily provide both as most modern service stacks handle either format with ease. –  Wyatt Barnett Apr 14 '11 at 13:09
    
@daniel.sedlacek See big edit. Not all server setups zip everything by the way, sometimes just because it's a bad setup, but sometimes because the CPU overhead from live zipping is deemed more expensive than the extra bandwidth. –  eBusiness Apr 14 '11 at 14:50
    
@daniel.sedlacek and everyone else. I'd really like to get some feedback on this answer, I'm afraid that it might be a bit too heavy or that people will not take it serious because it's not in line with what is being parroted. –  eBusiness Apr 15 '11 at 23:12
add comment

The big kicker is how are the clients connecting. Servers are easy to get fat, responsive pipes to and setup fancy caching/map reducing/precomputing to get around scale issues. Clients are alot trickier. If you are targeting businesses with big, fat pipes and loads of hardware that could push towards something more chatty. If you are worried about 3rd world mobile phones on substandard 2.5g connections, you'll have to play careful games about balancing chattyness versus payload size. But keep in mind that it is oftentimes more expensive to open up that HTTP connection than it is to transfer the payload. I would err towards the side of less chatty because of this -- bandwidth scales, number of connections don't.

share|improve this answer
    
bandwidth scales, number of connections don't - thank's, that's what I needed to hear. (+1) –  daniel.sedlacek Apr 14 '11 at 12:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.