Sharding and Cloud computing

Question

I've read that Facebook uses this technique DB sharding to manage its data volume and that the idea of cloud computing is elastic resources. So I'm wondering does the cloud instance take care of sharding for you automatically? Or do you still have to do that part manually? And if you do have to do it manually - then the DB doesn't seem very elastic to me.

Answer 1

You sure can have automatic sharding - try out ScaleBase (disclosure - I work there). You can install it on any cloud environment (or physical environment). BTW - Sharding is not something that any RDBMS provides - Oracle and MS-SQL solutions are not sharding - and at least Oracle's RAC solution doesn't really scale, nor does it run on cloud environments. MySQL has MySQL cluster - which does scale, but of course it's not free...

Answer 2

Sharding in practice is complicated because its rarely possible to automatically (and thus elastically in the cloud) partition your DB to load balance queries effectively. In practice you always land up with hot zones and one of the few ways around is to architect the db schema anticipating such hot zones (which is something cloud elasticity won't give you).

The other problem with sharding is the loss of availability of a shard on failure of the node carrying the shard. Typically, you have to give up either strict consistency or availability when you are sharding. You may want to look at Brewer's CAP theorem which points out the trade-offs you have to make when building a distributed system. In practice, distributed systems tend to get complicated with intertwined systems with different availability and consistency properties.

Bottomline... there's little automatic here. Proper design of distributed components can get you very far.

Answer 3

This feature is not of the cloud instance but of the DBMS. Oracle & SQL Server do it by free MySQL uses plugins.

You can use this on any kind of app not just cloud based ones.