Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Instead of a database I just serialize my data to JSON, saving and loading it to disk when necessary. All the data management is made on the program itself, which is faster AND easier than using SQL queries. For that reason I have never understood why databases are necessary at all.

Why should one use a database instead of just saving the data to disk?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

50  
If managing the relationships of your data in your application is actually faster than doing it in a database (which I find extremely hard to believe) then you need to read up on SQL and database normalization. What you are experiencing is most probably the side-effect of a horribly designed database. –  Yannis Rizos Mar 14 '13 at 3:09
59  
You don't need a database in the scenario you are describing because your data set is trivial. Databases are meant for more complex data sets, if all you do is read and show a list, your approach works. –  Yannis Rizos Mar 14 '13 at 3:24
14  
What race conditions could you encounter, and are you ready for that? Will you want to to scale past a single webserver? What is your backup plan if your server fails? Your answer to all of these questions is likely to be better if you have a database than if you don't. Also if you ever went over the hump of learning how to use databases, my guess is that you'd find your "easier than using SQL queries" should be amended to "easier than using SQL queries if you don't understand SQL." –  btilly Mar 14 '13 at 5:21
31  
Database stores data to disk anyway. It's just the end result of a natural evolution of systems for storing structured data to file. Chances are if you set out to use files to store your structured data you are going to find yourself reinventing features that have already been developed in databases. So why not just use a database from the start? –  Benedict Mar 14 '13 at 10:00
10  
Depending on how your project evolves, you may find yourself having to deal with things like concurrent access and rollbacks. They sound trivial, but aren't. By the time you get done solving them, you will find you have basically written a database. Do you really want to be in the database business, or another business? –  jwernerny Mar 14 '13 at 12:35

13 Answers 13

  1. You can query data in a database (ask it questions).
  2. You can look up data from a database relatively rapidly.
  3. You can relate data from two different tables together using JOINs.
  4. You can create meaningful reports from data in a database.
  5. Your data has a built-in structure to it.
  6. Information of a given type is always stored only once.
  7. Databases are ACID.
  8. Databases are fault-tolerant.
  9. Databases can handle very large data sets.
  10. Databases are concurrent; multiple users can use them at the same time without corrupting the data.
  11. Databases scale well.

In short, you benefit from a wide range of well-known, proven technologies developed over many years by a wide variety of very smart people.

If you're worried that a database is overkill, check out SQLite.

share|improve this answer
17  
6. Normalization, 7. See the link, 8. Read up on fault-tolerance. Oh, and before you get sucked up into the NoSQL craze, learn about SQL databases; get to know them on their own terms. You will understand. If you're just talking about simple configuration data, JSON may be all you need. But there are many other types of data out there besides program settings. –  Robert Harvey Mar 14 '13 at 3:04
21  
As far as it not being safe to have two programs editing the data at once, well, that's partly why databases exist. If you ever have this need (and some or all of the other needs I mentioned), you're going to be very glad that you don't have to re-invent all this. –  Robert Harvey Mar 14 '13 at 3:07
20  
@Dokkat It's not necessary, nothing is. If your approach works for you, by all means go for it. I should mention however that most half decent rdbms support memory based storages, you can load everything you need in memory when your app wakes up (as you already do), and query them as you would a typical database (keeping all the benefits Robert mentioned). –  Yannis Rizos Mar 14 '13 at 3:13
23  
To put it another way, sometimes you need a tent, but sometimes you need a house, and building a house is a whole different ball game than pitching a tent. –  Robert Harvey Mar 14 '13 at 3:13
38  
@Dokkat when people are referring to crashes, they mean stuff like... your CPU blew up halfway through writing your "database" file. What happens now? Most likely your file is corrupt / unreadable (at least, it may no longer conform to your own format), and you need to restore form a backup (while most "real" DBs would only lose the last transaction). Of course, you can write code to make it handle this. Then you can write code for all the other stuff. And then you realise you've spent 6 months writing a DB, which you could have used from the start, for very little effort. –  Daniel B Mar 14 '13 at 6:44

Whilst I agree with everything Robert said, he didn't tell you when you should use a database as opposed to just saving the data to disk.

So take this in addition to what Robert said about scalability, reliability, fault tolerance, etc.

For when to use a RDBMS, here are some points to consider:

  • You have relational data, i.e. you have a customer who purchases your products and those products have a supplier and manufacturer
  • You have large amounts of data and you need to be able to locate relevant information quickly
  • You need to start worrying about the previous issues identified: scalability, reliability, ACID compliance
  • You need to use reporting or intelligence tools to work out business problems

As for when to use a NoSQL

  • You have lots of data that needs to be stored which is unstructured
  • Scalability and speed needs
  • You generally don't need to define your schema up front, so if you have changing requirements this might be a good point

Finally, when to use files

  • You have unstructured data in reasonable amounts that the file system can handle
  • You don't care about structure, relationships
  • You don't care about scalability or reliability (although these can be done, depending on the file system)
  • You don't want or can't deal with the overhead a database will add
  • You are dealing with structured binary data that belongs in the file system, for example: images, PDFs, documents, etc.
share|improve this answer
3  
Great advices, very helpful, thanks. –  Dokkat Mar 14 '13 at 3:24
9  
+1, I think its important that you pointed out there's times when files actually are suitable for storage. –  GrandmasterB Mar 14 '13 at 4:13
11  
You could add another example to your third list: When data actually is files, e.g. uploaded images, pdf documents and such. It may seem obvious but I did see cases where images were stored in a database blob for no good reason whatsoever. –  Goran Jovic Mar 14 '13 at 9:46
3  
Well, there was never any explicit mention made to it being a web app but I did infer it from the JSON comment. However, sometimes something will only be used by a few people and you can justify the scope of the application to not worry about scalability and reliability. By this I mean, not worrying about things such as clustering and redundancy. –  Sam Mar 14 '13 at 11:16
5  
@GoranJovic it sometimes makes sense. Store 10,000+ images in a directory and some filesystems will grind to a halt - a DB might be easier than a manual sub-directory partition scheme. –  Martin Beckett Mar 14 '13 at 19:48

When you have simple data, like a list of things as you describe in the comments of your question, then an SQL database won't give you much. A lot of people still use them, because they know their data can get more complicated over time, and there are a lot of libraries that make working with database trivial.

But even with a simple list that you load, hold in memory, then write when needed, can suffer from a number of problems:

Abnormal program termination can lose data, or while writing data to disk something goes wrong, and you can end up killing the whole file. You can roll your own mechanisms to handle this, but databases handle this for you using battle-proven techniques.

If your data starts growing too big and updating too often, serializing all your data and saving is going to be a big resource hog and slow everything down. You'd have to start working out how to partition things, so it won't be so expensive. Databases are optimized to save just the things that change to disk in a fault tolerant way. Also they are designed, so you can quickly just load the little bits of data you need at any given time.

Also, you don't have to use SQL databases. You can use NoSQL "databases" which many do, just use JSON to store the data. But it is done in a fault-tolerant way, and in a way where the data can intelligent split up, queried, and intelligently split across multiple computers.

Also, some people mix things up. They might use a NoSQL data store like Redis for storing login information. Then use relational databases to store more complex data where they need to do more interesting queries.

share|improve this answer

One thing that no one seems to have mentioned is indexing of records. Your approach is fine at the moment, and I assume that you have a very small data set and very few people accessing it.

As you get more complex, you're actually creating a database. Whatever you want to call it, a database is just a set of records stored to disk. Whether you're creating the file, or MySQL, SQLite or whatever is creating the file(s), they're both databases.

What you're missing is the complex functionality that has been built into the database systems to make them easier to use.

The main thing that springs to mind is indexing. OK, so you can store 10 or 20 or even 100 or 1000 records in a serialised array, or a JSON string and pull it out of your file and iterate it relatively quickly.

Now, imagine you have 10,000, 100,000, or even 1,000,000 records. When someone tries to log in you're going to have to open a file which is now several hundred megabytes large, load it into memory in your program, pull out a similarly sized array of information and then iterate 100s of thousands of records just to find the one record you want to access.

A proper database will allow you to set up indexes on certain fields in records allowing you to query the database and receive a response very quickly, even with huge data sets. Combine that with something like Memcached, or even a home-brew caching system (for example, store the results of a search in a separate table for 10 minutes and load those results in case someone else searches for the same thing soon afterwards), and you'll have blazing fast queries, something you won't get with such a large dataset when you're manually reading/writing to files.

Another thing loosely related to indexing is transfer of information. As I said above, when you've got files of hundreds or thousands of megabytes you're having to load all of that information into memory, iterate it manually (probably on the same thread) and then manipulate your data.

With a database system it will run on its own thread(s), or even on its own server. All that is transmitted between your program and the database server is an SQL query and all that is transmitted back is the data you want to access. You're not loading the whole dataset into memory - all you're sending and receiving is a tiny fraction of your total data set.

share|improve this answer
1  
1. Please don't ever load all of your user information into client side code! (I'm sure it was only an example) 2. Loading that in the first place from a file 100s of MB large will take a while. 3. You're example is correct, however it assume that you're only ever going to search by username. What happens if you want to store more data about a user? e.g. Age. Now you want to search for all users who are between the ages of 20-30. Or even simpler, find a user by address when your json looks like this: {login:{pass:pass,add1:"123 sasd",city:"Wherever"}}. –  Thomas Clayson Mar 14 '13 at 9:33
1  
Your last point is potentially correct, but then I could be working from old data - specifically, if I open your program, load the current database then 5 mins later someone else logs on and edits something, my database is now a later version until I quit the program and start it again. If I then edit my database and save it again I'll overwrite any changes that the other user made. When you've got a user's database this could be anything from just changing your password. If two users change their password during each others sessions then one user will have their change reversed. –  Thomas Clayson Mar 14 '13 at 9:37
3  
I've learned a lot after searching some things about indexing. It was really enlightening. Databases make a little more sense now. There are still some things I don't understand, but that's a big progress. Thanks for that answer! –  Dokkat Mar 14 '13 at 10:06
2  
About indices, no, database doesn't index everything automatically. Only few things are automatically indexed while the rest require explicit "please make this indexed". And indices reduce search to logarithmic time, O(log(n)) which is slightly slower than constant. –  Emperor Orionii Mar 14 '13 at 12:46
1  
Worrying about the difference between a hash-based and b-tree based implementation is a premature optimization. If data is in the index, it'll still be a dozen times faster than reading it off of disk. –  SilverbackNet May 18 '13 at 19:16

As soon as the data are updated concurrently, the approach using a database (it could well be an in memory database) will likely be more correct and more performant, while at the same time your code remains easy, because you simply don't have to worry about concurrent updates, transactions, caching, asynchronous I/O and all that.

share|improve this answer
    
Concurrent modification within a process will be more efficient using in-process locks rather than IPC to a database daemon that acquires a bunch of locks. But you're presumably talking about multiple processes modifying the data. –  dhasenan Mar 15 '13 at 16:18
    
@dhasenan - This is another advantage of good database systems. You get the concurrency, and it works in all cases: Multi-threaded, multi-process, multiple clients on different servers, or any combination thereof. Your well though out multi-threaded program may be "more efficient" in certain cases, yet it simply won't scale. –  Ingo Mar 15 '13 at 21:16

A database is needed if you have multiple processes (users/servers) modifying the data. Then the database serves to prevent them from overwriting each others changes.

You also need a database when your data is larger than memory. Nowadays with the memory we have available, this does indeed makes the use of databases in many applications obsolete.

Your approach is definitely better than the nonsense of "in-memory databases". Which are essentially your approach, but with a lot of overhead added.

share|improve this answer
    
To be honest I love this answer and would like that to be true, but I'm not sure that's the case. For example, some users (and you) raised a concern about memory. Of course, if I'm storing GBs worth of data I can't keep it all on memory. But what if I'm certain the data would never be that large, should I just use memory? Well, there are other things too. For example, I've learned about CouchDB's incremental views. That is certainly something that, differently from indexing, would NOT be trivial to implement yourself, and is certainly a huge speedup when you are using a view model, –  Dokkat Mar 14 '13 at 13:37
    
which I guess I am. For example, when I transform data from "player list" to "ranking", this is nothing but a map reduce operation. When creating a game or an interactive site, pretty much everything you present is a mapReduce operation from your core data! So having that kind of optimization could be really desirable. Well, I have no idea if any of what I'm talking proceeds, but that makes sense. Learning a lot today, and I'm really liking the NoSQL concepts. Thanks for the answer (: –  Dokkat Mar 14 '13 at 13:42

I see a lot of answers focus on the problem of concurrency and reliability. Databases provide other benefits beside concurrency, reliability and performance. They allow to not to bother how bytes and chars are represented in the memory. In other words, databases allow programmer to focus himself on "what" rather than "how".

One of the answers mentions queries. "Asking SQL database a question" scales well with the complexity of a question. As code evolves during the development simple queries such as "fetch all" can easily expand to "fetch all where property1 equals this value and then sort by property2" without making it programmer's concern to optimize data structure for such query. Performance of most queries can be speed up by making index for a certain property.

Other benefit are relations. With queries it's cleaner to cross-reference data from different data sets then having nested loops. For example searching for all forum posts from users that have less then 3 posts in a system where users and posts are different data sets (or DB tables or JSON objects) can be done with a single query without sacrificing readability.

All in all, SQL databases are better then plain arrays if data volume can be big (let's say more than 1000 objects), data access in non-trivial and different parts of code access to different subset of data.

share|improve this answer
    
I'm a little leery about the idea that you can just ignore how stuff is represented. While you can ignore this, if you do, and esp. if you do write a slightly more complex query, it's exceedingly likely your application can no longer scale. "Adding an index" is not always possible - you've got writes to contend with, and it simply doesn't help that much with queries whose complexity spans multiple tables. When indexes are necessary that implies you've lost the benefit of interactive queryability since only specifically structured queries are answerable in reasonable time. –  Eamon Nerbonne May 19 '13 at 7:42

You should always ask yourself if a particular application needs an RDBMS. Too many applications are built with a design process that automatically assumes all the required tools and frameworks at the beginning. Relational databases are so common and many developers have worked on similar applications as before, that they're automatically included before the project starts. Many projects can get away with this, so don't judge too harshly.

You started your project without one, and it works. It was easier for you to get this up and running without waiting until you SQL. There is nothing wrong with that.

As this project expands and the requirements become more complicated, some things are going to become difficult to build. Until you research and test alternate methods, how do you know which is better? You can ask on Programmers and weed through the flames and 'it depends' to answer this question. Once you learn it, you can consider how many lines of code you're willing to write in your language to handle some of the benefits of a database. At some point, you're reinventing the wheel.

Easy is often relative. There are some frameworks that can build a web page and connect a form to a database table without requiring the user to write any code. I guess if you struggle with the mouse, this could be a problem. Everyone knows, this isn't scalable or flexible because god forbid you've tightly coupled everything to the GUI. A non-programmer just built a prototype; lots of YAGNI to be found here.

If you'd rather learn an ORM manipulated by your language of choice instead of learning SQL, go for it, but try to install, create a table and pull some data out of a popular database with SQL (Select * From ; isn't mindblowing stuff). It's easy to do. That's why someone created them in the first place. It doesn't seem like such a huge investment in order to make an informed decision. You could probably do a performance test as well.

share|improve this answer
    
Just to note, I've actually used mysql for years when I hosted an "otserv". Guess what? All it brought was problems. People could "clone" items using a dirty trick after they realized their characters was saved when they logged out but not when the server crashed. This is a serious problem for otservs. And the otserv community is HUGE. That wouldn't happen if they just stored data on memory and serialized it periodically. So I modified the source by myself, those long C++ files and started saving to mysql periodically, instead of when characters logged out. Guess what? It was SLOW! –  Dokkat Mar 15 '13 at 1:18
    
Mysql simply couldn't handle fully saving state each 2 minutes or so. It was pretty clear when the saving happened - the whole server "lagged" for a second. Now I'd really appreciate if people posting here had an answer for that one! –  Dokkat Mar 15 '13 at 1:20
1  
Don't judge RDBMSs by what happened with a single application that was probably coded poorly. Especially when the modifications to support a database were made by someone with no database experience. –  alroc Mar 15 '13 at 15:37
1  
@Dokkat, I hope that nobody kicks the power cord in between depositing funds in your bank account and "periodically" writing the account balance to disk. You've described a guaranteed data loss architecture. That is fine for some applications, but most database applications give users the power to choose. You can run a single database node with backups and risk some data loss or use replication to eliminate data loss if a single node fails. –  mikerobi May 18 '13 at 22:45
    
@Dokkat so you don;t use MySql or any other full-featured "server" style DB. You use Sqlite (or similar) and it will persist to disk every time, whilst giving you a DB embedded in your app (so no need for a separate install) and still giving you sql access, transactional integrity and disk persistence. –  gbjbaanb May 19 '13 at 12:58

You need a databse to store / retrieve QAs like the ones we are posting here ! A simple file is unable to organize data related to different topics.

share|improve this answer
2  
No, "topics" could be folders, and the "posts" on the site could be files. It's definitely possible to run a site like this off a filesystem. It is not efficient: slow and complicated to develop, run queries, insert new data, etc. –  Chris S Mar 15 '13 at 17:12
    
slow + complicated = unable ? –  joe Mar 15 '13 at 17:13
    
Slow and complicated to build != slow and complicated to function –  joe Mar 15 '13 at 17:22
1  
@joe, it's really not true that a file (maybe not a "simple" file, but what does that mean?) can't be used to organize data related to different topics. You could use JSON, as Dokkat suggests, or XML, or mixed-record files like we used to do back in the pre-XML days, or whatever file format you can dream up. I wouldn't recommend any of these approaches for most scenarios, but that doesn't mean they can't be done. –  John M Gant Mar 15 '13 at 17:30
    
@John M Gant: totally agree with you, databases cannot replace single (since you do not like simple) files, and vice versa, for the only reason that a car cannot replace a bicycle. i speak 3 "human" languages, and my choice of words and vocabulary is the reason why i was misunderstood...i guess –  joe Mar 15 '13 at 17:38

Saving the data to disk IS writing it to a database, especially if you put each object in its own file with the name of the file being the key to the record. And to minimize lookup times for reading the file, create subdirectories based on the first few characters of the key.

For instance key=ghostwriter would go in g/ho/stwriter.json or g/h/o/stwriter.json or g/ho/ghostwriter.json or g/h/o/ghostwriter.json. Choose your naming scheme based on the distribution of your keys. If they are sequence numbers then 5/4/3/12345.json is better than the other way around.

That is a database and if it does all that you need, then do it that way. Nowadays that would be called a NoSQL database like GDBM, or Berkeley db. So many choices. First figure out what you need, then build an interface library to deal with the details, perhaps a get/set interface like memcached or a CRUD interface, and then you will be able to swap libraries if you need to change the database format for one with different characteristics.

Note that some SQL databases like PostgreSQL and Apache Derby DB, will allow you to do SQL queries on top of many NoSQL formats including your own homegrown databases. Not sure about MyBatis but it may be similar.

Avoid NoSQL hype. Read about the features, test the performance and capability and then choose based upon how well it matches your application needs.

http://www.hdfgroup.org/HDF5/ is yet another interesting and widely used datastore format that people do not often consider.

share|improve this answer

File systems are a type of database. Maybe not a RDBMS like everyone else is talking about, but certainly a DB in the strictest sense. You're provide keys (file name) to look-up data (file contents), which has abstracted storage and an API by which your program communicates.

So, you are using a Database. The other posts can argue about the virtues of different types of database...

share|improve this answer
1  
database and storage can't really be used interchangeably. A database is a type of storage, but a file systems is certainly not a type of database –  Gaz_Edge Mar 15 '13 at 15:57
1  
"storage" is where bits and bytes are held. A database does not necessarily use files on a file system. A file system is most definitely a type of database in the strictest sense of the term. –  Chris S Mar 15 '13 at 15:59
4  
For someone who is arguing that there's no use in databases when they're alternative is to use a database; yes. It seems helpful to explain to them that their argument is based on a preconceived notion that is wrong. Once they have a better understanding of their initial situation we can help them move forward with more complete understanding of available technologies. File systems are hierarchical databases, there's good reasons relation and object database systems have supplanted them as faster, better organized, and more efficient data storage/retrieval. –  Chris S Mar 15 '13 at 17:10
1  
@Gaz_Edge The data is already in an inefficient "database" of sorts by being stored in a bunch of files whose structure and content are both managed by the OP's application. Trying to get the OP to understand and accept that is a useful first step to getting them to understand the use case for a "real" database system; once they understand that a "database" of some kind is happening anyway, it's easier to start talking about where a properly structured and managed service is more efficient than letting the app do its own thing. I'd suggest this answer does help, very much so. –  RobM Mar 17 '13 at 10:29

A file system fits the description of a NoSQL database, so I'd say you should definitely consider using that when deciding on how to store your data and not just dismiss it off hand in favor of RDBMS, like some answers seems to suggest here.

One issue with file systems (and NoSQL in general) is handling relationships between data. If that is not major blocker here, then I'd say skip the RDBMS for now. Also remember the positive sides of using a file system as storage:

  • Zero administration
  • Low complexity, easy to set up
  • Works with any operating system, language, platform, libraries etc
  • Only configuration setting is the directory
  • Trivial to test
  • Trivial to examine with existing tools, backup, modify etc
  • Good performance characteristics and well tuned by the operating system
  • Easy for any developer to understand
  • No dependencies, no extra drivers
  • Security model is trivial to understand and is a base part of operating system
  • Data is not externally accessible

(source)

share|improve this answer

TLDR

It sounds like you made an essentially valid, short term data-store technical decision for your application - you chose to write a custom data store management tool.

You're sitting on a continuum, with options to move in either direction.

In the long term, you'll likely (almost, but not 100% certainly) find yourself running into trouble, and may be better off to change to using existing data store solutions. There are specific, very common, predictable, performance problems you will be forced to deal with, and you're better off using existing tools instead of rolling your own.


It sounds like you've written a (small) custom-purpose database, built into and directly used by your application. I assume you're relying on an OS and file system to manage the actual disk writing and reading, and treating the combination as a data-store.

When to do what you did

You're sitting at a sweet-spot for data storage. An OS and file system data store is incredibly convenient, accessible, and cross-platform portable. The combination has been around for so long, that you're certain to be supported, and have your application run, on almost any standard deployment configuration.

It's also an easy combination to write code for - the API is fairly straight-forward and basic, and it takes relatively few lines of code to get it working.

Generally, it's ideal to do what you've done when:

  • Prototyping new ideas
  • Building applications which are highly unlikely to need to scale, performance wise
  • Constrained by unusual circumstances, such as lack of resources for installing a database

Alternatives

You're on a continuum of options, and there are two 'directions' you can go from here, what I think of as 'down' and 'up':

Down

This is the least likely option to apply, but it's here for completeness sake:

You can, if you want, go down, that is, bypass the OS and filesystem altogether and really write and read directly from disk. This choice is usually relevant only in cases where extreme efficiency is required - think, for example, of a minimal/tiny MP3 player device, without enough RAM for a fully functional OS, or of something like the Wayback Machine, which requires incredibly efficient mass data write operations (most data stores trade off slower writes for faster reads, since that's the overwhelmingly more common use case for almost all applications).

Up

There are several sub-categories here - these aren't exactly exclusive, though. Some tools span both, providing some functionality in each, some can completely switch from working in one mode to working in the other, and some can be layered on top of each other, providing different functionality to different parts of your application.

More powerful data stores

You may find yourself needing to store higher and higher volumes of data, while still relying on your own application for managing the data manipulation complexity. A whole range of key-value stores are available to you, with varying extents of support for related functions. NoSQL tools fall into this category, as well as others.

This is the obvious path to scale up on when the following describe your application:

  • It is unusually heavy read reliant
  • You're OK with trading off higher performance for lower (short term) consistency guarantees (many offer "eventual consistency").
  • Is "directly" managing most of the data manipulation and lack of consistency (in practice, you'll probably end up using a third party tool at first, though eventually you'll bring this into your application or into a custom written intermediate layer).
  • You're looking to massively scale the amount of data you're storing and/or your ability to search through it, with "relatively simple" data manipulation requirements.

There is some wiggle room here - you can force better read consistency, for slower reads. Various tools and options provide data manipulation apis, indexing and other options, which may be more or less suited for easily writing your specific application. So if the above points almost completely describe your application, you might be "close enough" to work with a more powerful data store solution.

Well-known examples: CouchDB, MongoDB, Redis, cloud storage solutions like Microsoft's Azure, Google App Data Store and Amazon's ECE.

More complex data manipulation engines

The "SQL" family of data storage application, as well as a range of others, are better described as data manipulation tools, than pure storage engines. They provide a wide range of additional functionality, beyond storage of data, and often beyond what's available in the key-value store side of things. You'll want to take this path when:

  • You absolutely have to have read consistency, even if it means you'll take a performance hit.
  • You're looking to efficiently perform highly complex data manipulation - think of very complex JOIN and UPDATE operations, data cubes and slicing, etc...
  • You're OK with trading off rigidity for performance (think forced, fixed data storage formats, such as tables, which cannot easily and/or efficiently be altered).
  • You have the resources to deal with an often times more complex set of tools and interfaces.

This is the more "traditional" way of thinking of a database or data store, and has been around for much longer - so there is a lot that's available here, and there's often a lot of complexity to deal with. It's possible, though it takes some expertise and knowledge, and build simple solutions/avoid much of the complexity - you most likely will end up using third-party tools and libraries to manage most of it for you, though.

Well known examples are MySQL, SQL Server, Oracle's Database, and DB2.

Outsource the work

There are several, modern, third-party tools and libraries, which interpose themselves between your data storage tools and your application, to help you manage the complexity.

They attempt to initially take away most or all of the work that goes into managing and manipulating data stores, and, ideally, allow you to make a smooth transition into complexity only when and if it is required. This is an active area of entrepreneurship and research, with a few recent results that are immediately accessible and useable.

Well-known examples are MVC tools (Django, Yii), Ruby on Rails, and Datomic. It is hard to be fair here as there are literally dozens of tools and libraries which act as wrappers around the APIs of various data stores.


PS: if you prefer videos to text, you might want to watch some of Rich Hickey's database related videos; he does a good job of elucidating most of the thinking that goes into choosing, designing and using a data store.

share|improve this answer

protected by maple_shaft Mar 15 '13 at 17:31

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.