Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Hypothetical scenario: Let's say we are downloading JSON from Facebook with details of a user's friend's checkins, posts, etc... These come in as one document per friend per activity, so with 8 activities a user with 300 friends will cause our system to make 2400 requests to Facebook, downloading 2400 JSON documents.

Let's say we want to merge these 2400 documents together, sort the activities by date_created descending and then page through them in a sort of pseudo newsfeed. Please do not comment on the wisdom of recreating a facebook newsfeed in this way.

Let's also suppose that we want to re-download all of this data whenever we are notified that it has changed by Facebook. (FB has an update service you can subscribe to for users of your app). For argument sake let's assume all of the data has to be refreshed every 5 minutes, and further assume we want to be able to support 1000 simultaneous users, and that the average JSON document size is 25kb.

I am curious as to how NoSQL techniques would be better than parsing the JSON on ingestion into a relational database? To me it seems like map/reduce are just synonyms for parse/aggregate and that both approaches will require the same thing to occur. What advantages would I get from using NoSQL?

share|improve this question
add comment

2 Answers

What advantages would I get from using NoSQL?

NoSQL will scale better as the number of users grows.

Traditional RDBMS don't really scale well. All that you can do is throw bigger machines at the problem. They aren't really suited for distributed systems (cloud e.g.).

NoSQL is (under given circumstances) better at handling hierarchical structures like documents/JSON.

The key point to understand is that these storage mechanisms are key-value based and thus can retrieved data that is stored together very fast, as opposed to data that is "merely related" (what RDBMS were built for).

In your case that would mean, that you can easily retrieve all records for a certain user very fast for example. In traditional relational databases you would either have to denormalize your schema for performance or keep the schema clean but potentially suffer performance penalties caused by joins or heavy aggregations.

Look at it this way: Why is a hash map (key value store) fast? You can retrieve items from a hashmap in almost O(1) as the hash directly translates to a memory address (simplified). Looking up a binary index in contrast to that would yield O(log(n));

For your case, MongoDB or CouchDB might be good solutions, as it's already based on JSON.

In my opinion, using a NoSQL solution here is a good choice. You want to retrieve all the activities of a user as a feed. If they're properly written to your data storage, then NoSQL should, in theory, excell at this, without the need for joining anything or worrying about proper indexes. @Earlz also mentioned that you have no ACID guarantee for NoSQL databases. This makes NoSQL fast and you probably don't need ACID properties for your application. Give it a try!

Moreover, there's a good article from Martin Fowler on the subject. He's made a nice diagram that I really like:

enter image description here

Go check out his pages to read some deep thoughts about NoSQL.

share|improve this answer
Another thing which is good and bad is there is no ACID guarantee. This enables NoSQL databases to be much faster at the expense of requiring special handling of "all-or-nothing" transactions –  Earlz Feb 5 '13 at 18:09
Good point, Earlz, I will incorporate that in my answer. –  Falcon Feb 5 '13 at 18:11
Problem #1 is something nuodb is trying to solve. –  JeffO Feb 5 '13 at 18:12
How will I merge the JSON documents together, which are each an array of activities and then sort and page through the activities based on the created_date, which is just a JSON attribute all activities have in common? Then, when I refresh one of the 2400 JSON documents for the user, how will the updates to that document make it into the merged/sorted view? In RDBMS using a de-normalized table this part is trivial. –  Joshiatto Feb 5 '13 at 18:29
@Earlz no ACID != NoSQL. There are plenty of great NoSQL solutions that are fast and guarantee ACID like RavenDB and you can run a traditional RDBMS without using transactions. –  Benjamin Gruenbaum Feb 5 '13 at 20:21
show 9 more comments

First of all a NoSQL database is a database that does not use an SQL interface. What all NoSQL databases have in common is that they do not use an SQL interface. Did I just repeat myself? Yes, but there is nothing else I can say about NoSQL databases as a group. Anything else being said about NoSQL databases around the internet is either wrong for some members of the group, or likely to become so at some point in the future with the release of a new database or a feature upgrade of an existing one.

All this to say that asking if a NoSQL database is a good choice for a particular job is really not an answerable question since different NoSQL databases have hugely different characteristics.

In the scenario that you describe the biggest issue would definitely be that you are pounding Facebook with 8000 HTTP requests per second, but let's ignore that and focus on the quite common issue of having a large amount of tiny data pieces.

Data handling

All other things being equal, what is the performance difference between fetching an 8 byte string and a 16 byte string from a database? It is insignificant, and barring some obscure counterexample that is true for any database, SQL or not, the overhead of everything else that goes on in a request dwarfs the time it takes to copy 8 more bytes. If you want to shift data through a database fast then sorting it into big blocks that fit your use case is one of the most significant thing you can do, often way more important than what database software you use.

Of course there are cases where your use does not fit having big chunks of data, in some cases a caching strategy where data is kept both in the original split form and in chunks may work well, in other cases there is not a lot to do but keeping the small pieces separated.

Data manipulation

Databases are slow, that is to say, if you implement a data manipulation function in a common program, for instance taking a lot of small strings and joining them into one, and you implement similar functionality through a database request, then the database version will typically take 100 to 1000 times as long to do the operation. The exact figure of course depend on the database, some databases will not be able to do it, so you'd have to write a program that fetch all the data, perform the operation, and then write the result to the database, which is also a pretty slow method.

In general, don't do on the database what you could reasonably do to the data before it's written to the database.

What database to choose

Once you have taken all these considerations, what requirements for a database do you have left? Did you manage to come up with a structure that doesn't need any of the fancy/slow features offered by some databases? If you did then an SQL database may be like a Swiss knife with a dull blade, lots of cool features, but not particularly good at what you need it for. Some of the NoSQL databases are simply faster and better when you only need the simple features, others fit the job just as bad as an SQL database.

The big question

Despite being written last in this post it is the question you should ask before all the other questions that I mentioned. Do you actually need a database?

It is a pretty common assumption that when you handle a significant amount of data, you should use a database. But with a modern computer you can store several gigabytes of data in application memory. This gives you fast and easy access, and the good tools for manipulation are right at hand. The one thing that it does not give you is persistence, if the program crash of there is a power loss the data is lost. In a lot of cases that is however perfectly acceptable, your example has data with a life span of ~5 minutes, it doesn't need persistence, it doesn't need a database.

share|improve this answer
Here's a problem with that: Facebook doesn't support obtaining all the data we are looking to assemble together about the user's friends in a single, high performing HTTP call. For example, if I want to just show the user a list of their friend's birthdays, I still have to loop through their friend's list and get each birthday separately. For more than a dozen friends this can take significantly longer than a request, it can even take up to 5, 10, 20 minutes! Also, when the user returns to our site we want to show the last data we acquired for them immediately. –  Joshiatto Feb 5 '13 at 22:56
@Joshiatto It is definitely problematic, stupid Facebook. In any case, for your practical problem you may be best off starting out with tools you are used to and focus on making a good data structure. As for application memory versus database it doesn't have to be either or, you can decide that some data doesn't need persistence while other does, or you can degrade the database to basically just being a backup tool, storing data that also reside in application memory. –  eBusiness Feb 5 '13 at 23:19
I guess we can just use JavaScript from the browser to loop through the friends, and then display/store results as we obtain them and persist them in some sort of session state? HTML 5 local storage? –  Joshiatto Feb 5 '13 at 23:21
I'm starting to get it. Databases are often optional. In my scenario, I could potentially just store the JSON documents in a folder and name them by convention: <userID>_<type>. This document would contain the most recent data for that user of that type. I could then load and transform this data either on the server as part of the request handling, or, could even let JavaScript in the browser merge/sort/page/filter through the data. Why would I even need to use a NoSQL database or key/value store when I could just use the filenames to look up the data? Am I getting it? Thanks. –  Joshiatto Feb 6 '13 at 18:56
@Joshiatto You have certainly begun thinking in the right patterns, beware that disk access can also be pretty slow, especially if you use an old-school harddisk. Effectively the file system is just a key/value database. I'd probably keep a copy of the data in memory if possible, but it may be a viable option simply to rely on the OS keeping some of these files in memory for you. It is not something I have experience with, but it is a feature of pretty much all modern desktop and server OSs. Usually enabled by default, but if you want to rely on it you better check that the settings are good. –  eBusiness Feb 6 '13 at 20:21
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.