Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data model, it looked like a tree structure similar to a relational model. Here is the dilemma for availability and partition tolerance we need NoSQL, but our data model is relational.

The intended file system must be able to handle filtered search based on date, name etc. as fast as possible. So what path should i take? Stick to relational with some indexing mechanism backed by 3 rd tools like Apache Solr or dig deeper into NoSQL and find a suitable model and database satisfying the model?

P.S: Currently from NoSQL Cassandra or MongoDB are choices proposed by my colleagues.

share|improve this question
    
Is a real file system out of the question? You can index it anywhere --just make sure the thing remains in sync with the index. –  Monster Truck Dec 4 '12 at 12:45
    
CAP theorem says you can't have Consistency, Availability and Partitioning Tolerance all at the same time. –  Emmad Kareem Dec 4 '12 at 12:50
    
@Emmand Yes i know we have also a heavy discussion what to pick two out of three. –  fga Dec 4 '12 at 12:57
    
@Monster well the real file system we use is an object file system, so by virtual i meant to represent objects as files. It is not out of question, but that would mean to change the architecture all together :) –  fga Dec 4 '12 at 13:01

1 Answer 1

up vote 3 down vote accepted

As I could see from your questions is, that you first should think about what you need and what is NoSql made for.

NoSQL - made for huge amount of distributed data. Scales well from performance an amount of data. and depending in the Type of NoSQL system you can just put and get objects very fast, or also do long running jobs on the distributed data, it's not made for fast search/queries.

Search - it's just made for searching references to data fast. Depending on system it scales well with data and performance. its not made for query huge amount of data nor data relations.

RDBMS - it's made to store data which have relations to each other. System scales depending on system itself and based on your data design. means even the fastest rdbms solutions could perform out with wrong data/query design. It's not made for fast search on huge amount of data. And scaling is depending of rdmbs product, it's not a feature by default.

So if you want to search data from a huge amount, choose the system that is made for it: Search Engine.

If you want to store huge amount of data, where data needs to be distributed (because of amount) and performance of getting data should be independent from amount of data: choose a NoSQL System.

If you don't have that much data,but data needs to have relations to each other, then choose a RDBMS, and think well of you data design.

If you need a Search which stored the index distributed, combine a search product with distributed storage (filesystem).

If you need a distributed filesystem, have a look at Apache Hadoop.

If you need a NoSQL System like Google Big Table, which is somehow compareable to a RDBMS, have a look at Apache Hbase: hbase.apache.org or Hypertable.

Cassandra is more like Amazon Dynamo. It's distributed, but more far away from rdbms.
and MongoDB, is more close to LotusNotes. It's a good storage for Documents/Objects.

And MAIN POINT to solve the problem, think solution dependent, means: do NOT think like RDBMS when using a NoSQL System, you need to think in that specific NoSQL System (they are all very different from each other).

share|improve this answer
    
Thank you for insightful answer.A good summary of cases and corresponding models. –  fga Dec 4 '12 at 14:04
    
Could you be a bit more precise on what you mean by "huge amount of data"? It would make it clearer. –  Klaim Dec 4 '12 at 19:47
    
huge amount of data means for me: from size more then 500 GB, from amount of entries more then 100.000.000, they are related to each other, so what comes first. and mostly it depends how this values grows. (so in time) 500 GB in 5 years, is not that much, means with that grow rate you may have time to scale / change / optimize your RDBMS, 500 GB in one year, it is huge, you should now better go for a system that scales fast with this amount. so short, huge amount mostly means: system increases more in shorter times (with bad RDBMS Design you will get bad results with much smaler numbers) –  fmt.Println.MKO Dec 4 '12 at 21:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.