Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Currently we are working on some storage problems for log data from various servers and communication message logs (HTTP(S), XMPP). There will be many write operations and for read operations we will use search queries with filters.

Should we stick with the classic relation database schema solution or concentrate on Cassandra with ColumnFamily based indexes?

share|improve this question
add comment

5 Answers

You can stay relational and be noSQL at the same time if you can partition data(perhaps by time) with playOrm/cassandra so that you can do "scalable JQL" like so

@NoSqlQuery(name="findJoinOnNullPartition", query="PARTITIONS t(:partId) select t FROM TABLE as t INNER JOIN t.security as s where s.securityType = :type and t.numShares = :shares")

It also supports of course the full ManyToOne, OneToMany, etc. etc. but they work slightly differently than hibernate as this is noSQL after all.

share|improve this answer
add comment

Hadoop, hbase, hive/pig is generally used for these kind of log analysis or to handle huge data produces logs & messages. This llnk can give give u more details http://www.cloudera.com/blog/2009/06/analyzing-apache-logs-with-pig/. But this has huge learning curve.

share|improve this answer
    
Thank you for the answer. Actually before storing the logs we plan to execute analysis tasks with hadoop map-reduce. However after the analysis task we need store them in database and choices are mysql or cassandra. –  fga Oct 14 '11 at 12:25
add comment

Hadoop plays nice with NoSQL stores like Cassandra. I'm doing development on a side project that uses Cassandra and Hadoop. It was a bit of a pain to get set up, as there are some extra jars needed in the Hadoop installation to support Cassandra and not much documentation about how to do it. The Thrift API is a little awkward but still manageable once you're writing map-reduce code.

I think the decision comes down to the queries you need to run and the volume of processed data you're looking to store. Complex queries on small volumes of data push more toward MySQL. Simple queries or larger volumes of data push more toward Cassandra or HBase.

share|improve this answer
add comment

If all you're going to be doing is searching the records once they're stored, then I don't think there's going to be much difference. If you need to re-process them (or a large chunk thereof) they you might see some advantage from the NoSQL approach. If you know what fields you're going to be searching on, I'd recommend compressing the data into a BLOB and storing that along with the searchable fields in a relational table, in order to cut down on your storage requirements (I don't know if that's possible with Cassandra).

share|improve this answer
add comment

If you are only going to store logs, you've strayed into one of the few places were relational databases do not make much sense.

SQL is surely not going to be the ideal medium to perform your work- no joins, not many different tables, etc. Transactions and UPDATEs probably won't be necessary.

I would choose something that let me run map-reduce jobs easily- it might well be the case that a single-threaded algorithm be all you need, but if it ever becomes too slow for your purposes, being able to throw cores at the problem will come in handy.

If you are going to be searching a lot (i.e. perform calculations on small subsets of data), indexing is going to be another defining factor- having a store which supports indexing which handles your searches well will save plenty of time.

On the other hand, depending on what you do, it might make sense to "summarize" your logs using a non-relational data store, but put in the massaged data in a RDBMS. If there's structured/relational data hidden in your logs, the ability to perform ad-hoc queries using SQL is invaluable- aggregates, window functions, etc. can be made to run pretty quickly in a decent RDBMS, perhaps even more efficiently than with map-reduce and highly-parallel algorithms; definitely, if you know your SQL, implementation is usually much quicker.

For instance, say the logs you are really interested all look like:

[timestamp] add student xxxxx
[timestamp] create class yyyy at [date-time] with professor zzzz
[timestamp] student xxxx books class yyyy
[timestamp] student xxxx cancels class yyyy

then dumping them into student, class, student_booking and performing aggregates on them makes plenty of sense.

(of course, I would contend that you are not parsing logs then...)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.