Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am doing some investigating and research into programming "big data", and am particularly interested in QuantCast (QFS) because it is written in my own favourite programming language, C++.

Looking at how this may play a part in the solution to certain "big data" problems, I know this can manage implementation of peta-byte file systems. I would like to know however how scalable it is for big-data systems that are not as hugely big.

I have seen a lot of comparisons between QFS and Hadoop, and how QFS can outperform Hadoop and so anyone who uses Hadoop may consider using Quantcast (if they just want mega performance).

Essentially is it scalable for medium to large projects? (Someone asked the same about BDD but not Quantcast)

(Also I have seen Hadoop and thus Quantcast being called a "database". I would call it a filesystem, on top of which you might build a database but wouldn't call it a database itself).

share|improve this question

1 Answer 1

up vote 2 down vote accepted

DISCLAIMER - I work at Quantcast

The "speed at scale" benefits of QFS certainly become more pronounced the larger the scale of the deployment and the more intense the usage, but that's not to say there isn't some benefit for more modest deployments. However, I would grant that if you aren't running your cluster at full capacity 24/7 (like we are at Quantcast) and your cluster is relatively modest, speed isn't your primary motivator for using QFS.

The benefit that is available at any scale is an increase of logical storage capacity for a given hardware deployment over what HDFS offers for the same level of data integrity due to QFS using Reed-Solomon error correction rather than straight duplication (as in HDFS with 3-way duplication). The QFS approach to error correction also increases the failure tolerance for any given data set from 2 nodes (with 3-way data replication) to 3. More fault tolerant and half the physical space translate in to real dollars when it comes to operating a cluster. The value of this can't be understated, as it basically multiplies the usefulness of your capital expenditure. This financial benefits are the reason why we added the Reed-Solomon feature to QFS.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.