Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The title basically says it all. I keep hearing about Hadoop, and from the wikipedia entry, it certainly sounds interesting, but I'm curious if there are some good examples where it makes sense to pursue a solution using that technology?

Put another way, if I pick up a new toy, I like to at least know how I'm supposed to play with it!

share|improve this question

closed as unclear what you're asking by gnat, Kilian Foth, Dan Pichelman, MichaelT, Yusubov Jul 17 '13 at 18:11

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Sharing your research helps everyone. Tell us what you've tried and why it didn’t meet your needs. This demonstrates that you’ve taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask –  gnat Jul 16 '13 at 6:46
add comment

1 Answer

up vote 32 down vote accepted

Hadoop is an open-source, Java-based implementation of Google's MapReduce framework. Hadoop is designed for any application which can take advantage of massively parallel distributed-processing, particularly with clusters composed of unreliable hardware.

For example, suppose you have 10 terabytes of data, and you want to process it somehow, (suppose you need to sort it). Using a single computer, this could take a very long time. Traditionally, a high-end super-computer with exotic hardware would be required to do this in a reasonable amount of time.

Hadoop provides a framework to process data of this size using a computing cluster made from normal, commodity hardware. There are two major components to Hadoop: the file system, which is a distributed file system that splits up large files onto multiple computers, and the MapReduce framework, which is an application framework used to process large data stored on the file system.

Of course, there are many distributed computing frameworks, but what is particularly notable about Hadoop (and Google's MapReduce) is the built-in fault tolerance. It is designed to run on commodity hardware, and therefore it expects computers to be breaking frequently. The underlying file system is highly-redundant (blocks of data are replicated across multiple computers) and the MapReduce processing framework automatically handles computer failures which occur during a processing job by reassigning the processing to another computer in the cluster.

share|improve this answer
2  
Excellent synopsis! –  Brandon Tilley Apr 3 '11 at 21:03
1  
That's the kind of answer that makes SE relevant in relation to wikipedia - concise actionable answers. –  ted.strauss Feb 20 '12 at 23:14
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.