Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm trying to understand better how Hadoop works, and I'm reading

The NameNode is a Single Point of Failure for the HDFS Cluster. HDFS is not currently a High Availability system. When the NameNode goes down, the file system goes offline. There is an optional SecondaryNameNode that can be hosted on a separate machine. It only creates checkpoints of the namespace by merging the edits file into the fsimage file and does not provide any real redundancy. Hadoop 0.21+ has a BackupNameNode that is part of a plan to have an HA name service, but it needs active contributions from the people who want it (i.e. you) to make it Highly Available.

from http://wiki.apache.org/hadoop/NameNode

So why is the NameNode a single point of failure? What is bad or difficult about having a complete duplicate of the NameNode running as well?

share|improve this question
1  
There are always trade-offs in almost any system. You could see CAP theorem to get the basic idea. –  Ubermensch May 4 '12 at 5:50

3 Answers 3

Why does the design of HDFS have a single name node? Simplicity. According to http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#NameNode+and+DataNodes:

The existence of a single NameNode in a cluster greatly simplifies the architecture of the system. The NameNode is the arbitrator and repository for all HDFS metadata.

You can have a secondary name node that can take over when the primary fails (see http://hadoop.apache.org/common/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailability.html) and there are design proposals for distributed name nodes but, as far as I know, none are implemented reliably at this time.

share|improve this answer

Update:

There is an option of High Availability now.

The HDFS High Availability feature addresses the SPOF problem by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby.

This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.

share|improve this answer

The NameNode keeps track of what data is stored where. If it goes down, the data still exists, but it's impossible to find. It's a single point of failure because Hadoop does not support having two of them running cooperatively. Just having a secondary NameNode running isn't enough to provide high availability because there is no way to ensure consistency when the main one goes down (the secondary NameNode may be missing recent changes).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.