About the SecondaryNameNode
The NameNode is responsible for the reliable storage and interactive lookup and modification of the metadata for HDFS. To maintain interactive speed, the filesystem metadata is stored in the NameNode’s RAM. Storing the data reliably necessitates writing it to disk as well. To ensure that these writes do not become a speed bottleneck, instead of storing the current snapshot of the filesystem every time, a list of modifications is continually appended to a log file called the EditLog. Restarting the NameNode involves replaying the EditLog to reconstruct the final system state.The SecondaryNameNode periodically compacts the EditLog into a “checkpoint;” the EditLog is then cleared. A restart of the NameNode then involves loading the most recent checkpoint and a shorter EditLog containing only events since the checkpoint. Without this compaction process, restarting the NameNode can take a very long time. Compaction ensures that restarts do not incur unnecessary downtime.
The duties of the SecondaryNameNode end there; it cannot take over the job of serving interactive requests from the NameNode. Although, in the event of the loss of the primary NameNode, an instance of the NameNode daemon could be manually started on a copy of the NameNode metadata retrieved from the SecondaryNameNode.
Why should this run on a separate machine?
- Scalability. Creating the system snapshot requires about as much memory as the NameNode itself occupies. Since the memory available to the NameNode process is a primary limit on the size of the distributed filesystem, a large-scale cluster will require most or all of the available memory for the NameNode.
- Durability. When the SecondaryNameNode creates a checkpoint, it does so in a separate copy of the filesystem metadata. Moving this process to another machine also creates a copy of the metadata file on an independent machine, increasing its durability.
Configuring the SecondaryNameNode on a remote host
An HDFS instance is started on a cluster by logging in to the NameNode machine and running $HADOOP_HOME/bin/start-dfs.sh (or start-all.sh). This script starts a local instance of the NameNode process, logs into every machine listed in the conf/slaves file and starts an instance of the DataNode process, and logs into every machine listed in the conf/masters file and starts an instance of the SecondaryNameNode process. The masters file does not govern which nodes become NameNodes or JobTrackers; those are started on the machine(s) where bin/start-dfs.sh and bin/start-mapred.sh are executed. A more accurate filename might be “secondaries,” but that’s not currently the case.- Put each machine where you intend to run a SecondaryNameNode in the conf/masters file, one per line. (Note: currently, only one SecondaryNameNode may be configured in this manner.)
- Modify the conf/hadoop-site.xml file on each of these machines to include the following property:
dfs.http.address namenode.host.address:50070 The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port.
Usually this setting could be placed in the hadoop-site.xml file used by all daemons on all nodes. In an environment such as Amazon EC2, though, where a node is known by multiple addresses (one public IP and one private IP), it is preferable to have the SecondaryNameNode connect to the NameNode over the private (unmetered bandwidth) IP address, while you connect to the public IP address for status pages. Specifying dfs.http.address as anything other than 0.0.0.0 on the NameNode will cause it to bind to only one address instead of all available ones.
In conclusion, larger deployments of HDFS will require a remote SecondaryNameNode, but doing so requires a subtle configuration tweak, to ensure that the SecondaryNameNode can communicate back to the remote NameNode.
