Step by step Installation of hadoop on linux

1)First of all install java jdk 1.6

In my case ,JAVA_HOME= /usr/java/jdk1.6.0_33

2)Create a dedicated hadoop user group to separate the hadoop installation from other software applications.

$ sudo addgroup hadoop

$ sudo adduser --ingroup hadoop hduser

This will add the user hduser and the group hadoop to your local machine.

3)Configure SSH

Name the node as master and slave so it becomes convenient to distinguish the two

master's IP:192.168.0.106

Slave's IP:192.168.0.122

For Master

root@master:~$ su - hduser

hduser@master:~$ ssh-keygen -t rsa -P ""

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hduser/.ssh/id_rsa):

Created directory '/home/hduser/.ssh/id_rsa'

Your identification has been saved in /home/hduser/.ssh/id_rsa.

Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

The key fingerprint is:

f6:de:f8:ad:ff:b4:aa:06:63:57:2f:ae:f3:bf:ce:93 hduser@master

Copy the key in authorized keys of both the machines

[hduser@master ~]$ scp /home/hduser/.ssh/id_rsa.pub 192.168.0.106:$HOME/.ssh/authorized_keys

id_rsa.pub                                    100%  394     0.4KB/s   00

[hduser@master ~]$ scp /home/hduser/.ssh/id_rsa.pub 192.168.0.122:$HOME/.ssh/authorized_keys

id_rsa.pub                                    100%  394     0.4KB/s   00

For Slave

[root@slave ~]# su - hduser

[hduser@slave ~]$ ssh-keygen -t rsa -P ""

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hduser/.ssh/id_rsa):

Created Directory '/home/hduser/.ssh/id_rsa '

Your identification has been saved in /home/hduser/.ssh/id_rsa.

Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.

The key fingerprint is:

f6:de:f8:ad:ff:b4:aa:06:63:57:2f:ae:f3:bf:ce:93 hduser@slave

Copy the key to master's authorized key file

[hduser@slave ~]$ scp /home/hduser/.ssh/id_rsa.pub 192.168.0.106:$HOME/.ssh/authorized_keys2

hduser@192.168.0.106's password:

id_rsa.pub                                    100%  394     0.4KB/s   00:00  

4)In /etc/hosts of both the machines

192.168.0.106      master

192.168.0.122      slave

Caution:Avoid using localhost ,it creates poblems later.

5) You need to disable IPv6

Check the status

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

A return value of zero indicates it is enabled but it should be disabled.

We will disable it in conf/hadoop-env.sh by adding

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

6)Download Hadoop and extract its content to /usr/local/hadoop by

$ cd /usr/local
$ sudo tar xzf hadoop-1.0.3.tar.gz
$ sudo mv hadoop-1.0.3 hadoop
$ sudo chown -R hduser:hadoop hadoop

export HADOOP_HOME=/usr/local/hadoop

export JAVA_HOME= /usr/java/jdk1.6.0_33 

Make a directory /app/hadoop/tmp to store temporary data

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp 

7)In /usr/local/hadoop/conf/hadoop-env.sh

 For Masters,add or uncomment these

export JAVA_HOME=/usr/java/jdk1.6.0_33

# Extra Java CLASSPATH elements.  Optional.
 export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.92.1.jar:$HBASE_HOME/conf:$HBASE_HOME/lib/zookeeper-3.4.3.jar


# The maximum amount of heap to use, in MB. Default is 1000.
 export HADOOP_HEAPSIZE=2000
 export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENO$
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADO$
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANO$
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANC$
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBT$
# export HADOOP_TASKTRACKER_OPTS=

For Slaves,add or uncomment these 

export JAVA_HOME=/usr/java/jdk1.6.0_33/

export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.92.1.jar:$HBASE_HOME/conf:$HBASE_H$

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OP$
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SE$
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OP$
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OP$
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKE$
# export HADOOP_TASKTRACKER_OPTS=

8) In /usr/local/hadoop/conf/core-site.xml

 For master,add these properties in configuration

  hadoop.tmp.dir
  /app/hadoop/tmp
  A base for other temporary directories.


  fs.default.name
  hdfs://master:54310

For slaves,add these properties in configuration

 

  hadoop.tmp.dir
  /app/hadoop/tmp
  A base for other temporary directories.

  fs.default.name
  hdfs://master:54310
  The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.

9)In /usr/local/hadoop/conf/mapred-site.xml

For master,add

  mapred.job.tracker
  master:54311
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
 

  mapred.map.tasks
  20
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
 

  mapred.reduce.tasks
  4
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
 

  mapred.local.dir
  /app/hadoop/tmp/mapred
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
 

For slaves,add

  mapred.job.tracker
  master:54311
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  

10)In /usr/local/hadoop/conf/hdfs-site.xml

For master

  dfs.replication
  2
  Default block replication.
  The actual number of replications can be specified when the file is
created.
  The default is used if replication is not specified in create time.
 
 
            dfs.support.append
            true
       


  dfs.name.dir
  /app/hadoop/tmp/dfs/name

  dfs.data.dir
  /app/hadoop/tmp/dfs/data

  dfs.datanode.max.xcievers
  4096

  dfs.permissions
  /false

  hadoop.tmp.dir
  /app/hadoop/tmp/hadoop

  mapred.system.dir
  /hadoop/mapred/system
 true

For Slaves 

    
          dfs.support.append
          true
   
  
         dfs.datanode.max.xcievers
         4096
  
  
           dfs.replication
            2
           Default block replication.
               The actual number of replications can be specified when the file is created.
               The default is used if replication is not specified in create time.
           
 






11)The first step to starting up your Hadoop installation is formatting the Hadoop filesystem which is implemented on top of the local filesystem of your “cluster” . You need to do this the first time you set up a Hadoop cluster.

Caution:Do not format a running Hadoop filesystem as you will lose all the data currently in the cluster (in HDFS).

To format the filesystem (which simply initializes the directory specified by the dfs.name.dir variable), run the command on master only

hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format

The output will look like this:

hduser@ubuntu:/usr/local/hadoop$ bin/hadoop namenode -format
10/05/08 16:59:56 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/08 16:59:56 INFO namenode.FSNamesystem: fsOwner=hduser,hadoop
10/05/08 16:59:56 INFO namenode.FSNamesystem: supergroup=supergroup
10/05/08 16:59:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/05/08 16:59:56 INFO common.Storage: Image file of size 96 saved in 0 seconds.
10/05/08 16:59:57 INFO common.Storage: Storage directory .../hadoop-hduser/dfs/name has been successfully formatted.
10/05/08 16:59:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
************************************************************/
hduser@ubuntu:/usr/local/hadoop$

For installing hbase refer
http://biforbeginners.blogspot.in/2012/07/step-by-step-installation-of-hbase.html

Add the following line to your hbase-env.sh file:


export JAVA_HOME=/usr/lib/jvm/java-6-sun


In the same file change this line:


export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode


to look like:


export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Djava.net.preferIPv4Stack=true"


Now, modify your “regionservers” file to list all of the machines you want to host regions. Think of an Hbase region as a small chunk of the data in your database. The more regionservers you have, the more data you can reliably serve. In my cluster, the regionservers are the same nodes as all of my datanodes, and all of my tasktrackers. So, essentially, the “regionservers” file should be identical to your “slaves” file from the hadoop tutorial.
Next, modify the hbase-site.xml file. The settings in this file over-write those in hbase-default.xml, so if you want to see a list of available settings to configure, then study that file, but only make changes to your hbase-site.xml. Add the following settings to hbase-site.xml:



    hbase.rootdir
    hdfs://$master$/hbase

    hbase.cluster.distributed
    true

    hbase.zookeeper.quorum
    $slave1$,$slave2$,$slave3$

    hbase.zookeeper.property.dataDir
    /hadoop/zookeeper/data


Please remember to replace $master$ and $slaveX$ with your master and slave host names respectively. You may have read that Hbase 0.20 now requires zookeeper, but fear not, the above configuration directives allow hbase to completely manage zookeper on it’s own, you never have to mess with it. Now, it is typically recommended to always run zookeeper on dedicated zookeeper only servers. If you are running a small cluster, then this is hardly efficient, because you want as many nodes “working” as possible. While I can’t give you recommendations of the maximum cluster size you can have before requiring dedicated zk nodes, I can tell you that my 6 slave nodes run datanode, tasktracker, regionserver, and zookeeper without too much of a problem. I would imagine that if you have over 10 nodes in your cluster, then you shouldn’t have a problem dedicating a few for zookeeper. They also recommend (maybe even require) that zookeeper runs on an odd number of machines. I don’t completely understand how zookeeper works, but basically as long as you still have more than half of your “quorum” in tact, then your cluster won’t fail. In essence, if your zk quorum has 7 nodes, you can lose 3 nodes without any adverse affects, a 35 node quorum could theoretically lose 17 nodes, and still operate. I think basically zookeeper is used to keep track of the locations of regions, so your quorum will notify any clients, and fellow regionservers where to find the data they are looking for. If zk becomes overloaded, then your regionservers can time out and crash, and potentially lose data if they haven’t flushed to disk yet. So make sure you have enough horsepower for your application. In my cluster, the hbase.zookeeper.quorum directive is simply a comma separated list of all of my slave nodes, including my master. If you have an odd number of slaves (even number counting your master), then just leave the master out of the list. If you have more than ten slaves, then consider dedicating 3 of them to zookeeper if you have problems with regionservers timing out. The logs will tell you if that is the case.

mkdir -p /hadoop/zookeeper/data && echo 'X' > /hadoop/zookeeper/data/myid

It is imperative that you replace the ‘X’ with ’0′, on the first node in your quorum, ’1′ on the second, ’2′ on the third and so on. This file allows the node to identify itself in the zk quorum.
Once all that per node work is done, you can finally start your hbase instance. From the master /hadoop/hbase directory run:


bin/start-hbase.sh

Interview Questions & Answers for Hadoop MapReduce developers

What is a JobTracker in Hadoop? How many instances of JobTracker run on a Hadoop Cluster?

JobTracker is the daemon service for submitting and tracking MapReduce jobs in Hadoop. There is only One Job Tracker process run on any hadoop cluster. Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine. Each slave node is configured with job tracker node location. The JobTracker is single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. JobTracker in Hadoop performs following actions(from Hadoop Wiki:)
• Client applications submit jobs to the Job tracker.
• The JobTracker talks to the NameNode to determine the location of the data
• The JobTracker locates TaskTracker nodes with available slots at or near the data
• The JobTracker submits the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.
• A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.
• When the work is completed, the JobTracker updates its status.


• Client applications can poll the JobTracker for information.



How JobTracker schedules a task?

The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated. When the JobTracker tries to find somewhere to schedule a task within the MapReduce operations, it first looks for an empty slot on the same server that hosts the DataNode containing the data, and if not, it looks for an empty slot on a machine in the same rack.

What is a Task Tracker in Hadoop? How many instances of TaskTracker run on a Hadoop Cluster

A TaskTracker is a slave node daemon in the cluster that accepts tasks (Map, Reduce and Shuffle operations) from a JobTracker. There is only One Task Tracker process run on any hadoop slave node. Task Tracker runs on its own JVM process. Every TaskTracker is configured with a set of slots, these indicate the number of tasks that it can accept. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these task instances, capturing the output and exit codes. When the Task instances finish, successfully or not, the task tracker notifies the JobTracker. The TaskTrackers also send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated.

What is a Task instance in Hadoop? Where does it run?

Task instances are the actual MapReduce jobs which are run on each slave node. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. Each Task Instance runs on its own JVM process. There can be multiple processes of task instance running on a slave node. This is based on the number of slots configured on task tracker. By default a new task instance JVM process is spawned for a task.

How many Daemon processes run on a Hadoop system?

Hadoop is comprised of five separate daemons. Each of these daemon run in its own JVM. Following 3 Daemons run on Master nodes NameNode - This daemon stores and maintains the metadata for HDFS. Secondary NameNode - Performs housekeeping functions for the NameNode. JobTracker - Manages MapReduce jobs, distributes individual tasks to machines running the Task Tracker. Following 2 Daemons run on each Slave nodes DataNode – Stores actual HDFS data blocks. TaskTracker - Responsible for instantiating and monitoring individual Map and Reduce tasks.

What is configuration of a typical slave node on Hadoop cluster? How many JVMs run on a slave node?

• Single instance of a Task Tracker is run on each Slave node. Task tracker is run as a separate JVM process.
• Single instance of a DataNode daemon is run on each Slave node. DataNode daemon is run as a separate JVM process.
• One or Multiple instances of Task Instance is run on each slave node. Each task instance is run as a separate JVM process. The number of Task instances can be controlled by configuration. Typically a high end machine is configured to run more task instances.

How NameNode Handles data node failures?

NameNode periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode. When NameNode notices that it has not recieved a hearbeat message from a data node after a certain amount of time, the data node is marked as dead. Since blocks will be under replicated the system begins replicating the blocks that were stored on the dead datanode. The NameNode Orchestrates the replication of data blocks from one datanode to another. The replication data transfer happens directly between datanodes and the data never passes through the namenode.

Does MapReduce programming model provide a way for reducers to communicate with each other? In a MapReduce job can a reducer communicate with another reducer?

Nope, MapReduce programming model does not allow reducers to communicate with each other. Reducers run in isolation.

Can I set the number of reducers to zero?

Yes, Setting the number of reducers to zero is a valid configuration in Hadoop. When you set the reducers to zero no reducers will be executed, and the output of each mapper will be stored to a separate file on HDFS. [This is different from the condition when reducers are set to a number greater than zero and the Mappers output (intermediate data) is written to the Local file system(NOT HDFS) of each mappter slave node.]

Where is the Mapper Output (intermediate key-value data) stored ?

The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.

What are combiners? When should I use a combiner in my MapReduce Job?

Combiners are used to increase the efficiency of a MapReduce program. They are used to aggregate intermediate map output locally on individual mapper outputs. Combiners can help you reduce the amount of data that needs to be transferred across to the reducers. You can use your reducer code as a combiner if the operation performed is commutative and associative. The execution of combiner is not guaranteed, Hadoop may or may not execute a combiner. Also, if required it may execute it more then 1 times. Therefore your MapReduce jobs should not depend on the combiners execution.

What is Writable & WritableComparable interface?

• org.apache.hadoop.io.Writable is a Java interface. Any key or value type in the Hadoop Map-Reduce framework implements this interface. Implementations typically implement a static read(DataInput) method which constructs a new instance, calls readFields(DataInput) and returns the instance.
• org.apache.hadoop.io.WritableComparable is a Java interface. Any type which is to be used as a key in the Hadoop Map-Reduce framework should implement this interface. WritableComparable objects can be compared to each other using Comparators.

What is the Hadoop MapReduce API contract for a key and value Class?

• The Key must implement the org.apache.hadoop.io.WritableComparable interface.
• The value must implement the org.apache.hadoop.io.Writable interface.

What is a IdentityMapper and IdentityReducer in MapReduce ?

• org.apache.hadoop.mapred.lib.IdentityMapper Implements the identity function, mapping inputs directly to outputs. If MapReduce programmer do not set the Mapper Class using JobConf.setMapperClass then IdentityMapper.class is used as a default value.
• org.apache.hadoop.mapred.lib.IdentityReducer Performs no reduction, writing all input values directly to the output. If MapReduce programmer do not set the Reducer Class using JobConf.setReducerClass then IdentityReducer.class is used as a default value.



What is the meaning of speculative execution in Hadoop? Why is it important?

Speculative execution is a way of coping with individual Machine performance. In large clusters where hundreds or thousands of machines are involved there may be machines which are not performing as fast as others. This may result in delays in a full job due to only one machine not performaing well. To avoid this, speculative execution in hadoop can run multiple copies of same map or reduce task on different slave nodes. The results from first node to finish are used.

When the reducers are started in a MapReduce job?

In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.

If reducers do not start before all mappers finish then why does the progress on MapReduce job shows something like Map(50%) Reduce(10%)? Why reducers progress percentage is displayed when mapper is not finished yet?

Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The progress calculation also takes in account the processing of data transfer which is done by reduce process, therefore the reduce progress starts showing up as soon as any intermediate key-value pair for a mapper is available to be transferred to reducer. Though the reducer progress is updated still the programmer defined reduce method is called only after all the mappers have finished.

What is HDFS ? How it is different from traditional file systems?

HDFS, the Hadoop Distributed File System, is responsible for storing huge data on the cluster. This is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant.
• HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
• HDFS provides high throughput access to application data and is suitable for applications that have large data sets.
• HDFS is designed to support very large files. Applications that are compatible with HDFS are those that deal with large data sets. These applications write their data only once but they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS supports write-once-read-many semantics on files.

What is HDFS Block size? How is it different from traditional file system block size?

In HDFS data is split into blocks and distributed across multiple nodes in the cluster. Each block is typically 64Mb or 128Mb in size. Each block is replicated multiple times. Default is to replicate each block three times. Replicas are stored on different nodes. HDFS utilizes the local file system to store each HDFS block as a separate file. HDFS Block size can not be compared with the traditional file system block size.

What is a NameNode? How many instances of NameNode run on a Hadoop Cluster?

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself. There is only One NameNode process run on any hadoop cluster. NameNode runs on its own JVM process. In a typical production cluster its run on a separate machine. The NameNode is a Single Point of Failure for the HDFS Cluster. When the NameNode goes down, the file system goes offline. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives.

What is a DataNode? How many instances of DataNode run on a Hadoop Cluster?

A DataNode stores data in the Hadoop File System HDFS. There is only One DataNode process run on any hadoop slave node. DataNode runs on its own JVM process. On startup, a DataNode connects to the NameNode. DataNode instances can talk to each other, this is mostly during replicating data.

How the Client communicates with HDFS?

The Client communication to HDFS happens using Hadoop HDFS API. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file on HDFS. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data.

How the HDFS Blocks are replicated?

HDFS is designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are write-once and have strictly one writer at any time. The NameNode makes all decisions regarding replication of blocks. HDFS uses rack-aware replica placement policy. In default configuration there are total 3 copies of a datablock on HDFS, 2 copies are stored on datanodes on same rack and 3rd copy on a different rack. 

24.    What is the purpose of the secondary name-node?

The term "secondary name-node" is somewhat misleading. It is not a name-node in the sense that data-nodes cannot connect to the secondary name-node, and in no event it can replace the primary name-node in case of its failure.

The only purpose of the secondary name-node is to perform periodic checkpoints. The secondary name-node periodically downloads current name-node image and edits log files, joins them into new image and uploads the new image back to the (primary and the only) name-node.

So if the name-node fails and you can restart it on the same physical node then there is no need to shutdown data-nodes, just the name-node need to be restarted. If you cannot use the old node anymore you will need to copy the latest image somewhere else. The latest image can be found either on the node that used to be the primary before failure if available; or on the secondary name-node. The latter will be the latest checkpoint without subsequent edits logs, that is the most recent name space modifications may be missing there. You will also need to restart the whole cluster in this case.

 

1st,2nd,3rd,BCNF and 4th normal form in one example

All-in-One Example

Many of you asked for a "complete" example that would run through all of the normal forms from beginning to end using the same tables. This is tough to do, but here is an attempt: Example relation:
EMPLOYEE ( Name, Project, Task, Office, Phone )
Note: Keys are underlined.
Example Data:

Name	Project	Task	Office	Floor	Phone
Bill	100X	T1	400	4	1400
Bill	100X	T2	400	4	1400
Bill	200Y	T1	400	4	1400
Bill	200Y	T2	400	4	1400
Sue	100X	T33	442	4	1442
Sue	200Y	T33	442	4	1442
Sue	300Z	T33	442	4	1442
Ed	100X	T2	588	5	1588

• Name is the employee's name
• Project is the project they are working on. Bill is working on two different projects, Sue is working on 3.
• Task is the current task being worked on. Bill is now working on Tasks T1 and T2. Note that Tasks are independent of the project. Examples of a task might be faxing a memo or holding a meeting.
• Office is the office number for the employee. Bill works in office number 400.
• Floor is the floor on which the office is located.
• Phone is the phone extension. Note this is associated with the phone in the given office.

First Normal Form

• Assume the key is Name, Project, Task.
• Is EMPLOYEE in 1NF ?

Second Normal Form

• List all of the functional dependencies for EMPLOYEE.
• Are all of the non-key attributes dependant on all of the key ?
• Split into two relations EMPLOYEE_PROJECT_TASK and EMPLOYEE_OFFICE_PHONE. EMPLOYEE_PROJECT_TASK (Name, Project, Task)
  

  Name	Project	Task
  Bill	100X	T1
  Bill	100X	T2
  Bill	200Y	T1
  Bill	200Y	T2
  Sue	100X	T33
  Sue	200Y	T33
  Sue	300Z	T33
  Ed	100X	T2

  EMPLOYEE_OFFICE_PHONE (Name, Office, Floor, Phone)
  
  

  Name	Office	Floor	Phone
  Bill	400	4	1400
  Sue	442	4	1442
  Ed	588	5	1588

Third Normal Form

• Assume each office has exactly one phone number.
• Are there any transitive dependencies ?
• Where are the modification anomalies in EMPLOYEE_OFFICE_PHONE ?
• Split EMPLOYEE_OFFICE_PHONE. EMPLOYEE_PROJECT_TASK (Name, Project, Task)
  
  

  Name	Project	Task
  Bill	100X	T1
  Bill	100X	T2
  Bill	200Y	T1
  Bill	200Y	T2
  Sue	100X	T33
  Sue	200Y	T33
  Sue	300Z	T33
  Ed	100X	T2

  EMPLOYEE_OFFICE (Name, Office, Floor)
  
  

  Name	Office	Floor
  Bill	400	4
  Sue	442	4
  Ed	588	5

  EMPLOYEE_PHONE (Office, Phone)
  
  

  Office	Phone
  400	1400
  442	1442
  588	1588

Boyce-Codd Normal Form

• List all of the functional dependencies for EMPLOYEE_PROJECT_TASK, EMPLOYEE_OFFICE and EMPLOYEE_PHONE. Look at the determinants.
• Are all determinants candidate keys ?

Forth Normal Form

• Are there any multivalued dependencies ?
• What are the modification anomalies ?
• Split EMPLOYEE_PROJECT_TASK. EMPLOYEE_PROJECT (Name, Project )
  
  

  Name	Project
  Bill	100X
  Bill	200Y
  Sue	100X
  Sue	200Y
  Sue	300Z
  Ed	100X

  EMPLOYEE_TASK (Name, Task )
  
  

  Name	Task
  Bill	T1
  Bill	T2
  Sue	T33
  Ed	T2

  EMPLOYEE_OFFICE (Name, Office, Floor)
  
  

  Name	Office	Floor
  Bill	400	4
  Sue	442	4
  Ed	588	5

  R4 (Office, Phone)
  
  

  Office	Phone
  400	1400
  442	1442
  588	1588

At each step of the process, we did the following:

1. Write out the relation
2. (optionally) Write out some example data.
3. Write out all of the functional dependencies
4. Starting with 1NF, go through each normal form and state why the relation is in the given normal form.

BCNF(Boyce Codd Normal Form)

Boyce-Codd Normal Form (BCNF)

• A relation is in BCNF if every determinant is a candidate key.
• Recall that not all determinants are keys.
• Those determinants that are keys we initially call candidate keys.
• Eventually, we select a single candidate key to be the primary key for the relation.
• Consider the following example:
  Funds consist of one or more Investment Types.
  Funds are managed by one or more Managers
  Investment Types can have one more Managers
  Managers only manage one type of investment.
  
  

  FundID	InvestmentType	Manager
  99	Common Stock	Smith
  99	Municipal Bonds	Jones
  33	Common Stock	Green
  22	Growth Stocks	Brown
  11	Common Stock	Smith

  
  

  FundID, InvestmentType -> Manager
  FundID, Manager        -> InvestmentType
  Manager                -> InvestmentType
  

• In this case, the combination FundID and InvestmentType form a candidate key because we can use FundID,InvestmentType to uniquely identify a tuple in the relation.
• Similarly, the combination FundID and Manager also form a candidate key because we can use FundID, Manager to uniquely identify a tuple.
• Manager by itself is not a candidate key because we cannot use Manager alone to uniquely identify a tuple in the relation.
• Is this relation R(FundID, InvestmentType, Manager) in 1NF, 2NF or 3NF ?
  Given we pick FundID, InvestmentType as the Primary Key: 1NF for sure.
  2NF because all of the non-key attributes (Manager) is dependant on all of the key.
  3NF because there are no transitive dependencies.
• Consider what happens if we delete the tuple with FundID 22. We loose the fact that Brown manages the InvestmentType "Growth Stocks."
• The following are steps to normalize a relation into BCNF:
  1. List all of the determinants.
  2. See if each determinant can act as a key (candidate keys).
  3. For any determinant that is not a candidate key, create a new relation from the functional dependency. Retain the determinant in the original relation.
• For our example:
  Rorig(FundID, InvestmentType, Manager)
  1. The determinants are:
     FundID, InvestmentType
     FundID, Manager
     Manager
  2. Which determinants can act as keys ?
     FundID, InvestmentType YES
     FundID, Manager YES
     Manager NO
  3. Create a new relation from the functional dependency:
     Rnew(Manager, InvestmentType)
     Rorig(FundID, Manager)
     
     In this last step, we have retained the determinant "Manager" in the original relation Rorig.

Candidate Key,Super Key

A candidate key for a relation is a set of its attributes that satisfy:
• Uniqueness. The values of the attributes uniquely identify a
tuple.
• Minimality. No proper subset of the attributes has the uniqueness
property.
If uniqueness is satisfied (but not necessarily minimality) the attributes
are said to form a superkey.

Third Normal Form

Prior to Normalization

Item	Colors	Price	Tax
Pen	red, blue	2.0	0.20
Scale	red, yellow	2.0	0.20
Pen	red, blue	2.0	0.20
Bag	blue, black	150.00	7.80

This table is not in first normal form because :
  A. There are multiple fields in color lab.
  B. Records are repeating (Duplicate records) or no primary key.

First Normal Form (1NF)

Item	Colors	Price	Tax
Pen	red	2.0	0.20
Pen	blue	2.0	0.20
Scale	red	2.0	0.20
Scale	yellow	2.0	0.20
Bag	blue	150.00	7.80
Bag	black	150.00	7.80

This table is now in first normal form.

Second Normal Form (2NF)
The concept of remove the delicacy of data comes in the Second Normal Form (2NF).

A.  It should meet all the requirements of the first normal form.
B.  It should remove subsets of data that apply to multiple rows of a table and place them in separate tables.
C.  It create relationships between these new tables and their predecessors through the use of foreign keys. 

The First Normal form deals with the atomicity whereas the Second Normal Form deals with the relationship between the composite key columns and non-key columns. To achieve the next progressive level your table should satisfy the requirement of First Normal Form then move towards the Second Normal Form.
Let's introduce a Review table as an example :

Item	Colors	Price	Tax
Pen	red	2.0	0.20
Pen	blue	2.0	0.20
Scale	red	2.0	0.20
Scale	yellow	2.0	0.20
Bag	blue	150.00	7.80
Bag	black	150.00	7.80

Table is not in Second Normal Form because the price and tax depends on the item, but not color.

Item	Colors
Pen	red
Pen	blue
Scale	red
Scale	yellow
Bag	blue
Bag	black

Item	Price	Tax
Pen	2.0	0.20
Scale	2.0	0.20
Bag	150.00	7.80

Tables are now in Second Normal Form.


Third Normal Form (1NF)
The Third Normal Form has one more additional requirement :
        A. It should meet all the requirements of the second normal form.
      B. It should remove columns that are not dependent upon the primary key.
In the Third Normal Form all columns depend upon the primary key. When one column depends upon the other column,  table break the rule and turns into the dependency on the primary key.      

Item	Colors
Pen	red
Pen	blue
Scale	red
Scale	yellow
Bag	blue
Bag	black

Item	Price	Tax
Pen	2.0	0.20
Scale	2.0	0.20
Bag	150.00	7.80

Tables are not in Second Normal Form because tax depends on price, not item.
Tables are now in Third Normal Form with tables

Item	Colors
Pen	red
Pen	blue
Scale	red
Scale	yellow
Bag	blue
Bag	black

Item	Price
Pen	2.0
Scale	2.0
Bag	150.00

Price	Tax
2.0	0.20
150.00	7.80