This document describes how to set up and configure a single-node Apache Hadoop 2.7 installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS). Let you start working with Hadoop with basic configuration.

Requirement:

Java

Java is the primary requirement for running hadoop on any system, So make sure you have Java installed on your system using following command.

# java -version 

java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Install proper HadoopJavaVersions on your server If you don’t have Java installed on your system.

ssh

Hadoop framework itself doesn't need ssh, the administration tools like start.dfs.sh and stop.dfs.sh etc.. need it to start/stop various daemons. Thus, ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.

Step 1. Creating Hadoop User

We recommend to create a normal (nor root) account for hadoop working. So create a system account using following command.

# adduser hadoop
# passwd hadoop

Step 2. ssh key pairs for password free access

After creating account, it also required to set up key based ssh to its own account. To do this use execute following commands.

# su - hadoop
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Lets verify key based login. Below command should not ask for password but first time it will prompt for adding RSA to the list of known hosts.

$ ssh localhost
$ exit

Step 3. Downloading Hadoop 2.7.3

Now download hadoop 2.7.3 binary file using below command. You can also select alternate download mirror. Source code package is also available, download and recompile it to fit your environment.

$ cd /opt
$ wget http://www-us.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
$ tar xzf hadoop-2.7.3.tar.gz
$ mv hadoop-2.7.3 hadoop

Step 4. Setup Environment Variables

First we need to set environment variable uses by hadoop. Edit ~/.bashrc file and append following values at end of file.

export HADOOP_HOME=/opt/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Now apply the changes in current running environment

$ source ~/.bashrc

Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable. Change the JAVA path as per install on your system.

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-oracle.x86_64/

Step 5. Edit Configuration Files

Hadoop has many of configuration files, which need to configure as per requirements of your hadoop infrastructure. Lets start with the configuration with basic hadoop single node cluster setup. first navigate to below location

$ cd $HADOOP_HOME/etc/hadoop

Edit core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

Or, replace the localhost with the node hostname

Edit hdfs-site.xml

Suppose you have two separated filesystem dedicated for hadoop namespace and data

/hadoop  --- > hadoop namespace

/hadoopdata  --- > hadoop data

<configuration>
<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
    <value>file:///hadoop/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
    <value>file:///hadoopdata/hdfs/datanode</value>
</property>
</configuration>

You can also define multiple entries for dfs.name.dir and dfs.data.dir

Edit mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
</configuration>

Edit yarn-site.xml

<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>

Step 6. Format Namenode

Now format the namenode using following command, make sure that Storage directory is

$ hdfs namenode -format

Step 7. Start Hadoop services

Lets start your hadoop using the scripts provides by hadoop. Just navigate to your hadoop sbin directory and execute scripts one by one.

$ cd $HADOOP_HOME/sbin/

Now run start-dfs.sh script.

$ start-dfs.sh

Now run start-yarn.sh script.

$ start-yarn.sh

Step 8. Access Hadoop Services in Browser

Hadoop NameNode started on port 50070 default. Access your server on port 50070 in your favorite web browser.

http://<hadoopnode>:50070/

Also, you can access port 8088 for getting the information about cluster and all applications

http://<hadoopnode>:8088/

Access port 50090 for getting details about secondary namenode.

http://<hadoopnode>:50090/

Access port 50075 to get details about DataNode

http://<hadoopnode>:50075/

Step 9. Test Hadoop Single Node Setup

Make the HDFS directories required using following commands.

$ hdfs dfs -mkdir /user
$ hdfs dfs -mkdir /user/hadoop

Now copy all files from local file system /home/hadoop/etc/hadoop/hadoop-env.sh to hadoop distributed file system using below command

$ hdfs dfs -put /home/hadoop/hadoop-2.7.3.tar /user/hadoop

Now browse hadoop distributed file system by opening below url in browser.

http://<hadoopnode>:50070/explorer.html#/user/hadoop/

Now copy logs directory for hadoop distributed file system to local file system.

$ hdfs dfs -get /user/hadoop/hadoop-2.7.3.tar /home/test

Ref:

http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html