Hadoop cluster setup in Centos Operating System explained in this post. So you can install CentOs either in your Laptop or in Virtual Machine.
sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-jre sun-java6-plugin
sudo update-java-alternatives -s java-6-sun
Step 2: Create Hadoop User. Commands to execute for the same:
$sudo addgroup hadoop
$sudo adduser —ingroup hadoop hduser
Step 3: Install SSH Server if not already present. Commands are:
$ sudo apt-get install openssh-server
$ su - hduser
$ ssh-keygen -t rsa -P ""
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Step 4: Installing Hadoop. Commands for the same are:
$wget http://www.eng.lsu.edu/mirrors/apache/hadoop/core/hadoop-0.22.0/hadoop-0.22.0.tar.gz
$ cd /home/hduser
$ tar xzf hadoop-0.22.2.tar.gz
$ mv hadoop-0.22.2 hadoop
Step 5: Edit .bashrc. Commands:
# Set Hadoop-related environment variables
export HADOOP_HOME=/home/hduser/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun
Step 6: Update hadoop-env.sh
Here we need to only update the JAVA_HOME variable, we just need to open the file from text editor.
$vi /home/hduser/hadoop/conf/hadoop-env.sh
And update the following:
export JAVA_HOME=/usr/lib/jvm/java-6-sun
Step 7: Update the files residing in the conf folder. The three main files are:
Core-site.xml
Mapred-site.xml
Hdfs-site.xml
Here we need to change the value of namenode and jobtracker to where we want to run.
Step 8: Start the hadoop cluster, using command.
Start-all.sh
Step 9: Check if all the processes are up and running, using command:
Jps
That’s it. The cluster should be up and running with the following 5 processes:
 |
| Hadoop Cluster Setup Process |
9 Steps Process to Setup Hadoop Cluster
Step 1: Installing Sun Java on Linux. Commands to execute for the same:sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-jre sun-java6-plugin
sudo update-java-alternatives -s java-6-sun
Step 2: Create Hadoop User. Commands to execute for the same:
$sudo addgroup hadoop
$sudo adduser —ingroup hadoop hduser
Step 3: Install SSH Server if not already present. Commands are:
$ sudo apt-get install openssh-server
$ su - hduser
$ ssh-keygen -t rsa -P ""
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Step 4: Installing Hadoop. Commands for the same are:
$wget http://www.eng.lsu.edu/mirrors/apache/hadoop/core/hadoop-0.22.0/hadoop-0.22.0.tar.gz
$ cd /home/hduser
$ tar xzf hadoop-0.22.2.tar.gz
$ mv hadoop-0.22.2 hadoop
Step 5: Edit .bashrc. Commands:
# Set Hadoop-related environment variables
export HADOOP_HOME=/home/hduser/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun
Step 6: Update hadoop-env.sh
Here we need to only update the JAVA_HOME variable, we just need to open the file from text editor.
$vi /home/hduser/hadoop/conf/hadoop-env.sh
And update the following:
export JAVA_HOME=/usr/lib/jvm/java-6-sun
Step 7: Update the files residing in the conf folder. The three main files are:
Core-site.xml
Mapred-site.xml
Hdfs-site.xml
Here we need to change the value of namenode and jobtracker to where we want to run.
Step 8: Start the hadoop cluster, using command.
Start-all.sh
Step 9: Check if all the processes are up and running, using command:
Jps
That’s it. The cluster should be up and running with the following 5 processes:
- NameNode
- SecondaryNameNode
- DataNode
- JobTracker
- TaskTracker