Skip to main content

9 Recommended Steps to create Hadoop Cluster

hadoop cluster
9 Steps to perform to setup hadoop cluster on your PC/Laptop in your centos either in local or virtual machine.

Step 1:  Installing Sun Java on Linux. Commands to execute for the same:

sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-jre sun-java6-plugin
sudo update-java-alternatives -s java-6-sun

Step 2:  Create Hadoop User. Commands to execute for the same:

$sudo addgroup hadoop
$sudo adduser —ingroup hadoop hduser

Step 3:  Install SSH Server if not already present. Commands are:

$ sudo apt-get install openssh-server
$ su - hduser
$ ssh-keygen -t rsa -P ""
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Step 4:  Installing Hadoop. Commands for the same are:
$wget http://www.eng.lsu.edu/mirrors/apache/hadoop/core/hadoop-0.22.0/hadoop-0.22.0.tar.gz
$ cd /home/hduser
$ tar xzf hadoop-0.22.2.tar.gz
$ mv hadoop-0.22.2 hadoop

Step 5:  Edit .bashrc. Commands:
# Set Hadoop-related environment variables
export HADOOP_HOME=/home/hduser/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun

Step 6:  Update hadoop-env.sh
Here we need to only update the JAVA_HOME variable, we just need to open the file from text editor.

$vi /home/hduser/hadoop/conf/hadoop-env.sh
And update the following:
export JAVA_HOME=/usr/lib/jvm/java-6-sun

Step 7:  Update the files residing in the conf folder.  The three main files are:
Core-site.xml
Mapred-site.xml
Hdfs-site.xml
Here we need to change the value of namenode and jobtracker to where we want to run.

Step 8:  Start the hadoop cluster, using command.
Start-all.sh

Step 9:  Check if all the processes are up and running, using command:

Jps
That’s it. The cluster should be up and running with the following 5 processes:

  • NameNode
  • SecondaryNameNode
  • DataNode
  • JobTracker
  • TaskTracker
If we see all these processes, it means our installation is successful and the cluster is up and running.

Comments

Popular posts from this blog

Blue Prism complete tutorials download now

Blue prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career. The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews.
RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation.
RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism BlogsVi…

Hyperledger Fabric Real Interview Questions Read Today

I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…

Data analysis tools top demand in the job market to read today

Data analytics is the job role hot in demand in each organization. The digital skills such as Mobile development, Full stack development, and Data Science, and Cloud computing are successful because those are very user-friendly to the end users.
Predictive Analytics Digital devices enabled with digital technologies cause to generate more data. You need different kinds of tools to analyze data of different format.

You need the right tools. Else you cannot predict user mind. User search data is the source for big retail markets. Based on these search words, they start selling the products.

The motto behind data analytics is to get the benefit to all stakeholders.
Cloud Computing Let us take a cloud computing the main advantage is cost-effective and scalability. Top Data Analytics Tools in DemandR ProgrammingSASExcelTableauQlikViewTop Magazines in Data AnalyticsAnalytics InsightAnalytics MagazineAnalytics India Magazine Related PostsR Vs SAS Top Differences6 Top IT Skills that have Huge D…

Three popular RPA tools functional differences

Robotic process automation is growing area and many IT developers across the board started up-skill in this popular area. I have written this post for the benefit of Software developers who are interested in RPA also called Robotic Process Automation.

In my previous post, I have described that total 12 tools are available in the market. Out of those 3 tools are most popular. Those are Automation anywhere, BluePrism and Uipath. Many programmers asked what are the differences between these tools. I have given differences of all these three RPA tools.

BluePrism Blue Prism has taken a simple concept, replicating user activity on the desktop, and made it enterprise strength. The technology is scalable, secure, resilient, and flexible and is supported by a comprehensive methodology, operational framework and provided as packaged software.The technology is developed and deployed within a “corridor of IT governance” and has sophisticated error handling and process modelling capabilities to ens…

R Vs SAS differences to read today

Statistical analysis should know by every software engineer. R is an open source statistical programming language. SAS is licensed analysis suite for statistics. The two are very much popular in Machine learning and data analytics projects.
SAS is analysis suite software and R is a programming language R ProgrammingR supports both statistical analysis and GraphicsR is an open source project.R is 18th most popular LanguageR packages are written in C, C++, Java, Python and.NetR is popular in Machine learning, data mining and Statistical analysis projects. SASSAS is a statistical analysis suite. Developed to process data sets in mainframe computers.Later developed to support multi-platforms. Like  Mainframe, Windows, and LinuxSAS has multiple products. SAS/ Base is very basic level.SAS is popular in data related projects. Learn SAS vs R Top Differences between SAS Vs R Programming SAS AdvantagesThe data integration from any data source is faster in SAS.The licensed software suite, so you…

Automation developer these are top Skills you need to learn

Robotic process automation is an upcoming IT skill. Three tools are popular. It is difficult to learn all three tool. So, learn anyone tool to start your career in automation.
To get a job in this line, I found in my research that some programming skills and Hand-on training on any one of the tools is required. Also, try to know the differences between popular RPA tools.
All big companies looking for candidates having experience in Automation anywhere, Blue Prism and UIPath. It is not possible to learn all tools. Learn anyone tool and do practice well.

Ok.

You may ask a question about how to do it. Join in good training institute and learn one tool.  Take online classes to learn faster.

To learn Uipath try here. Also, you can enroll online course to learn UiPath.

The list of IT skills you need
Automation anywhere/Blue Prism/Uipath .Net/C#/Java/SQL skills MS-Visio Power Builder Python scripts/Unix Scripts/Perl Scripts HTML/CSS/JavaScript