Featured Post

Scraping Website: How to Write a Script in Python

Image
Here's a python script that you can use as a model to scrape a website. Python script The below logic uses BeautifulSoup Package for web scraping. import requests from bs4 import BeautifulSoup url = 'https://www.example.com' response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') # Print the title of the webpage print(soup.title.text) # Print all the links in the webpage for link in soup.find_all('a'):     print(link.get('href')) In this script, we first import the Requests and Beautiful Soup libraries. We then define the URL we want to scrape and use the Requests library to send a GET request to that URL. We then pass the response text to Beautiful Soup to parse the HTML contents of the webpage. We then use Beautiful Soup to extract the title of the webpage and print it to the console. We also use a for loop to find all the links in the webpage and print their href attributes to the console. This is just a basic example, but

Here is Hadoop MapReduce DataFlow Tutorial

Here are the six stages of MapReduce. The MapReduce is critical for your data processing needs. Traditionally, the whole file needs to read once then divided manually, but it is not convenient. With that respect, Hadoop provides the facility to read files (ignoring their size) line-for-line by using offset and key-value.

Explained the dataflow in Hadoop MapReducer

MapReduce dataflow Quick Tutorial


1. Dataflow Diagram



How a Mapreduce process in Hadoop divides input and processes it, you will learn in this post.


2. MapReduce Stages


MapReduce receives input and processes it. Here are the six stages of processing. It is helpful for your interviews and project.


MapReduce Stage-1


Take the file as input for processing purposes. Any file will consist of a group of lines. These lines containing key-value pairs of data. The whole file can be read out with this method.

MapReduce Stage-2


In the next step, the file will be in "splitting" mode. This mode will divide the file into key, value pair of data. This time key will be offset and data will be a valuable part of the program. Each line will be read individually so there is no need to split data manually.

MapReduce Stage-3


The further step is to process the value of each line with an associate from counting numbers. Each individual that is separated from a space counted with the number and that number is written with each key. This is the logic of "mapping" that programmers need to write.

MapReduce Stage-4


After that shuffling is performed and with this, each key gets associated with the group of numbers that are involved in the mapping section. Now scenario becomes key with string and value will be a list of numbers. This will go as input to the reducer.

MapReduce Stage-5


In the reducer phase, whole numbers are counted and each key associated with final counting is the sum of all numbers which leads to the final result.

MapReduce Stage-6


Output of the reducer phase will lead to the final result. This final result will have counting of individual word count. This is independent of the size of the file used for processing.


Keep Reading
  1. Big Data and Hadoop: Learn by Example

Comments

Popular posts from this blog

7 AWS Interview Questions asked in Infosys, TCS

How to Decode TLV Quickly