Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

Here is Hadoop MapReduce DataFlow Tutorial

Here are the six stages of MapReduce. The MapReduce is critical for your data processing needs. Traditionally, the whole file needs to read once then divided manually, but it is not convenient. With that respect, Hadoop provides the facility to read files (ignoring their size) line-for-line by using offset and key-value.

Explained the dataflow in Hadoop MapReducer

MapReduce dataflow Quick Tutorial

1. Dataflow Diagram

How a Mapreduce process in Hadoop divides input and processes it, you will learn in this post.

2. MapReduce Stages

MapReduce receives input and processes it. Here are the six stages of processing. It is helpful for your interviews and project.

MapReduce Stage-1

Take the file as input for processing purposes. Any file will consist of a group of lines. These lines containing key-value pairs of data. The whole file can be read out with this method.

MapReduce Stage-2

In the next step, the file will be in "splitting" mode. This mode will divide the file into key, value pair of data. This time key will be offset and data will be a valuable part of the program. Each line will be read individually so there is no need to split data manually.

MapReduce Stage-3

The further step is to process the value of each line with an associate from counting numbers. Each individual that is separated from a space counted with the number and that number is written with each key. This is the logic of "mapping" that programmers need to write.

MapReduce Stage-4

After that shuffling is performed and with this, each key gets associated with the group of numbers that are involved in the mapping section. Now scenario becomes key with string and value will be a list of numbers. This will go as input to the reducer.

MapReduce Stage-5

In the reducer phase, whole numbers are counted and each key associated with final counting is the sum of all numbers which leads to the final result.

MapReduce Stage-6

Output of the reducer phase will lead to the final result. This final result will have counting of individual word count. This is independent of the size of the file used for processing.

Keep Reading
  1. Big Data and Hadoop: Learn by Example


Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

Explained Ideal Structure of Python Class

How to Check Kafka Available Brokers