Featured Post

Mastering flat_map in Python with List Comprehension

Image
Introduction In Python, when working with nested lists or iterables, one common challenge is flattening them into a single list while applying transformations. Many programming languages provide a built-in flatMap function, but Python does not have an explicit flat_map method. However, Python’s powerful list comprehensions offer an elegant way to achieve the same functionality. This article examines implementation behavior using Python’s list comprehensions and other methods. What is flat_map ? Functional programming  flatMap is a combination of map and flatten . It transforms the collection's element and flattens the resulting nested structure into a single sequence. For example, given a list of lists, flat_map applies a function to each sublist and returns a single flattened list. Example in a Functional Programming Language: List(List(1, 2), List(3, 4)).flatMap(x => x.map(_ * 2)) // Output: List(2, 4, 6, 8) Implementing flat_map in Python Using List Comprehension Python’...

Apache Yarn to Manage Resources a Solution

Apache Hadoop is one of the most popular tools for big data processing. It has been successfully deployed in production by many companies for several years. 

Though Hadoop is considered a reliable, scalable, and cost-effective solution, it is constantly being improved by a large community of developers. As a result, the 2.0 version offers several revolutionary features, including Yet Another Resource Negotiator (YARN), HDFS Federation, and a highly available NameNode, which make the Hadoop cluster much more efficient, powerful, and reliable. 

Apache Yarn

Apache Hadoop 2.0 includes YARN, which separates the resource management and processing components. The YARN-based architecture is not constrained to MapReduce.
  • New developmens in Hadoop 2.0 Architecture with YARN: 
  • ResourceManager instead of a cluster manager 
  • ApplicationMaster instead of a dedicated and short-lived JobTracker 
  • NodeManager instead of TaskTracker 
  • A distributed application instead of a MapReduce job 

Basic changes in Hadoop 2.0 architecture

  • The ResourceManager, the NodeManager, and a container are not concerned about the type of application or task.
  • All application framework-specific code is simply moved to its ApplicationMaster so that any distributed framework can be supported by YARN — as long as someone implements an appropriate ApplicationMaster for it.
  • Thanks to this generic approach, the dream of a Hadoop YARN cluster running many various workloads comes true. Imagine: a single Hadoop cluster in your data center that can run MapReduce, Giraph, Storm, Spark, Tez/Impala, MPI, and more.

Read More

Comments

Popular posts from this blog

SQL Query: 3 Methods for Calculating Cumulative SUM

Big Data: Top Cloud Computing Interview Questions (1 of 4)

5 SQL Queries That Popularly Used in Data Analysis