Apache Yarn to Manage Resources a Solution

-

Apache Hadoop is one of the most popular tools for big data processing. It has been successfully deployed in production by many companies for several years. 

Though Hadoop is considered a reliable, scalable, and cost-effective solution, it is constantly being improved by a large community of developers. As a result, the 2.0 version offers several revolutionary features, including Yet Another Resource Negotiator (YARN), HDFS Federation, and a highly available NameNode, which make the Hadoop cluster much more efficient, powerful, and reliable. 

Apache Yarn

Apache Hadoop 2.0 includes YARN, which separates the resource management and processing components. The YARN-based architecture is not constrained to MapReduce.
  • New developmens in Hadoop 2.0 Architecture with YARN: 
  • ResourceManager instead of a cluster manager 
  • ApplicationMaster instead of a dedicated and short-lived JobTracker 
  • NodeManager instead of TaskTracker 
  • A distributed application instead of a MapReduce job 

Basic changes in Hadoop 2.0 architecture

  • The ResourceManager, the NodeManager, and a container are not concerned about the type of application or task.
  • All application framework-specific code is simply moved to its ApplicationMaster so that any distributed framework can be supported by YARN — as long as someone implements an appropriate ApplicationMaster for it.
  • Thanks to this generic approach, the dream of a Hadoop YARN cluster running many various workloads comes true. Imagine: a single Hadoop cluster in your data center that can run MapReduce, Giraph, Storm, Spark, Tez/Impala, MPI, and more.

Read More

Comments

Post a Comment

Thanks for your message. We will get back you.

Popular Posts

How to Fix Python Syntax Errors Quickly

-
Image
In Python, you can call parsing errors Syntax errors. How to fix the syntax errors quickly, I can show in this post. Also, check Python Syntax Errors Cheat Sheet. Python Syntax Error. If you run the Python code, the interpreter first parses the logic. If any errors found, it is the developer's responsibility to fix them. Courtesy: Python documentation. >>> while True print('Hello world')   File "<stdin>", line 1     while True print('Hello world')                    ^ SyntaxError: invalid syntax How to Fix the Syntax Error First, check for the line number. In the above case, it is '1'. Then look for the symbol '^'. The issue is at the PRINT function. The reason is before the PRINT function you need a colon ':'. That is the reason, you are getting error. This is a syntax error. Find below the corrected logic: >>> while True : print('Hello world')   File "<stdin>", line 1     while Tr
Read more

7 AWS Interview Questions asked in Infosys, TCS

-
Image
I have shared 7 amazing AWS interview questions for your quick reference. 1). What is AWS? Amazon Web Services (AWS) provides on-demand computing resources and services in the cloud, with pay-as-you-go pricing. For example, you can run a server on AWS that you can log on to, configure, secure, and run just as you would a server that's sitting in front of you 2). What you can do with AWS? Store public or private data. Host a static website. These websites use client-side technologies (such as HTML, CSS, and JavaScript) to display content that doesn't change frequently. A static website doesn't require server-side technologies (such as PHP and ASP.NET). Host a dynamic website or web app. These websites include classic three-tier applications, with web, application, and database tiers. Support students or online training programs. Process business and scientific data. Handle peak loads. AWS Questions 3). What is the Security group? A security group acts as a virtual fi
Read more

Python 'getsizeof' Command the Real Purpose

-
Image
In Python, there are many objects. Out of those List (arrays) is one type.  To get the number of elements present in a List is not easy. So how to get this? To do you need the 'getsizeof' command. Import 'sys' Library. You need to import the 'sys' library to use the SIZE commands.  >>> import sys Congrats to you since you have now imported the SYS library. The command name is " getsizeof" . Check the below example. print("An Integer:") i = int(100) print(sys.getsizeof(i)) The result will be: 100 What is the size command?  - is to get the size of elements. It gives the result based on the object you are being used. Suppose, if you use Strings, you will get String length. How to use the  'getsizeof' command? Example:1 print("Empty string:") empty_string = "" print(sys.getsizeof(empty_string)) Example:2 print("A normal string") s = "Hello world" print(sys.getsizeof(s)) Example:3 print(&quo
Read more

How to Decode TLV Quickly

-
Image
In TLV, the format is Tag, Length, and Value. The TLV protocol needs this type of data. Here you will know how to decode TLV data.  According to IBM , TLV data is three parts. The tag tells what type of data it is. The length field denotes the length of the value. The Value-field denotes the actual value. Structure of TLV. TLV comprises three field values.  Tag Length Value EMV formulated different tags. They have their meanings. Usually, the Tag and Length together takes 1 to 4 bytes. The Best example for TLV. In the below example, you can find the sample TAG, LENGTH, and VALUE fields. [Tag][Value Length][Value] (ex. " 9F4005F000F0A001 ") where Tag Name =  9F40 Value Length (in bytes) =  05  Value (Hex representation of bytes. Example, "F0" – 1-byte) =  F000F0A001 In the above message, tag 9F40 has some meaning designed by EMV company. Here  you can find a list of EMV Tags. How to read the TLV Tag: 1 or 2 bytes Length: Length of the Value. F0-00-F0-A0-01 ==> 5 B
Read more

Hyperledger Fabric: 20 Real Interview Questions

-
Image
I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview. Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications. The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions 1). What are Nodes? In Hyperledger the communication entities are called Nodes. 2). What are the three different types of Nodes? - Client Node - Peer Node - Order Node The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery. 3). What is Channel? A channel in Hyperledger is the subnet of
Read more

AWS Vs Azure Load Balancers Top Insights

-
Image
In AWS, you have three types of load-balancers. In the case of Azure, you have only two load balancers. This post tells you the comparison between these two. AWS Load Balancers: Application Load balancer Network Load balancer Classic Load balancers. Application Load Balancer Balancing of HTTP and HTTPS traffic and provides advanced request routing targeted at the delivery of modern application architectures, including microservices and containers.  Operating at the individual request level (Layer 7), Application Load Balancer routes traffic to targets within Amazon Virtual Private Cloud (Amazon VPC) based on the content of the request. Network Load Balancer Network Load Balancer is best suited for load balancing of Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Transport Layer Security (TLS) traffic where extreme performance is required. Classic Load Balancer The Classic Load Balancer provides the basic load-balancing across multiple Amazon EC2 instances.  It op
Read more

Oozie - Concepts And Architecture

-
Image
Oozie is a workflow/coordination system that you can use to manage Apache Hadoop jobs. It is one of the main components of Oozie is the Oozie server — a web application that runs in a Java servlet container (the standard Oozie distribution is using Tomcat). Oozie is a workflow management-server that works on the Oozie server. Role of Oozie in Workflow Management in Hadoop Jobs This server supports reading and executing Workflows, Coordinators, Bundles, and SLA definitions . It implements a set of remote Web Services APIs that can be invoked from Oozie client components and third-party applications. Add a note where the execution of the server leverages a customizable database. This database contains Workflow, Coordinator, Bundle, and SLA definitions, as well as execution states and process variables. The list of currently supported databases includes MySQL, Oracle, and Apache Derby. The Oozie shared library component is located in the Oozie HOME directory and contains code u
Read more

QlikView Server vs Publisher Top Differences Really Useful to Your Project

-
Image
The differences between QlikView Server and publisher, I have given for your reference. These are related to internally how differ each other. QLIKVIEW SERVER The QVS is a server-side product.  It contains an  in-memory analytics engine , which handles all client/server communication between a QlikView client (Desktop, IE plugin, AJAX, or Mobile) and the server. The Administrator can have access to the management console to control all aspects of the server deployments (including security, clustering, distribution, etc.)  It also includes a web server to provide front-end access to the documents within. The web server’s user portal is known as Access Point. (It’s important to note that while the QVS contains its web server, one can utilize Microsoft IIS (Internet Information Server) for this purpose, too). The QVS handles client authorization against existing directory providers (Microsoft Active Directory, eDirectory) It also performs read and write to ACLs (access control lists) for
Read more

Telecom Domain Knowledge Useful For a Data Analyst

-
Image
Telecom companies provide five kinds of services. Those are Voice, Data, Toll-free, internet, and wireless. The given Telecom domain knowledge useful to work on data analysis projects. 5 TopTelecom Services. Voice Toll-free Data service Internet Wireless  Telecom Domain Knowledge Useful for IT developers, Data Analysts, and Software Testers   In addition to the above, the latest services are Internet and wireless. Domain knowledge is crucial to design and develop the code. 1. Voice service This is the standard phone service that you have been using since the first time you called your Grandma to thank her for a birthday present. You need voice service. 2. Toll-free service A toll-free number is a special-number; that enables calls to come in from anywhere without incurring a charge to the caller. The called party pays a toll tariff for the Call. How much the called party pays for a call depends on the arrangements made with the long-distance carrier.  If you look at all of your
Read more

Payment Cards Complete List

-
Image
In this post, I have shared top payment cards, which are alternatives to money. The list is Credit, Debit, Fleet, Co-branded, Contactless cards.   Types of Plastic cards Many plastic cards are present to carry out financial transactions. The purpose is to buy items within the limit prescribed by banks to the cardholder. Two types of cards in this category are Magnetic-stripe and Chip cards. Credit cards In today's world, all banks issue credit cards with CHIP and PIN. After entering the PIN by the cardholder, the transaction kicks in for further processing. Types of Credit cards Security deposited cards: The cardholder should pay some front money to get this type of card. When the cardholder goes unpaid, then the card issuer pay from it. Credit Union Cards : The private Credit Union owners issue these cards. They usually charge less than what banks will. Pre-paid cards: They technically equal to Debit cards. They pre-loaded with a certain amount. You can use the cards until
Read more