Skip to main content

Featured post

8 Top Data Science Platform Developers in the World

Top data science tools and platforms providers across the world. Useful information for data science and data analytics developers.

Data Science is a combination of multiple skills. AI and Machine Learning are part of data science. You can create AI and Machine Learning products with data. 

Related Posts

Top Skills You Need for Data Science CareerData Science Sample Project an Example

Top 100 Hadoop Complex Interview Questions (Part 2 of 4)

I am giving series Hadoop interview questions. This is my 2nd set of questions. You can get quick benefit by reading these questions start to end.

hadoop part 2
1). If a data Node is full how it’s identified?
Ans). When data is stored in a data node, then the metadata of that data will be stored in the Namenode. So Namenode will identify if the data node is full.

2). If data nodes increase, then do we need to upgrade Namenode?
Ans). While installing the Hadoop system, Namenode is determined based on the size of the clusters. Most of the time, we do not need to upgrade the Namenode because it does not store the actual data, but just the metadata, so such a requirement rarely arise.

3). Are job tracker and task trackers present in separate machines?Ans). Yes, job tracker and task tracker are present in different machines. The reason is job tracker is a single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted.

4). When we send a data to a node, do we allow settling in time, before sending another data to that node?
Ans). Yes, we do.

Related: Hadoop Complex Questions part-1

5). Does Hadoop always require digital data to process?
Ans). Yes. Hadoop always require digital data to be processed.

6).On what basis Namenode will decide which data node to write on?
As the Namenode has the metadata (information) related to all the data nodes, it knows which data node is free.

7). Doesn’t Google have its very own version of DFS?
Ans). Yes, Google owns a DFS known as “Google File System (GFS)” developed by Google Inc. for its own use.

8). Who is a ‘user’ in HDFS?
Ans). A user is like you or me, who has some query or who needs some kind of data.

9) Is client the end user in HDFS?
Ans). No, Client is an application which runs on your machine, which is used to interact with the Namenode (job tracker) or data node (task tracker).

10). What is the communication channel between the client and the name node/data node?
Ans). The mode of communication is SSH.

11). What is a rack?
Ans). A rack is a storage area with all the data nodes put together. These data nodes can be physically located in different places. The rack is a physical collection of data nodes which are stored at a single location. There can be multiple racks in a single location.

12). On what basis data will be stored on a rack?
Ans). When the client is ready to load a file into the cluster, the content of the file will be divided into blocks. Now the client consults the Namenode and gets 3 data nodes for every block of the file which indicates where the block should be stored. While placing the data nodes, the key rule followed is “for every block of data, two copies will exist in one rack, third copy in a different rack“. This rule is known as “Replica Placement Policy“.

13). Do we need to place 2nd and 3rd data in rack 2 only?
Ans). Yes, this is to avoid data node failure.

14). What if rack 2 and data node fails?
Ans). If both rack2 and data node present in rack 1 fails then there is no chance of getting data from it. In order to avoid such situations, we need to replicate that data number of times instead of replicating only thrice. This can be done by changing the value in replication factor which is set to 3 by default.

15). What is a Secondary Namenode? Is it a substitute for the Namenode?
Ans). The secondary Namenode constantly reads the data from the RAM of the Namenode and writes it into the hard disk or the file system. It is not a substitute for the Namenode, so if the Namenode fails, the entire Hadoop system goes down.

16). What is the difference between Gen1 and Gen2 Hadoop with regards to the Namenode?
Ans). In Gen 1 Hadoop, Namenode is the single point of failure. In Gen 2 Hadoop, we have what is known as Active and Passive Namenodes kind of a structure. If the active Namenode fails, passive Namenode takes over the charge.

17). What is MapReduce?
Ans). Map Reduce is the ‘heart‘ of Hadoop that consists of two parts – ‘map’ and ‘reduce’. Maps and reduces are programs for processing data. ‘Map’ processes the data first to give some intermediate output which is further processed by ‘Reduce’ to generate the final output. Thus, MapReduce allows for distributed processing of the map and reduction operations.

18). Can you explain how do ‘map’ and ‘reduce’ work?
Ans). Namenode takes the input and divide it into parts and assign them to data nodes. These data nodes process the tasks assigned to them and make a key-value pair and return the intermediate output to the Reducer. The reducer collects this key-value pairs of all the data nodes and combines them and generates the final output.

19). What is ‘Key value pair’ in HDFS?
Ans). The key value pair is the intermediate data generated by maps and sent to reduces for generating the final output.

20). What is the difference between MapReduce engine and HDFS cluster?
Ans). HDFS cluster is the name given to the whole configuration of master and slaves where data is stored. Map Reduce Engine is the programming module which is used to retrieve and analyze data.

21). Is map like a pointer?
Ans). No, Map is not like a pointer.

22). Do we require two servers for the Namenode and the data nodes?
Ans). Yes, we need two different servers for the Namenode and the data nodes. This is because Namenode requires highly configurable system as it stores information about the location details of all the files stored in different data nodes and on the other hand, data nodes require low configuration system.

23). Why is the number of splits equal to the number of maps?
Ans). The number of maps is equal to the number of input splits because we want the key and value pairs of all the input splits.

24). Is a job split into maps?
Ans). No, a job is not split into maps. Spilled is created for the file. The file is placed on data nodes in blocks. For each split, a map is needed.

25). Which are the two types of ‘writes’ in HDFS?
Ans). There are two types of writes in HDFS: posted and non-posted write. Posted Write is when we write it and forget about it, without worrying about the acknowledgment. It is similar to our traditional Indian post. In a Non-posted Write, we wait for the acknowledgment. It is similar to today’s courier services. Naturally, non-posted write is more expensive than the posted write. It is much more expensive, though both writes are asynchronous.

26). Why ‘Reading‘ is done in parallel and ‘Writing‘ is not in HDFS?
Ans). Reading is done in parallel because by doing so we can access the data fast. But we do not perform the write operation in parallel. The reason is that if we perform the write operation in parallel, then it might result in data inconsistency. For example, you have a file and two nodes are trying to write data into the file in parallel, then the first node does not know what the second node has written and vice-versa. So, this makes it confusing which data to be stored and accessed.

27). Can Hadoop be compared to NoSQL database like Cassandra?
Ans). Though NoSQL is the closest technology that can be compared to Hadoop, it has its own pros and cons. There is no DFS in NOSQL. Hadoop is not a database. It’s a filesystem (HDFS) and distributed programming framework (MapReduce).

28). How can I install Cloudera VM in my system?
Ans). When you enroll for the Hadoop course at Eureka, you can download the Hadoop Installation steps.pdf file from our dropbox. This will be shared with you by an e-mail.

29). Which are the three modes in which Hadoop can be run?
Ans). The three modes in which Hadoop can be run are:

  1. standalone (local) mode
  2. Pseudo-distributed mode
  3. Fully distributed mode

Related Posts

Comments

Popular posts from this blog

Hyperledger Fabric Real Interview Questions Read Today

I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…

Blue Prism complete tutorials download now

Blue prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career.
The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews.
RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation.

RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism Blogs

PL/SQL: Popular Reserved Words

Perfect in PL/SQL is an art. To become this you need to understand top reserved words and their meanings. The below list is useful for your projects.


Top List of PL/SQL Reserved Words.. Before you start knowing reserved words, wait one moment. The reserved words all are similar to words that you use in normal SQL. ALL*DESC*ISOLATIONOUTSQLERRMALTER*DISTINCT*JAVAPACKAGESTART*AND*DOLEVEL*PARTITIONSTDDEVANY*DROP*LIKE*PCTFREE*SUBTYPEARRAYELSE*LIMITEDPLS_INTEGERSUCCESSFUL*AS*ELSIFLOCK*POSITIVESUMASC*ENDLONG*POSITIVENSYNONYM*AUTHIDEXCEPTIONLOOPPRAGMASYSDATE*AVGEXCLUSIVE*MAXPRIOR*TABLE*BEGINEXECUTEMINPRIVATETHEN*BETWEEN*EXISTS*MINUS*PROCEDURETIMEBINARY_INTEGEREXITMINUTEPUBLIC*TIMESTAMPINTEGEREXTENDSMLSLABEL*RAISE

Automation developer these are top Skills you need to learn

Robotic process automation is an upcoming IT skill. Three tools are popular. It is difficult to learn all three tool. So, learn anyone tool to start your career in automation.
To get a job in this line, I found in my research that some programming skills and Hand-on training on any one of the tools is required. Also, try to know the differences between popular RPA tools.
Skills Companies Looking in Automation Engineers All big companies looking for candidates having experience in Automation anywhere, Blue Prism and UIPath. It is not possible to learn all tools. Learn anyone tool and do practice well.

Ok.

You may ask a question about how to do it. Join in good training institute and learn one tool.  Take online classes to learn faster.

To learn Uipath try here. Also, you can enroll online course to learn UiPath.

UiPath GO The list of IT skills you needAutomation anywhere/Blue Prism/Uipath .Net/C#/Java/SQL skills MS-Visio Power Builder Python scripts/Unix Scripts/Perl Scripts HTML/CSS/J…

Three popular RPA tools functional differences

Robotic process automation is growing area and many IT developers across the board started up-skill in this popular area. I have written this post for the benefit of Software developers who are interested in RPA also called Robotic Process Automation.


In my previous post, I have described that total 12 tools are available in the market. Out of those 3 tools are most popular. Those are Automation anywhere, BluePrism and Uipath. Many programmers asked what are the differences between these tools. I have given differences of all these three RPA tools.

BluePrism Blue Prism has taken a simple concept, replicating user activity on the desktop, and made it enterprise strength. The technology is scalable, secure, resilient, and flexible and is supported by a comprehensive methodology, operational framework and provided as packaged software.The technology is developed and deployed within a “corridor of IT governance” and has sophisticated error handling and process modelling capabilities to ens…

8 Top Data Science Platform Developers in the World

Top data science tools and platforms providers across the world. Useful information for data science and data analytics developers.

Data Science is a combination of multiple skills. AI and Machine Learning are part of data science. You can create AI and Machine Learning products with data. 

Related Posts

Top Skills You Need for Data Science CareerData Science Sample Project an Example

Top 100 Hadoop Complex Interview Questions (Part 3 of 4)

These are complex Hadoop interview questions. This is my 3rd set of questions useful for your interviews. 1). What are the features of Standalone (local) mode? Ans). In stand-alone mode there are no daemons, everything runs on a single JVM. It has no DFS and utilizes the local file system. Stand-alone mode is suitable only for running MapReduce programs during development. It is one of the least used environments.

2). What are the features of Pseudo mode?
Ans). The pseudo mode is used both for development and in the QA environment. In the Pseudo mode, all the daemons run on the same machine.

3). Can we call VMs as pseudos?
Ans). No, VMs are not pseudos because VM is something different and pseudo is very specific to Hadoop.

4). What are the features of Fully Distributed mode?

Ans). The fully Distributed mode is used in the production environment, where we have ‘n’ number of machines forming a Hadoop cluster. Hadoop daemons run on a cluster of machines. There is one host onto which Namenod…

Tokenization story you need Vault based Vs Vault-less

The term tokenization refers to create a numeric or alphanumeric number in place of the original card number. It is difficult for hackers to get original card numbers.

Vault-Tokenization is a concept a Vault server create a new Token for each transaction when Customer uses Credit or Debit Card at Merchant outlets 
Let us see an example,  data analysis. Here, card numbers masked with other junk characters for security purpose.

Popular Tokenization ServersThere are two kinds of servers currently popular for implementing tokenization.
Vault-based Vault-less Video Presentation on Tokenization
Vault-based server The term vault based means both card number and token will be stored in a Table usually Teradata tables. During increasing volume of transactions, the handling of Table is a big challenge.
Every time during tokenization it stores a record for each card and its token. When you used a card multiple times, each time it generates multiple tokens. It is a fundamental concept.
So the challe…