Skip to main content

Featured post

8 Popular Encryption methods real Usage

Encryption works as it replaces original data with some special characters and stores the data. You need a private key to decrypt the data. Else it is not possible to understand encrypted data even for hackers.

Private keys are two types. Sender uses one private key and receiver uses one other private key.
I have given in this post about 8 encryption methods.

Flow Diagram Encryption
 Different Encryption Methods Classic CipherRandom Key GenerationRSA TokenHash FunctionsMac value - Message Authentication codeKey HashTLS ProtocolAuthenticated Cipher  1. Classic Cipher The best example is this kind of algorithm was used in world war -II communication machines. 2. Random Key Generations One kind of security. They use an algorithm that sends random numbers. People use that number as a password.
3. RSA Token Every 60 seconds a token is generated. Mostly people use for financial applications. An extra identity mechanism along with user id and password. 4. Hash Functions It assigns a hash va…

Top 100 Hadoop Complex Interview Questions (Part 3 of 4)

These are complex Hadoop interview questions. This is my 3rd set of questions useful for your interviews.

hadoop part 3

 1). What are the features of Standalone (local) mode?

Ans). In stand-alone mode there are no daemons, everything runs on a single JVM. It has no DFS and utilizes the local file system. Stand-alone mode is suitable only for running MapReduce programs during development. It is one of the least used environments.

2). What are the features of Pseudo mode?
Ans). The pseudo mode is used both for development and in the QA environment. In the Pseudo mode, all the daemons run on the same machine.

3). Can we call VMs as pseudos?
Ans). No, VMs are not pseudos because VM is something different and pseudo is very specific to Hadoop.

4). What are the features of Fully Distributed mode?

Ans). The fully Distributed mode is used in the production environment, where we have ‘n’ number of machines forming a Hadoop cluster. Hadoop daemons run on a cluster of machines. There is one host onto which Namenode is running and another host on which data node is running and then there are machines on which task tracker is running. We have separate masters and separate slaves in this distribution.

5). Does Hadoop follow the UNIX pattern?
Ans). Yes, Hadoop closely follows the UNIX pattern. Hadoop also has the ‘conf‘ directory as in the case of UNIX.

6). In which directory Hadoop is installed?
Ans). Cloudera and Apache have the same directory structure. Hadoop is installed in cd /usr/lib/hadoop-0.20/.

7). What are the port numbers of Namenode, job tracker and task tracker?
Ans). The port number for Namenode is ’70′, for job tracker is ’30′ and for task tracker is ’60′.

8). What is the Hadoop-core configuration?
Hadoop core is configured by two xml files:
1. hadoop-default.xml which was renamed to 
2. hadoop-site.xml.These files are written in xml format. We have certain properties in these xml files, which consist of name and value. But these files do not exist now.

9). What are the Hadoop configuration files at present?
There are 3 configuration files in Hadoop:1. core-site.xml2. hdfs-site.xml3. mapred-site.xmlThese files are located in the conf/ subdirectory.

10). How to exit the Vi editor?
Ans). To exit the Vi Editor, press ESC and type :q and then press enter.

11). What is a spill factor with respect to the RAM?
Ans). Spill factor is the size after which your files move to the temp file. Hadoop-temp directory is used for this.

12). Is fs.mapr.working.dir a single directory?
Ans). Yes, fs.mapr.working.dir it is just one directory.

13). Which are the three main hdfs-site.xml properties?
Ans). The three main hdfs-site.xml properties are:1. dfs.name.dir which gives you the location on which metadata will be stored and where DFS is located – on disk or onto the remote.2. dfs.data.dir which gives you the location where the data is going to be stored.3. fs.checkpoint.dir which is for secondary Namenode.

14). How to come out of the insert mode?
Ans). To come out of the insert mode, press ESC, type :q (if you have not written anything) OR type :wq (if you have written anything in the file) and then press ENTER.

15). What is Cloudera and why it is used?
Ans). Cloudera is the distribution of Hadoop. It is a user created on VM by default. Cloudera belongs to Apache and is used for data processing.

14). What happens if you get a ‘connection refused java exception’ when you type Hadoop fsck /?
Ans). It could mean that the Namenode is not working on your VM.

15). We are using the Ubuntu operating system with Cloudera, but from where we can download Hadoop or does it come by default with Ubuntu?
Ans). This is a default configuration of Hadoop that you have to download from Cloudera or from Eureka's dropbox and the run it on your systems. You can also proceed with your own configuration but you need a Linux box, be it Ubuntu or Red Hat. There are installation steps present at the Cloudera location or in Eureka's Dropbox. You can go either way.

16). What does ‘jps’ command do?
Ans). This command checks whether your Namenode, data node, task tracker, job tracker, etc are working or not.

17). How can I restart Namenode?
Ans). Click on stop-all.sh and then click on start-all.sh OR2. Write sudo hdfs (press enter), su-hdfs (press enter), /etc/init.d/ha (press enter) and then /etc/init.d/hadoop-0.20-namenode start (press enter).

18). What is the full form of fsck?
Ans). Full form of fsck is File System Check.

19). How can we check whether Namenode is working or not?
Ans). To check whether Namenode is working or not, use the command /etc/init.d/hadoop-0.20-namenode status or as simple as jps.

20). What does the command mapred.job.tracker do?
The command mapred.job.tracker lists out which of your nodes is acting as a job tracker.What does /etc /init.d do?/etc /init.d specifies where daemons (services) are placed or to see the status of these daemons. It is very LINUX specific, and nothing to do with Hadoop.

21). How can we look for the Namenode in the browser?
Ans). If you have to look for Namenode in the browser, you don’t have to give localhost:8021, the port number to look for Namenode in the browser is 50070.

22). How to change from SU to Cloudera?
Ans). To change from SU to Cloudera just type exit.

23). Which files are used by the startup and shutdown commands?
Ans). Slaves and Masters are used by the startup and the shutdown commands.

24). What do slaves consist of?
Ans). Slaves consist of a list of hosts, one per line, that host data node and task tracker servers.

25). What do masters consist of?
Ans). Masters contain a list of hosts, one per line, that are to host secondary name node servers.

26). What does hadoop-env.sh do?
Ans). hadoop-env.sh provides the environment for Hadoop to run. JAVA_HOME is set over here.

27). Can we have multiple entries in the master files?
Ans). Yes, we can have multiple entries in the Master files.

28). Where is hadoop-env.sh file present?
Ans). hadoop-env.sh file is present in the conf location.

29). In Hadoop_PID_DIR, what does PID stands for?
Ans). PID stands for ‘Process ID’.What does /var/hadoop/pids do?It stores the PID.

30). What does hadoop-metrics.properties file do?
Ans). hadoop-metrics.properties is used for ‘Reporting‘ purposes. It controls the reporting for Hadoop. The default status is ‘not to report‘.

31). What are the network requirements for Hadoop?
Ans). The Hadoop core uses Shell (SSH) to launch the server processes on the slave nodes. It requires password-less SSH connection between the master and all the slaves and the secondary machines.

32). Why do we need a password-less SSH in a Fully Distributed environment?
Ans). We need a password-less SSH in a Fully-Distributed environment because when the cluster is LIVE and running in a fully distributed environment, the communication is too frequent. The job tracker should be able to send a task to task tracker quickly.

33). Does this lead to security issues?
Ans). No, not at all. Hadoop cluster is an isolated cluster. And generally, it has nothing to do with the internet. It has a different kind of a configuration. We needn’t worry about that kind of a security breach, for instance, someone hacking through the internet, and so on. Hadoop has a very secure way to connect to other machines to fetch and to process data.

34). On which port does SSH work?
Ans). SSH works on Port No. 22, though it can be configured. 22 is the default Port number.Can you tell us more about SSH?SSH is nothing but a secure shell communication, it is a kind of a protocol that works on a Port No. 22, and when you do an SSH, what you really require is a password.

35). Why password is needed in SSH localhost?
Ans). The password is required in SSH for security and in a situation where password-less communication is not set. Do we need to give a password, even if the key is added in SSH? Yes, the password is still required even if the key is added in SSH.

36). What if a Namenode has no data?
Ans) If a Namenode has no data it is not a Namenode. Practically, Namenode will have some data.

37). What happens to job tracker when Namenode is down?
Ans). When Namenode is down, your cluster is OFF, this is because Namenode is the single point of failure in HDFS.

38). What happens to a Namenode, when job tracker is down?
Ans). When a job tracker is down, it will not be functional but Namenode will be present. So, the cluster is accessible if Namenode is working, even if the job tracker is not working.

39). Can you give us some more details about SSH communication between Masters and the Slaves?
Ans). SSH is a password-less secure communication where data packets are sent across the slave. It has some format into which data is sent across. SSH is not only between masters and slaves but also between two hosts.

40). What is the formatting of the DFS?
Ans). Just like we do for Windows, DFS is formatted for proper structuring. It is not usually done as it formats the Namenode too.

41). Does the HDFS client decide the input split or Namenode?
Ans). No, the Client does not decide. It is already specified in one of the configurations through which input split is already configured.

42). In Cloudera there is already a cluster, but if I want to form a cluster on Ubuntu can we do it?
Ans). Yes, you can go ahead with this! There are installation steps for creating a new cluster. You can uninstall your present cluster and install the new cluster.

43). Can we create a Hadoop cluster from scratch?
Ans). Yes, we can do that also once we are familiar with the Hadoop environment.

44). Can we use Windows for Hadoop?
Ans). Actually, Red Hat Linux or Ubuntu is the best Operating Systems for Hadoop. Windows is not used frequently for installing Hadoop as there are many support problems attached to Windows. Thus, Windows is not a preferred environment for Hadoop.

Related Posts

Comments

Popular posts from this blog

Python for Loop Print rows using List concept

For loop is crucial for your python programming. Out of all the For Loop options, the list option is so popular. If you know, how to use this concept, you can write any number of programs quickly and with less effort.

Using for loop you can get index for list items. You can print list elements using for loop.

If you take any language, the loops play a vital role.

The use of For Loop When you want to deal with data of multiple values, you need loops. So, For Loop is so famous because, you can control multiple input values.

Syntax For Loop for i in my_list: print(i)
The 'i' says variable that represents value in the 'my_list'. The for helps to iterate the list values one by one.
Example Program For Loop List in Python my_list=[2,3,4] for i in my_list: print(i)
Result 2 3 4 How to create Multiplication Table using Python with For Loop and List option Result of function that gener…

8 Popular Encryption methods real Usage

Encryption works as it replaces original data with some special characters and stores the data. You need a private key to decrypt the data. Else it is not possible to understand encrypted data even for hackers.

Private keys are two types. Sender uses one private key and receiver uses one other private key.
I have given in this post about 8 encryption methods.

Flow Diagram Encryption
 Different Encryption Methods Classic CipherRandom Key GenerationRSA TokenHash FunctionsMac value - Message Authentication codeKey HashTLS ProtocolAuthenticated Cipher  1. Classic Cipher The best example is this kind of algorithm was used in world war -II communication machines. 2. Random Key Generations One kind of security. They use an algorithm that sends random numbers. People use that number as a password.
3. RSA Token Every 60 seconds a token is generated. Mostly people use for financial applications. An extra identity mechanism along with user id and password. 4. Hash Functions It assigns a hash va…

Blue Prism complete tutorials download now

Blue prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career.
The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews.
RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation.

RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism Blogs

Hyperledger Fabric Real Interview Questions Read Today

I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…

Block storage vs object storage real differences

Interesting questions, I am giving on the two different types of storage popular in Amazon aws. Those are Block and Object.
Object vs Block Storage in AWS Why these names are different. You need to understand storage available in AWS. You alreday know that AWS is leading provider in cloud computing.
Object StorageObject means it is a single object. You are not dividing here.In the context of AWS, object storage helps your file to store as -is. How big it is does not matter.Suppose, your file size is 10MB, it stores as 10 MB file.What happens when you update a 30MB file, simply, it deletes old object and create a brand new object.Even, for small change, you need to update whole file. So, it utlilizes lot of resources. Object storage is much better for big file and very few changes.The object storage is well managed by AWS .AWS has full control over Object storage.Block StorageBlock storage divides your file into blocksYou have selected a block size as 512 bytes, then you want to upload…

Three popular RPA tools functional differences

Robotic process automation is growing area and many IT developers across the board started up-skill in this popular area. I have written this post for the benefit of Software developers who are interested in RPA also called Robotic Process Automation.


In my previous post, I have described that total 12 tools are available in the market. Out of those 3 tools are most popular. Those are Automation anywhere, BluePrism and Uipath. Many programmers asked what are the differences between these tools. I have given differences of all these three RPA tools.

BluePrism Blue Prism has taken a simple concept, replicating user activity on the desktop, and made it enterprise strength. The technology is scalable, secure, resilient, and flexible and is supported by a comprehensive methodology, operational framework and provided as packaged software.The technology is developed and deployed within a “corridor of IT governance” and has sophisticated error handling and process modelling capabilities to ens…

AWS EC2 Real Story on Elastic Cloud Computing

The short name for Amazon Elastic Computing Cloud is EC2. You can keep this point as an interview question. The computing capacity has an elastic property. Based on your requirement you can increase or decrease computing power. I am giving in this AWS EC2 real story for your quick reference.
You need to be very attentive when you enable Auto scaling feature. It is a responsibility on Admins. Amazon AWS EC2Making your existing hardware to the requirement, always is not so easy. So the EC2 feature in AWS helps you to allocate computing power according to your needs. AWS EC2 instance acts as your physical server.It has memory.You can increase the instance size in terms of CPU, Memory, Storage and GPU.EC2 auto scaling is a property, where it automatically increase your computing power. List of Top Security Features in EC2  1#. Virtual Private CloudThe responsibility of Virtual Private Cloud, is to safeguard each instance separately. That means, you cannot access others instance, which is al…

Python Syntax Rules Eliminate Errors Before you start debugging

In Python, if you know syntax rules, you can eliminate errors. The basic mistakes programmers do are missing semicolons, adding extra commas, and extra spaces. Python is case sensitive. So using the wrong identifier gives an error.
Indentation is unique to Python. You cannot find this kind of rule in any other programming languages Python Syntax Cheat Sheet These are the main areas you need to focus while writing a Python program. You need to learn rules. Else you need to waste a lot of time fixing the issues or errors.
Indentation or Syntax ErrorsExceptionsHandling Exceptions
1. Indentation If you do not follow proper order, you will get an error. The details of one block shroud follow in one vertical line. The sub-block should be inside of that.

In if loop, the if, elif, and else should have same indentation. Not only, the statement inside of them should have same indentation.Understand these examples a good material on indentation for you.   2. Exceptions  Python raises exception, wh…