Skip to main content

Featured post

8 Popular Encryption methods real Usage

Encryption works as it replaces original data with some special characters and stores the data. You need a private key to decrypt the data. Else it is not possible to understand encrypted data even for hackers.

Private keys are two types. Sender uses one private key and receiver uses one other private key.
I have given in this post about 8 encryption methods.

Flow Diagram Encryption
 Different Encryption Methods Classic CipherRandom Key GenerationRSA TokenHash FunctionsMac value - Message Authentication codeKey HashTLS ProtocolAuthenticated Cipher  1. Classic Cipher The best example is this kind of algorithm was used in world war -II communication machines. 2. Random Key Generations One kind of security. They use an algorithm that sends random numbers. People use that number as a password.
3. RSA Token Every 60 seconds a token is generated. Mostly people use for financial applications. An extra identity mechanism along with user id and password. 4. Hash Functions It assigns a hash va…

Top 11 complex PIG Interview Questions to Read Now

PIG Questions

1). What is PIG?
PIG is a platform for analyzing large data sets that consist of high level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. PIG’s infrastructure layer consists of a compiler that produces sequence of MapReduce Programs.

2). What is the difference between logical and physical plans?
Pig undergoes some steps when a Pig Latin Script is converted into MapReduce jobs. After performing the basic parsing and semantic checking, it produces a logical plan. The logical plan describes the logical operators that have to be executed by Pig during execution. After this, Pig produces a physical plan. The physical plan describes the physical operators that are needed to execute the script.

3). Does ‘ILLUSTRATE’ run MR job?
No, illustrate will not pull any MR, it will pull the internal data. On the console, illustrate will not do any job. It just shows output of each stage and not the final output.

4). Is the keyword ‘DEFINE’ like a function name?
Yes, the keyword ‘DEFINE’ is like a function name. Once you have registered, you have to define it. Whatever logic you have written in Java program, you have an exported  jar and also a jar registered by you. Now the compiler will check the function in exported jar. When the function is not present in the library, it looks into your jar.

5). Is the keyword ‘FUNCTIONAL’ a User Defined Function (UDF)?
No, the keyword ‘FUNCTIONAL’ is not a User Defined Function (UDF). While using UDF, we have to override some functions. Certainly you have to do your job with the help of these functions only. But the keyword ‘FUNCTIONAL’ is a built-in function i.e a pre-defined function, therefore it does not work as a UDF.

6). Why do we need MapReduce during Pig programming?
Pig is a high-level platform that makes many Hadoop data analysis issues easier to execute. The language we use for this platform is: Pig Latin. A program written in Pig Latin is like a query written in SQL, where we need an execution engine to execute the query. So, when a program is written in Pig Latin, Pig compiler will convert the program into MapReduce jobs. Here, MapReduce acts as the execution engine.

7). Are there any problems which can only be solved by MapReduce and cannot be solved by PIG? In which kind of scenarios MR jobs will be more useful than PIG?

Let us take a scenario where we want to count the population in two cities. I have a data set and sensor list of different cities. I want  to count the population by using one mapreduce for two cities. Let us assume that one is Bangalore and the other is Noida. So I need to consider key of  Bangalore city  similar to Noida through which I can bring the population data of these two cities to one reducer. The idea behind this is some how I have to instruct map reducer program – whenever you find city with the name ‘Bangalore‘ and city with the name ‘Noida’,  you create the alias name which will be the common name for these two cities so that  you create a common key for both the cities and it get passed to the same reducer. For this, we have to write  custom partitioner.

  • In mapreduce when you create a ‘key’ for city,  you have to consider ’city’ as the key. So, whenever the framework comes across a different city, it considers it as a different key. Hence, we need to use customized partitioner. 
  • There is a provision in mapreduce only, where you can write your custom partitioner and mention if city = bangalore or noida then pass similar hashcode.  However, we cannot create custom partitioner in Pig. As Pig is not a framework, we cannot direct execution engine to customize the partitioner. In such scenarios, MapReduce works better than Pig.

8). Does Pig give any warning when there is a type mismatch or missing field?
No, Pig will not show any warning if there is no matching field or a mismatch. If you assume that Pig gives such a warning, then it is difficult to find in log file. If any mismatch is found, it assumes a null value in Pig.

9). What co-group does in Pig?
Co-group joins the data set by grouping one particular data set only. It groups the elements by their common field and then returns a set of records containing two separate bags. The first bag consists of the record of the first data set with the common  data set and the second bag consists of the records of the second data set with the common data set.

10). Can we say cogroup is a group of more than 1 data set?
Cogroup is a group of one data set. But in the case of more than one data sets, cogroup will group all the data sets and join them based on the common field. Hence, we can say that cogroup is a group of more than one data set and join of that data set as well.

11). What does FOREACH do?

FOREACH is used to apply transformations to the data and to generate new data items. The name itself is indicating that for each element of a data bag, the respective action will be performed.

Syntax : FOREACH bagname GENERATE expression1, expression2, …..

The meaning of this statement is that the expressions mentioned after GENERATE will be applied to the current record of the data bag.
Bonus Question:

What is bag?

A bag is one of the data models present in Pig. It is an unordered collection of tuples with possible duplicates. Bags are used to store collections while grouping. The size of bag is the size of the local disk, this means that the size of the bag is limited. When the bag is full, then Pig will spill this bag into local disk and keep only some parts of the bag in memory. There is no necessity that the complete bag should fit into memory. We represent bags with “{}”.

Related:


Comments

Popular posts from this blog

Python for Loop Print rows using List concept

For loop is crucial for your python programming. Out of all the For Loop options, the list option is so popular. If you know, how to use this concept, you can write any number of programs quickly and with less effort.

Using for loop you can get index for list items. You can print list elements using for loop.

If you take any language, the loops play a vital role.

The use of For Loop When you want to deal with data of multiple values, you need loops. So, For Loop is so famous because, you can control multiple input values.

Syntax For Loop for i in my_list: print(i)
The 'i' says variable that represents value in the 'my_list'. The for helps to iterate the list values one by one.
Example Program For Loop List in Python my_list=[2,3,4] for i in my_list: print(i)
Result 2 3 4 How to create Multiplication Table using Python with For Loop and List option Result of function that gener…

8 Popular Encryption methods real Usage

Encryption works as it replaces original data with some special characters and stores the data. You need a private key to decrypt the data. Else it is not possible to understand encrypted data even for hackers.

Private keys are two types. Sender uses one private key and receiver uses one other private key.
I have given in this post about 8 encryption methods.

Flow Diagram Encryption
 Different Encryption Methods Classic CipherRandom Key GenerationRSA TokenHash FunctionsMac value - Message Authentication codeKey HashTLS ProtocolAuthenticated Cipher  1. Classic Cipher The best example is this kind of algorithm was used in world war -II communication machines. 2. Random Key Generations One kind of security. They use an algorithm that sends random numbers. People use that number as a password.
3. RSA Token Every 60 seconds a token is generated. Mostly people use for financial applications. An extra identity mechanism along with user id and password. 4. Hash Functions It assigns a hash va…

Blue Prism complete tutorials download now

Blue prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture and creating new bots. Useful if you are a new learner and trying to enter into automation career.
The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews.
RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation.

RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time.

The Books Available on Blue Prism 
Blue Prism resourcesDavid chappal PDF bookBlue Prism Blogs

Hyperledger Fabric Real Interview Questions Read Today

I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…

Block storage vs object storage real differences

Interesting questions, I am giving on the two different types of storage popular in Amazon aws. Those are Block and Object.
Object vs Block Storage in AWS Why these names are different. You need to understand storage available in AWS. You alreday know that AWS is leading provider in cloud computing.
Object StorageObject means it is a single object. You are not dividing here.In the context of AWS, object storage helps your file to store as -is. How big it is does not matter.Suppose, your file size is 10MB, it stores as 10 MB file.What happens when you update a 30MB file, simply, it deletes old object and create a brand new object.Even, for small change, you need to update whole file. So, it utlilizes lot of resources. Object storage is much better for big file and very few changes.The object storage is well managed by AWS .AWS has full control over Object storage.Block StorageBlock storage divides your file into blocksYou have selected a block size as 512 bytes, then you want to upload…

Three popular RPA tools functional differences

Robotic process automation is growing area and many IT developers across the board started up-skill in this popular area. I have written this post for the benefit of Software developers who are interested in RPA also called Robotic Process Automation.


In my previous post, I have described that total 12 tools are available in the market. Out of those 3 tools are most popular. Those are Automation anywhere, BluePrism and Uipath. Many programmers asked what are the differences between these tools. I have given differences of all these three RPA tools.

BluePrism Blue Prism has taken a simple concept, replicating user activity on the desktop, and made it enterprise strength. The technology is scalable, secure, resilient, and flexible and is supported by a comprehensive methodology, operational framework and provided as packaged software.The technology is developed and deployed within a “corridor of IT governance” and has sophisticated error handling and process modelling capabilities to ens…

AWS EC2 Real Story on Elastic Cloud Computing

The short name for Amazon Elastic Computing Cloud is EC2. You can keep this point as an interview question. The computing capacity has an elastic property. Based on your requirement you can increase or decrease computing power. I am giving in this AWS EC2 real story for your quick reference.
You need to be very attentive when you enable Auto scaling feature. It is a responsibility on Admins. Amazon AWS EC2Making your existing hardware to the requirement, always is not so easy. So the EC2 feature in AWS helps you to allocate computing power according to your needs. AWS EC2 instance acts as your physical server.It has memory.You can increase the instance size in terms of CPU, Memory, Storage and GPU.EC2 auto scaling is a property, where it automatically increase your computing power. List of Top Security Features in EC2  1#. Virtual Private CloudThe responsibility of Virtual Private Cloud, is to safeguard each instance separately. That means, you cannot access others instance, which is al…

Python Syntax Rules Eliminate Errors Before you start debugging

In Python, if you know syntax rules, you can eliminate errors. The basic mistakes programmers do are missing semicolons, adding extra commas, and extra spaces. Python is case sensitive. So using the wrong identifier gives an error.
Indentation is unique to Python. You cannot find this kind of rule in any other programming languages Python Syntax Cheat Sheet These are the main areas you need to focus while writing a Python program. You need to learn rules. Else you need to waste a lot of time fixing the issues or errors.
Indentation or Syntax ErrorsExceptionsHandling Exceptions
1. Indentation If you do not follow proper order, you will get an error. The details of one block shroud follow in one vertical line. The sub-block should be inside of that.

In if loop, the if, elif, and else should have same indentation. Not only, the statement inside of them should have same indentation.Understand these examples a good material on indentation for you.   2. Exceptions  Python raises exception, wh…