Featured post

The Ultimate Cheat Sheet On Hadoop

Top 20 frequently asked questions to test your Hadoop knowledge given in the below Hadoop cheat sheet. Try finding your own answers and match the answers given here.




Question #1 

You have written a MapReduce job that will process 500 million input records and generate 500 million key-value pairs. The data is not uniformly distributed. Your MapReduce job will create a significant amount of intermediate data that it needs to transfer between mappers and reducers which is a potential bottleneck. A custom implementation of which of the following interfaces is most likely to reduce the amount of intermediate data transferred across the network?



A. Writable
B. WritableComparable
C. InputFormat
D. OutputFormat
E. Combiner
F. Partitioner
Ans: e




Question #2 

Where is Hive metastore stored by default ?


A. In HDFS
B. In client machine in the form of a flat file.
C. In client machine in a derby database
D. In lib directory of HADOOP_HOME, and requires HADOOP_CLASSPATH to be modified.
Ans: c




Question…
Post a Comment

6 Advantages of Columnar Databases over Traditional RDBMS

In traditional RDBMS, when a data source is accessed by multi users at single time, then database will go into deadlock state.

One of the advantages of a columnar model is that if two or more users want to use a different subset of columns, they do not have to lock out each other.

(Superior benefits for NoSQL Jobs)
        (Superior benefits for NoSQL Jobs)
This design is made easier because of a disk storage method known as RAID (redundant array of independent disks, originally redundant array of inexpensive disks), which combines multiple disk drives into a logical unit. Data is stored in several patterns called levels that have different amounts of redundancy. The idea of the redundancy is that when one drive fails, the other drives can take over. When a replacement disk drive in put in the array, the data is replicated from the other disks in the array and the system is restored.

The following are the various levels of RAID:

RAID 0 (block-level striping without parity or mirroring) has no (or zero) redundancy. It provides improved performance and additional storage but no fault tolerance. It is a starting point for discussion.

In RAID 1 (mirroring without parity or striping) data is written identically to two drives, thereby producing a mirrored set; the read request is serviced by either of the two drives containing the requested data, whichever one involves the least seek time plus rotational latency.

In RAID 10 (mirroring and striping) data is written in stripes across primary disks that have been mirrored to the secondary disks.

In RAID 2 (bit-level striping with dedicated Hamming-code parity) all disk spindle rotation is synchronized, and data is striped such that each sequential bit is on a different drive. Hamming-code parity is calculated across corresponding bits and stored on at least one parity drive. This theoretical RAID level is not used in practice.

In RAID 3 (byte-level striping with dedicated parity) all disk spindle rotation is synchronized, and data is striped so each sequential byte is on a different drive.

RAID 4 (block-level striping with dedicated parity) is equivalent to RAID 5 except that all parity data is stored on a single drive. In this arrangement, files may be distributed between multiple drives.

RAID 5, RAID 6, and other patterns exist; many of them are marketing terms more than technology. The goal is to provide fault tolerance of drive failures, up to n disk drive failures or removals from the array. This makes larger RAID arrays practical, especially for high-availability systems. While this is nice for database people, we get more benefit from parallelism for queries.

Comments

Post a Comment

Thanks for your message. We will get back you.

Popular posts from this blog

AWS Vs Azure Load Balancers Top Insights

Image
Amazon Web Services has three types of load balancers. And Azure is having two types of load balancers. This post tells you the top comparison between these two load balancers.


Amazon Web Services Load Balancer Types
Application Load balancer Network Load balancer Classic Load balancers.
Application Load Balancer

Balancing of HTTP and HTTPS traffic and provides advanced request routing targeted at the delivery of modern application architectures, including microservices and containers. 

Operating at the individual request level (Layer 7), Application Load Balancer routes traffic to targets within Amazon Virtual Private Cloud (Amazon VPC) based on the content of the request.

Network Load Balancer
 Network Load Balancer is best suited for load balancing of Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Transport Layer Security (TLS) traffic where extreme performance is required.

Classic Load BalancerClassic Load Balancer provides basic load balancing across multiple Amazo…
Read more

Hadoop File System Basic Commands

Image
List of basic commands to execute on the Hadoop File system given in this post. These are handy commands. Basically, you always need to remember.





The Syntax for Command and the List and Usage.
Hadoop fs -cmd <args>

cmd is a specific command and arg is the variable name.

cat

Hadoop fs –cat FILE [FILE …]

Displays the files' content. For reading compressed files.




chgrp

Hadoop fs –chgrp [-R] GROUP PATH [PATH …]

Changes the group association for files and directories. The – R option applies the change recursively. The user must be the files' owner or a superuser.



chmod



Hadoop fs –chmod [-R] MODE[,MODE …] PATH [PATH …]

Changes the permissions of files and directories. Like, its Unix equivalent, MODE can be a 3-digit octal mode, or {augo}+/-{rwxX}. The -R option applies the change recursively. The user must be the files' owner or a superuser.



chown



Hadoop fs –chown [-R] [OWNER][:[GROUP]] PATH [PATH…]



Changes the ownership of files and directories. The –R option applies the …
Read more

4 Important Skills You Need for Data Scientists

Image
Data science is a combination of technical and general skills. As an analyst, you are responsible to provide useful information to the client. Below are the useful list of skills.

Top Skills You Need for Data Scientist

1. Paradigms and practices. This involves data scientists acquiring a grounding in core concepts of data science, analytics, and data management. 
Data scientists should easily grasp the data science life cycle, know their typical roles and responsibilities in every phase, and be able to work in teams and with business domain experts and stakeholders. 
Also, they should learn a standard approach for establishing, managing, and operationalizing data science projects in the business.
2. Algorithms and modeling. Here are the areas with which data scientists must become familiar:
linear algebra, basic statistics, linear and logistic regression, data mining, predictive modeling, cluster analysis, association rules, market-basket analysis, decision trees, time-series analysis, f…
Read more

Hyperledger Fabric: 20 Real Interview Questions

Image
I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview.

Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications.
The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions1). What are Nodes?
In Hyperledger the communication entities are called Nodes.

2). What are the three different types of Nodes?
- Client Node
- Peer Node
- Order Node
The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery.

3). What is Channel?
A channel in Hyperledger is the subnet of the main blockchain. You c…
Read more