Data lake Repository You Need to Know About it

-

In DataLake data stored internally in a repository. You can say this format as a blob. The data in DataLake does not have a particular Schema or Format.

data lake repository example
Photo Credit: Srini

SQL Database

  • Let us take a traditional database, here, a database design and Scheme are to be defined before you enter data. In data-lake, there is no format for this. It is like a dump. 
  • This dump you can send to Hadoop repository for data analysis. This repository can be incremental. Also, you can build a large database.

Data lake Vs Hadoop

Data-Lake is a dump of data with no format. There are many pre-formats required before it sends for analytics. One is data security and encryption. These techniques to be done before you send your data to Hadoop repository.

In real-time, Hadoop data analytics need lot other pre-processing of data required to proceed further.

Comments

Post a Comment

Thanks for your message. We will get back you.

Popular Posts

7 AWS Interview Questions asked in Infosys, TCS

-
Image
I have shared 7 amazing AWS interview questions for your quick reference. 1). What is AWS? Amazon Web Services (AWS) provides on-demand computing resources and services in the cloud, with pay-as-you-go pricing. For example, you can run a server on AWS that you can log on to, configure, secure, and run just as you would a server that's sitting in front of you 2). What you can do with AWS? Store public or private data. Host a static website. These websites use client-side technologies (such as HTML, CSS, and JavaScript) to display content that doesn't change frequently. A static website doesn't require server-side technologies (such as PHP and ASP.NET). Host a dynamic website or web app. These websites include classic three-tier applications, with web, application, and database tiers. Support students or online training programs. Process business and scientific data. Handle peak loads. AWS Questions 3). What is the Security group? A security group acts as a virtual fi
Read more

Hyperledger Fabric: 20 Real Interview Questions

-
Image
I am practicing Hyperledger. This is one of the top listed blockchains. This architecture follows R3 Corda specifications. Sharing the interview questions with you that I have prepared for my interview. Though Ethereum leads in the real-time applications. The latest Hyperledger version is now ready for production applications. It has now become stable for production applications. The Hyperledger now backed by IBM. But, it is still an open source. These interview questions help you to read quickly. The below set of interview questions help you like a tutorial on Hyperledger fabric. Hyperledger Fabric Interview Questions 1). What are Nodes? In Hyperledger the communication entities are called Nodes. 2). What are the three different types of Nodes? - Client Node - Peer Node - Order Node The Client node initiates transactions. The peer node commits the transaction. The order node guarantees the delivery. 3). What is Channel? A channel in Hyperledger is the subnet of
Read more

How to Fix Python Syntax Errors Quickly

-
Image
In Python, you can call parsing errors Syntax errors. How to fix the syntax errors quickly, I can show in this post. Also, check Python Syntax Errors Cheat Sheet. Python Syntax Error. If you run the Python code, the interpreter first parses the logic. If any errors found, it is the developer's responsibility to fix them. Courtesy: Python documentation. >>> while True print('Hello world')   File "<stdin>", line 1     while True print('Hello world')                    ^ SyntaxError: invalid syntax How to Fix the Syntax Error First, check for the line number. In the above case, it is '1'. Then look for the symbol '^'. The issue is at the PRINT function. The reason is before the PRINT function you need a colon ':'. That is the reason, you are getting error. This is a syntax error. Find below the corrected logic: >>> while True : print('Hello world')   File "<stdin>", line 1     while Tr
Read more

Python 'getsizeof' Command the Real Purpose

-
Image
In Python, there are many objects. Out of those List (arrays) is one type.  To get the number of elements present in a List is not easy. So how to get this? To do you need the 'getsizeof' command. Import 'sys' Library. You need to import the 'sys' library to use the SIZE commands.  >>> import sys Congrats to you since you have now imported the SYS library. The command name is " getsizeof" . Check the below example. print("An Integer:") i = int(100) print(sys.getsizeof(i)) The result will be: 100 What is the size command?  - is to get the size of elements. It gives the result based on the object you are being used. Suppose, if you use Strings, you will get String length. How to use the  'getsizeof' command? Example:1 print("Empty string:") empty_string = "" print(sys.getsizeof(empty_string)) Example:2 print("A normal string") s = "Hello world" print(sys.getsizeof(s)) Example:3 print(&quo
Read more

How to Check Log File in Kafka

-
Image
In Kafka, the log file's function is to store entries. Here, you can find entries for the producer's incoming messages. These log files you can call as topics. Again topics are divided into partitions. IN THIS PAGE Kafka Logs How Producer Messages Store Benefits of Kafka Logs How to check Logs in Kafka 1. Kafka Logs The mechanism underlying Kafka is the log. Most software engineers are familiar with this. It tracks what an application is doing.  If you have performance issues or errors in your application, the first place to check is the application logs. But it is a different sort of log.  In the context of Kafka (or any other distributed system), a log is "an append-only, totally ordered sequence of records - ordered by time. 2. How Producer Messages Store The producer writes the messages to Kafka, the records store in a log file. The records sored as 0,1,2,3 and so on. 4. Benefits of Kafka Logs Logs are a simple data abstraction with powerful implications. If you have r
Read more

5 HBase Vs. RDBMS Top Functional Differences

-
Image
HBASE in the Big data context has a lot of benefits over RDBMS. Clear difference makes you feel great why you need HBASE in Hadoop or Bigdata platform. Let us check one by one quickly. HBASE Vs. RDBMS HBase handles a large amount of data that is store in a distributed manner in the column-oriented format while RDBMS is systematic storage of a database that cannot support a random manner for accessing the database. RDBMS strictly follow Codd's 12 rules with fixed schemas and row-oriented manner of database and also follow ACID properties, while HBase follows BASE properties and implement complex queries. Secondary indexes, complex inner and outer joins, count, sum, sort, group, and data of page and table can easily be accessible by RDBMS. From small to medium storage application there is the use of RDBMS that provide the solution with MySQL and PostgreSQL whose size increase with concurrency and performance. Codd's rules always need to keep in mind while extending the size of th
Read more

Python Dictionary Vs List With Examples

-
Image
Dictionary and List we use interchangeably in Python to store values. For beginners, both look the same. In reality, they both differ. I want to share some key differences between Python Dictionary and List . Python Dictionary Vs List Top Differences. Values in lists are accessed by means of integers called indices, which indicate where in the list a given value is found. Dictionaries access values by means of integers, strings, or other Python objects called keys , which indicate where in the dictionary a given value is found.  In other words, both lists and dictionaries provide indexed access to arbitrary values, but the set of items that can be used as dictionary indices is much larger than, and contains, the set of items that can be used as list indices.  Also, the mechanism that dictionaries use to provide indexed access is quite different from that used by lists. Both lists and dictionaries can store objects of any type. Values stored in a list are implicitly ordered by their p
Read more

Blue Prism complete tutorials download now

-
Image
Blue Prism is an automation tool useful to execute repetitive tasks without human effort. To learn this tool you need the right material. Provided below quick reference materials to understand detailed elements, architecture, and creating new bots . Useful if you are a new learner and trying to enter into an automation career. The number one and most popular tool in automation is a Blue prism. In this post, I have given references for popular materials and resources so that you can use for your interviews. RPA Blue Prism RPA blue prism tutorial popular resources I have given in this post. You can download it quickly. Learning Blue Prism is a really good option if you are a learner of Robotic process automation . RPA Advantages The RPA is also called "Robotic Process Automation"- Real advantages are you can automate any business process and you can complete the customer requests in less time. The Books Available on Blue Prism  Blue Prism resources David chap
Read more

Linux Relative Vs. Absolute Path Top Differences

-
Image
Linux directories are helpful to store files. Many a time, the programmer would trade in directory paths. The paths are two types - absolute and relative. I have shared simple ideas to remember the differences between these two. Linux Absolute Vs. Relative Path in Linux Below are the differences between these two. 1. Absolute pathname An example is /usr/lib, which is an exact directory in the directory tree.  Think of the absolute pathname as being the complete mailing address for a package that the postal service will deliver to your next-door neighbor. 2. Relative pathname An example is cups, which represents the cups subdirectory of the current directory, whatever that may be.  Think of the relative directory name as giving the postal carrier directions from your house to the one next door so that the carrier can deliver the package. Useful differences, how to differentiate the differences between absolute and real paths. How to go back to the home directory If I type  cd cups  in
Read more

How to Use the ps Command in Linux

-
Image
The real use of the ps command is to know the process details. In fact, along with process details, you would know many other parameters about the process. How many processes are running in a current Session you will get from the below examples. ps command Linux ps Command in Linux PID (Process Id) - It is the process identification number. TTY(Terminal name) - associated with this process TIME(Time) - the format is hh: mm: ss (Hours, Minutes, and Seconds). Cumulated time of that particular process. CMD (Executable Command name) I will show you an example, which is very basic, how it appears when you issue the  ps command. ps command You May Also Like: PS Command Options in Linux What is Slave/Master Terminal in Linux The pts/0 is the slave terminal. The  pts  means pseudo terminal slave. Raise your knowledge by bringing in these quick definitions.  LINUX - DIFFERENCE BETWEEN TTY AND PTS. In reality, the terminals are two types. One is ptmx (Master) and pts (Slave). The master termi
Read more