Featured Post

SQL Interview Success: Unlocking the Top 5 Frequently Asked Queries

-
Image
 Here are the five top commonly asked SQL queries in the interviews. These you can expect in Data Analyst, or, Data Engineer interviews. Top SQL Queries for Interviews 01. Joins The commonly asked question pertains to providing two tables, determining the number of rows that will return on various join types, and the resultant. Table1 -------- id ---- 1 1 2 3 Table2 -------- id ---- 1 3 1 NULL Output ------- Inner join --------------- 5 rows will return The result will be: =============== 1  1 1   1 1   1 1    1 3    3 02. Substring and Concat Here, we need to write an SQL query to make the upper case of the first letter and the small case of the remaining letter. Table1 ------ ename ===== raJu venKat kRIshna Solution: ========== SELECT CONCAT(UPPER(SUBSTRING(name, 1, 1)), LOWER(SUBSTRING(name, 2))) AS capitalized_name FROM Table1; 03. Case statement SQL Query ========= SELECT Code1, Code2,      CASE         WHEN Code1 = 'A' AND Code2 = 'AA' THEN "A" | "A
Post a Comment

5 Top Data warehousing Skills in the age of Big data

-
5 Top Data warehousing Skills in the age of Big data
#5 Top Data warehousing Skills in the age of Big data:
A data warehouse is a home for "secondhand" data that originates in either other corporate applications, such as the one your company uses to fill customer orders for its products, or some data source external to your company, such as a public database that contains sales information gathered from all your competitors.

What is Data warehousing

If your company's data warehouse were advertised as a used car, for example, it may be described this way: "Contains late-model, previously owned data, all of which has undergone a 25-point quality check and is offered to you with a brand-new warranty to guarantee hassle-free ownership."

Most organizations build a data warehouse in a relatively straightforward manner:
  • The data warehousing team selects a focus area, such as tracking and reporting the company's product sales activity against that of its competitors.
  • The team in charge of building the data warehouse assigns a group of business users and other key individuals within the company to play the role of Subject-Matter Experts. Together, these people compile a list of different types of data that enable them to use the data warehouse to help track sales activity (or whatever the focus is for the project).
  • The group then goes through the list of data, item by item, and figures out where it can obtain that particular piece of information. In most cases, the group can get it from at least one internal (within the company) database or file, such as the one the application uses to process orders by mail or the master database of all customers and their current addresses. In other cases, a piece of information is not available from within the company's computer applications but could be obtained by purchasing it from some other company. Although the credit ratings and total outstanding debt for all of a bank's customers, for example, aren't known internally, that information can be purchased from a credit bureau.
  • After completing the details of where each piece of data comes from, the data warehousing team (usually computer analysts and programmers) create extraction programs. These programs collect data from various internal databases and files, copy certain data to a staging area (a work area outside the data warehouse), ensure that the data has no errors, and then copy it all into the data warehouse. Extraction programs are created either by hand (custom-coded) or by using specialized data warehousing products.
Different roles in Data warehousing projects:

Data modeling.: Design and implementation of data models are required for both the integration and presentation repositories. Relational data models are distinctly different from dimensional data models, and each has unique properties. Moreover, relational data modelers may not have dimensional modeling expertise and vice versa.

ETL development: ETL refers to the extraction of data from source systems into staging, the transformations necessary to recast source data for analysis, and the loading of transformed data into the presentation repository. ETL includes the selection criteria to extract data from source systems, performing any necessary data transformations or derivations needed, data quality audits, and cleansing.

Data cleansing: Source data is typically not perfect. Furthermore, merging data from multiple sources can inject new data quality issues. Data hygiene is an important aspect of data warehouse that requires specific skills and techniques.

OLAP design: Typically data warehouses support some variety of online analytical processing (HOLAP, MOLAP, or ROLAP). Each OLAP technique is different but requires special design skills to balance the reporting requirements against performance constraints.

Application development: Users commonly require an application interface into the data warehouse that provides an easy-to-use front end combined with comprehensive analytical capabilities, and one that is tailored to the way the users work. This often requires some degree of custom programming or commercial application customization.

Production automation: Data warehouses are generally designed for periodic automated updates when new and modified data is slurped into the warehouse so that users can view the most recent data available. These automated update processes must have built-in fail-over strategies and must ensure data consistency and correctness.

General systems and database administration: Data warehouse developers must have many of the same skills held by the typical network administrator and database administrator. They must understand the implications of efficiently moving possibly large volumes of data across the network, and the issues of effectively storing changing data.

Comments

Post a Comment

Thanks for your message. We will get back you.

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

-
Image
Here's a quick resolution for import datetime Python error . The reason is your .py python script name and datetime  are the same. I'll show you how this error happens and its resolution.   Here's the Resolution for ImportError I've created a script called 'datetime.py.' to check whether the minute value is 'odd' or not. During the import of my python script, I got the import error cannot import name datetime. My Script: datetime.py My script's intention is to find whether the minute value is odd or not. Python Logic from datetime import datetime odds = [ 1 , 3 , 5 , 7 , 9 , 11 , 13 , 15 , 17 , 19 , 21 , 23 , 25 , 27 , 29 , 31 , 33 , 35 , 37 , 39 , 41 , 43 , 45 , 47 , 49 , 51 , 53 , 55 , 57 , 59 ] right_this_minute = datetime.today().minute if right_this_minute in odds: print ( "This minute seems a little odd." ) else : print ( "Not an odd minute." ) ImportError I have imported my datetime.py from Linux. It gives an error
Read more

Explained Ideal Structure of Python Class

-
Image
When you are designing a class, you need to ensure that the classification of its critical parts is outlined at the beginning. The clearer the initial design, the more performant and scalable the class is. Some of the components in the order in which they should be defined in the class are mentioned as follows. Ideal structure of a class Class variables Constants or default variables are usually defined at the top of the class. For someone who is reading the code, it comes as an easy-to-view consolidated list, and for the interpreter it ensures that all such variables are processed before diving into the main logic of the class, including any other Instance method or constructor. The __init__ method The __init__ method provides information about inputs needed and how to instantiate the class. It is also the constructor of the class, which the very first method called while initializing the class . Special Python methods These methods change the functionality of the class or provide add
Read more

How to Check Kafka Available Brokers

-
Image
Here's the command to check the list of brokers present in a Kafka Cluster. You can say the Broker is the heart of the Kafka cluster. In simple terms, it works as a process. The main function is to receive messages from publishers and gives permission to access messages by consumers. How to Check Available Brokers in Kafka Here is the command: linux$  ./zookeeper-shell.sh zookeeper-IPaddress:2181 | "ls /brokers/ids" Just use the above command to get the number of brokers present in your host (Kafka Cluster). What is Default Broker-id in Kafka  The default broker id is -1 . When you create a new broker, it adds to -1. Then, it gives a new broker id. The broker ids will be generated from  reserved.broker.max.id + 1. Types of broker ids. Use assigned Zookeeper generated broker id. According to Wiki: Kafka runs on a cluster of one or more servers (called brokers), and the partitions of all topics are distributed across the cluster nodes.  Additionally, partitions are replicat
Read more