Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

Big Data:Top Hadoop Interview Questions (2 of 5)

Frequently asked Hadoop interview questions.

1. What is Hadoop?Hadoop is a framework that allows users the power of distributed computing.

2.What is the difference between SQL and Hadoop?

SQL is allowed to work with structured data. But SQL is most suitable for legacy technologies. Hadoop is suitable for unstructured data. And, it is well suited for modern technologis.

3. What is Hadoop framework?

It is distributed network of commodity servers(A server can contain multiple clusters, and a cluster can have multiple nodes)

4. What are 4 properties of Hadoop?

Accessible-Hadoop runs on large clusters of commodity machinesRobust-An assumption that low commodity machines cause many machine failures. But it handles these tactfully. Scalable-Hadoop scales linearly to handle larger data by adding more nodes to the cluster. Simple-Hadoop allows users to quickly write efficient parallel code

5. What kind of data Hadoop needs?

Traditional RDBMS having relational structure with data resides in tables. In Hadoop. data should be in Key,Value pair.

6. Is Hadoop suitable for on the fly processing?

Hadoop is not suitable. It is suitable only for off-line processing. That means, we can not use Hadoop on active web logs. We can use it on web logs data,which already generated. So, in this property Hadoop is matching to traditional data warehouses.

7. What is Map reduce?

Map reduce is a data processing model, which contain mappers, and reducers. It takes unstructred data as input, and create as Key,Value pairs for processing on Hadoop.


Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

How to Check Kafka Available Brokers

SQL Query: 3 Methods for Calculating Cumulative SUM