Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

HBASE: Top Features in Storing Big data

In this post explained top features added in HBase to handle the data. The Java implementation of Google's Big Table you can call it as HBASE.  In HBase, the data store as two parts.


hadoop hbase

Row Key : 00001 
Column : (Column Qualifier:Version:Value)       

Features of HBASE

  • HBase data stores consist of one or more tables, which are indexed by row keys.
  • Data is stored in rows with columns, and rows can have multiple versions.
  • By default, data versioning for rows is implemented with time stamps.
  • Columns are grouped into column families, which must be defined upfront during table creation. Column families are stored together on disk, which is why HBase is referred to as a column-oriented datastore
New features of HBASE check now

In addition...

HBase is a distributed data store, which leverages a network-attached cluster of low-cost commodity servers to store and persist data.HBase architecture is a little trick to know.

Region Servers...

RegionServers are the software processes (often called daemons) you activate to store and retrieve data in HBase.

The big difference...

  • HABSE handles growing data or big data. HBase automatically scales as you add data to the system. A huge benefit compared to most database management systems, which require manual intervention to scale the overall system beyond a single server. 
  • With HBase, as long as you have in the rack another spare server that's configured, scaling is automatic.

Read more

Comments

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

Explained Ideal Structure of Python Class

How to Check Kafka Available Brokers