Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

How to Retain data in Kafka and Get Additional Time for Analysis

The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights.


How to Retain data in Kafka and Get Additional Time for Analysis


Kafka retention period

  • The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period. 
  • The retention period setup you need in the broker. It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period.

Setup for the retention period

Below is the setup in the broker configuration file for the retention period.

log.retention.bytes


  • The most significant size threshold in bytes for deleting a log.


log.retention.ms

  • The length in milliseconds of a log will be maintained before being deleted.


log.retention.minutes


  • Length before deletion in minutes. log.retention.ms is used as well if both are set.


log.retention.hours

  • Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if both are set.

Steps to disable retention period

You can disable the retention period by setting both log.retention.bytes and log.retention.ms to –1. So that we can effectively turn off the data deletion.

Steps to store data outside the broker

If we want our data to stick around for a while, what is the option other than brokers?
  • Move the data outside of Kafka and not retain it internally to the Kafka brokers themselves. 
  • Before data is removed by retention from Kafka, we could store the data in a database, in a Hadoop Distributed File System (HDFS™), or upload our event messages into something like cloud storage.

Comments

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

Explained Ideal Structure of Python Class

How to Check Kafka Available Brokers