How to Retain data in Kafka and Get Additional Time for Analysis

The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights.

Kafka retention period

• The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period. 
• The retention period setup you need in the broker. It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period.

Setup for the retention period

Below is the setup in the broker configuration file for the retention period.

log.retention.bytes

• The most significant size threshold in bytes for deleting a log.

log.retention.ms

• The length in milliseconds of a log will be maintained before being deleted.

log.retention.minutes

• Length before deletion in minutes. log.retention.ms is used as well if both are set.

log.retention.hours

• Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if both are set.

Steps to disable retention period

You can disable the retention period by setting both log.retention.bytes and log.retention.ms to –1. So that we can effectively turn off the data deletion.

Steps to store data outside the broker

If we want our data to stick around for a while, what is the option other than brokers?

• Move the data outside of Kafka and not retain it internally to the Kafka brokers themselves. 
• Before data is removed by retention from Kafka, we could store the data in a database, in a Hadoop Distributed File System (HDFS™), or upload our event messages into something like cloud storage.