How to retain data in Kafka

The default retention period in Kafka is seven days. However, you can set retention period as you wish.

Kafka retention period

The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period. The retention period setup you need in the broker. It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period.

Setup for retention period

Below is the setup in broker configuration file for retention period.

log.retention.bytes

The largest size threshold in bytes for deleting a log.


log.retention.ms

The length in milliseconds a log will be maintained before being deleted.


log.retention.minutes

Length before deletion in minutes. log.retention.ms is used as well if both are set.


log.retention.hours

Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if either of those are set.

Steps to disable retention period

You can disable the retention period by setting both log.retention.bytes and log.retention.ms to –1. So that we can effectively turn off the data deletion.

Steps to store data outside the broker

What if we want our data to stick around for a while, but simply do not have the disk space to hold our data on brokers?

• Another option for long-term storage is to move the data outside of Kafka and not retain it internally to the Kafka brokers themselves. 
• Before data is removed by retention from Kafka, we could store the data in a database, in a Hadoop Distributed File System (HDFS™), or upload our event messages into something like cloud storage.