
Kafka retention period
The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period. The retention period setup you need in the broker. It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period.
Setup for retention period
Below is the setup in broker configuration file for retention period.
log.retention.bytes
The largest size threshold in bytes for deleting a log.
log.retention.ms
The length in milliseconds a log will be maintained before being deleted.
log.retention.minutes
Length before deletion in minutes. log.retention.ms is used as well if both are set.
log.retention.hours
Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if either of those are set.
log.retention.ms
The length in milliseconds a log will be maintained before being deleted.
log.retention.minutes
Length before deletion in minutes. log.retention.ms is used as well if both are set.
log.retention.hours
Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if either of those are set.
Steps to disable retention period
You can disable the retention period by setting both log.retention.bytes and log.retention.ms to –1. So that we can effectively turn off the data deletion.Steps to store data outside the broker
What if we want our data to stick around for a while, but simply do not have the disk space to hold our data on brokers?- Another option for long-term storage is to move the data outside of Kafka and not retain it internally to the Kafka brokers themselves.
- Before data is removed by retention from Kafka, we could store the data in a database, in a Hadoop Distributed File System (HDFS™), or upload our event messages into something like cloud storage.
0 Comments
Thanks for your message. We will get back you.