Posts

Showing posts with the label Data-storage

Featured Post

Mastering flat_map in Python with List Comprehension

Image
Introduction In Python, when working with nested lists or iterables, one common challenge is flattening them into a single list while applying transformations. Many programming languages provide a built-in flatMap function, but Python does not have an explicit flat_map method. However, Python’s powerful list comprehensions offer an elegant way to achieve the same functionality. This article examines implementation behavior using Python’s list comprehensions and other methods. What is flat_map ? Functional programming  flatMap is a combination of map and flatten . It transforms the collection's element and flattens the resulting nested structure into a single sequence. For example, given a list of lists, flat_map applies a function to each sublist and returns a single flattened list. Example in a Functional Programming Language: List(List(1, 2), List(3, 4)).flatMap(x => x.map(_ * 2)) // Output: List(2, 4, 6, 8) Implementing flat_map in Python Using List Comprehension Python’...

How to Retain data in Kafka and Get Additional Time for Analysis

Image
The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights. Kafka retention period The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period.  The retention period setup you need in the broker.  It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period. Setup for the retention period Below is the setup in the broker configuration file for the retention period. log.retention.bytes The most significant size threshold in bytes for deleting a log. log.retention.ms The length in milliseconds of a log will be maintained before being deleted. log.retention.minutes Length before deletion in minutes. log.retention.ms is used as well if bot...