Posts

Showing posts with the label Data-storage

Featured Post

Step-by-Step Guide to Reading Different Files in Python

Image
 In the world of data science, automation, and general programming, working with files is unavoidable. Whether you’re dealing with CSV reports, JSON APIs, Excel sheets, or text logs, Python provides rich and easy-to-use libraries for reading different file formats. In this guide, we’ll explore how to read different files in Python , with code examples and best practices. 1. Reading Text Files ( .txt ) Text files are the simplest form of files. Python’s built-in open() function handles them effortlessly. Example: # Open and read a text file with open ( "sample.txt" , "r" ) as file: content = file.read() print (content) Explanation: "r" mode means read . with open() automatically closes the file when done. Best Practice: Always use with to handle files to avoid memory leaks. 2. Reading CSV Files ( .csv ) CSV files are widely used for storing tabular data. Python has a built-in csv module and a powerful pandas library. Using cs...

How to Retain data in Kafka and Get Additional Time for Analysis

Image
The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights. Kafka retention period The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period.  The retention period setup you need in the broker.  It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period. Setup for the retention period Below is the setup in the broker configuration file for the retention period. log.retention.bytes The most significant size threshold in bytes for deleting a log. log.retention.ms The length in milliseconds of a log will be maintained before being deleted. log.retention.minutes Length before deletion in minutes. log.retention.ms is used as well if bot...