Featured Post

Python map() and lambda() Use Cases and Examples

-
Image
 In Python, map() and lambda functions are often used together for functional programming. Here are some examples to illustrate how they work. Python map and lambda top use cases 1. Using map() with lambda The map() function applies a given function to all items in an iterable (like a list) and returns a map object (which can be converted to a list). Example: Doubling Numbers numbers = [ 1 , 2 , 3 , 4 , 5 ] doubled = list ( map ( lambda x: x * 2 , numbers)) print (doubled) # Output: [2, 4, 6, 8, 10] 2. Using map() to Convert Data Types Example: Converting Strings to Integers string_numbers = [ "1" , "2" , "3" , "4" , "5" ] integers = list ( map ( lambda x: int (x), string_numbers)) print (integers) # Output: [1, 2, 3, 4, 5] 3. Using map() with Multiple Iterables You can also use map() with more than one iterable. The lambda function can take multiple arguments. Example: Adding Two Lists Element-wise list1 = [ 1 , 2 , 3 ]
Post a Comment

How to Retain data in Kafka and Get Additional Time for Analysis

-
The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights.


How to Retain data in Kafka and Get Additional Time for Analysis


Kafka retention period

  • The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period. 
  • The retention period setup you need in the broker. It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period.

Setup for the retention period

Below is the setup in the broker configuration file for the retention period.

log.retention.bytes


  • The most significant size threshold in bytes for deleting a log.


log.retention.ms

  • The length in milliseconds of a log will be maintained before being deleted.


log.retention.minutes

  • Length before deletion in minutes. log.retention.ms is used as well if both are set.


log.retention.hours

  • Length before deletion in hours. log.retention.ms and log.retention.minutes would be used before this value if both are set.

Steps to disable retention period

You can disable the retention period by setting both log.retention.bytes and log.retention.ms to –1. So that we can effectively turn off the data deletion.

Steps to store data outside the broker

If we want our data to stick around for a while, what is the option other than brokers?
  • Move the data outside of Kafka and not retain it internally to the Kafka brokers themselves. 
  • Before data is removed by retention from Kafka, we could store the data in a database, in a Hadoop Distributed File System (HDFS™), or upload our event messages into something like cloud storage.

Comments

Post a Comment

Thanks for your message. We will get back you.

Popular posts from this blog

How to Fix datetime Import Error in Python Quickly

-
Image
Here's a quick resolution for import datetime Python error . The reason is your .py python script name and datetime  are the same. I'll show you how this error happens and its resolution.   Here's the Resolution for ImportError I've created a script called 'datetime.py.' to check whether the minute value is 'odd' or not. During the import of my python script, I got the import error cannot import name datetime. My Script: datetime.py My script's intention is to find whether the minute value is odd or not. Python Logic from datetime import datetime odds = [ 1 , 3 , 5 , 7 , 9 , 11 , 13 , 15 , 17 , 19 , 21 , 23 , 25 , 27 , 29 , 31 , 33 , 35 , 37 , 39 , 41 , 43 , 45 , 47 , 49 , 51 , 53 , 55 , 57 , 59 ] right_this_minute = datetime.today().minute if right_this_minute in odds: print ( "This minute seems a little odd." ) else : print ( "Not an odd minute." ) ImportError I have imported my datetime.py from Linux. It gives an error
Read more

SQL Query: 3 Methods for Calculating Cumulative SUM

-
Image
SQL provides various constructs for calculating cumulative sums, offering flexibility and efficiency in data analysis. In this article, we explore three distinct SQL queries that facilitate the computation of cumulative sums. Each query leverages different SQL constructs to achieve the desired outcome, catering to diverse analytical needs and preferences. Using Window Functions (e.g., PostgreSQL, SQL Server, Oracle) SELECT id, value, SUM(value) OVER (ORDER BY id) AS cumulative_sum  FROM your_table; This query uses the SUM() window function with the OVER clause to calculate the cumulative sum of the value column ordered by the id column. Using Subqueries (e.g., MySQL, SQLite): SELECT t1.id, t1.value, SUM(t2.value) AS cumulative_sum FROM your_table t1 JOIN your_table t2 ON t1.id >= t2.id GROUP BY t1.id, t1.value ORDER BY t1.id; This query uses a self-join to calculate the cumulative sum. It joins the table with itself, matching rows where the id in the first table is greater than or
Read more

Python placeholder '_' Perfect Way to Use it

-
Image
What is placeholder in Python? The purpose of it is to mask the variable that you don't want to use in a function. In python, y ou can call the underscore ( _ ) operator placeholder. Below, you'll find how to use single and double placeholders in a function. What is placeholder in python The purpose of placeholder in Python is to mask variables that you don't want to use in a function. So that your code will be readable. Moreover, in future, if you want to use those variables you can replace the placeholders with the names you want. In This Page You'll know in three steps how to use placeholder correctly. Creating a function Logic to use single placeholder Logic to use two placeholders 1. Creating a function. def function_that_returns_multiple_values(x):        return x*2, x*3, x+1        for i in range(0,5):             square, cube, added_one = function_that_returns_multiple_values(i)             print(square, cube) Here, in print, it returns two variables. I will s
Read more