Showing posts with the label Kafka

Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif

2 Exclusive Ways to Start Kafka Services

The Kafka services start or stop you can do in two ways. Those are  Systemd  and  Systemctl  (sudo user). Below, you will find the commands for these two methods. How to start/stop Kafka service Here are exclusive ways. With these, you can start or stop Kafka services .   1. Systemd service The concept of unit files people who worked on Linux servers have familiarity with it. Also, they know creating the unit file to use by systemd. To summarize,  it initializes and maintains components throughout the system . This means that you can define ZooKeeper and Kafka as unit files, which then will be used by systems. Commands The first command starts the service, and the second command stops the service. ... [Service] ... ExecStart=/opt/kafkainaction/bin/ ExecStop= /opt/kafkainaction/bin/ 2. Systemctl by Sudo (root) user The root user can start or stop the Kafka services. This is more like front-end processing. This way of executing is called

How to Retain data in Kafka and Get Additional Time for Analysis

The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights. Kafka retention period The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period.  The retention period setup you need in the broker.  It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period. Setup for the retention period Below is the setup in the broker configuration file for the retention period. log.retention.bytes The most significant size threshold in bytes for deleting a log. The length in milliseconds of a log will be maintained before being deleted. log.retention.minutes Length before deletion in minutes. is used as well if both are set

Messages in Kafka the Types and Details

A message, also called a record, is the basic piece of data flowing through Kafka. Messages are how Kafka represents your data. Kafka producer Vs. consumer messages Kafka is an intermediate server that receives a message from a producer and sends them to the consumer. Here is a set of 10 Kafka Interview Questions. Kafka message format Each message has a timestamp, a value, and an optional key. Custom headers can be used if desired as well.  A simple example of a message could be something like the following: the machine with host ID “1234567” (a message key) failed with the message “Alert: Machine Failed” (a message value) at “2020-10-02T10:34:11.654Z” (a message timestamp). Here is Kafka's flowchart for dummies. Kafka record The above image shows probably the most important and common parts of a message that users deal with directly. Each key and value can interact in its own specific ways to serialize or deserialize its data. Now that we have a record, how do we let Kafka know ab

10 Kafka Interview Questions That Recently Asked

Kafka Interview Questions Here're ten interview questions that were asked during Kafka's interview.  These are useful to update your knowledge. 1. What is Kafka? Kafka is a framework of Publisher and Subscribe. It reads messages from the Producer and allows them to read by Subscribers. It keeps store all the producer messages in the form of topics (underlying partitions). It also maintains logs. 2. What is a Consumer group? Each consumer is part of some Consumer group. By adding more consumers to a Consumer group, you can balance the load. In general, the Consumer group reads data from the same topic. The number of partitions in a Topic always should be the same as Consumers in a particular CG (consumer group). 3. What is Fault-Tolerance? Each partition is replicated on multiple servers. So, when one partition is failed, the other backup will deliver. So this concept is called Fault-tolerance. 4. Can we decrease the partitions that we created? No, you can't decrease the par

How to Read Kafka Logs Quickly

In Kafka, the log file's function is to store entries. Here, you can find entries for the producer's incoming messages. You can call these topics. And,  topics are divided into partitions.

How to Check Kafka Available Brokers

Here's the command to check the list of brokers present in a Kafka Cluster. You can say the Broker is the heart of the Kafka cluster. In simple terms, it works as a process. The main function is to receive messages from publishers and gives permission to access messages by consumers. How to Check Available Brokers in Kafka Here is the command: linux$  ./ zookeeper-IPaddress:2181 | "ls /brokers/ids" Just use the above command to get the number of brokers present in your host (Kafka Cluster). What is Default Broker-id in Kafka  The default broker id is -1 . When you create a new broker, it adds to -1. Then, it gives a new broker id. The broker ids will be generated from + 1. Types of broker ids. Use assigned Zookeeper generated broker id. According to Wiki: Kafka runs on a cluster of one or more servers (called brokers), and the partitions of all topics are distributed across the cluster nodes.  Additionally, partitions are replicat

Kafka Flowchart Useful for Dummies

How Kafka Works Here're the prime points on Kafka stream-processing. In Mainframe, the data you receive/process in two methods (Batch and online). In Kafka, it receives data and sends it to consumers. Here're the details with Architecture, Logs, and applications that use Kafka. The streaming data is different (YouTube Live). When the data comes into Queue the data will then be processed. In the batch process, you need to wait till you get the Batch completes. In the case of stream processing, it is on the fly. 1. Architecture 2. Process Kafka is a publish/subscribe system , but it would be more precise to say that Kafka acts as a message broker. A broker is an intermediary that brings together two parties that don’t necessarily know each other for a mutually beneficial exchange or deal. Kafka stores messages in topics and retrieves messages from topics. There’s no direct connection between the producers and the consumers of the messages. Additionally, Kafka doesn’t keep any st