Showing posts with the label Kafka

10 Kafka Interview Questions That Recently Asked

Here're ten interview questions that were asked during Kafka's interview.  These are useful to update your knowledge. Kafka Interview Questions 1. What is Kafka? Kafka is a framework of Publisher and Subscribe. It reads messages from the Producer and allows them to read by Subscribers. It keeps store all the producer messages in the form of topics (underlying partitions). It also maintains logs. 2. What is a Consumer group? Each consumer is part of some Consumer group. By adding more consumers to a Consumer group, you can balance the load. In general, the Consumer group reads data from the same topic. The number of partitions in a Topic always should be the same as Consumers in a particular CG (consumer group). 3. What is Fault-Tolerance? Each partition is replicated on multiple servers. So, when one partition is failed, the other backup will deliver. So this concept is called Fault-tolerance. 4. Can we decrease the partitions that we created? No, you can't decrease the par

How to Read Kafka Logs Quickly

In Kafka, the log file's function is to store entries. Here, you can find entries for the producer's incoming messages. You can call these topics. And,  topics are divided into partitions. How to Read Logs in Kafka IN THIS PAGE Kafka Logs How Producer Messages Store Benefits of Kafka Logs How to check Logs in Kafka 1. Kafka Logs The mechanism underlying Kafka is the log . Most software engineers are familiar with this. It tracks what an application is doing.  If you have performance issues or errors in your application, the first place to check is the application logs. But it is a different sort of log.  In the context of Kafka (or any other distributed system), a log is "an append-only, totally ordered sequence of records - ordered by time. Kafka Basics [Video] 2. How Producer Messages Store The producer writes the messages to Broker, the records store in a log file. The records stored as 0,1,2,3 and so on. Each record will have one unique id. 4. Benefits of Kafka Logs L

How to Check Kafka Available Brokers

Here's the command to check the list of brokers present in a Kafka Cluster. You can say the Broker is the heart of Kafka-cluster. In simple terms, it works as a process. The main function is to receive messages from publishers and gives permission to access messages by consumers. How to Check Available Brokers in Kafka Here is the command: linux$ ./ zookeeper-IPaddress:2181 | "ls /brokers/ids" Just use the above command to get the number of brokers present in your host (Kafka Cluster). What is Default Broker-id in Kafka  The default broker id is -1 . When you create a new broker, it adds to -1. Then, it gives a new broker id. The broker ids will be generated from + 1. Types of broker ids. Use assigned Zookeeper generated broker id. According to Wiki: Kafka runs on a cluster of one or more servers (called brokers), and the partitions of all topics are distributed across the cluster nodes.  Additionally, partitions are replicated t

Kafka Flowchart Useful for Dummies

Here're the prime points on Kafka stream-processing. In Mainframe, the data you receive/process in two methods (Batch and online). In Kafka, it receives data and sends it to consumers. Here're the details with Architecture, Logs, and applications that use Kafka. The streaming data is different (YouTube Live). When the data comes into Queue the data will then be processed. In the batch process, you need to wait till you get the Batch completes. In the case of stream processing, it is on the fly. How Kafka Works 1. Architecture 2. Process Kafka is a publish/subscribe system , but it would be more precise to say that Kafka acts as a message broker. A broker is an intermediary that brings together two parties that don’t necessarily know each other for a mutually beneficial exchange or deal. Kafka stores messages in topics and retrieves messages from topics. There’s no direct connection between the producers and the consumers of the messages. Additionally, Kafka doesn’t keep any st