Featured Post

The Quick and Easy Way to Analyze Numpy Arrays

Image
The quickest and easiest way to analyze NumPy arrays is by using the numpy.array() method. This method allows you to quickly and easily analyze the values contained in a numpy array. This method can also be used to find the sum, mean, standard deviation, max, min, and other useful analysis of the value contained within a numpy array. Sum You can find the sum of Numpy arrays using the np.sum() function.  For example:  import numpy as np  a = np.array([1,2,3,4,5])  b = np.array([6,7,8,9,10])  result = np.sum([a,b])  print(result)  # Output will be 55 Mean You can find the mean of a Numpy array using the np.mean() function. This function takes in an array as an argument and returns the mean of all the values in the array.  For example, the mean of a Numpy array of [1,2,3,4,5] would be  result = np.mean([1,2,3,4,5])  print(result)  #Output: 3.0 Standard Deviation To find the standard deviation of a Numpy array, you can use the NumPy std() function. This function takes in an array as a par

How to Monitor Kafka-stream's Performance

Kafka Streams API is a part of Kafka, it goes without saying that monitoring your application will require some monitoring of Kafka as well.

Performance


The consumer and producer performance is one of the fundamental performance concerns for a producer and consumer.
 

Stream performance


The Kafka data flow diagram



Kafka data flow diagram


What is lag


For producers, we care mostly about how fast the producer is sending messages to the broker. Obviously, the higher the throughput, the better.

For consumers, we’re also concerned with performance, or how fast we can read messages from a broker.

we care about how much and how fast our producers can publish to a broker, and we simultaneously care about how quickly our consumers can read those messages from the broker. The difference between how fast the producers place records on the broker and when consumers read those messages is called consumer lag


How to check consumer lag


To check for consumer lag, Kafka provides a convenient command-line tool, kafka-consumer-groups.sh, found in the <kafka-install-dir>/bin directory. The script has a few options, but here we’ll focus on the list and describe options. These two options will give you the information you need about consumer group performance.

List command

<kafka-install-dir>/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --list


Describe command

<kafka-install-dir>/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group <GROUP-NAME> \ --describe


How to trace problem

  • A small lag or one that stays constant is OK, but a lag that continues to grow over time is an indication you’ll need to give your consumer more resources. 
  • For example, you might need to increase the partition count and hence increase the number of threads consuming from the topic. Or maybe your processing after reading the message is too heavyweight. After consuming a message, you could hand it off to an async queue, where another thread can pick up the message and do the processing.

Related

Comments

Popular posts from this blog

How to Decode TLV Quickly

7 AWS Interview Questions asked in Infosys, TCS