Apache Kafka

Supported by Dynatrace

Automatic and intelligent observability for Apache Kafka with trace and metric insights.

With Dynatrace, you can get observability for Apache Kafka without touching any code, thanks to automatic monitoring. Seamless end-to-end traces for connected producer and consumer services allow you to diagnose anomalies and pinpoint the root cause of the broken code before your customers are affected. Comprehensive metrics give you insight into the health and performance of Kafka brokers and topics. Events point you to critical anomalies, reducing the mean repair time.

use cases

Capture every message across tiers without blind spots.
Improve the performance of your producer and consumer services end-to-end.
Troubleshoot asynchronous service problems across your stack proactively.
Prevent message processing anomalies to reduce the mean time to repair.
Monitor the health and performance of all your brokers and topics.

Compatibility information

Apache Kafka
Apache Kafka Streams
Spring for Apache Kafka
Spring Cloud Stream

Get started

If your application runs on a virtual machine or bare-metal, install OneAgent on it to get started.

If your application runs as a workload in Kubernetes or OpenShift, set up Dynatrace on Kubernetes or OpenShift.

Activate the following OneAgent features to get tracing insight:

Java Kafka
Java Kafka Streams
Spring for Apache Kafka
Node.js KafkaJs
.NET Messaging Apache Kafka

To get metric insight:

Activate the OneAgent feature Java Metric Extensions 2.0 (JMX).
Activate the Kafka extension by selecting Add to environment.

Activate log monitoring to get log insight.

Extension

With the Kafka extension, you can get additional insight into your Kafka server with metrics for brokers, topics, producers, consumers, and more. The extension also provides alerts for the most critical metrics. It creates a custom topology and entities for brokers, topics, producers, and consumers. It provides a dashboard for easier access and configuration of the extension and its entities.

Requirements

OneAgent version 1.265+
Dynatrace version 1.265+
OneAgent feature Java Metric Extensions 2.0 (JMX) activated

Select Add to environment to get started.

Metrics

Name	Metric key	Unit	Description
Kafka Controller - Offline partitions	kafka.controller.KafkaController.OfflinePartitionsCount	Count	-
Kafka Controller - Active cluster controllers	kafka.controller.KafkaController.ActiveControllerCount.Value	Count	-

Name	Metric key	Unit	Description
Kafka Log - Log flush 95th percentile	kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Percentile95th	MilliSecond	-
Kafka Log - Log flush mean time	kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Mean	MilliSecond	-

Name	Metric key	Unit	Description
Kafka Server - ZooKeeper disconnects	kafka.server.SessionExpireListener.ZooKeeperDisconnectsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - ZooKeeper expires	kafka.server.SessionExpireListener.ZooKeeperExpiresPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Zookeeper Active Connections	kafka.server.active-connections	Count	-

Name	Metric key	Unit	Description
Kafka Server - Handler Pool Idle Percent Rate	kafka.server.handler.average-idle-percent.rate	PerSecond	-

Name	Metric key	Unit	Description
Kafka Server - Purgatory Produce Size	kafka.server.purgatory.produce-delay-size	Count	-
Kafka Server - Purgatory Fetch Size	kafka.server.purgatory.fetch-delay-size	Count	-

Name	Metric key	Unit	Description
Kafka Consumer - Requests	kafka.consumer.consumer-metrics.request-rate	PerSecond	-
Kafka Consumer - Request size	kafka.consumer.consumer-metrics.request-size-avg	Byte	-
Kafka Consumer - Incoming byte rate	kafka.consumer.consumer-metrics.incoming-byte-rate	BytePerSecond	-
Kafka Consumer - Outgoing byte rate	kafka.consumer.consumer-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Consumer - Request latency	kafka.consumer.consumer-metrics.request-latency-avg	MilliSecond	-
Kafka Consumer - Consumer lag average	kafka.consumer.consumer-metrics.records-lag-avg	Count	-
Kafka Consumer - Consumer lag maximum	kafka.consumer.consumer-metrics.records-lag-max	Count	-
Kafka Consumer - Messages consumed rate	kafka.consumer.consumer-metrics.records-consumed-rate	PerSecond	-
Kafka Consumer - Bytes consumed rate	kafka.consumer.consumer-metrics.bytes-consumed-rate	PerSecond	-
Kafka Consumer - Fetch latency	kafka.consumer.consumer-metrics.fetch-latency-avg	MilliSecond	-

Name	Metric key	Unit	Description
Kafka Network - Produce requests per second	kafka.network.RequestMetrics.RequestsPerSec.Produce.OneMinuteRate	PerSecond	-
Kafka Network - FetchConsumer requests per second	kafka.network.RequestMetrics.RequestsPerSec.FetchConsumer.OneMinuteRate	PerSecond	-
Kafka Network - FetchFollower requests per second	kafka.network.RequestMetrics.RequestsPerSec.FetchFollower.OneMinuteRate	PerSecond	-
Kafka Network - Total time per Produce request	kafka.network.RequestMetrics.TotalTimeMs.Produce.Count	MilliSecond	-
Kafka Network - Total time per FetchConsumer request	kafka.network.RequestMetrics.TotalTimeMs.FetchConsumer.Count	MilliSecond	-
Kafka Network - Total time per FetchFollower request	kafka.network.RequestMetrics.TotalTimeMs.FetchFollower.Count	MilliSecond	-
Kafka Network - Request queue size	kafka.network.RequestChannel.RequestQueueSize.Value	Count	-

Name	Metric key	Unit	Description
Kafka Server - Disk Read Rate	kafka.server.disk.read-bytes	PerSecond	-
Kafka Server - Disk Write Rate	kafka.server.disk.write-bytes	PerSecond	-

Name	Metric key	Unit	Description
Kafka Connect - Requests	kafka.connect.connect-metrics.request-rate	PerSecond	-
Kafka Connect - Outgoing byte rate	kafka.connect.connect-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Connect - Request size	kafka.connect.connect-metrics.request-size-avg	Byte	-
Kafka Connect - Incoming byte rate	kafka.connect.connect-metrics.incoming-byte-rate	BytePerSecond	-

Name	Metric key	Unit	Description
Kafka Broker - Incoming byte rate	kafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRate	BytePerSecond	-
Kafka Broker - Outgoing byte rate	kafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRate	BytePerSecond	-
Kafka Broker - Messages in rate	kafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Partitions	kafka.server.ReplicaManager.PartitionCount	Count	-
Kafka Broker - Under replicated partitions	kafka.server.ReplicaManager.UnderReplicatedPartitions	Count	-
Kafka Broker - Produce request rate	kafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Fetch request rate	kafka.server.BrokerTopicMetrics.TotalFetchRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Failed produce requests	kafka.server.BrokerTopicMetrics.FailedProduceRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Failed fetch requests	kafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Max follower lag	kafka.server.ReplicaFetcherManager.MaxLag.Replica.Value	Count	-
Kafka Server - Current follower lag	kafka.server.FetcherLagMetrics.CosumerLag.Value	Count	-
Kafka Server - Fetch Conversions Rate	kafka.server.FetchConversionsRate.OneMinuteRate	Count	-
Kafka Server - Produce Conversions Rate	kafka.server.ProduceConversionsRate.OneMinuteRate	Count	-

Name	Metric key	Unit	Description
Kafka Producer - Incoming byte rate	kafka.producer.producer-metrics.incoming-byte-rate	BytePerSecond	-
Kafka Producer - Outgoing byte rate	kafka.producer.producer-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Producer - I/O Wait time	kafka.producer.producer-metrics.io-wait-time-ns-avg	NanoSecond	-
Kafka Producer - Response rate	kafka.producer.producer-metrics.response-rate	PerSecond	-
Kafka Producer - Request latency	kafka.producer.producer-metrics.request-latency-avg	MilliSecond	-
Kafka Producer - Compression rate	kafka.producer.producer-metrics.compression-rate-avg	PerSecond	-
Kafka Producer - Request size	kafka.producer.producer-metrics.request-size-avg	Byte	-
Kafka Producer - Requests	kafka.producer.producer-metrics.request-rate	PerSecond	-
Kafka Producer - Byte rate	kafka.producer.producer-topic-metrics.byte-rate	BytePerSecond	-
Kafka Producer - Compression rate	kafka.producer.producer-topic-metrics.compression-rate	BytePerSecond	-
Kafka Producer - Failed Requests Rate	kafka.producer.producer-topic-metrics.record-error-rate	PerSecond	-
Kafka Producer - Requests Sent Rate	kafka.producer.producer-topic-metrics.record-send-rate	PerSecond	-

Name	Metric key	Unit	Description
Kafka Controller - Leader election rate	kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.OneMinuteRate	MilliSecond	-
Kafka Controller - Unclean election rate	kafka.controller.ControllerStats.UncleanLeaderElectionsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Leader count	kafka.server.ReplicaManager.LeaderCount.Value	Count	-