Apache Kafka

Supported by Dynatrace

Automatic and intelligent observability for Apache Kafka with trace and metric insights.

With Dynatrace, you can get observability for Apache Kafka without touching any code, thanks to automatic monitoring. Seamless end-to-end traces for connected producer and consumer services allow you to diagnose anomalies and pinpoint the root cause of the broken code before your customers are affected. Comprehensive metrics give you insight into the health and performance of Kafka brokers and topics. Events point you to critical anomalies, reducing the mean repair time.

use cases

Capture every message across tiers without blind spots.
Improve the performance of your producer and consumer services end-to-end.
Troubleshoot asynchronous service problems across your stack proactively.
Prevent message processing anomalies to reduce the mean time to repair.
Monitor the health and performance of all your brokers and topics.

Compatibility information

Apache Kafka
Apache Kafka Streams
Spring for Apache Kafka
Spring Cloud Stream

Get started

If your application runs on a virtual machine or bare-metal, install OneAgent on it to get started.

If your application runs as a workload in Kubernetes or OpenShift, set up Dynatrace on Kubernetes or OpenShift.

Activate the following OneAgent features to get tracing insight:

Java Kafka
Java Kafka Streams
Spring for Apache Kafka
Node.js KafkaJs
.NET Messaging Apache Kafka

To get metric insight:

Activate the OneAgent feature Java Metric Extensions 2.0 (JMX).
Activate the Kafka extension by selecting Add to environment.

Activate log monitoring to get log insight.

Extension

With the Kafka extension, you can get additional insight into your Kafka server with metrics for brokers, topics, producers, consumers, and more. The extension also provides alerts for the most critical metrics. It creates a custom topology and entities for brokers, topics, producers, and consumers. It provides a dashboard for easier access and configuration of the extension and its entities.

Requirements

OneAgent version 1.265+
Dynatrace version 1.265+
OneAgent feature Java Metric Extensions 2.0 (JMX) activated

Select Add to environment to get started.

Metrics

Name	Metric key	Unit	Description
Kafka Controller - Offline partitions	kafka.controller.KafkaController.OfflinePartitionsCount	Count	-
Kafka Controller - Active cluster controllers	kafka.controller.KafkaController.ActiveControllerCount.Value	Count	-

Name	Metric key	Unit	Description
Kafka Log - Log flush 95th percentile	kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Percentile95th	MilliSecond	-
Kafka Log - Log flush mean time	kafka.log.LogFlushStats.LogFlushRateAndTimeMs.Mean	MilliSecond	-

Name	Metric key	Unit	Description
Kafka Server - ZooKeeper disconnects	kafka.server.SessionExpireListener.ZooKeeperDisconnectsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - ZooKeeper expires	kafka.server.SessionExpireListener.ZooKeeperExpiresPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Zookeeper Active Connections	kafka.server.active-connections	Count	-

Name	Metric key	Unit	Description
Kafka Server - Handler Pool Idle Percent Rate	kafka.server.handler.average-idle-percent.rate	PerSecond	-

Name	Metric key	Unit	Description
Kafka Server - Purgatory Produce Size	kafka.server.purgatory.produce-delay-size	Count	-
Kafka Server - Purgatory Fetch Size	kafka.server.purgatory.fetch-delay-size	Count	-

Name	Metric key	Unit	Description
Kafka Consumer - Requests	kafka.consumer.consumer-metrics.request-rate	PerSecond	-
Kafka Consumer - Request size	kafka.consumer.consumer-metrics.request-size-avg	Byte	-
Kafka Consumer - Incoming byte rate	kafka.consumer.consumer-metrics.incoming-byte-rate	BytePerSecond	-
Kafka Consumer - Outgoing byte rate	kafka.consumer.consumer-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Consumer - Request latency	kafka.consumer.consumer-metrics.request-latency-avg	MilliSecond	-
Kafka Consumer - Consumer lag average	kafka.consumer.consumer-metrics.records-lag-avg	Count	-
Kafka Consumer - Consumer lag maximum	kafka.consumer.consumer-metrics.records-lag-max	Count	-
Kafka Consumer - Messages consumed rate	kafka.consumer.consumer-metrics.records-consumed-rate	PerSecond	-
Kafka Consumer - Bytes consumed rate	kafka.consumer.consumer-metrics.bytes-consumed-rate	PerSecond	-
Kafka Consumer - Fetch latency	kafka.consumer.consumer-metrics.fetch-latency-avg	MilliSecond	-

Name	Metric key	Unit	Description
Kafka Network - Produce requests per second	kafka.network.RequestMetrics.RequestsPerSec.Produce.OneMinuteRate	PerSecond	-
Kafka Network - FetchConsumer requests per second	kafka.network.RequestMetrics.RequestsPerSec.FetchConsumer.OneMinuteRate	PerSecond	-
Kafka Network - FetchFollower requests per second	kafka.network.RequestMetrics.RequestsPerSec.FetchFollower.OneMinuteRate	PerSecond	-
Kafka Network - Total time per Produce request	kafka.network.RequestMetrics.TotalTimeMs.Produce.Count	MilliSecond	-
Kafka Network - Total time per FetchConsumer request	kafka.network.RequestMetrics.TotalTimeMs.FetchConsumer.Count	MilliSecond	-
Kafka Network - Total time per FetchFollower request	kafka.network.RequestMetrics.TotalTimeMs.FetchFollower.Count	MilliSecond	-
Kafka Network - Request queue size	kafka.network.RequestChannel.RequestQueueSize.Value	Count	-

Name	Metric key	Unit	Description
Kafka Server - Disk Read Rate	kafka.server.disk.read-bytes	PerSecond	-
Kafka Server - Disk Write Rate	kafka.server.disk.write-bytes	PerSecond	-

Name	Metric key	Unit	Description
Kafka Connect - Requests	kafka.connect.connect-metrics.request-rate	PerSecond	-
Kafka Connect - Outgoing byte rate	kafka.connect.connect-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Connect - Request size	kafka.connect.connect-metrics.request-size-avg	Byte	-
Kafka Connect - Incoming byte rate	kafka.connect.connect-metrics.incoming-byte-rate	BytePerSecond	-

Name	Metric key	Unit	Description
Kafka Broker - Incoming byte rate	kafka.server.BrokerTopicMetrics.BytesInPerSec.OneMinuteRate	BytePerSecond	-
Kafka Broker - Outgoing byte rate	kafka.server.BrokerTopicMetrics.BytesOutPerSec.OneMinuteRate	BytePerSecond	-
Kafka Broker - Messages in rate	kafka.server.BrokerTopicMetrics.MessagesInPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Partitions	kafka.server.ReplicaManager.PartitionCount	Count	-
Kafka Broker - Under replicated partitions	kafka.server.ReplicaManager.UnderReplicatedPartitions	Count	-
Kafka Broker - Produce request rate	kafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Fetch request rate	kafka.server.BrokerTopicMetrics.TotalFetchRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Failed produce requests	kafka.server.BrokerTopicMetrics.FailedProduceRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Broker - Failed fetch requests	kafka.server.BrokerTopicMetrics.FailedFetchRequestsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Max follower lag	kafka.server.ReplicaFetcherManager.MaxLag.Replica.Value	Count	-
Kafka Server - Current follower lag	kafka.server.FetcherLagMetrics.CosumerLag.Value	Count	-
Kafka Server - Fetch Conversions Rate	kafka.server.FetchConversionsRate.OneMinuteRate	Count	-
Kafka Server - Produce Conversions Rate	kafka.server.ProduceConversionsRate.OneMinuteRate	Count	-

Name	Metric key	Unit	Description
Kafka Producer - Incoming byte rate	kafka.producer.producer-metrics.incoming-byte-rate	BytePerSecond	-
Kafka Producer - Outgoing byte rate	kafka.producer.producer-metrics.outgoing-byte-rate	BytePerSecond	-
Kafka Producer - I/O Wait time	kafka.producer.producer-metrics.io-wait-time-ns-avg	NanoSecond	-
Kafka Producer - Response rate	kafka.producer.producer-metrics.response-rate	PerSecond	-
Kafka Producer - Request latency	kafka.producer.producer-metrics.request-latency-avg	MilliSecond	-
Kafka Producer - Compression rate	kafka.producer.producer-metrics.compression-rate-avg	PerSecond	-
Kafka Producer - Request size	kafka.producer.producer-metrics.request-size-avg	Byte	-
Kafka Producer - Requests	kafka.producer.producer-metrics.request-rate	PerSecond	-
Kafka Producer - Byte rate	kafka.producer.producer-topic-metrics.byte-rate	BytePerSecond	-
Kafka Producer - Compression rate	kafka.producer.producer-topic-metrics.compression-rate	BytePerSecond	-
Kafka Producer - Failed Requests Rate	kafka.producer.producer-topic-metrics.record-error-rate	PerSecond	-
Kafka Producer - Requests Sent Rate	kafka.producer.producer-topic-metrics.record-send-rate	PerSecond	-

Name	Metric key	Unit	Description
Kafka Controller - Leader election rate	kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.OneMinuteRate	MilliSecond	-
Kafka Controller - Unclean election rate	kafka.controller.ControllerStats.UncleanLeaderElectionsPerSec.OneMinuteRate	PerSecond	-
Kafka Server - Leader count	kafka.server.ReplicaManager.LeaderCount.Value	Count	-

Versions

Keep up-to-date with newest releases

To have more information on how to install the downloaded package, please follow the instructions on this page.

No release notes

Screenshots

Keywords

apache
kafka
message-broker
message-queue
queues
spring
stream-processing
topics

Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:

Home hero bg

Dynatrace Hub

Hub homeBuild your own app

All (664)Log Management and Analytics Kubernetes Infrastructure Monitoring Automation Applications & Microservices Application Security Digital Experience Business Analytics

Dynatrace Developer

Discover recent additions to Dynatrace

Business Flow

Track, analyze, and optimize your critical business processes.

Carbon Impact

Track, analyze, and reduce the carbon footprint of your hybrid cloud environment.

DevSecOps Lifecycle Coverage with Snyk

Connect Snyk container scans and Dynatrace runtime insights for strong security.

Salesforce Insights

Monitor Salesforce performance, user experience, adoption, and threats.

Site Reliability Guardian

Get automated change impact analysis for release and change validation.

Analyze your data

Understand your data better with deep insights and clear visualizations.

Dashboards

Transform complex data into clear visualizations with custom dashboards.

Notebooks

Create powerful, data-driven documents for custom analytics and collaboration.

Automate your processes

Turn data and answers into actions, securely, and at scale.

Jira for Workflows

Create Jira tickets, comment, and resolve tickets. Populate custom fields and query Jira for enriched workflow functionality.

Slack for Workflows

Send Slack messages, manage communication within threads, and more.

Workflows

Automate tasks in your IT landscape, remediate problems, and visualize processes.

Secure your cloud application

See vulnerabilities and attacks in your environment.

Attack Protection

Get a real-time overview of all the attacks on your environment.

Code-level vulnerabilities

Detect vulnerabilities in your code in real time.

Security overview

Get a comprehensive overview of the security of your applications.

Third-party vulnerabilities

Detect vulnerabilities in your environment in real time, with full context and automated risk assessment.

Are you looking for something different?We have hundred of apps, extensions, and other technologies to customize your environment

Leverage our newest innovations of Dynatrace Saas

Kick-start your app creation

Kick-start your app creation

Whether you’re a beginner or a pro, Dynatrace Developer has the tools and support you need to create incredible apps with minimal effort.

Go to Dynatrace Developer

Upgrading from Dynatrace Managed to SaaS

Upgrading from Dynatrace Managed to SaaS

Drive innovation, speed, and agility in your organization by seamlessly and securely upgrading.

Log Management and Analytics

Log Management and Analytics

Innovate faster and more efficiently with unified log management and log analytics for actionable insights and automation.