Confluent Cloud (Kafka)

Supported by Dynatrace

Remotely monitor your Confluent Cloud Kafka Cluster and other resources with this Prometheus based Extension 2.0.

This extension provides the ability to remotely monitor your Confluent Cloud Kafka Clusters, Connectors, Schema Registries, and KSQL DB Applications. Every minute using the API provided by Confluent it ingests data about how your different Confluent Resources are performing.

This is intended for users, who:

Would like to monitor the health state and performance of their Confluent Cloud Resources.
Look for analysis support for Ops, IT and Network Admins.

This enables you to:

Monitor infrastructure with a comprehensive dashboard
Detect usage anomalies and alert on them

Compatibility Requirements Confluent Cloud Resource(s) and API User/Token

use cases

This extension provides monitoring of Confluent Cloud Resources via their public API (see details tab).
Also supported, via the Kafka Lag Partition Metrics and Kafka Lag Consumer Group Metrics feature sets, are metrics provided by the Kafka Lag Exporter.
- NOTE: This exporter is not supported by Dynatrace and needs to be set up and run independently from this extension.
- Currently the extension only supports ingesting metrics from this exporter.

Detail

This extension uses the Confluent Metric Export API to gather metrics.

First you will need to create either a Cloud/Cluster API Key and Secret. This can be done via the Confluent UI or via their CLI. The MetricsViewer role is required to access the Confluent API. It is suggested to use the Organization scope for this role so it can be used as clusters are created or destroyed.

In Dynatrace, then create a new Monitoring Configuration and select "Monitor Remotely without OneAgent" near the bottom on the Monitoring Source Screen.

In the Dynatrace Monitoring Configuration the Confluent Cloud API Key and API Secret are used as the Basic Auth User (API Key) /Password (API secret) combination.

Next you'll create a URL with your resource types and IDs at the end, similar to what is shown below. This URL supports multiple resources but it is recommended to not have more than 5 to 10 per URL.

https://api.telemetry.confluent.cloud/v2/metrics/cloud/export?resource.kafka.id=lkc-XXXXX&resource.connector.id=lcc-XXXX1&resource.connector.id=lcc-XXXX2

Base URL https://api.telemetry.confluent.cloud/v2/metrics/cloud/export?

Confluent Kafka Cluster
- resource.kafka.id=lkc-XXXXX
Confluent Kafka Schema Registry
- resource.schema_registry.id=lsrc=XXXXX
Confluent Kafka Connector
- resource.connector.id=lcc-XXXXX
Confluent Kafka KSQL DB Application
- resource.ksql.id=lksqlc-XXXXX

Get started

Simply activate the extension in your environment using the in-product Hub, provide the necessary device configuration and you’re all set up.

Read more in the Prometheus Extension Documentation

Metrics

Name	Metric key	Unit	Description
Kafka Consumer Group Group Topic Sum Lag	kafka_consumergroup_group_topic_sum_lag	Count	Sum of group offset lag across topic partitions
Kafka Consumer Group Poll Time (ms)	kafka_consumergroup_poll_time_ms	MilliSecond	Group poll time
Kafka Consumer Group Group Offset	kafka_consumergroup_group_offset	Count	Last group consumed offset of a partition
Kafka Consumer Group Group Sum Lag	kafka_consumergroup_group_sum_lag	Count	Sum of group offset lag
Kafka Consumer Group Group Lag	kafka_consumergroup_group_lag	Count	Group offset lag of a partition
Kafka Consumer Group Group Lag Seconds	kafka_consumergroup_group_lag_seconds	Second	Group time lag of a partition
Kafka Consumer Group Group Max Lag	kafka_consumergroup_group_max_lag	Count	Max group offset lag
Kafka Consumer Group Group Max Lag Seconds	kafka_consumergroup_group_max_lag_seconds	Second	Max group time lag

Name	Metric key	Unit	Description
Kafka Schema Registry Schema Count	confluent_kafka_schema_registry_schema_count.gauge	Count	The number of registered schemas.
Kafka Schema Registry Request Count	confluent_kafka_schema_registry_request_count.gauge	Count	The delta count of requests received by the schema registry server. Each sample is the number of requests received since the previous data point. The count sampled every 60 seconds.

Name	Metric key	Unit	Description
Kafka Cluster Received Bytes	confluent_kafka_server_received_bytes	Byte	The number of bytes of the customer's data received from the network.
Kafka Cluster Sent Bytes	confluent_kafka_server_sent_bytes	Byte	The number of bytes of the customer's data sent over the network.
Kafka Cluster Received Records	confluent_kafka_server_received_records	Count	The number of records received.
Kafka Cluster Sent Records	confluent_kafka_server_sent_records	Count	The number of records sent.
Kafka Cluster Retained Bytes	confluent_kafka_server_retained_bytes	Byte	The current number of bytes retained by the cluster.
Kafka Cluster Partition Count	confluent_kafka_server_partition_count.gauge	Count	The number of partitions.
Kafka Cluster Load Raw	confluent_kafka_server_cluster_load_percent	Count	A measure of the utilization of the cluster. The value is between 0.0 and 1.0.

Name	Metric key	Unit	Description
Kafka Partition Earliest Offset	kafka_partition_earliest_offset	Count	Earliest offset of a partition
Kafka Partition Latest Offset	kafka_partition_latest_offset	Count	Latest offset of a partition

Name	Metric key	Unit	Description
Kafka Ksql Streaming Unit Count	confluent_kafka_ksql_streaming_unit_count.gauge	Count	The count of Confluent Streaming Units (CSUs) for this KSQL instance. The implicit time aggregation for this metric is MAX.
Kafka Ksql Query Saturation	confluent_kafka_ksql_query_saturation	Count	The maximum saturation for a given ksqlDB query across all nodes. Returns a value between 0 and 1, a value close to 1 indicates that ksqlDB query processing is bottlenecked on available resources.
Kafka Ksql Task Stored Bytes	confluent_kafka_ksql_task_stored_bytes	Byte	The size of a given task's state stores in bytes.
Kafka Ksql Storage Utilization	confluent_kafka_ksql_storage_utilization	Percent	The total storage utilization for a given ksqlDB application.

Name	Metric key	Unit	Description
Kafka Cluster Request Bytes	confluent_kafka_server_request_bytes	Byte	The delta count of total request bytes from the specified request types sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
Kafka Cluster Response Bytes	confluent_kafka_server_response_bytes	Byte	The delta count of total response bytes from the specified response types sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
Kafka Cluster Active Connection Count	confluent_kafka_server_active_connection_count.gauge	Count	The count of active authenticated connections.
Kafka Cluster Request Count	confluent_kafka_server_request_count.gauge	Count	The number of requests received over the network.
Kafka Cluster Successful Authentication Count	confluent_kafka_server_successful_authentication_count.gauge	Count	The number of successful authentications.

Name	Metric key	Unit	Description
Kafka Server Consumer Lag Offsets	confluent_kafka_server_consumer_lag_offsets	Count	The lag between a group member's committed offset and the partition's high watermark

Name	Metric key	Unit	Description
Kafka Server Cluster Link Destination Response Bytes	confluent_kafka_server_cluster_link_destination_response_bytes	Byte	The delta count of cluster linking response bytes from all request types. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
Kafka Server Cluster Link Source Response Bytes	confluent_kafka_server_cluster_link_source_response_bytes	Byte	The delta count of cluster linking source response bytes from all request types. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
Kafka Server Cluster Link Count	confluent_kafka_server_cluster_link_count.gauge	Count	The current count of cluster links. The count is sampled every 60 seconds. The implicit time aggregation for this metric is MAX.
Kafka Server Cluster Link Mirror Topic Count	confluent_kafka_server_cluster_link_mirror_topic_count.gauge	Count	The cluster linking mirror topic count for a link. The count is sampled every 60 seconds.
Kafka Server Cluster Link Mirror Topic Offset Lag	confluent_kafka_server_cluster_link_mirror_topic_offset_lag	Count	The cluster linking mirror topic offset lag maximum across all partitions. The lag is sampled every 60 seconds.
Kafka Server Cluster Link Mirror Topic Bytes	confluent_kafka_server_cluster_link_mirror_topic_bytes	Byte	The delta count of cluster linking mirror topic bytes. The count is sampled every 60 seconds.

Name	Metric key	Unit	Description
Kafka Connect Sent Records	confluent_kafka_connect_sent_records	Count	The delta count of total number of records sent from the transformations and written to Kafka for the source connector. Each sample is the number of records sent since the previous data point.
Kafka Connect Received Records	confluent_kafka_connect_received_records	Count	The delta count of total number of records received by the sink connector. Each sample is the number of records received since the previous data point.
Kafka Connect Sent Bytes	confluent_kafka_connect_sent_bytes	Byte	The delta count of total bytes sent from the transformations and written to Kafka for the source connector. Each sample is the number of bytes sent since the previous data point.
Kafka Connect Received Bytes	confluent_kafka_connect_received_bytes	Byte	The delta count of total bytes received by the sink connector. Each sample is the number of bytes received since the previous data point.
Kafka Connect Dead Letter Queue Records	confluent_kafka_connect_dead_letter_queue_records	Count	The delta count of dead letter queue records written to Kafka for the sink connector.

Versions

Keep up-to-date with newest releases

To have more information on how to install the downloaded package, please follow the instructions on this page.

v2.0.0

IMPORTANT : Updated Dynatrace metric keys to match metric keys from Prometheus.
- This will cause existing Dashboards & Alerts (or anything that relies on the old metric keys) to stop working! Please update them accordingly.
- Please immediately update your Monitoring Configurations once this new version is activated
- You can use still view the old metrics either via the Confluent Kafka Overview (Deprecated Dashboard) or the Data Explorer
Added new Cluster, Schema Registry and ksqlDB metrics.

Download