Remotely monitor your Confluent Cloud Kafka Cluster and other resources with this Prometheus based Extension 2.0.
This extension provides the ability to remotely monitor your Confluent Cloud Kafka Clusters, Connectors, Schema Registries, and KSQL DB Applications. Every minute using the API provided by Confluent it ingests data about how your different Confluent Resources are performing.
This is intended for users, who:
This enables you to:
Compatibility Requirements Confluent Cloud Resource(s) and API User/Token
This extension provides monitoring of Confluent Cloud Resources via their public API (see details tab).
Also supported, via the
Kafka Lag Partition Metrics and
Kafka Lag Consumer Group Metrics feature sets, are metrics provided by the Kafka Lag Exporter.
This extension uses the Confluent Metric Export API to gather metrics.
First you will need to create either a Cloud/Cluster API Key and Secret. This can be done via the Confluent UI or via their CLI. The MetricsViewer role is required to access the Confluent API. It is suggested to use the Organization scope for this role so it can be used as clusters are created or destroyed.
In Dynatrace, then create a new Monitoring Configuration and select "Monitor Remotely without OneAgent" near the bottom on the Monitoring Source Screen.
In the Dynatrace Monitoring Configuration the Confluent Cloud API Key and API Secret are used as the Basic Auth User (API Key) /Password (API secret) combination.
Next you'll create a URL with your resource types and IDs at the end, similar to what is shown below. This URL supports multiple resources but it is recommended to not have more than 5 to 10 per URL.
Confluent Kafka Cluster
Confluent Kafka Schema Registry
Confluent Kafka Connector
Confluent Kafka KSQL DB Application
Simply activate the extension in your environment using the in-product Hub, provide the necessary device configuration and you’re all set up.
Read more in the Prometheus Extension Documentation
|Kafka Consumer Group Group Topic Sum Lag||kafka_consumergroup_group_topic_sum_lag||Count||Sum of group offset lag across topic partitions|
|Kafka Consumer Group Poll Time (ms)||kafka_consumergroup_poll_time_ms||MilliSecond||Group poll time|
|Kafka Consumer Group Group Offset||kafka_consumergroup_group_offset||Count||Last group consumed offset of a partition|
|Kafka Consumer Group Group Sum Lag||kafka_consumergroup_group_sum_lag||Count||Sum of group offset lag|
|Kafka Consumer Group Group Lag||kafka_consumergroup_group_lag||Count||Group offset lag of a partition|
|Kafka Consumer Group Group Lag Seconds||kafka_consumergroup_group_lag_seconds||Second||Group time lag of a partition|
|Kafka Consumer Group Group Max Lag||kafka_consumergroup_group_max_lag||Count||Max group offset lag|
|Kafka Consumer Group Group Max Lag Seconds||kafka_consumergroup_group_max_lag_seconds||Second||Max group time lag|
|Kafka Schema Registry Schema Count||confluent_kafka_schema_registry_schema_count.gauge||Count||The number of registered schemas.|
|Kafka Schema Registry Request Count||confluent_kafka_schema_registry_request_count.gauge||Count||The delta count of requests received by the schema registry server. Each sample is the number of requests received since the previous data point. The count sampled every 60 seconds.|
|Kafka Cluster Received Bytes||confluent_kafka_server_received_bytes||Byte||The number of bytes of the customer's data received from the network.|
|Kafka Cluster Sent Bytes||confluent_kafka_server_sent_bytes||Byte||The number of bytes of the customer's data sent over the network.|
|Kafka Cluster Received Records||confluent_kafka_server_received_records||Count||The number of records received.|
|Kafka Cluster Sent Records||confluent_kafka_server_sent_records||Count||The number of records sent.|
|Kafka Cluster Retained Bytes||confluent_kafka_server_retained_bytes||Byte||The current number of bytes retained by the cluster.|
|Kafka Cluster Partition Count||confluent_kafka_server_partition_count.gauge||Count||The number of partitions.|
|Kafka Cluster Load Raw||confluent_kafka_server_cluster_load_percent||Count||A measure of the utilization of the cluster. The value is between 0.0 and 1.0.|
|Kafka Partition Earliest Offset||kafka_partition_earliest_offset||Count||Earliest offset of a partition|
|Kafka Partition Latest Offset||kafka_partition_latest_offset||Count||Latest offset of a partition|
|Kafka Ksql Streaming Unit Count||confluent_kafka_ksql_streaming_unit_count.gauge||Count||The count of Confluent Streaming Units (CSUs) for this KSQL instance. The implicit time aggregation for this metric is MAX.|
|Kafka Ksql Query Saturation||confluent_kafka_ksql_query_saturation||Count||The maximum saturation for a given ksqlDB query across all nodes. Returns a value between 0 and 1, a value close to 1 indicates that ksqlDB query processing is bottlenecked on available resources.|
|Kafka Ksql Task Stored Bytes||confluent_kafka_ksql_task_stored_bytes||Byte||The size of a given task's state stores in bytes.|
|Kafka Ksql Storage Utilization||confluent_kafka_ksql_storage_utilization||Percent||The total storage utilization for a given ksqlDB application.|
|Kafka Cluster Request Bytes||confluent_kafka_server_request_bytes||Byte||The delta count of total request bytes from the specified request types sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.|
|Kafka Cluster Response Bytes||confluent_kafka_server_response_bytes||Byte||The delta count of total response bytes from the specified response types sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.|
|Kafka Cluster Active Connection Count||confluent_kafka_server_active_connection_count.gauge||Count||The count of active authenticated connections.|
|Kafka Cluster Request Count||confluent_kafka_server_request_count.gauge||Count||The number of requests received over the network.|
|Kafka Cluster Successful Authentication Count||confluent_kafka_server_successful_authentication_count.gauge||Count||The number of successful authentications.|
|Kafka Server Consumer Lag Offsets||confluent_kafka_server_consumer_lag_offsets||Count||The lag between a group member's committed offset and the partition's high watermark|
|Kafka Server Cluster Link Destination Response Bytes||confluent_kafka_server_cluster_link_destination_response_bytes||Byte||The delta count of cluster linking response bytes from all request types. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.|
|Kafka Server Cluster Link Source Response Bytes||confluent_kafka_server_cluster_link_source_response_bytes||Byte||The delta count of cluster linking source response bytes from all request types. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.|
|Kafka Server Cluster Link Count||confluent_kafka_server_cluster_link_count.gauge||Count||The current count of cluster links. The count is sampled every 60 seconds. The implicit time aggregation for this metric is MAX.|
|Kafka Server Cluster Link Mirror Topic Count||confluent_kafka_server_cluster_link_mirror_topic_count.gauge||Count||The cluster linking mirror topic count for a link. The count is sampled every 60 seconds.|
|Kafka Server Cluster Link Mirror Topic Offset Lag||confluent_kafka_server_cluster_link_mirror_topic_offset_lag||Count||The cluster linking mirror topic offset lag maximum across all partitions. The lag is sampled every 60 seconds.|
|Kafka Server Cluster Link Mirror Topic Bytes||confluent_kafka_server_cluster_link_mirror_topic_bytes||Byte||The delta count of cluster linking mirror topic bytes. The count is sampled every 60 seconds.|
|Kafka Connect Sent Records||confluent_kafka_connect_sent_records||Count||The delta count of total number of records sent from the transformations and written to Kafka for the source connector. Each sample is the number of records sent since the previous data point.|
|Kafka Connect Received Records||confluent_kafka_connect_received_records||Count||The delta count of total number of records received by the sink connector. Each sample is the number of records received since the previous data point.|
|Kafka Connect Sent Bytes||confluent_kafka_connect_sent_bytes||Byte||The delta count of total bytes sent from the transformations and written to Kafka for the source connector. Each sample is the number of bytes sent since the previous data point.|
|Kafka Connect Received Bytes||confluent_kafka_connect_received_bytes||Byte||The delta count of total bytes received by the sink connector. Each sample is the number of bytes received since the previous data point.|
|Kafka Connect Dead Letter Queue Records||confluent_kafka_connect_dead_letter_queue_records||Count||The delta count of dead letter queue records written to Kafka for the sink connector.|
Confluent Kafka Overview (Deprecated Dashboard)or the Data Explorer
Extend the platform,
empower your team.