Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
Google DataflowGoogle Dataflow
Google Dataflow

Google Dataflow

Get insights into Google Dataflow service metrics collected from the Google Operations API to ensure health of your cloud infrastructure.

Extension
Free trialDocumentation
  • Product information
  • Release notes

Overview

This Dynatrace extension leverages data collected from the Google Operations API to constantly monitor health and performance of Google Dataflow services. This extension combines all relevant data into pre-configured dashboards and provides alerting and event tracking.

This is intended for users, who want to:

  • enrich their monitoring data with metrics and logs from Google Operations API

This enables you to:

  • View and analyze 23 metrics that are specific to Google Dataflow, like Billable shuffle data processed, Current number of vCPUs in use, Current shuffle slots in use, and more.

  • Build custom dashboards for your cloud infrastructure.

  • Analyze Google Dataflow logs.

  • Set custom alerts that trigger remediation workflows.

Google Dataflow metric and log ingestion requires advanced GCP integration.

Compatibility requirements

  • Dynatrace GCP integration version 1.0.0+. To enable metric and log ingestion, see Dynatrace Documentation.
  • Dynatrace version 1.230+

Details

This extension package contains:

  • Configuration for Google Dataflow metric ingest based on your selected feature set (see full list of feature sets and metrics)

To provide correlation and causation analysis all ingested metrics and logs are analyzed by the Dynatrace Davis AI engine, which consumes DDUs.

Get started

To add this extension to your environment:

  • Follow the instructions detailed in Dynatrace Documentation. (Ignore the “Add to environment” button below.)
  • Ensure that you have GCP integration running in your environment and that Google Dataflow service is configured.

Following GCP integration and Google Dataflow configuration:

  • The first data points will be ingested by Dynatrace Davis within ~5 minutes.
  • Explore Google Dataflow metrics in Data Explorer and create custom charts.
  • Set custom events for alerting
Dynatrace
Documentation
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Extension content

Content typeNumber of items included
metric metadata
44
generic type
1

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
Per-stage backlog in bytescloud.gcp.dataflow_googleapis_com.job.backlog_bytes-Byte
Per-stage backlog in elementscloud.gcp.dataflow_googleapis_com.job.backlog_elements-Count
BigQueryIO.Write Requestscloud.gcp.dataflow_googleapis_com.job.bigquery.write_count-Count
Billable shuffle data processedcloud.gcp.dataflow_googleapis_com.job.billable_shuffle_data_processed-Byte
Bundle user processing latenciescloud.gcp.dataflow_googleapis_com.job.bundle_user_processing_latencies-MilliSecond
Current number of vCPUs in usecloud.gcp.dataflow_googleapis_com.job.current_num_vcpus-Count
Current shuffle slots in usecloud.gcp.dataflow_googleapis_com.job.current_shuffle_slots-Count
Data watermark lagcloud.gcp.dataflow_googleapis_com.job.data_watermark_age-Second
Disk Space Capacitycloud.gcp.dataflow_googleapis_com.job.disk_space_capacity-Byte
Duplicate message count per stagecloud.gcp.dataflow_googleapis_com.job.duplicates_filtered_out_count-Count
Elapsed timecloud.gcp.dataflow_googleapis_com.job.elapsed_time-Second
Element countcloud.gcp.dataflow_googleapis_com.job.element_count.gauge-Count
Elements Producedcloud.gcp.dataflow_googleapis_com.job.elements_produced_count-Count
Estimated byte countcloud.gcp.dataflow_googleapis_com.job.estimated_byte_count.gauge-Byte
Estimated Bytes Producedcloud.gcp.dataflow_googleapis_com.job.estimated_bytes_produced_count-Count
Failedcloud.gcp.dataflow_googleapis_com.job.is_failed-Count
Memory Capacitycloud.gcp.dataflow_googleapis_com.job.memory_capacity-Byte
Per-stage data watermark lagcloud.gcp.dataflow_googleapis_com.job.per_stage_data_watermark_age-Second
Per-stage system lagcloud.gcp.dataflow_googleapis_com.job.per_stage_system_lag-Second
The approximate number of parallel processing keyscloud.gcp.dataflow_googleapis_com.job.processing_parallelism_keys-Count
Pub/Sub Late Messages Countcloud.gcp.dataflow_googleapis_com.job.pubsub.late_messages_count-Count
Pub/Sub Published Messages Countcloud.gcp.dataflow_googleapis_com.job.pubsub.published_messages_count-Count
Pub/Sub Pulled Message Agescloud.gcp.dataflow_googleapis_com.job.pubsub.pulled_message_ages-MilliSecond
PubsubIO.Read requests from Dataflow jobscloud.gcp.dataflow_googleapis_com.job.pubsub.read_count-Count
Pub/Sub Pull Request Latenciescloud.gcp.dataflow_googleapis_com.job.pubsub.read_latencies-MilliSecond
Percentage of active/terminated Streaming Pull connectionscloud.gcp.dataflow_googleapis_com.job.pubsub.streaming_pull_connection_status-Percent
Pub/Sub Publish Requestscloud.gcp.dataflow_googleapis_com.job.pubsub.write_count-Count
Pub/Sub Publish Request Latenciescloud.gcp.dataflow_googleapis_com.job.pubsub.write_latencies-MilliSecond
Statuscloud.gcp.dataflow_googleapis_com.job.status-Unspecified
Storage bytes readcloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.read_bytes_count-Count
Storage bytes writtencloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.write_bytes_count-Count
Storage write latenciescloud.gcp.dataflow_googleapis_com.job.streaming_engine.persistent_state.write_latencies-MilliSecond
Per stage end to end latencies.cloud.gcp.dataflow_googleapis_com.job.streaming_engine.stage_end_to_end_latencies-MilliSecond
System lagcloud.gcp.dataflow_googleapis_com.job.system_lag-Second
Thread Timecloud.gcp.dataflow_googleapis_com.job.thread_time-MilliSecond
Timers pending count per stagecloud.gcp.dataflow_googleapis_com.job.timers_pending_count-Count
Timers processed count per stagecloud.gcp.dataflow_googleapis_com.job.timers_processed_count-Count
Total memory usage timecloud.gcp.dataflow_googleapis_com.job.total_memory_usage_time-GigaByte
Total PD usage timecloud.gcp.dataflow_googleapis_com.job.total_pd_usage_time-GigaByte
Total SECU usagecloud.gcp.dataflow_googleapis_com.job.total_secu_usage-Unspecified
Total shuffle data processedcloud.gcp.dataflow_googleapis_com.job.total_shuffle_data_processed-Byte
Total streaming data processedcloud.gcp.dataflow_googleapis_com.job.total_streaming_data_processed-Byte
Total vCPU timecloud.gcp.dataflow_googleapis_com.job.total_vcpu_time-Second
User Countercloud.gcp.dataflow_googleapis_com.job.user_counter-Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

Version 1.0.5

Updated

  • Updated set of supported metrics

Full version history

Version 1.0.4

Updated

  • Feature sets & metrics section in HUB Product Information page

Full version history

Version 1.0.3

Added

  • Release notes to published extension
  • Auto publishing extensions to Hub

Full version history

No release notes

Full version history

No release notes

Dynatrace Hub
Get data into DynatraceBuild your own app
All (762)Log Management and AnalyticsKubernetesAI and LLM ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication ObservabilityApplication SecurityDigital ExperienceBusiness Analytics
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

Discover recent additions to Dynatrace

Problems logo

Problems

Analyze abnormal system behavior and performance problems detected by Davis AI.

Logs logo

Logs

Explore all your logs without writing a single query.

Security Investigator logo

Security Investigator

Fast and precise forensics for security and logs on Grail data with DQL queries.

Business Flow logo

Business Flow

Track, analyze, and optimize your critical business processes.

Cost & Carbon Optimization logo

Cost & Carbon Optimization

Track, analyze, and optimize your IT carbon footprint and public cloud costs.

Davis Anomaly Detection logo

Davis Anomaly Detection

Detect anomalies in timeseries using the Davis AI

Analyze your data

Understand your data better with deep insights and clear visualizations.

Notebooks logo

Notebooks

Create powerful, data-driven documents for custom analytics and collaboration.

Dashboards logo

Dashboards

Transform complex data into clear visualizations with custom dashboards.

Automate your processes

Turn data and answers into actions, securely, and at scale.

Workflows logo

Workflows

Automate tasks in your IT landscape, remediate problems, and visualize processes

Jira logo

Jira

Create, query, comment, transition, and resolve Jira tickets within workflows.

Slack logo

Slack

Automate Slack messaging for security incidents, attacks, remediation, and more.

Secure your cloud application

See vulnerabilities and attacks in your environment.

Security Overview logo

Security Overview

Get a comprehensive overview of the security of your applications.

Code-Level Vulnerabilities logo

Code-Level Vulnerabilities

Detect vulnerabilities in your code in real time.

Threats & Exploits logo

Threats & Exploits

Understand, triage, and investigate application security findings and alerts.

Are you looking for something different?

We have hundreds of apps, extensions, and other technologies to customize your environment

Leverage our newest innovations of Dynatrace Saas

Kick-start your app creation

Kick-start your app creation

Whether you’re a beginner or a pro, Dynatrace Developer has the tools and support you need to create incredible apps with minimal effort.
Go to Dynatrace Developer
Upgrading from Dynatrace Managed to SaaS

Upgrading from Dynatrace Managed to SaaS

Drive innovation, speed, and agility in your organization by seamlessly and securely upgrading.
Learn More
Log Management and Analytics

Log Management and Analytics

Innovate faster and more efficiently with unified log management and log analytics for actionable insights and automation.
Learn more