Google Dataflow monitoring
Dynatrace GCP integration leverages data collected from the Google Operation API to constantly monitor health and performance of Google Cloud Platform Services. While combining all relevant data into dashboards, it also enables alerting and event tracking.
Prerequisites
Add services and feature sets optional
After integration, Dynatrace automatically monitors a number of preset GCP services and feature sets (metrics). Besides these, you can add more services or feature sets to monitoring. For details, see Add or remove services.
For a list of feature sets available for this service, see Metric table.
View metrics
After deploying the integration, you can see metrics from monitored services in the Metrics browser, the Data explorer, and your dashboard tiles.
Metric table
The following feature sets are available for Google Dataflow.
| Feature set | Name | Unit | GCP metric identifier |
|---|---|---|---|
| dataflow_job/default_metrics | Billable shuffle data processed | Byte | dataflow.googleapis.com/job/billable_shuffle_data_processed |
| dataflow_job/default_metrics | Current number of vCPUs in use | Count | dataflow.googleapis.com/job/current_num_vcpus |
| dataflow_job/default_metrics | Current shuffle slots in use | Count | dataflow.googleapis.com/job/current_shuffle_slots |
| dataflow_job/default_metrics | Data watermark lag | Second | dataflow.googleapis.com/job/data_watermark_age |
| dataflow_job/default_metrics | Elapsed time | Second | dataflow.googleapis.com/job/elapsed_time |
| dataflow_job/default_metrics | Element count | Count | dataflow.googleapis.com/job/element_count |
| dataflow_job/default_metrics | Elements Produced | Count | dataflow.googleapis.com/job/elements_produced_count |
| dataflow_job/default_metrics | Estimated byte count | Byte | dataflow.googleapis.com/job/estimated_byte_count |
| dataflow_job/default_metrics | Estimated Bytes Produced | Count | dataflow.googleapis.com/job/estimated_bytes_produced_count |
| dataflow_job/default_metrics | Failed | Count | dataflow.googleapis.com/job/is_failed |
| dataflow_job/default_metrics | Per-stage data watermark lag | Second | dataflow.googleapis.com/job/per_stage_data_watermark_age |
| dataflow_job/default_metrics | Per-stage system lag | Second | dataflow.googleapis.com/job/per_stage_system_lag |
| dataflow_job/default_metrics | PubsubIO.Read requests from Dataflow jobs | Count | dataflow.googleapis.com/job/pubsub/read_count |
| dataflow_job/default_metrics | Pub/Sub Pull Request Latencies | MilliSecond | dataflow.googleapis.com/job/pubsub/read_latencies |
| dataflow_job/default_metrics | Pub/Sub Publish Requests | Count | dataflow.googleapis.com/job/pubsub/write_count |
| dataflow_job/default_metrics | Pub/Sub Publish Request Latencies | MilliSecond | dataflow.googleapis.com/job/pubsub/write_latencies |
| dataflow_job/default_metrics | dataflow.googleapis.com/job/status | ||
| dataflow_job/default_metrics | System lag | Second | dataflow.googleapis.com/job/system_lag |
| dataflow_job/default_metrics | Total memory usage time | GigaByte | dataflow.googleapis.com/job/total_memory_usage_time |
| dataflow_job/default_metrics | Total PD usage time | GigaByte | dataflow.googleapis.com/job/total_pd_usage_time |
| dataflow_job/default_metrics | Total shuffle data processed | Byte | dataflow.googleapis.com/job/total_shuffle_data_processed |
| dataflow_job/default_metrics | Total streaming data processed | Byte | dataflow.googleapis.com/job/total_streaming_data_processed |
| dataflow_job/default_metrics | Total vCPU time | Second | dataflow.googleapis.com/job/total_vcpu_time |
| dataflow_job/default_metrics | User Counter | Count | dataflow.googleapis.com/job/user_counter |