Azure HDInsight

Dynatrace ingests metrics from Azure Metrics API for Azure HDInsight. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.
On the Azure HDInsights dashboard, you get holistic insights into your Hadoop, Spark, and Kafka resources and can cover all angles of your big data monitoring in one place.

Prerequisites

  • Dynatrace version 1.196+
  • Environment ActiveGate version 1.195+

Enable monitoring

To enable monitoring for Azure HDInsight, you first need to set up integration with Azure Monitor.

Optional Install OneAgent

For additional Hadoop, Spark, and Kafka insights, you can install OneAgent on Azure HDInsight cluster nodes.

Add the service to monitoring

In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.

Configure service metrics

Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics.

Recommended metrics:

  • Are enabled by default
  • Can't be disabled
  • Can have recommended dimensions (enabled by default, can't be disabled)
  • Can have optional dimensions (disabled by default, can be enabled).

Apart from the recommended metrics, most services have the possibility of enabling optional metrics.

Optional metrics:

  • Can be added and configured manually

View service metrics

Once you add the service to monitoring, you can view the service metrics in your Dynatrace environment either on your dashboard page or on the custom device overview page.

Import preset dashboards

Dynatrace provides preset Azure dashboards that you can import from GitHub to your environment's Dashboards page.
Note: To save a preset dashboard locally, create a new JSON file on your local machine and copy-paste the content of the JSON file from GitHub into the new file.
Once you save a preset dashboard locally, there are two ways to import it.

hdinsights-azure

Set up a management zone

To import a dashboard for Azure HDInsight, you need to set up a management zone to limit the entities displayed on the dashboard to cluster nodes only and exclude other hosts not relevant to the service.

When you create a management zone for this dashboard:

  1. Create a rule that identifies hosts based on a common property:
    • Host name contains the hdi string
  2. Create a rule that identifies custom devices based on a common property:
    • Custom device group contains the HDInsight string.
  3. Create a rule that identifies services based on a common property:
    • Service technology: Apache Hadoop
    • Service technology: Apache Hadoop Distributed File System
    • Service technology: Spark

After you create the management zone, assign it to your dashboard (from the dashboard, select Edit > Settings > Default management zone). For more information, see Dashboard timeframe and management zone.

Available metrics

Name Description Dimensions Unit Recommended
CategorizedGatewayRequests Number of gateway requests by categories (1xx/2xx/3xx/4xx/5xx) HttpStatus Count
GatewayRequests Number of gateway requests HttpStatus Count ✔️
NumActiveWorkers Number of active workers MetricName Count ✔️