Azure HDInsight monitoring

Dynatrace ingests metrics from Azure Metrics API for Azure HDInsight. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.
On the Azure HDInsights dashboard, you get holistic insights into your Hadoop, Spark, and Kafka resources and can cover all angles of your big data monitoring in one place.

Prerequisites

  • Dynatrace version 1.196+
  • Environment ActiveGate version 1.195+

Enable monitoring

To enable monitoring for Azure HDInsight, you first need to set up integration with Azure Monitor.

Install OneAgent optional

For additional Hadoop, Spark, and Kafka insights, you can install OneAgent on Azure HDInsight cluster nodes.

Add the service to monitoring

In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.

Monitor resources based on tags

You can choose to monitor resources based on existing Azure tags, as Dynatrace automatically imports them from service instances.

To monitor resources based on tags

  1. Go to Settings > Cloud and virtualization > Azure and select the Azure instance.
  2. For Resource monitoring method, select Monitor resources based on tags.
  3. Enter the Key and Value.
  4. Select Save.

Note: To import the Azure tags automatically into Dynatrace, enable Capture Azure tags automatically.

Configure service metrics

Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics.

Recommended metrics:

  • Are enabled by default
  • Can't be disabled
  • Can have recommended dimensions (enabled by default, can't be disabled)
  • Can have optional dimensions (disabled by default, can be enabled).

Apart from the recommended metrics, most services have the possibility of enabling optional metrics.

Optional metrics:

  • Can be added and configured manually

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

View metrics on the custom device overview page

To access the custom device overview page

  1. Go to Technologies on the Dynatrace navigation menu.
  2. Filter by service name and select the relevant custom device group.
  3. Once you select the custom device group, you're on the custom device group overview page.
  4. The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

Once you add a service to monitoring, a preset dashboard for the respective service containing all recommended metrics is automatically created on your Dashboards page. You can look for specific dashboards by filtering by Preset and then by Name.

Note: For existing monitored services, you might need to resave your credentials for the preset dashboard to appear on the Dashboards page. To resave your credentials, go to Settings > Cloud and virtualization > Azure, select the desired Azure instance, then select Save.

You can't make changes on a preset dashboard directly, but you can clone and edit it. To clone a dashboard, open the browse menu (...) and select Clone.
To remove a dashboard from the dashboards list, you can hide it. To hide a dashboard, open the browse menu (...) and select Hide.
Note: Hiding a dashboard doesn't affect other users.
clone-hide-azure

hdinsights-azure

Set up a management zone

To import a dashboard for Azure HDInsight, you need to set up a management zone to limit the entities displayed on the dashboard to cluster nodes only and exclude other hosts not relevant to the service.

When you create a management zone for this dashboard:

  1. Create a rule that identifies hosts based on a common property:
    • Host name contains the hdi string
  2. Create a rule that identifies custom devices based on a common property:
    • Custom device group contains the HDInsight string.
  3. Create a rule that identifies services based on a common property:
    • Service technology: Apache Hadoop
    • Service technology: Apache Hadoop Distributed File System
    • Service technology: Spark

After you create the management zone, assign it to your dashboard (from the dashboard, select Edit > Settings > Default management zone). For more information, see Dashboard timeframe and management zone.

Available metrics

Name Description Dimensions Unit Recommended
CategorizedGatewayRequests Number of gateway requests by categories (1xx/2xx/3xx/4xx/5xx) HttpStatus Count
GatewayRequests Number of gateway requests HttpStatus Count ✔️
NumActiveWorkers Number of active workers MetricName Count ✔️