Azure HDInsight monitoring
Dynatrace ingests metrics from Azure Metrics API for Azure HDInsight. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.
On the Azure HDInsights dashboard, you get holistic insights into your Hadoop, Spark, and Kafka resources and can cover all angles of your big data monitoring in one place.
Prerequisites
- Dynatrace version 1.196+
- Environment ActiveGate version 1.195+
Enable monitoring
To enable monitoring for this service, you first need to set up integration with Azure Monitor.
Install OneAgent optional
For additional Hadoop, Spark, and Kafka insights, you can install OneAgent on Azure HDInsight cluster nodes.
Add the service to monitoring
In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.
Monitor resources based on tags
You can choose to monitor resources based on existing Azure tags, as Dynatrace automatically imports them from service instances.
To monitor resources based on tags
- In the Dynatrace menu, go to Settings > Cloud and virtualization > Azure and select Edit for the desired Azure instance.
- For Resources to be monitored, select Monitor resources selected by tags.
- Enter the Key and Value.
- Select Save to save your configuration.
Note: To import the Azure tags automatically into Dynatrace, enable Capture Azure tags automatically.
Configure service metrics
Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics.
Recommended metrics:
- Are enabled by default
- Can't be disabled
- Can have recommended dimensions (enabled by default, can't be disabled)
- Can have optional dimensions (disabled by default, can be enabled).
Apart from the recommended metrics, most services have the possibility of enabling optional metrics.
Optional metrics:
- Can be added and configured manually
View service metrics
You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.
View metrics on the custom device overview page
To access the custom device overview page
- In the Dynatrace menu, go to Technologies.
- Filter by service name and select the relevant custom device group.
- Once you select the custom device group, you're on the custom device group overview page.
- The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.
View metrics on your dashboard
Once you add a service to monitoring, a preset dashboard for the respective service containing all recommended metrics is automatically created on your Dashboards page. You can look for specific dashboards by filtering by Preset and then by Name.
Note: For existing monitored services, you might need to resave your credentials for the preset dashboard to appear on the Dashboards page. To resave your credentials, go to Settings > Cloud and virtualization > Azure, select the desired Azure instance, then select Save.
You can't make changes on a preset dashboard directly, but you can clone and edit it. To clone a dashboard, open the browse menu (…) and select Clone.
To remove a dashboard from the dashboards list, you can hide it. To hide a dashboard, open the browse menu (…) and select Hide.
Note: Hiding a dashboard doesn't affect other users.
Set up a management zone
To import a dashboard for Azure HDInsight, you need to set up a management zone to limit the entities displayed on the dashboard to cluster nodes only and exclude other hosts not relevant to the service.
When you create a management zone for this dashboard:
- Create a rule that identifies hosts based on a common property:
- Host name contains the
hdi
string
- Host name contains the
- Create a rule that identifies custom devices based on a common property:
- Custom device group contains the
HDInsight
string.
- Custom device group contains the
- Create a rule that identifies services based on a common property:
- Service technology:
Apache Hadoop
- Service technology:
Apache Hadoop Distributed File System
- Service technology:
Spark
- Service technology:
After you create the management zone, assign it to your dashboard (from the dashboard, select Edit > Settings > Default management zone). For more information, see Dashboard timeframe and management zone.
Available metrics
Name | Description | Dimensions | Unit | Recommended |
---|---|---|---|---|
CategorizedGatewayRequests | Number of gateway requests by categories (1xx/2xx/3xx/4xx/5xx) | HttpStatus | Count | |
GatewayRequests | Number of gateway requests | HttpStatus | Count | ✔️ |
NumActiveWorkers | Number of active workers | MetricName | Count | ✔️ |