• Home
  • Deploy Dynatrace
  • Set up Dynatrace on cloud platforms
  • Amazon Web Services
  • Integrations
  • Amazon Web Services
  • Amazon Elastic MapReduce (EMR) monitoring

Amazon Elastic MapReduce (EMR) monitoring

Dynatrace ingests metrics for multiple preselected namespaces, including Amazon Elastic MapReduce (EMR). You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.

Prerequisites

To enable monitoring for this service, you need

  • ActiveGate version 1.181+, as follows:
    • For Dynatrace SaaS deployments, you need an Environment ActiveGate or a Multi-environment ActiveGate.
    • For Dynatrace Managed deployments, you can use any kind of ActiveGate.
      Note: For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an Amazon EC2 host.
  • Dynatrace version 1.182+
  • An updated AWS monitoring policy to include the additional AWS services.
    To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.
JSON predefined policy for all supporting services
json
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "acm-pca:ListCertificateAuthorities", "apigateway:GET", "apprunner:ListServices", "appstream:DescribeFleets", "appsync:ListGraphqlApis", "athena:ListWorkGroups", "autoscaling:DescribeAutoScalingGroups", "cloudformation:ListStackResources", "cloudfront:ListDistributions", "cloudhsm:DescribeClusters", "cloudsearch:DescribeDomains", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "codebuild:ListProjects", "datasync:ListTasks", "dax:DescribeClusters", "directconnect:DescribeConnections", "dms:DescribeReplicationInstances", "dynamodb:ListTables", "dynamodb:ListTagsOfResource", "ec2:DescribeAvailabilityZones", "ec2:DescribeInstances", "ec2:DescribeNatGateways", "ec2:DescribeSpotFleetRequests", "ec2:DescribeTransitGateways", "ec2:DescribeVolumes", "ec2:DescribeVpnConnections", "ecs:ListClusters", "eks:ListClusters", "elasticache:DescribeCacheClusters", "elasticbeanstalk:DescribeEnvironmentResources", "elasticbeanstalk:DescribeEnvironments", "elasticfilesystem:DescribeFileSystems", "elasticloadbalancing:DescribeInstanceHealth", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeRules", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:DescribeTargetHealth", "elasticmapreduce:ListClusters", "elastictranscoder:ListPipelines", "es:ListDomainNames", "events:ListEventBuses", "firehose:ListDeliveryStreams", "fsx:DescribeFileSystems", "gamelift:ListFleets", "glue:GetJobs", "inspector:ListAssessmentTemplates", "kafka:ListClusters", "kinesis:ListStreams", "kinesisanalytics:ListApplications", "kinesisvideo:ListStreams", "lambda:ListFunctions", "lambda:ListTags", "lex:GetBots", "logs:DescribeLogGroups", "mediaconnect:ListFlows", "mediaconvert:DescribeEndpoints", "mediapackage-vod:ListPackagingConfigurations", "mediapackage:ListChannels", "mediatailor:ListPlaybackConfigurations", "opsworks:DescribeStacks", "qldb:ListLedgers", "rds:DescribeDBClusters", "rds:DescribeDBInstances", "rds:DescribeEvents", "rds:ListTagsForResource", "redshift:DescribeClusters", "robomaker:ListSimulationJobs", "route53:ListHostedZones", "route53resolver:ListResolverEndpoints", "s3:ListAllMyBuckets", "sagemaker:ListEndpoints", "sns:ListTopics", "sqs:ListQueues", "storagegateway:ListGateways", "sts:GetCallerIdentity", "swf:ListDomains", "tag:GetResources", "tag:GetTagKeys", "transfer:ListServers", "workmail:ListOrganizations", "workspaces:DescribeWorkspaces" ], "Resource": "*" } ] }

If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for all services (All monitored Amazon services) and, for each supporting service, a list of optional permissions specific to that service.

Complete list of permissions for cloud services
NameAdditional permissions
AWS Certificate Manager Private Certificate Authority"acm-pca:ListCertificateAuthorities"
All monitored Amazon services"cloudwatch:GetMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics",
"sts:GetCallerIdentity",
"tag:GetResources",
"tag:GetTagKeys",
"ec2:DescribeAvailabilityZones"
Amazon MQ
Amazon API Gateway"apigateway:GET"
AWS App Runner"apprunner:ListServices"
Amazon AppStream"appstream:DescribeFleets"
AWS AppSync"appsync:ListGraphqlApis"
Amazon Athena"athena:ListWorkGroups"
Amazon Aurora"rds:DescribeDBClusters"
Amazon EC2 Auto Scaling"autoscaling:DescribeAutoScalingGroups"
Amazon EC2 Auto Scaling (built-in)"autoscaling:DescribeAutoScalingGroups"
AWS Billing
Amazon Keyspaces
AWS Chatbot
Amazon CloudFront"cloudfront:ListDistributions"
AWS CloudHSM"cloudhsm:DescribeClusters"
Amazon CloudSearch"cloudsearch:DescribeDomains"
AWS CodeBuild"codebuild:ListProjects"
Amazon Cognito
Amazon Connect
Amazon Elastic Kubernetes Service (EKS)"eks:ListClusters"
AWS DataSync"datasync:ListTasks"
Amazon DynamoDB Accelerator (DAX)"dax:DescribeClusters"
Amazon Database Migration Service"dms:DescribeReplicationInstances"
Amazon DocumentDB"rds:DescribeDBClusters"
AWS Direct Connect"directconnect:DescribeConnections"
Amazon DynamoDB (built-in)"dynamodb:ListTables",
"dynamodb:ListTagsOfResource"
Amazon EBS (built-in)"ec2:DescribeVolumes"
Amazon EC2 API
Amazon EC2 (built-in)"ec2:DescribeInstances"
Amazon EC2 Spot Fleet"ec2:DescribeSpotFleetRequests"
Amazon Elastic Container Service (ECS)"ecs:ListClusters"
Amazon ECS ContainerInsights"ecs:ListClusters"
Amazon ElastiCache (EC)"elasticache:DescribeCacheClusters"
AWS Elastic Beanstalk"elasticbeanstalk:DescribeEnvironments"
Amazon Elastic File System (EFS)"elasticfilesystem:DescribeFileSystems"
Amazon Elastic Inference
Amazon Elastic Map Reduce (EMR)"elasticmapreduce:ListClusters"
Amazon Elasticsearch Service (ES)"es:ListDomainNames"
Amazon Elastic Transcoder"elastictranscoder:ListPipelines"
AWS Elastic Load Balancing (ELB) (built-in)"elasticloadbalancing:DescribeInstanceHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetHealth"
Amazon EventBridge"events:ListEventBuses"
Amazon FSx"fsx:DescribeFileSystems"
Amazon GameLift"gamelift:ListFleets"
AWS Glue"glue:GetJobs"
Amazon Inspector"inspector:ListAssessmentTemplates"
AWS Internet of Things (IoT)
AWS IoT Analytics
Amazon Managed Streaming for Kafka"kafka:ListClusters"
Amazon Kinesis Data Analytics"kinesisanalytics:ListApplications"
Amazon Kinesis Data Firehose"firehose:ListDeliveryStreams"
Amazon Kinesis Data Streams"kinesis:ListStreams"
Amazon Kinesis Video Streams"kinesisvideo:ListStreams"
AWS Lambda (built-in)"lambda:ListFunctions",
"lambda:ListTags"
Amazon Lex"lex:GetBots"
AWS Application and Network Load Balancer (built-in)"elasticloadbalancing:DescribeInstanceHealth",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetHealth"
Amazon CloudWatch Logs"logs:DescribeLogGroups"
AWS Elemental MediaConnect"mediaconnect:ListFlows"
Amazon MediaConvert"mediaconvert:DescribeEndpoints"
Amazon MediaPackage Live"mediapackage:ListChannels"
Amazon MediaPackage Video on Demand"mediapackage-vod:ListPackagingConfigurations"
Amazon MediaTailor"mediatailor:ListPlaybackConfigurations"
Amazon VPC NAT Gateways"ec2:DescribeNatGateways"
Amazon Neptune"rds:DescribeDBClusters"
AWS OpsWorks"opsworks:DescribeStacks"
Amazon Polly
Amazon QLDB"qldb:ListLedgers"
Amazon RDS (built-in)"rds:DescribeDBInstances",
"rds:DescribeEvents",
"rds:ListTagsForResource"
Amazon Redshift"redshift:DescribeClusters"
Amazon Rekognition
AWS RoboMaker"robomaker:ListSimulationJobs"
Amazon Route 53"route53:ListHostedZones"
Amazon Route 53 Resolver"route53resolver:ListResolverEndpoints"
Amazon S3"s3:ListAllMyBuckets"
Amazon S3 (built-in)"s3:ListAllMyBuckets"
Amazon SageMaker Batch Transform Jobs
Amazon SageMaker Endpoint Instances"sagemaker:ListEndpoints"
Amazon SageMaker Endpoints"sagemaker:ListEndpoints"
Amazon SageMaker Ground Truth
Amazon SageMaker Processing Jobs
Amazon SageMaker Training Jobs
AWS Service Catalog
Amazon Simple Email Service (SES)
Amazon Simple Notification Service (SNS)"sns:ListTopics"
Amazon Simple Queue Service (SQS)"sqs:ListQueues"
AWS Systems Manager - Run Command
AWS Step Functions
AWS Storage Gateway"storagegateway:ListGateways"
Amazon SWF"swf:ListDomains"
Amazon Textract
AWS IoT Things Graph
Amazon Transfer Family"transfer:ListServers"
AWS Transit Gateway"ec2:DescribeTransitGateways"
Amazon Translate
AWS Trusted Advisor
AWS API Usage
AWS Site-to-Site VPN"ec2:DescribeVpnConnections"
Amazon WAF Classic
Amazon WAF
Amazon WorkMail"workmail:ListOrganizations"
Amazon WorkSpaces"workspaces:DescribeWorkspaces"

Example of JSON policy for one single service.

JSON policy for Amazon API Gateway
json
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "apigateway:GET", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", "ec2:DescribeAvailabilityZones" ], "Resource": "*" } ] }

In this example, from the complete list of permissions you need to select

  • "apigateway:GET" for Amazon API Gateway
  • "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", and "ec2:DescribeAvailabilityZones" for All monitored Amazon services.

Enable monitoring

To enable monitoring for this service, you first need to integrate Dynatrace with Amazon Web Services:

  • Set up Dynatrace SaaS integration
  • Set up Dynatrace Managed integration

Add the service to monitoring

In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.

To add a service to monitoring
  1. In the Dynatrace menu, go to Settings > Cloud and virtualization and select AWS.
  2. On the AWS overview page, scroll down and select the desired AWS instance. Select the Edit button.
  3. Scroll down and select Add service. Choose the service name from the drop-down and select Add service.
  4. Select Save changes.

Note: Once AWS cloud services are added to monitoring, you might have to wait 15-20 minutes before the metric values are displayed.

Cloud-service monitoring consumption

All cloud services consume Davis data units (DDUs). The amount of DDU consumption per service instance depends on the number of monitored metrics and their dimensions (each metric dimension results in the ingestion of 1 data point; 1 data point consumes 0.001 DDUs).

Monitor resources based on tags

You can choose to monitor resources based on existing AWS tags, as Dynatrace automatically imports them from service instances. Nevertheless, the transition from AWS to Dynatrace tagging isn't supported for all AWS services. Expand the table below to see which cloud services are filtered by tagging.

Tag filtering per service
NameTags monitoring & filtering
AWS Certificate Manager Private Certificate Authorityyes
Amazon MQ-
Amazon API Gatewayyes
AWS App Runneryes
Amazon AppStreamyes
AWS AppSyncyes
Amazon Athenayes
Amazon Aurorayes
Amazon EC2 Auto Scaling-
Amazon EC2 Auto Scaling (built-in)yes
AWS Billing-
Amazon Keyspacesyes
AWS Chatbot-
Amazon CloudFrontyes
AWS CloudHSMyes
Amazon CloudSearch-
AWS CodeBuildyes
Amazon Cognito-
Amazon Connect-
Amazon Elastic Kubernetes Service (EKS)yes
AWS DataSyncyes
Amazon DynamoDB Accelerator (DAX)yes
Amazon Database Migration Serviceyes
Amazon DocumentDByes
AWS Direct Connectyes
Amazon DynamoDB (built-in)yes
Amazon EBS (built-in)yes
Amazon EC2 API-
Amazon EC2 (built-in)yes
Amazon EC2 Spot Fleet-
Amazon Elastic Container Service (ECS)yes
Amazon ECS ContainerInsightsyes
Amazon ElastiCache (EC)yes
AWS Elastic Beanstalkyes
Amazon Elastic File System (EFS)yes
Amazon Elastic Inferenceyes
Amazon Elastic Map Reduce (EMR)yes
Amazon Elasticsearch Service (ES)yes
Amazon Elastic Transcoder-
AWS Elastic Load Balancing (ELB) (built-in)yes
Amazon EventBridgeyes
Amazon FSxyes
Amazon GameLift-
AWS Glueyes
Amazon Inspectoryes
AWS Internet of Things (IoT)-
AWS IoT Analytics-
Amazon Managed Streaming for Kafkayes
Amazon Kinesis Data Analyticsyes
Amazon Kinesis Data Firehoseyes
Amazon Kinesis Data Streamsyes
Amazon Kinesis Video Streamsyes
AWS Lambda (built-in)yes
Amazon Lexyes
AWS Application and Network Load Balancer (built-in)yes
Amazon CloudWatch Logsyes
AWS Elemental MediaConnect-
Amazon MediaConvertyes
Amazon MediaPackage Liveyes
Amazon MediaPackage Video on Demandyes
Amazon MediaTailoryes
Amazon VPC NAT Gatewaysyes
Amazon Neptuneyes
AWS OpsWorksyes
Amazon Polly-
Amazon QLDByes
Amazon RDS (built-in)yes
Amazon Redshiftyes
Amazon Rekognition-
AWS RoboMakeryes
Amazon Route 53-
Amazon Route 53 Resolveryes
Amazon S3yes
Amazon S3 (built-in)yes
Amazon SageMaker Batch Transform Jobs-
Amazon SageMaker Endpoint Instancesyes
Amazon SageMaker Endpointsyes
Amazon SageMaker Ground Truth-
Amazon SageMaker Processing Jobs-
Amazon SageMaker Training Jobs-
AWS Service Catalog-
Amazon Simple Email Service (SES)-
Amazon Simple Notification Service (SNS)yes
Amazon Simple Queue Service (SQS)yes
AWS Systems Manager - Run Command-
AWS Step Functions-
AWS Storage Gatewayyes
Amazon SWF-
Amazon Textract-
AWS IoT Things Graph-
Amazon Transfer Familyyes
AWS Transit Gatewayyes
Amazon Translate-
AWS Trusted Advisor-
AWS API Usage-
AWS Site-to-Site VPNyes
Amazon WAF Classic-
Amazon WAF-
Amazon WorkMailyes
Amazon WorkSpacesyes

To monitor resources based on tags

  1. In the Dynatrace menu, go to Settings > Cloud and virtualization > AWS and select Edit for the desired AWS instance.
  2. For Resources to be monitored, select Monitor resources selected by tags.
  3. Enter the Key and Value.
  4. Select Save.

Configure service metrics

Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics. Apart from the recommended metrics, most services have the possibility of enabling optional metrics. You can remove or edit any of the existing metrics or any of their dimensions, where there are multiple dimensions available. Metrics consisting of only one dimension can't be edited. They can only be removed or added.

Service-wide metrics are metrics for the whole service across all regions. Typically, these metrics include dimensions containing Region in their name. If selected, these metrics are displayed on a separate chart when viewing your AWS deployment in Dynatrace. Keep in mind that available dimensions differ among services.

To change a metric's statistics, you have to recreate that metric by choosing different statistics. You can choose among the following statistics: Sum, Minimum, Maximum, Average, and Sample count. The Average + Minimum + Maximum statistics enable you to collect all three statistics as one metric instead of one statistic for three metrics separately. This can reduce your expenses for retrieving metrics from your AWS deployment.

To be able to save a newly added metric, you need to select at least one statistic and one dimension.

How to add and configure metrics
  1. In the Dynatrace menu, go to Settings > Cloud and virtualization and select AWS.
  2. On the AWS overview page, scroll down and select the desired AWS instance. Select the Edit button.
  3. Scroll down to the Services section and select Manage services.
  4. To add a metric, select the service for which you want to add metrics.
  5. Select Add new metric.
  6. From the menu, select the metric you want.
  7. Select Add metric to add the metric to monitoring.
  8. To configure a metric, select the Edit button associated to it.
  9. Select Apply to save your configuration.

Note: Once AWS cloud services are configured, you might have to wait 15-20 minutes before the metric values are displayed.

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

View metrics on the custom device overview page

To access the custom device overview page

  1. In the Dynatrace menu, go to Technologies and processes.
  2. Filter by service name and select the relevant custom device group.
  3. Once you select the custom device group, you're on the custom device group overview page.
  4. The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

You can also view metrics in the Dynatrace web UI on dashboards. There is no preset dashboard available for this service, but you can create your own dashboard.

To check the availability of preset dashboards for each AWS service, see the list below.

Preset dashboard availability list
AWS servicePreset dashboard
AWS Certificate Manager Private Certificate Authorityno
Amazon MQyes
Amazon API Gatewayno
AWS App Runnerno
Amazon AppStreamyes
AWS AppSyncyes
Amazon Athenayes
Amazon Aurorano
Amazon EC2 Auto Scalingyes
Amazon EC2 Auto Scaling (built-in)no
AWS Billingyes
Amazon Keyspacesyes
AWS Chatbotyes
Amazon CloudFrontno
AWS CloudHSMyes
Amazon CloudSearchyes
AWS CodeBuildyes
Amazon Cognitono
Amazon Connectyes
Amazon Elastic Kubernetes Service (EKS)yes
AWS DataSyncyes
Amazon DynamoDB Accelerator (DAX)yes
Amazon Database Migration Serviceyes
Amazon DocumentDByes
AWS Direct Connectyes
Amazon DynamoDB (built-in)no
Amazon EBS (built-in)no
Amazon EC2 APIyes
Amazon EC2 (built-in)no
Amazon EC2 Spot Fleetno
Amazon Elastic Container Service (ECS)no
Amazon ECS ContainerInsightsyes
Amazon ElastiCache (EC)no
AWS Elastic Beanstalkyes
Amazon Elastic File System (EFS)no
Amazon Elastic Inferenceyes
Amazon Elastic Map Reduce (EMR)no
Amazon Elasticsearch Service (ES)no
Amazon Elastic Transcoderyes
AWS Elastic Load Balancing (ELB) (built-in)no
Amazon EventBridgeyes
Amazon FSxyes
Amazon GameLiftyes
AWS Glueno
Amazon Inspectoryes
AWS Internet of Things (IoT)no
AWS IoT Analyticsyes
Amazon Managed Streaming for Kafkayes
Amazon Kinesis Data Analyticsno
Amazon Kinesis Data Firehoseno
Amazon Kinesis Data Streamsno
Amazon Kinesis Video Streamsno
AWS Lambda (built-in)no
Amazon Lexyes
AWS Application and Network Load Balancer (built-in)no
Amazon CloudWatch Logsyes
AWS Elemental MediaConnectyes
Amazon MediaConvertyes
Amazon MediaPackage Liveyes
Amazon MediaPackage Video on Demandyes
Amazon MediaTailoryes
Amazon VPC NAT Gatewaysno
Amazon Neptuneyes
AWS OpsWorksyes
Amazon Pollyyes
Amazon QLDByes
Amazon RDS (built-in)no
Amazon Redshiftno
Amazon Rekognitionyes
AWS RoboMakeryes
Amazon Route 53yes
Amazon Route 53 Resolveryes
Amazon S3no
Amazon S3 (built-in)no
Amazon SageMaker Batch Transform Jobsno
Amazon SageMaker Endpoint Instancesno
Amazon SageMaker Endpointsno
Amazon SageMaker Ground Truthno
Amazon SageMaker Processing Jobsno
Amazon SageMaker Training Jobsno
AWS Service Catalogyes
Amazon Simple Email Service (SES)no
Amazon Simple Notification Service (SNS)no
Amazon Simple Queue Service (SQS)no
AWS Systems Manager - Run Commandyes
AWS Step Functionsyes
AWS Storage Gatewayyes
Amazon SWFyes
Amazon Textractyes
AWS IoT Things Graphyes
Amazon Transfer Familyyes
AWS Transit Gatewayyes
Amazon Translateyes
AWS Trusted Advisoryes
AWS API Usageyes
AWS Site-to-Site VPNyes
Amazon WAF Classicyes
Amazon WAFyes
Amazon WorkMailyes
Amazon WorkSpacesyes

Available metrics

NameDescriptionUnitStatisticsDimensionsRecommended
AppsCompletedThe number of applications submitted to YARN that have completedCountSumJobFlowId; JobFlowId, JobId
AppsFailedThe number of applications submitted to YARN that have failed to completeCountSumJobFlowId; JobFlowId, JobId
AppsKilledThe number of applications submitted to YARN that have been killedCountSumJobFlowId; JobFlowId, JobId
AppsPendingThe number of applications submitted to YARN that are in a Pending stateCountSumJobFlowId; JobFlowId, JobId
AppsRunningThe number of applications submitted to YARN that are runningCountSumJobFlowId✔️
AppsRunningCountSumJobFlowId, JobId✔️
AppsSubmittedThe number of applications submitted to YARNCountSumJobFlowId; JobFlowId, JobId
BackupFailedShows if the last backup failed. Set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters.CountSumJobFlowId; JobFlowId, JobId
CapacityRemainingGBThe amount of remaining HDFS disk capacityCountSumJobFlowId; JobFlowId, JobId
ContainerAllocatedThe number of resource containers allocated by the resource managerCountSumJobFlowId; JobFlowId, JobId
ContainerPendingThe number of containers in the queue that have not yet been allocatedCountSumJobFlowId; JobFlowId, JobId
ContainerPendingRatioThe ratio (in numbers) of pending containers to containers allocated (ContainerPendingRatio = ContainerPending / ContainerAllocated). If ContainerAllocated = 0, then ContainerPendingRatio = ContainerPending.CountSumJobFlowId; JobFlowId, JobId
ContainerReservedThe number of containers reservedCountSumJobFlowId; JobFlowId, JobId
CoreNodesPendingThe number of core nodes waiting to be assigned (pending requests)CountSumJobFlowId; JobFlowId, JobId
CoreNodesRunningThe number of working core nodesCountSumJobFlowId; JobFlowId, JobId
CorruptBlocksThe number of blocks that HDFS reports as corruptedCountSumJobFlowId; JobFlowId, JobId
DfsPendingReplicationBlocksThe status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requestsCountSumJobFlowId; JobFlowId, JobId
HDFSBytesReadThe number of bytes read from HDFSCountSumJobFlowId; JobFlowId, JobId
HDFSBytesWrittenThe number of bytes written to HDFSCountSumJobFlowId; JobFlowId, JobId
HDFSUtilizationThe percentage of HDFS storage currently usedPercentAverageJobFlowId✔️
HDFSUtilizationPercentAverageJobFlowId, JobId✔️
HbaseBackupFailedShows if the last backup failed. Set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters.CountMinimumJobFlowId; JobFlowId, JobId
IsIdleIndicates that a cluster is no longer performing work, but is still alive and accruing charges. Set to 1 if no tasks are running and no jobs are running, and to 0 otherwise. This value is checked at five-minute intervals and a value of 1 indicates only that the cluster was idle when checked, not that it was idle for the entire five minutes.CountMinimumJobFlowId; JobFlowId, JobId✔️
IsIdleCountMinimumJobFlowId, JobId✔️
JobsFailedThe number of jobs in the cluster that have failedCountSumJobFlowId; JobFlowId, JobId
JobsRunningThe number of jobs in the cluster that are currently runningCountSumJobFlowId; JobFlowId, JobId
LiveDataNodesThe percentage of data nodes that are receiving work from HadoopCountSumJobFlowId; JobFlowId, JobId
LiveTaskTrackersThe percentage of task trackers that are functionalPercentAverageJobFlowId; JobFlowId, JobId
MRActiveNodesThe number of nodes presently running MapReduce tasks or jobs. Equivalent to YARN metric mapred.resourcemanager.NoOfActiveNodesCountSumJobFlowId; JobFlowId, JobId
MRDecommissionedNodesThe number of nodes allocated to MapReduce applications that have been marked in a Decommissioned stateCountSumJobFlowId; JobFlowId, JobId
MRLostNodesThe number of nodes allocated to MapReduce that have been marked in a Lost stateCountSumJobFlowId; JobFlowId, JobId
MRRebootedNodesThe number of nodes available to MapReduce that have been rebooted and marked in a Rebooted stateCountSumJobFlowId; JobFlowId, JobId
MRTotalNodesThe number of nodes presently available to MapReduce jobsCountSumJobFlowId; JobFlowId, JobId
MRUnhealthyNodesThe number of nodes available to MapReduce jobs marked in an Unhealthy stateCountSumJobFlowId; JobFlowId, JobId
MapSlotsOpenThe unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster.CountSumJobFlowId; JobFlowId, JobId
MapTasksRemainingThe number of remaining map tasks for each jobCountSumJobFlowId; JobFlowId, JobId
MapTasksRunningThe number of running map tasks for each jobCountSumJobFlowId✔️
MapTasksRunningCountSumJobFlowId, JobId✔️
MemoryAllocatedMBThe amount of memory allocated to the clusterCountSumJobFlowId; JobFlowId, JobId
MemoryAvailableMBThe amount of memory available for allocationCountSumJobFlowId; JobFlowId, JobId
MemoryReservedMBThe amount of memory reserved for allocationCountSumJobFlowId; JobFlowId, JobId
MemoryTotalMBThe total amount of memory in the clusterCountSumJobFlowId; JobFlowId, JobId
MissingBlocksThe number of blocks in which HDFS has no replicas. These might be corrupt blocks.CountSumJobFlowId; JobFlowId, JobId
MostRecentBackupDurationThe amount of time it took the previous backup to complete. This metric is set regardless of whether the last completed backup succeeded or failed. While the backup is ongoing, this metric returns the number of minutes after the backup started. This metric is only reported for HBase clusters.CountSumJobFlowId; JobFlowId, JobId
PendingDeletionBlocksThe number of blocks marked for deletionCountSumJobFlowId; JobFlowId, JobId
ReduceSlotsOpenUnused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster.CountSumJobFlowId; JobFlowId, JobId
ReduceTasksRemainingThe number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.CountSumJobFlowId; JobFlowId, JobId
ReduceTasksRunningThe number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated.CountSumJobFlowId; JobFlowId, JobId
RemainingMapTasksPerSlotThe ratio of the total map tasks remaining to the total map slots available in the clusterPercentAverageJobFlowId; JobFlowId, JobId
S3BytesReadThe number of bytes read from Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on EMR.CountSumJobFlowId; JobFlowId, JobId
S3BytesWrittenThe number of bytes written to Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on EMR.CountSumJobFlowId; JobFlowId, JobId
TaskNodesPendingThe number of task nodes waiting to be assigned (pending requests)CountSumJobFlowId; JobFlowId, JobId
TaskNodesRunningThe number of working task nodesCountSumJobFlowId; JobFlowId, JobId
TimeSinceLastSuccessfulBackupThe number of elapsed minutes after the last successful HBase backup started on your cluster. This metric is only reported for HBase clusters.CountSumJobFlowId; JobFlowId, JobId
TotalLoadThe total number of concurrent data transfersCountSumJobFlowId; JobFlowId, JobId
UnderReplicatedBlocksThe number of blocks that need to be replicated one or more timesCountSumJobFlowId; JobFlowId, JobId
YARNMemoryAvailablePercentageThe percentage of remaining memory available to YARN (YARNMemoryAvailablePercentage = MemoryAvailableMB / MemoryTotalMB)PercentAverageJobFlowId; JobFlowId, JobId