Amazon MSK (Kafka) monitoring

Dynatrace ingests metrics for multiple preselected namespaces, including Amazon MSK (Kafka). You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.

Prerequisites

To enable monitoring for this service, you need

• An Environment or Cluster ActiveGate version 1.197+
  Note: For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an AWS EC2 host.
• Dynatrace version 1.203+
• An updated AWS monitoring policy to include the additional AWS services.

To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"acm-pca:ListCertificateAuthorities",
				"apigateway:GET",
				"appstream:DescribeFleets",
				"appsync:ListGraphqlApis",
				"athena:ListWorkGroups",
				"autoscaling:DescribeAutoScalingGroups",
				"cloudformation:ListStackResources",
				"cloudfront:ListDistributions",
				"cloudhsm:DescribeClusters",
				"cloudsearch:DescribeDomains",
				"cloudwatch:GetMetricData",
				"cloudwatch:GetMetricStatistics",
				"cloudwatch:ListMetrics",
				"codebuild:ListProjects",
				"datasync:ListTasks",
				"dax:DescribeClusters",
				"directconnect:DescribeConnections",
				"dms:DescribeReplicationInstances",
				"dynamodb:ListTables",
				"dynamodb:ListTagsOfResource",
				"ec2:DescribeAvailabilityZones",
				"ec2:DescribeInstances",
				"ec2:DescribeNatGateways",
				"ec2:DescribeSpotFleetRequests",
				"ec2:DescribeTransitGateways",
				"ec2:DescribeVolumes",
				"ec2:DescribeVpnConnections",
				"ecs:ListClusters",
				"eks:ListClusters",
				"elasticache:DescribeCacheClusters",
				"elasticbeanstalk:DescribeEnvironmentResources",
				"elasticbeanstalk:DescribeEnvironments",
				"elasticfilesystem:DescribeFileSystems",
				"elasticloadbalancing:DescribeInstanceHealth",
				"elasticloadbalancing:DescribeListeners",
				"elasticloadbalancing:DescribeLoadBalancers",
				"elasticloadbalancing:DescribeRules",
				"elasticloadbalancing:DescribeTags",
				"elasticloadbalancing:DescribeTargetHealth",
				"elasticmapreduce:ListClusters",
				"elastictranscoder:ListPipelines",
				"es:ListDomainNames",
				"events:ListEventBuses",
				"firehose:ListDeliveryStreams",
				"fsx:DescribeFileSystems",
				"gamelift:ListFleets",
				"glue:GetJobs",
				"inspector:ListAssessmentTemplates",
				"kafka:ListClusters",
				"kinesis:ListStreams",
				"kinesisanalytics:ListApplications",
				"kinesisvideo:ListStreams",
				"lambda:ListFunctions",
				"lambda:ListTags",
				"lex:GetBots",
				"logs:DescribeLogGroups",
				"mediaconnect:ListFlows",
				"mediaconvert:DescribeEndpoints",
				"mediapackage-vod:ListPackagingConfigurations",
				"mediapackage:ListChannels",
				"mediatailor:ListPlaybackConfigurations",
				"opsworks:DescribeStacks",
				"qldb:ListLedgers",
				"rds:DescribeDBClusters",
				"rds:DescribeDBInstances",
				"rds:DescribeEvents",
				"rds:ListTagsForResource",
				"redshift:DescribeClusters",
				"robomaker:ListSimulationJobs",
				"route53:ListHostedZones",
				"route53resolver:ListResolverEndpoints",
				"s3:ListAllMyBuckets",
				"sagemaker:ListEndpoints",
				"sns:ListTopics",
				"sqs:ListQueues",
				"storagegateway:ListGateways",
				"sts:GetCallerIdentity",
				"swf:ListDomains",
				"tag:GetResources",
				"tag:GetTagKeys",
				"transfer:ListServers",
				"workmail:ListOrganizations",
				"workspaces:DescribeWorkspaces"
			],
			"Resource": "*"
		}
	]
}

If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for all services (All monitored Amazon services) and, for each supporting service, a list of optional permissions specific to that service.

Example of JSON policy for one single service.

{
  "Version": "2012-10-17",
  "Statement": [
          {
                  "Sid": "VisualEditor0",
                  "Effect": "Allow",
                  "Action": [
                          "apigateway:GET",
                          "cloudwatch:GetMetricData",
                          "cloudwatch:GetMetricStatistics",
                          "cloudwatch:ListMetrics",
                          "sts:GetCallerIdentity",
                          "tag:GetResources",
                          "tag:GetTagKeys",
                          "ec2:DescribeAvailabilityZones"
                  ],
                  "Resource": "*"
          }
      ]
}

In this example, from the complete list of permissions you need to select

• "apigateway:GET" for Amazon API Gateway
• "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", and "ec2:DescribeAvailabilityZones" for All monitored Amazon services.

Enable monitoring

To enable monitoring for this service, you first need to integrate Dynatrace with Amazon Web Services:

• Set up Dynatrace SaaS integration
• Set up Dynatrace Managed integration

Add the service to monitoring

In order to view the service metrics, you must add the service to monitoring in your Dynatrace environment.

Monitor resources based on tags

You can choose to monitor resources based on existing AWS tags, as Dynatrace automatically imports them from service instances. Nevertheless, the transition from AWS to Dynatrace tagging isn't supported for all AWS services. Expand the table below to see which supporting services are filtered by tagging.

To monitor resources based on tags

1. In the Dynatrace menu, go to Settings > Cloud and virtualization > AWS and select the AWS instance.
2. For Resource monitoring method, select Monitor resources based on tags.
3. Enter the Key and Value.
4. Select Save.

Configure service metrics

Once you add a service, Dynatrace starts automatically collecting a suite of metrics for this particular service. These are recommended metrics.

Recommended metrics:

• Are enabled by default
• Can't be disabled
• Can have recommended dimensions (enabled by default, can't be disabled)
• Can have optional dimensions (disabled by default, can be enabled)

Apart from the recommended metrics, most services have the possibility of enabling optional metrics.

Optional metrics:

• Can be added and configured manually

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

View metrics on the custom device overview page

To access the custom device overview page

1. In the Dynatrace menu, go to Technologies.
2. Filter by service name and select the relevant custom device group.
3. Once you select the custom device group, you're on the custom device group overview page.
4. The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

After you add the service to monitoring, a preset dashboard containing all recommended metrics is automatically listed on your Dashboards page. To look for specific dashboards, filter by Preset and then by Name.

Note: For existing monitored services, you might need to resave your credentials for the preset dashboard to appear on the Dashboards page. To resave your credentials, go to Settings > Cloud and virtualization > AWS, select the desired AWS instance, and then select Save.

You can't make changes on a preset dashboard directly, but you can clone and edit it. To clone a dashboard, open the browse menu (…) and select Clone.
To remove a dashboard from the dashboards page, you can hide it. To hide a dashboard, open the browse menu (…) and select Hide.
Note: Hiding a dashboard doesn't affect other users.

Available metrics