CloudWatch Logs

AWS log forwarding allows you to stream logs from Amazon CloudWatch into Dynatrace logs via an ActiveGate.

Resources needed

To enable AWS log forwarding, you need to deploy our special-purpose CloudFormation stack into your AWS account. The stack consists of a Kinesis Firehose instance and a Lambda function. These resources incur AWS costs according to standard AWS billing policy. The same applies to included self-monitoring resources (CloudWatch dashboards and metrics).

Supported services

Service name	CloudWatch log forwarding	Log enrichment	Entity linking
AWS Lambda
AWS App Runner
AWS CloudTrail 1			-
Amazon API Gateway			-
Amazon SNS
Amazon RDS
All services that write to CloudWatch			-
1 AWS CloudTrail log group name is chosen by user. For log enrichment, start the log group name with aws-cloudtrail-logs.

Limitations

AWS log forwarder supports maximum 1 GB of data processing per hour.

Prerequisites

Dynatrace

• If you are using Log Monitoring v1, enable the latest version of Dynatrace Log Monitoring
• Enable generic log ingestion
• Create an API token with Ingest logs (API v2) permission

CLI

• You can run the deployment from AWS CloudShell or from any machine with AWS CLI installed that supports Bash script execution.

• Use a different profile - in this case, you need to update your configuration file.
• Overwrite the default profile temporarily (this option is limited to your shell session) using environment variables.

Permission policy

You need the following permissions to run the deployment script:

jsoncopydownload
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:CreateChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DescribeStacks",
        "cloudformation:GetTemplateSummary",
        "ec2:DescribeImages",
        "s3:CreateBucket",
        "s3:PutLifecycleConfiguration",
        "s3:PutBucketPublicAccessBlock",
        "iam:GetRole",
        "iam:CreateRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:GetRolePolicy",
        "iam:PassRole",
        "lambda:CreateFunction",
        "lambda:UpdateFunctionCode",
        "lambda:GetFunction",
        "lambda:GetFunctionCodeSigningConfig",
        "cloudwatch:PutDashboard",
        "cloudwatch:GetDashboard",
        "firehose:DescribeDeliveryStream",
        "firehose:CreateDeliveryStream",
        "firehose:ListTagsForDeliveryStream",
        "logs:DeleteSubscriptionFilter",
        "logs:DescribeLogGroups",
        "logs:PutSubscriptionFilter",
        "ssm:GetParameters"
      ],
      "Resource": "*"
    }
  ]
}

Deploy

1. Set the following environment variables, making sure to replace the placeholders (<...>) with your own values.

   • For TARGET_URL, enter your environment URL: https://<your_environment_ID>.live.dynatrace.com. To learn how to determine your environment ID for the SaaS or Managed deployment, see environment ID.
   • For TARGET_API_TOKEN, enter your API token. For instructions, see Prerequisites.
   • optional For STACK_NAME, the default value is dynatrace-aws-logs. To provide another name for the CloudFormation stack where you want to deploy the resources, replace the default value with your own.

   shellcopydownload
   TARGET_URL=<your_environment_URL>
   TARGET_API_TOKEN=<your_API_token>
   STACK_NAME=dynatrace-aws-logs

2. Download the script and deploy the infrastructure.

   shellcopydownload
   wget -O dynatrace-aws-log-forwarder.zip https://github.com/dynatrace-oss/dynatrace-aws-log-forwarder/releases/latest/download/dynatrace-aws-log-forwarder.zip \
   && unzip -qo dynatrace-aws-log-forwarder.zip \
   && ./dynatrace-aws-logs.sh deploy --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --stack-name $STACK_NAME --require-valid-certificate true

Subscribe to log groups

After deploying the infrastructure, you need to subscribe to the log groups whose logs you want to forward to Dynatrace.

To subscribe to log groups, you have the options described below.

Subscribe by listing log group names

Usage recommendation: Use this option if the number of log groups you'd like to subscribe to is small.

To subscribe: Run the command below, making sure to replace <your_log_group_list> with a space-separated list of the log group names you want to subscribe to.

Example list: /aws/lambda/my-lambda /aws/apigateway/my-api

shellcopydownload
./dynatrace-aws-logs.sh subscribe --log-groups <your_log_group_list>

Subscribe by reading log groups from file

Usage recommendation: Use this option if the number of log groups you'd like to subscribe to is large.

1. Create a file and enter each log group name on a separate line.
2. Save the file.
3. Run the command below, making sure to replace <your_file_name> with the actual file name.

   shellcopydownload
   ./dynatrace-aws-logs.sh subscribe --log-groups-from-file <your_file_name>

shellcopydownload
./dynatrace-aws-logs.sh discover-log-groups > <your_log_groups_file>

Subscribe with a subscription filter pattern

Usage recommendation: By default, you subscribe to all the logs in the log group. Use this option if you want to restrict the logs you subscribe to. See Filter and Pattern Syntax for details on the pattern syntax.

Limitation: You can use only two subscription filters per log group, so the possibility of creating multiple filters with different patterns is limited. If you create a subscription filter that exceeds the limit, an AWS LimitExceededException occurs.

To subscribe: Run the command below, making sure to replace <your_log_group_list> and <your_filter_pattern> with your own values.

shellcopydownload
./dynatrace-aws-logs.sh subscribe --log-groups <your_log_group_list> --filter-pattern <your_filter_pattern>

shellcopydownload
dynatrace-aws-logs.sh subscribe {--log-groups <your_log_group_list> | --log-groups-from-file <your_file_name>}
                    [--stack-name <your_stack_name>] [--filter-pattern <your_filter_pattern>] [--role-arn ROLE_ARN] [--firehose-arn FIREHOSE_ARN]

Subscription table

Unsubscribe from log groups

If you don't want to forward logs to Dynatrace anymore, use one of the two options below to unsubscribe from log groups.

Unsubscribe by listing the log group names

Run the command below, making sure to replace <your_log_group_list> with a space-separated list of the log group names you want to unsubscribe from.

shellcopydownload
./dynatrace-aws-logs.sh unsubscribe --log-groups <your_log_group_list>

Unsubscribe by reading log groups from a file

Run the command below, making sure to replace <your_file_name> with the file name you created to subscribe by reading log groups from file.

shellcopydownload
./dynatrace-aws-logs.sh unsubscribe --log-groups-from-file <your_file_name>

shellcopydownload
dynatrace-aws-logs.sh unsubscribe {--log-groups <your_log_group_list> | --log-groups-from-file <your_file_name>} [--stack-name <your_stack_name>]

Unsubscribe table

Update to new version

To replace your old stack with a new version of the AWS log forwarder stack, deploy the new stack using the same parameters that you used before (especially the stack name, if you changed it from the default value).

Alternative deployments

If you don't want to use direct ingest through the Cluster API, you need to use an existing ActiveGate (version 1.217+). The ActiveGate must be available publicly. The stack will be created without a dedicated ActiveGate if you choose this option.

See below for instructions.

Deploy with existing ActiveGate

jsoncopydownload
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:CreateChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DescribeStacks",
        "cloudformation:GetTemplateSummary",
        "ec2:DescribeImages",
        "s3:CreateBucket",
        "s3:PutLifecycleConfiguration",
        "s3:PutBucketPublicAccessBlock",
        "iam:GetRole",
        "iam:CreateRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:GetRolePolicy",
        "iam:PassRole",
        "lambda:CreateFunction",
        "lambda:UpdateFunctionCode",
        "lambda:GetFunction",
        "lambda:GetFunctionCodeSigningConfig",
        "cloudwatch:PutDashboard",
        "cloudwatch:GetDashboard",
        "firehose:DescribeDeliveryStream",
        "firehose:CreateDeliveryStream",
        "firehose:ListTagsForDeliveryStream",
        "logs:DeleteSubscriptionFilter",
        "logs:DescribeLogGroups",
        "logs:PutSubscriptionFilter",
        "ssm:GetParameters"
      ],
      "Resource": "*"
    }
  ]
}

1. Set the following environment variables, making sure to replace the placeholders (<...>) with your own values, as follows.

   • For TARGET_URL, enter the API URL of your ActiveGate endpoint: https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID>. To learn how to determine your environment ID, see environment ID.
   • For TARGET_API_TOKEN, enter your API token. For instructions, see Prerequisites.

   If you want Dynatrace to verify the SSL certificate of your Dynatrace environment URL, you can set REQUIRE_VALID_CERTIFICATE to true.

   shellcopydownload
   TARGET_URL=<your_API_URL>
   TARGET_API_TOKEN=<your_API_token>
   REQUIRE_VALID_CERTIFICATE=false

2. Download the script and deploy the infrastructure.

   shellcopydownload
   wget -O dynatrace-aws-log-forwarder.zip https://github.com/dynatrace-oss/dynatrace-aws-log-forwarder/releases/latest/download/dynatrace-aws-log-forwarder.zip \
   && unzip -qo dynatrace-aws-log-forwarder.zip \
   && ./dynatrace-aws-logs.sh deploy --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --require-valid-certificate $REQUIRE_VALID_CERTIFICATE

shellcopydownload
dynatrace-aws-logs.sh deploy --target-url <your_API_URL> --target-api-token <your_API_token> [--target-paas-token <your_PaaS_token>] [--require-valid-certificate {true|false}] [--stack-name <your_stack_name>] [--max-log-length <max_log_content_length>] [--tags <value> [<value>...] ]

Deploy table

Troubleshooting

Deployment status

To verify deployment correctness

1. In AWS Console, go to CloudFormation.
2. Select your log forwarder stack from the list on the left by stack name (the default value is dynatrace-aws-logs).
3. If you find any issues or discrepancies in any of the fields below, select Delete to delete the stack, and then repeat the deployment process.
   • In Stack info, check the stack status; it should be CREATE_COMPLETE.
   • In Parameters, check if the parameter values are consistent with the values you provided during deployment.
   • In Events, look for any events with a failed status.

Connectivity status

To verify AWS log forwarder connectivity and inspect operational logs

1. In AWS Console, go to CloudWatch Dashboards.
2. Find the self-monitoring dashboard for AWS log forwarding. It will have a name like DynatraceLogForwarder-SelfMonitoring-eu-north-1-dynatrace-aws-logs, where the middle part is the AWS region and the last part is the stack name you chose (the default is dynatrace-aws-logs).
3. Inspect the dashboard for any obvious issues.
4. Go to CloudFormation.
5. Select your log forwarder stack from the list on the left by stack name (the default value is dynatrace-aws-logs).
6. Select the Resources tab and then select the link next to Lambda.
7. On the Lambda screen, select the Monitor tab and then select Logs.
8. Select one of the listed log streams and look for exceptions in the logs.

Uninstall AWS log forwarding

To uninstall AWS log forwarding

1. Unsubscribe all log groups. You can use the log groups auto-discovery method described in Subscribe by reading log groups from file and pass the auto-discovery output file to the unsubscribe command. See Unsubscribe from log groups for details.
2. In AWS Console, go to your CloudFormation stack.
3. Select the Resources tab and navigate to the DeliveryBucket S3 bucket.
4. Remove all objects in the bucket.
5. Go back to the CloudFormation stack and select Delete.