Set up AWS log forwarding

AWS log forwarding allows you to stream logs from Amazon CloudWatch into Dynatrace logs via an ActiveGate.

DDU consumption for Log Monitoring

DDU pricing applies to cloud Log Monitoring. See DDUs for Log Monitoring for details.

To enable AWS log forwarding, you need to deploy our special-purpose CloudFormation stack into your AWS account. The stack consists of a Kinesis Firehose instance and a Lambda function, and, optionally, an EC2 instance for ActiveGate. These resources incur AWS costs according to standard AWS billing policy. The same applies to included self-monitoring resources (CloudWatch dashboards and metrics).

Supported services

Service name CloudWatch log forwarding Log enrichment Entity linking
AWS Lambda ✔️ ✔️ ✔️
AWS App Runner ✔️ ✔️ ✔️
AWS CloudTrail1 ✔️ ✔️ -
Amazon API Gateway ✔️ ✔️ -
Amazon SNS ✔️ ✔️ ✔️
Amazon RDS ✔️ ✔️ ✔️
All services that write to CloudWatch ✔️ ✔️ -
1

AWS CloudTrail log group name is chosen by user. For log enrichment, start the log group name with aws-cloudtrail-logs.

Prerequisites

Dynatrace

ActiveGate

You have two options:

  • Use the dedicated ActiveGate that is included in our CloudFormation stack. The ActiveGate will be installed on a new EC2 instance and will be placed in a VPC network with AWS log forwarder Lambda. recommended

    Note: Be sure you haven't reached your VPCs limit. The default limit is five VPCs per region. See Amazon VPC quotas for details.

  • Use an existing ActiveGate (version 1.217+ required). The ActiveGate has to be available publicly. The stack will be created without a dedicated ActiveGate if you choose this option.

API URL

Your API URL depends on whether you want to use the stack-included ActiveGate:

  • If you choose to use the stack-included ActiveGate, set it to your Dynatrace SaaS environment: https://<your_environment_ID>.live.dynatrace.com
  • Otherwise, set it to your ActiveGate endpoint: https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID>

Note: To determine <your_environment_ID>, see environment ID.

PaaS token

If you choose to use the stack-included ActiveGate, you need to create a PaaS token.

CLI

  • You can run the deployment from AWS CloudShell or from any machine with AWS CLI installed that supports Bash script execution.

Note: The deployment script uses the default AWS CLI profile configuration. The profile will determine the AWS account and region. To change the account or region:

Permissions

You need the following permissions to run the deployment script:

Deploy

Select one of the options below to deploy AWS log forwarding.

  1. Set the following environment variables in your environment.

Note: TARGET_URL and TARGET_API_TOKEN are required environment variables, and you need to replace the placeholders with your own values. Do not change REQUIRE_VALID_CERTIFICATE, as certificates aren't being provided on the new ActiveGate. See Deploy table for details.

TARGET_URL=<your_API_URL>
TARGET_API_TOKEN=<your_API_token>
TARGET_PAAS_TOKEN=<your_PaaS_token>
REQUIRE_VALID_CERTIFICATE=false
  1. Download the script and deploy the infrastructure.
wget -O dynatrace-aws-log-forwarder.zip https://github.com/dynatrace-oss/dynatrace-aws-log-forwarder/releases/latest/download/dynatrace-aws-log-forwarder.zip \
 && unzip -qo dynatrace-aws-log-forwarder.zip \
 && ./dynatrace-aws-logs.sh deploy --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --use-existing-active-gate false --require-valid-certificate $REQUIRE_VALID_CERTIFICATE --target-paas-token $TARGET_PAAS_TOKEN
  1. After deployment, search in the terminal output for the EC2ActiveGateHostname value to determine the name of your new ActiveGate and save it for later.
  2. In the Dynatrace menu, go to Deployment status.
  3. On the ActiveGates list, search for the name of the new ActiveGate you determined previously.
  4. Make sure that AutoUpdate is enabled.
  5. Select Save to save your configuration.
Deployment usage and options

For additional deployment options, see the command below.
Note: --target-url and --target-api-token are required parameters, and you need to replace the placeholders with your own values, while --require-valid-certificate and --stack-name are optional. See Deploy table for details.

dynatrace-aws-logs.sh deploy --target-url <your_API_URL> --target-api-token <your_API_token> --use-existing-active-gate {true|false} [--target-paas-token <your_PaaS_token>] [--require-valid-certificate {true|false}] [--stack-name <your_stack_name>]

Deploy table

Command-line parameter Environment variable Description Default value
--target-url TARGET_URL required The API URL to your Dynatrace SaaS environment logs ingest target.
See Prerequisites for instructions.
--target-api-token TARGET_API_TOKEN required Your API token. See Prerequisites for instructions.
--use-existing-active-gate USE_EXISTING_ACTIVE_GATE required If true, you are expected to provide your own ActiveGate. If false, a new EC2 with ActiveGate will be included in deployment.
--require-valid-certificate REQUIRE_VALID_CERTIFICATE optional If true, Dynatrace verifies the SSL certificate of your Dynatrace environment URL. false
--taget-paas-token TARGET_PAAS_TOKEN optional Your PaaS token. Required for deployment with a new ActiveGate. See Prerequisites for instructions.
--stack-name STACK_NAME optional The name of the CloudFormation stack where you want to deploy the resources. dynatrace-aws-logs

Subscribe to log groups

After deploying the infrastructure, you need to subscribe to the log groups whose logs you want to forward to Dynatrace.

To subscribe to log groups, you have the options described below.

Subscribe by listing log group names

Usage recommendation: Use this option if the number of log groups you'd like to subscribe to is small.

To subscribe: Run the command below, making sure to replace <your_log_group_list> with a space-separated list of the log group names you want to subscribe to.

Example list: /aws/lambda/my-lambda /aws/apigateway/my-api

dynatrace-aws-logs.sh subscribe --log-groups <your_log_group_list>

Subscribe by reading log groups from file

Usage recommendation: Use this option if the number of log groups you'd like to subscribe to is large.

  1. Create a file and enter each log group name on a separate line.
  2. Save the file.
  3. Run the command below, making sure to replace <your_file_name> with the actual file name.
    ./dynatrace-aws-logs.sh subscribe --log-groups-from-file <your_file_name>
    
Log groups auto-discovery

To simplify file creation, you can use the auto-discovery command below to list the names of all log groups in your account. You can adjust the list manually before subscribing.
Note: Be sure to replace <your_log_groups_file> with the name of the file to which you want to redirect the output.

./dynatrace-aws-logs.sh discover-log-groups > <your_log_groups_file>

Subscribe with a subscription filter pattern

Usage recommendation: By default, you subscribe to all the logs in the log group. Use this option if you want to restrict the logs you subscribe to. See Filter and Pattern Syntax for details on the pattern syntax.

Limitation: You can use only two subscription filters per log group, so the possibility of creating multiple filters with different patterns is limited. If you create a subscription filter that exceeds the limit, an AWS LimitExceededException occurs.

To subscribe: Run the command below, making sure to replace <your_log_group_list> and <your_filter_pattern> with your own values.

./dynatrace-aws-logs.sh subscribe --log-groups <your_log_group_list> --filter-pattern <your_filter_pattern>
Subscription usage and options

For additional subscription options, see the commands below.
Note: Consult the Subscription table for the commands below when replacing placeholders (<...>) with your own values.

dynatrace-aws-logs.sh subscribe {--log-groups <your_log_group_list> | --log-groups-from-file <your_file_name>}
                    [--stack-name <your_stack_name>] [--filter-pattern <your_filter_pattern>] [--role-arn ROLE_ARN] [--firehose-arn FIREHOSE_ARN]

Subscription table

Command-line parameter Environment variable Description Default value
--log-groups LOG_GROUPS_LIST A space-separated list of log group names you want to subscribe to. For example:
/aws/lambda/my-lambda /aws/apigateway/my-api.
--log-groups-from-file LOG_GROUPS_FILE A file listing the log groups you want to subscribe to. The file should contain each log group name on a separate line.
--filter-pattern FILTER_PATTERN If set, it allows you to subscribe to a filtered stream of logs. You subscribe to all logs in the log group.
--stack-name STACK_NAME The name of the CloudFormation stack where you have deployed the resources. dynatrace-aws-logs
--firehose-arn FIREHOSE_ARN Specify to which AWS Kinesis Data Firehose the logs should be streamed by providing its ARN (Amazon Resource Name).
Usage recommendation: Set this option if you have permission or performance issues with CloudFormation.
It will be extracted from the output of the CloudFormation stack used in the deploy step: either the $DEFAULT_STACK_NAME default value or the one specified with the --stack-name <your_stack_name> option.
--role-arn ROLE_ARN The ARN of an IAM role that grants CloudWatch Logs permission to deliver ingested log events to the destination stream.
Usage recommendation: Set this option if you have permission or performance issues with CloudFormation.
It will be extracted from the output of the CloudFormation stack used in the deploy step: either the $DEFAULT_STACK_NAME default value or the one specified with the --stack-name <your_stack_name> option.

Unsubscribe from log groups

If you don't want to forward logs to Dynatrace anymore, use one of the two options below to unsubscribe from log groups.

Unsubscribe by listing the log group names

Run the command below, making sure to replace <your_log_group_list> with a space-separated list of the log group names you want to unsubscribe from.

./dynatrace-aws-logs.sh unsubscribe --log-groups <your_log_group_list>

Unsubscribe by reading log groups from a file

Run the command below, making sure to replace <your_file_name> with the file name you created to subscribe by reading log groups from file.

./dynatrace-aws-logs.sh unsubscribe --log-groups-from-file <your_file_name>
Unsubscribe usage and options

For additional unsubscribe options, see the commands below.

Note: Consult the Unsubscribe table for the commands below when replacing the placeholders (<...>) with your own values.

dynatrace-aws-logs.sh unsubscribe {--log-groups <your_log_group_list> | --log-groups-from-file <your_file_name>} [--stack-name <your_stack_name>]

Unsubscribe table

Command-line parameter Environment variable Description Default value
--log-groups LOG_GROUPS_LIST A space-separated list of log group names you want to unsubscribe from. For example:
/aws/lambda/my-lambda /aws/apigateway/my-api.
--log-groups-from-file LOG_GROUPS_FILE A file listing log groups you want to unsubscribe from, with each log group name on a separate line.
--stack-name STACK_NAME The name of the CloudFormation stack where you have deployed the resources. dynatrace-aws-logs

Update to new version

To replace your old stack with a new version of the AWS log forwarder stack, deploy the new stack using the same parameters that you used before (especially the stack name, if you changed it from the default value).

Troubleshooting

Deployment status

To verify deployment correctness

  1. In AWS Console, go to CloudFormation.
  2. Select your log forwarder stack from the list on the left by stack name (the default value is dynatrace-aws-logs).
  3. If you find any issues or discrepancies in any of the fields below, select Delete to delete the stack, and then repeat the deployment process.
    • In Stack info, check the stack status; it should be CREATE_COMPLETE.
    • In Parameters, check if the parameter values are consistent with the values you provided during deployment.
    • In Events, look for any events with a failed status.

Connectivity status

To verify AWS log forwarder connectivity and inspect operational logs

  1. In AWS Console, go to CloudWatch Dashboards.
  2. Find the self-monitoring dashboard for AWS log forwarding. It will have a name like DynatraceLogForwarder-SelfMonitoring-eu-north-1-dynatrace-aws-logs, where the middle part is the AWS region and the last part is the stack name you chose (the default is dynatrace-aws-logs).
  3. Inspect the dashboard for any obvious issues.
  4. Go to CloudFormation.
  5. Select your log forwarder stack from the list on the left by stack name (the default value is dynatrace-aws-logs).
  6. Select the Resources tab and then select the link next to Lambda.
  7. On the Lambda screen, select the Monitor tab and then select Logs.
  8. Select one of the listed log streams and look for exceptions in the logs.

Uninstall AWS log forwarding

To uninstall AWS log forwarding

  1. Unsubscribe all log groups. You can use the log groups auto-discovery method described in Subscribe by reading log groups from file and pass the auto-discovery output file to the unsubscribe command. See Unsubscribe from log groups for details.
  2. In AWS Console, go to your CloudFormation stack.
  3. Select the Resources tab and navigate to the DeliveryBucket S3 bucket.
  4. Remove all objects in the bucket.
  5. Go back to the CloudFormation stack and select Delete.