How to configure AWS S3 log sources

Importing AWS logs into Dynatrace works as follows:

  1. CloudTrail logs are sent to S3 bucket and S3 events are created.
  2. S3 events are queued in the SQS queue.
  3. Dynatrace, based on its AWS log source configuration, reads the CloudTrail logs into Dynatrace Log Analytics.

Configuration within AWS

To configure AWS as a log source, you need to have access to the following AWS services: CloudTrail, S3, SQS.

  1. Set up Dynatrace permission to AWS.
  2. Create an S3 bucket for CloudTrail logs.
  3. Create an SQS queue for S3 events.
  4. Set up AWS as a log source in Dynatrace.

Give Dynatrace permission to AWS

Using your AWS account, create a policy using the following credentials for Dynatrace to access AWS. In addition to standard permissions for CloudWatch access, you will need additional permissions for SQS and S3. You can use the provided AWS credentials example file.

Create S3 bucket for logs

To examine CloudTrail log files for a period longer than the past seven days, you must deliver the files to the S3 bucket.

Using the CloudTrail console, configure CloudTrail for an AWS account:

In your AWS console, navigate to the CloudTrail dashboard.

Click Create trail.
Type in the Trail name and keep the default settings:

  • Apply trail to all regions: Yes
  • Read/Write events: All

In the Storage location section, either use an existing S3 bucket or create a new S3 bucket and type in its name.
Make sure to name the S3 bucket with a globally unique name.

Options:

  • You can turn on bucket encryption using the KMS key (see the Security panel in AWS).
  • You can define retention for the S3 bucket (use S3 object life cycle management).
    By default, retention is unlimited.

Click Create to create the S3 bucket. All logs from all regions for this account will feed into this S3 bucket.

Optionally enhance S3 security

By default, the log files delivered to the CloudTrail S3 bucket are encrypted with SSE-S3 keys (Amazon server-side encryption S3-managed). You can add an additional layer of security by using SSE-KMS keys (also known as CMK - customer master key), in which case you can read logs only if you're granted the decrypt permission by the CMK policy. For more information, see Encrypting CloudTrail log files with AWS KMS–managed keys (SSE-KMS) (Amazon documentation).

Create log queue

To create an SQS queue for the S3 bucket that has been configured to obtain CloudTrail logs:

In your AWS console, navigate to the SQS dashboard.

Click Create new Queue. Type in the Queue Name and Region.

On the Standard Queue tab, click Configure Queue. Set the Message Retention period to 1 hour.

Click Create Queue. Your newly created queue will appear on the SQS dashboard.

Make a note of the ARN.
On the SQS dashboard, highlight the new SQS queue for Dynatrace and, on the Details tab, note the ARN. You'll need this ARN when setting up AWS as a log source in Dynatrace.

Assign S3 bucket to the log queue

In your AWS console, navigate to the S3 dashboard.

Select the S3 bucket that you created earlier when creating a trail, and then select the Properties tab.

Click the Events box in the Advanced settings section.

Click Add notification.

  • Type in the Name.
  • Select the All object create events option.
  • Enter gz for the Suffix option.
  • Set Send to to SQS Queue.
  • Set SQS to the new SQS queue you created earlier.

Allow the S3 event to connect to SQS. Use the following policy file:

Click Save.
Optionally, you can go to the SQS dashboard and check to see if messages are coming in.
(AWS may send one "Welcome" SQS notification when you define a new event.)

Avoid creating duplicates

Don't define two events for the same bucket (for example, to push events to two different SQSes).

Configuration within Dynatrace

After you finish the AWS portion of configuration, Dynatrace configuration is as follows.

In Dynatrace, open the AWS dashboard: Settings > Cloud and virtualization > AWS.

Click Connect new instance or edit an existing AWS instance.

Provide the AWS authentication method.
See Connect your Amazon account to Dynatrace

Set the Import CloudTrail logs from SQS-based S3 buckets switch to the On position.

In the SQS ARN box, type the SQS ARN.

Locating the ARN

To get the SQS ARN, sign in to your AWS account, open the SQS dashboard, and highlight the SQS queue for Dynatrace. The ARN is displayed on the Details tab.