Overview
Fluent Bit is an open source and multiplatform log processor and forwarder that allows you to collect data/logs from different sources, unify and send them to multiple destinations and is fully compatible with Docker and Kubernetes environments.
While Fluent Bit has configurable HTTP output, Dyntrace can be configured as the target log management and analytics environment for your data.
This is intended for users who want to:
- Complement observability data in Dynatrace, extend the Davis AI engine, and enable App Owners, DevOps and SRE teams to centrally analyze all log data.
This enables you to:
- Leverage any of Fluent Bit input plugins, which connect to numerous data sources to get logs and events from your application into Dynatrace where you can immediately benefit from Dynatrace AI-driven root cause analysis.
Quickly search, explore, and analyze log and event data from a multitude of sources.
Use cases
Fluent Bit is the preferred solution when resource consumption is critical because it is a lightweight component.
In this example, we look at the AWS Fargate context, where FireLens makes it easy to set up Fluent Bit more quickly.
Ingest AWS Fargate logs with Fluent Bit
When creating a new task definition using the AWS Management Console, the FireLens integration section makes it easy to add a log router container. Just pick the built-in Fluent Bit image.
Next, edit the container in which your app-generating logs are running. In the โStorage and Loggingโ section, select โawsfirelensโ as the log driver.
The settings for the log driver should point to the log ingest API of your SaaS tenant. Note that you normally need to provide two headers for Fluent Bit: content type and authorization token. As FireLens supports only one header, you can pass the token as part of the URL. Your configuration for AWS FireLens should have the following:
- Name: http
- TLS: on
- TLS.verify: Off
- Format: json
- Header: Content-Type application/json; charset=utf-8
- Host: {your-environment-id}.live.dynatrace.com
- Port: 443
- URI: /api/v2/logs/ingest?api-token={your-API-token-here}
- Allow_Duplicated_Headers: false
- Json_Date_Format: iso8601
- Json_Date_Key: timestamp
To avoid publishing the token in plaintext, use AWS Secrets Manager to manage the token.
Referr to the AWS samples repository for task definition json with Dynatrace configuration.
As your application starts publishing logs, you can view them in Dynatrace.
Get started
The Fluent Bit http output plugin allows to forward your logs into a Dynatrace Generic logs ingest endpoint.
See Details section for configuration instructions.
Details
In order to send logs into a Dynatrace Generig logs ingest endpoint, run the HTTP output plugin through the configuration file.
In your main Fluent Bit configuration file, append the Output section with the following configurations:
[OUTPUT]
name http
match *
header Content-Type application/json; charset=utf-8
header Authorization Api-Token {your-API-token-here}
allow_duplicated_headers false
host {your-environment-id}.live.dynatrace.com
Port 443
URI /api/v2/logs/ingest
Format json
json_date_format iso8601
json_date_key timestamp
tls On
tls.verify Off
Note: You can place your-API-token in the header or as GET variable in URI (see example in Use cases tab)
Refer to the Fluent Bit HTTP output plugin documentation for detailed configuration description: https://docs.fluentbit.io/manual/pipeline/outputs/http
Compatibility information
- A Dynatrace API token with the logs.ingest (Ingest Logs) scope
- For Dynatrace SaaS Generic logs ingest endpoint is available in you environement. For Dynatrace managed or if Environment ActiveGate is your choice for an endpoint in local environment, install AG instance (From the main menu in the product go to Manage > Deploy Dynatarce > Install ActiveGate). Generic log ingest API v2 is automatically enabled on ActiveGate.
