Set up Azure log forwarding
DDU pricing applies to cloud Log Monitoring. See DDUs for Log Monitoring for details.
Azure log forwarding allows you to stream Azure logs from Azure Event Hubs into Dynatrace logs via an Azure Function App instance. It supports both Azure resource logs and activity logs.
Resources to be deployed
Azure log forwarding is performed directly through Cluster API. If you don't want to use direct ingest through the Cluster API, you have two options:
- Use Azure log forwarding setup to automatically deploy and configure a dedicated containerized ActiveGate.
- Use an existing ActiveGate for log ingestion.
The following resources will be created by the Azure logs forwarder deployment script:
- Storage account (
Microsoft.Storage/storageAccounts
) - Storage Account Blob Service (
Microsoft.Storage/storageAccounts/blobServices
) - Azure App Service plan (
Microsoft.Web/serverfarms
) - Azure Function App (
Microsoft.Web/sites
)
If you choose to use a new ActiveGate, the following resources will also be created, besides the ones already mentioned:
- Virtual Network (
Microsoft.Network/VirtualNetworks
) - Subnet (
Microsoft.Network/virtualNetworks/subnets
) - Network Profile (
Microsoft.Network/networkProfiles
) - Container Instance (
Microsoft.ContainerInstance/containerGroups
)
For details about the resources created, see the Azure Resource Manager file on GitHub
Prerequisites
Dynatrace version 1.230+Note: If you're using an earlier version of Dynatrace, see Alternative deployments for instructions.
See below the list of requirements for setting up Azure log forwarding. Some are needed before you start deployment, others during the deployment process.
Dynatrace
- Enable generic log ingestion
- The latest version of Dynatrace log monitoring
- Create an API token and enable the Ingest logs permission
Azure
- Set up an Azure Event Hubs instance in each Azure location from where you want to pull logs.
- Select the resource group where you want the deployment to run, or create a new one for the deployment. To create a new Azure resource group, run the command below, making sure to replace the placeholders with your actual values.
The Event Hubs instances and the resource group in which the deployment will run need to be in the same region to be able to send logs.
az group create --name <your_resource_group> --location <your_resource_group_region>
- Create an authorization rule with the listen permission for the Event Hubs instance that is configured for receiving logs:
az eventhubs eventhub authorization-rule create --resource-group <your_resource_group> --namespace-name <your_event_hub_namespace> --eventhub-name <your_event_hub_instance> --name <authorization_rule_name> --rights Listen
- Get an Event Hubs connection string for the authorization rule created above:
az eventhubs eventhub authorization-rule keys list --resource-group <your_resource_group> --namespace-name <your_event_hub_namespace> --eventhub-name <your_event_hub_instance> --name <your_authorization_rule_name>
- Configure diagnostic settings for resources from which you want to stream logs so that the diagnostic settings point to Azure Event Hubs instances.
CLI
- You can run Azure log forwarding deployment using Azure Portal Cloud Shell (Bash) or from any machine with Azure CLI and Bash shell (Linux or Windows WSL).
Deploy
- Set the following environment variables, making sure to replace the placeholders (
<...>
) with your own values.
- For
DEPLOYMENT_NAME
, enter your deployment name (lowercase only). - For
TARGET_URL
, enter your environment URL:https://<your_environment_ID>.live.dynatrace.com
. To learn how to determine your environment ID, see environment ID. - For
TARGET_API_TOKEN
, enter your API token. See Dynatrace requirements for details. - For
RESOURCE_GROUP
, enter the name of the Azure resource group in which deployment will run. See Azure requirements for details. - For
EVENT_HUB_CONNECTION_STRING
, enter the connection string for the Azure Event Hubs instances configured for receiving logs. See Azure requirements for details.
Note: optional You can enable self-monitoring and/or log filtering during or after deployment.
DEPLOYMENT_NAME=<your_deployment_name>
TARGET_URL=<your_environment_URL>
TARGET_API_TOKEN=<your_API_token>
RESOURCE_GROUP=<your_resource_group>
EVENT_HUB_CONNECTION_STRING=<your_Event_Hub_connection_string>
- Download the
azure-log-forwarder-function
script and deploy the infrastructure.
wget -q https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-logs.sh -O dynatrace-azure-logs.sh && chmod +x ./dynatrace-azure-logs.sh \
&& ./dynatrace-azure-logs.sh --deployment-name $DEPLOYMENT_NAME --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --resource-group $RESOURCE_GROUP --event-hub-connection-string $EVENT_HUB_CONNECTION_STRING --use-existing-active-gate true --require-valid-certificate true
View Azure logs
After deploying the script, you can view and analyze Azure logs in Dynatrace: go to Analyze > Logs and filter for cloud.provider: azure
.
Self-monitoring optional
Self-monitoring allows quick diagnosis to see if your function processes and sends logs to Dynatrace properly.
Enable self-monitoring
To enable self-monitoring, you have two options:
-
During deployment: Set the
--enable-self-monitoring
parameter (or theSFM_ENABLED
environment variable) totrue
. -
After deployment: In Azure Portal, go to the configuration of your deployed Function App instance and set
SELF_MONITORING_ENABLED
totrue
.
After enabling self-monitoring, you need to enable managed identity for your Function App instance created during deployment, and configure it to allow pushing metrics to the resource.
Self-monitoring metrics
Once you enable self-monitoring, you can view the following metrics in your dynatrace_logs_self_monitoring
namespace of the newly deployed Function App instance.
Metric name | Description | Dimension |
---|---|---|
all_requests | All requests sent to Dynatrace. | |
dynatrace_connectivity_failures | Reported when any Dynatrace connectivity issues occurred. | connectivity_status |
parsing_errors | Reported when any parsing errors occurred during log processing. | |
processing_time | Time needed to process all logs. | |
sending_time | Time needed to send all requests. | |
too_long_content_size | Reported when content of log is too long. The content will be trimmed. | |
too_old_records | Reported when logs received from Event Hubs are too old. |
Log filtering optional
To reduce the number of logs that are sent to Dynatrace, you can apply filters.
To apply filters you have two options:
-
During deployment: Set the
FILTER_CONFIG
environment variable in Azure Portal Cloud Shell (Bash) before running the deployment script.-
Add the
FILTER_CONFIG
environment variable to the list of environment variables needed for the deployment script.Note: Be sure to replace placeholders with your values. See Filter options for details.
FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=<log_level>;FILTER.GLOBAL.CONTAINS_PATTERN=<pattern>;FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.<resource_type>=<log_level>;FILTER.RESOURCE_TYPE.CONTAINS_PATTERN.<resource_type>=<pattern>;FILTER.RESOURCE_ID.MIN_LOG_LEVEL.<resource_id>=<log_level>;FILTER.RESOURCE_ID.CONTAINS_PATTERN.<resource_id>=<pattern>"
-
Set the environment variables.
-
Download the
azure-log-forwarder-function
script and deploy the infrastructure.
-
-
After deployment: Add
FILTER_CONFIG
in Azure Portal.-
In Azure Portal, go to the Configuration of your deployed Function App instance.
-
In Application settings, search and select FILTER_CONFIG.
Note: FILTER_CONFIG will appear in Azure after running the deployment script.
-
Select Edit to add a Value for your filter.
Note: Alternatively, you can select Advanced edit to enter your value in the JSON.
-
Select OK.
-
Restart your Function App instance.
-
Filter options
FILTER_CONFIG
is a key-value pair variable. You can set two types of filters (MIN_LOG_LEVEL
and/or CONTAINS_PATTERN
) for three filter groups (GLOBAL
, RESOURCE_TYPE
, and/or RESOURCE_ID
).
Filter type: MIN_LOG_LEVEL
This filter type allows you to filter out logs with unwanted levels. Possible log levels are:
- Critical (or
1
) - Error (or
2
) - Warning (or
3
) - Informational (or
4
)
Example:
FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=Warning"
In the example above, Informational logs will be skipped, and only Warning, Error, and Critical logs will be sent to Dynatrace.
Syntax options are:
FILTER.GLOBAL.MIN_LOG_LEVEL=<log_level>
FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.<resource_type>=<log_level>
FILTER.RESOURCE_ID.MIN_LOG_LEVEL.<resource_id>=<log_level>
You can have one global-level filter and additional filters for a particular resource type/ID.
Example:
FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=Error;FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.MICROSOFT.WEB/SITES=Informational"
In the example above, all logs from instances with resource type MICROSOFT.WEB/SITES
will be sent to Dynatrace, while for all other resources, Informational and Warning logs will be filtered out.
Filter type: CONTAINS_PATTERN
This filter type allows you to collect logs containing a particular text. We use fnmatch, which provides support for Unix shell–style wildcards. See Unix filename pattern matching for details.
Syntax options are:
FILTER.GLOBAL.CONTAINS_PATTERN=<log_pattern>
FILTER.RESOURCE_TYPE.CONTAINS_PATTERN.<resource_type>=<log_pattern>
FILTER.RESOURCE_ID.CONTAINS_PATTERN.<resource_id>=<log_pattern>
Filter group: GLOBAL
This filter is set for all logs.
Filter group: RESOURCE_TYPE
This filter is used only for logs coming from resources of the given Azure resource type, such as Microsoft.Compute/virtualMachines
.
You can find the resource type in Azure Portal, in your resource's Properties.
Note: If the Type field doesn't appear in Properties, you can extract it from the resource ID string.
Resource ID string syntax:
/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/<resourceType>/<resourceName>
The resource type will be the part between /providers/
and /resourceName/
.
Filter group: RESOURCE_ID
This filter is used only for logs coming from the given resource that is identified by the Azure resource ID.
You can look for the resource type in Azure Portal, in your resource's Properties.
Filter rules
-
If you set two filter types for the same group, both conditions need to be met, so the second filter will have to match the first filter.
For example, if you set
MIN_LOG_LEVEL
to Warning andCONTAINS_PATTERN
to<some_important_message>
, you will get only Warning logs containing<some_important_message>
, and all other warning logs that don't contain that specific message will be filtered out. -
If you set one filter type for one group, and another filter type for another group, the two conditions do not overlap.
For example, if you set
MIN_LOG_LEVEL
to Warning forGLOBAL
, andCONTAINS_PATTERN
to<some_important_message>
forRESOURCE_TYPE
, you will get all Warning, Error, and Critical logs fromGLOBAL
, and all logs containing<some_important_message>
fromRESOURCE_TYPE
. -
If you set more than one pair of filter types (
MIN_LOG_LEVEL
andCONTAINS_PATTERN
) for the same group (global or resource type/ID), only the last pair of filter types will apply; all the others will be ignored.
Update Azure log forwarding
To update Azure log forwarding
-
Download the latest Dynatrace Azure log forwarder.
wget https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-log-forwarder.zip
-
Deploy the new version, making sure to replace the placeholders with the required values.
az webapp deployment source config-zip -g <your_resource_group_name> -n <application_name> --src <zip_file_path>
Alternative deployments
If you don't want to use direct ingest through the Cluster API, you have two options.
- Use Azure log forwarding setup to automatically deploy and configure a dedicated containerized ActiveGate.
- Use an existing ActiveGate for log ingestion.
See below for instructions for each of these options.
- The latest version of Dynatrace log monitoring
- Create an API token and enable the Ingest logs permission
- Create a PaaS token
- Azure requirements
- CLI requirements
- Set the following environment variables, making sure to replace the placeholders (
<...>
) with your own values, as follows.
- For
DEPLOYMENT_NAME
, enter your deployment name (lowercase only). - For
TARGET_URL
, enter the API URL of your Dynatrace SaaS environment:https://<your_environment_ID>.live.dynatrace.com
To learn how to determine your environment ID, see environment ID. - For
TARGET_API_TOKEN
, enter your API token. For details, see the prerequisites above. - For
TARGET_PAAS_TOKEN
, enter your PaaS token. For instructions, see PaaS token. - For
RESOURCE_GROUP
, enter the name of the Azure resource group in which deployment will run. See Azure requirements for details. - For
EVENT_HUB_CONNECTION_STRING
, enter the connection string for the Azure Event Hubs instances configured to receive logs. See Azure requirements for details. - optional For
REQUIRE_VALID_CERTIFICATE
, entertrue
if you want Dynatrace to verify the SSL certificate of your ActiveGate. By default, certificates aren't validated.
Note: optional You can enable self-monitoring and/or log filtering during or after deployment.
DEPLOYMENT_NAME=<your_deployment_name>
TARGET_URL=<your_environment_URL>
TARGET_API_TOKEN=<your_API_token>
TARGET_PAAS_TOKEN=<your_PaaS_token>
RESOURCE_GROUP=<your_resource_group>
EVENT_HUB_CONNECTION_STRING=<your_Event_Hub_connection_string>
- Download the
azure-log-forwarder-function
script and deploy the infrastructure.
Note: Be sure to check whether you want to set other optional parameters as well. For details, see Deploy table.
wget -q https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-logs.sh -O dynatrace-azure-logs.sh && chmod +x ./dynatrace-azure-logs.sh \
&& ./dynatrace-azure-logs.sh --deployment-name $DEPLOYMENT_NAME --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --target-paas-token $TARGET_PAAS_TOKEN --resource-group $RESOURCE_GROUP --event-hub-connection-string $EVENT_HUB_CONNECTION_STRING --use-existing-active-gate false
- Set the following environment variables, making sure to replace the placeholders (
<...>
) with your own values.
- For
DEPLOYMENT_NAME
, enter your deployment name (lowercase only). - For
TARGET_URL
, enter the API URL of your ActiveGate endpoint:https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID>
. To learn how to determine your environment ID, see environment ID. - For
TARGET_API_TOKEN
, enter your API token. For details, see the prerequisites above. - For
RESOURCE_GROUP
, enter the name of the Azure resource group in which deployment will run. See Azure requirements for details. - For
EVENT_HUB_CONNECTION_STRING
, enter the connection string for the Azure Event Hubs instances configured for receiving logs. See Azure requirements for details. - For
USE_EXISTING_ACTIVE_GATE
, entertrue
. - optional For
REQUIRE_VALID_CERTIFICATE
, entertrue
if you want Dynatrace to verify the SSL certificate of your ActiveGate. By default, certificates aren't validated.
Note: optional You can enable self-monitoring and/or log filtering during or after deployment.
DEPLOYMENT_NAME=<your_deployment_name>
TARGET_URL=<your_environment_URL>
TARGET_API_TOKEN=<your_API_token>
RESOURCE_GROUP=<your_resource_group>
EVENT_HUB_CONNECTION_STRING=<your_Event_Hub_connection_string>
REQUIRE_VALID_CERTIFICATE=false
- Download the
azure-log-forwarder-function
script and deploy the infrastructure.
Note: Be sure to check whether you want to set other optional parameters as well. All parameters between brackets ([...]
) are optional. For details, see Deploy table.
wget -q https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-logs.sh -O dynatrace-azure-logs.sh && chmod +x ./dynatrace-azure-logs.sh \
&& ./dynatrace-azure-logs.sh --deployment-name $DEPLOYMENT_NAME --target-url $TARGET_URL --target-api-token $TARGET_API_TOKEN --resource-group $RESOURCE_GROUP --event-hub-connection-string $EVENT_HUB_CONNECTION_STRING --use-existing-active-gate true --require-valid-certificate $REQUIRE_VALID_CERTIFICATE
Deploy table
For a complete list of parameters, see the deploy table below.
Command-line parameter | Environment variable | Description |
---|---|---|
--deployment-name | DEPLOYMENT_NAME | required Your deployment name. Lowercase only. |
--target-url | TARGET_URL | required Your Dynatrace SaaS environment where you want to set up generic log ingestion. |
--target-paas-token | TARGET_PAAS_TOKEN | required Your PaaS token. Required only for deployment with a new ActiveGate. |
--target-api-token | TARGET_API_TOKEN | required Your API token. |
--resource-group | RESOURCE_GROUP | required Name of the Azure resource group in which deployment will run. |
--event-hub-connection-string | EVENT_HUB_CONNECTION_STRING | required The connection string for the Azure Event Hubs instance configured for receiving logs. |
--use-existing-active-gate | USE_EXISTING_ACTIVE_GATE | required If set to true , use an existing ActiveGate. By default, ActiveGate will be deployed as a container in Azure Container Instances. |
--require-valid-certificate | REQUIRE_VALID_CERTIFICATE | optional If set to true , Dynatrace verifies the SSL certificate of your ActiveGate. By default, certificates aren't validated. |
--enable-self-monitoring | SFM_ENABLED | optional If set to true , Dynatrace sends custom metrics to Azure. See Enable self-monitoring for details. By default, custom metrics aren't sent to Azure. |
--filter-config | FILTER_CONFIG | optional Apply filters to reduce the number of logs sent to Dynatrace. See Log filtering for details. |
--tags | TAGS | optional Apply Azure tags to newly created resources in comma-separated key:value pair format (for example, "tag:value,tag2:value2" ). The following characters are not supported in a tag key: ,:<>%&\?/ |
Uninstall Azure log forwarding
To uninstall the Dynatrace Azure log forwarder
-
In Azure Portal go to the resource group used for installation
-
Filter resources by tag.
Note: The deployment script tags all created resources with
LogsForwarderDeployment = <your_deployment_name>
. -
Delete the resources.
Troubleshoot
If you encounter issues removing the virtual network created for the containerized ActiveGate
-
Run the command below, making sure to replace the placeholders with your values.
az network profile delete --name <your_deployment_name>networkProfile --resource-group <your_resource_group_name>
-
Retry removing the virtual network in Azure Portal.