Set up Azure log forwarding

Azure log forwarding allows you to stream Azure logs from Azure Event Hubs into Dynatrace logs via an Azure Function App instance. It supports both Azure resource logs and activity logs.

Prerequisites

See below the list of requirements for setting up Azure log forwarding. Some are needed before you start deployment, others during the deployment process.

ActiveGate

Azure log forwarding requires an ActiveGate configured for generic log ingestion. You have two options:

Dynatrace

  • Dynatrace version 1.217+
  • Dynatrace Log Monitoring v2. To enable Dynatrace Log Monitoring v2, contact a Dynatrace ONE product specialist by selecting the chat button in the upper-right corner of the Dynatrace menu bar.
  • Create an API token and enable the Ingest logs permission
  • If you want to install a containerized ActiveGate during the Azure log forwarding deployment, you need to create a PaaS token
  • Determine your API URL, which depends on where you want to set up generic log ingestion.
    • If you want to install a containerized ActiveGate, set it to your Dynatrace SaaS environment: https://<your_environment_ID>.live.dynatrace.com
    • If you want to use an existing ActiveGate, set it to your ActiveGate endpoint: https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID> Note: To determine <your_environment_ID>, see environment ID.

Azure

  • Set up an Azure Event Hubs instance in each Azure location from where you want to pull logs.

  • Select the resource group where you want deployment to run, or create a new one for deployment.
    To create a new Azure resource group, run the command below, making sure to replace the placeholders with your actual values.

    The Event Hubs instance(s) and the resource group in which deployment will run need to be in the same region to be able to send logs.

    az group create --name <your_resource_group> --location <your_resource_group_region>
    
  • Get an Event Hubs connection string for the Event Hubs instance that is configured for receiving logs, and enable listen permission.

  • Configure diagnostic settings for resources from which you want to stream logs so that the diagnostic settings point to Azure Event Hubs instance(s).

CLI

  • You can run Azure log forwarding deployment using Azure Portal Cloud Shell (Bash) or from any machine with Azure CLI and Bash shell (Linux or Windows WSL).

Deploy Azure log forwarding

To set up Azure log forwarding, you need to run the azure-log-forwarder-function script, which will create the infrastructure needed and deploy the function code into the infrastructure.

  1. Download the script in Cloud Shell (Bash).

    wget -q https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-logs.sh -O dynatrace-azure-logs.sh && chmod +x ./dynatrace-azure-logs.sh
    
  2. Run the script.

You have three options.

Option 1. Use environment variables

  1. In an editor, replace the placeholders in the export list below with your own values. See Deploy table for details.

    export DEPLOYMENT_NAME="<your_deployment_name>"
    export TARGET_PAAS_TOKEN="<your_PaaS_token>"
    export TARGET_API_TOKEN="<your_API_token>"
    export RESOURCE_GROUP="<your_resource_group>"
    export EVENT_HUB_CONNECTION_STRING="<your_Event_Hub_connection_string>"
    export EVENT_HUB_NAME="<your_Event_Hub_name>"
    export USE_EXISTING_ACTIVE_GATE="false"
    export REQUIRE_VALID_CERTIFICATE="false"
    export SFM_ENABLED="false"
    export TARGET_URL="<your_environment_URL>"
    
  2. Run the export commands in Azure Portal Cloud Shell (Bash).

  3. Run the deployment script.

    ./dynatrace-azure-logs.sh
    

Option 2. Use CLI parameters

Run the command below, making sure to replace all placeholders (<...>) with your actual values, and check whether you want to set other optional parameters as well. All parameters between brackets ([...]) are optional. See Deploy table for details.

./dynatrace-azure-logs.sh --deployment-name <your_deployment_name> --target-url <your_environment_URL> --target-api-token <your_API_token> --resource-group <your_resource_group> --event-hub-connection-string <your_Event_Hub_connection_string> --event-hub-name <your_Event_Hub_name> [--target-paas-token <your_PaaS_token>] [--use-existing-active-gate] [--require-valid-certificate] [--enable-self-monitoring] [--filter-config]

Option 3. Use interactive mode

Run the script in interactive mode, which will prompt you for all the needed parameters. See Deploy table for details.

./dynatrace-azure-logs.sh -i

Deploy table

Command-line parameter Environment variable Description ActiveGate requirement
--deployment-name DEPLOYMENT_NAME Your deployment name. Lowercase only. Required for both containerized ActiveGate/existent ActiveGate
--target-url TARGET_URL Your Dynatrace SaaS environment where you want to set up generic log ingestion. See Dynatrace requirements for details. Required for both containerized ActiveGate/existent ActiveGate (different values in each case)
--target-paas-token TARGET_PAAS_TOKEN Your PaaS token. Required for the containerized ActiveGate deployment. See Dynatrace requirements for details. Required for containerized ActiveGate
--target-api-token TARGET_API_TOKEN Your API token. See Dynatrace requirements for details. Required for both containerized ActiveGate/existent ActiveGate
--resource-group RESOURCE_GROUP Name of the Azure resource group in which deployment will run. See Azure requirements for details. Required for both containerized ActiveGate/existent ActiveGate
--event-hub-connection-string EVENT_HUB_CONNECTION_STRING The connection string for the Azure Event Hubs instance(s) that is configured for receiving logs. See Azure requirements for details. Required for both containerized ActiveGate/existent ActiveGate
--event-hub-name EVENT_HUB_NAME Name of the Azure Event Hubs instance(s) configured for receiving logs. Required for both containerized ActiveGate/existent ActiveGate
--use-existing-active-gate USE_EXISTING_ACTIVE_GATE Decide if you want to use an existing ActiveGate. By default, ActiveGate will be deployed as a container in Azure Container Instances. Required for existent ActiveGate
--require-valid-certificate REQUIRE_VALID_CERTIFICATE If set to true, Dynatrace verifies the SSL certificate of your ActiveGate. By default, certificates aren't validated. Optional for both containerized ActiveGate/existent ActiveGate
--enable-self-monitoring SFM_ENABLED If set to true, it sends custom metrics to Azure. See Enable self-monitoring for details. By default, custom metrics aren't sent to Azure. Optional for both containerized ActiveGate/existent ActiveGate
--filter-config FILTER_CONFIG Apply filters to reduce the numbers of logs sent to Dynatrace. See Log filtering for details. Optional for both containerized ActiveGate/existent ActiveGate

View Azure logs

After deploying the script, you can view and analyze Azure logs in Dynatrace: Go to Analyze > Logs and filter for cloud.provider: azure.

Self-monitoring Optional

Self-monitoring allows quick diagnosis to see if your function processes and sends logs to Dynatrace properly.

Enable self-monitoring

To enable self-monitoring, you have two options:

Enable managed identity

After enabling self-monitoring, you need to enable managed identity for your Function App instance created during deployment, and configure it to allow pushing metrics to the resource.

Self-monitoring metrics

Once you enable self-monitoring, you can view the following metrics in your dynatrace_logs_self_monitoring namespace of the newly deployed Function App instance.

Metric name Description Dimension
all_requests All requests sent to Dynatrace.
dynatrace_connectivity_failures Reported when any Dynatrace connectivity issues occurred. connectivity_status
parsing_errors Reported when any parsing errors occurred during log processing.
processing_time Time needed to process all logs.
sending_time Time needed to send all requests.
too_long_content_size Reported when content of log is too long. The content will be trimmed.
too_old_records Reported when logs received from Event Hubs are too old.

Log filtering Optional

To reduce the number of logs that are sent to Dynatrace, you can apply filters.
To apply filters you have two options:

  • During deployment: Set the FILTER_CONFIG environment variable in Azure Portal Cloud Shell (Bash) before running the deployment script.

    1. Add the FILTER_CONFIG environment variable to the export list needed for the deployment script.

      Note: Be sure to replace placeholders with your values. See Filter options for details.

      export FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=<log_level>;FILTER.GLOBAL.CONTAINS_PATTERN=<pattern>;FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.<resource_type>=<log_level>;FILTER.RESOURCE_TYPE.CONTAINS_PATTERN.<resource_type>=<pattern>;FILTER.RESOURCE_ID.MIN_LOG_LEVEL.<resource_id>=<log_level>;FILTER.RESOURCE_ID.CONTAINS_PATTERN.<resource_id>=<pattern>"
      
    2. Run the export commands in Azure Portal Cloud Shell (Bash).

    3. Run the deployment script.

      ./dynatrace-azure-logs.sh
      
  • After deployment: Add FILTER_CONFIG in Azure Portal.

    1. In Azure Portal, go to the Configuration of your deployed Function App instance.

    2. In Application settings, search and select FILTER_CONFIG.

      Note: FILTER_CONFIG will appear in Azure after running the deployment script.

    3. Select Edit to add a Value for your filter.

      Note: Alternatively, you can select Advanced edit to enter your value in the JSON.

    4. Select OK.

    5. Restart your Function App instance.

Filter options

FILTER_CONFIG is a key-value pair variable. You can set two types of filters (MIN_LOG_LEVEL and/or CONTAINS_PATTERN) for three filter groups (GLOBAL, RESOURCE_TYPE, and/or RESOURCE_ID).

Filter type: MIN_LOG_LEVEL

This filter type allows you to filter out logs with unwanted levels. Possible log levels are:

  • Critical (or 1)
  • Error (or 2)
  • Warning (or 3)
  • Informational (or 4)

Example:
FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=Warning"
In the example above, Informational logs will be skipped, and only Warning, Error, and Critical logs will be sent to Dynatrace.

Syntax options are:

  • FILTER.GLOBAL.MIN_LOG_LEVEL=<log_level>
  • FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.<resource_type>=<log_level>
  • FILTER.RESOURCE_ID.MIN_LOG_LEVEL.<resource_id>=<log_level>

You can have one global-level filter and additional filters for a particular resource type/ID.

Example:
FILTER_CONFIG="FILTER.GLOBAL.MIN_LOG_LEVEL=Error;FILTER.RESOURCE_TYPE.MIN_LOG_LEVEL.MICROSOFT.WEB/SITES=Informational"
In the example above, all logs from instances with resource type MICROSOFT.WEB/SITES will be sent to Dynatrace, while for all other resources, Informational and Warning logs will be filtered out.

Filter type: CONTAINS_PATTERN

This filter type allows you to collect logs containing a particular text. We use fnmatch, which provides support for Unix shell–style wildcards. See Unix filename pattern matching for details.

Syntax options are:

  • FILTER.GLOBAL.CONTAINS_PATTERN=<log_pattern>
  • FILTER.RESOURCE_TYPE.CONTAINS_PATTERN.<resource_type>=<log_pattern>
  • FILTER.RESOURCE_ID.CONTAINS_PATTERN.<resource_id>=<log_pattern>

Filter group: GLOBAL

This filter is set for all logs.

Filter group: RESOURCE_TYPE

This filter is used only for logs coming from resources of the given Azure resource type, such as Microsoft.Compute/virtualMachines.

You can find the resource type in Azure Portal, in your resource's Properties.
Note: If the Type field doesn't appear in Properties, you can extract it from the resource ID string. Resource ID string syntax:
/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/<resourceType>/<resourceName>
The resource type will be the part between /providers/ and /resourceName/.

Filter group: RESOURCE_ID

This filter is used only for logs coming from the given resource that is identified by the Azure resource ID.

You can look for the resource type in Azure Portal, in your resource's Properties.

Filter rules

  • If you set two filter types for the same group, both conditions need to be met, so the second filter will have to match the first filter.

    For example, if you set MIN_LOG_LEVEL to Warning and CONTAINS_PATTERN to <some_important_message>, you will get only Warning logs containing <some_important_message>, and all other warning logs that don't contain that specific message will be filtered out.

  • If you set one filter type for one group, and another filter type for another group, the two conditions do not overlap.

    For example, if you set MIN_LOG_LEVEL to Warning for GLOBAL, and CONTAINS_PATTERN to <some_important_message> for RESOURCE_TYPE, you will get all Warning, Error, and Critical logs from GLOBAL, and all logs containing <some_important_message> from RESOURCE_TYPE.

  • If you set more than one pair of filter types (MIN_LOG_LEVEL and CONTAINS_PATTERN) for the same group (global or resource type/ID), only the last pair of filter types will apply; all the others will be ignored.

Update Azure log forwarding

To update Azure log forwarding

  1. Download the latest Dynatrace Azure log forwarder.

    wget https://github.com/dynatrace-oss/dynatrace-azure-log-forwarder/releases/latest/download/dynatrace-azure-log-forwarder.zip
    
  2. Deploy the new version, making sure to replace the placeholders with the required values.

    az webapp deployment source config-zip -g <your_resource_group_name> -n <application_name> --src <zip_file_path>
    

Uninstall Azure log forwarding

To uninstall the Dynatrace Azure log forwarder

  1. In Azure Portal go to the resource group used for installation

  2. Filter resources by tag.

    Note: The deployment script tags all created resources with LogsForwarderDeployment = <your_deployment_name>.

  3. Delete the resources.

Troubleshoot

If you encounter issues removing the virtual network created for the containerized ActiveGate

  1. Run the command below, making sure to replace the placeholders with your values.

    az network profile delete --name <your_deployment_name>networkProfile --resource-group <your_resource_group_name>
    
  2. Retry removing the virtual network in Azure Portal.