Sensitive data masking

Log Monitoring Classic

Dynatrace version 1.253+ OneAgent version 1.243+

Your log data contains information that may be considered sensitive. Specific log messages may include user names, email addresses, URL parameters, and other information that you may not want to disclose. Log Monitoring features the ability to mask any information by modifying the configuration file on each OneAgent that handles information you consider to be sensitive.

You can select the data that needs to be protected by applying a set of masking rules. Within each rule, you can decide what you need to hide and what to replace your hidden content with. If you need to address only specific attributes, such as predefined containers, log sources or process groups, you can achieve it by adding matchers to your rules.

Rule hierarchy

Masking rule execution occurs in a predefined hierarchy, from top to bottom. Each consecutive rule is applied to the result of a preceding rule. The hierarchy is as follows:

1. Host configuration rules
2. Host group configuration rules
3. Environment configuration rules

Host configuration rules

The host configuration rules can be accessed through the Host settings for a specific host.

1. In the Dynatrace menu, go to Hosts and select your host.
2. Select More (…) > Settings to open the Host settings page.
3. On the Host settings page, select Log Monitoring > Sensitive data masking.
4. Configure data masking by adding rules with a set of matchers that identify your sensitive data.

Host group configuration rules

The host group configuration rules can be accessed via the Host page.

1. In the Dynatrace menu, go to Hosts and select your host.
2. In the Properties and tags section, select the Host group.
3. On the Settings page, select Log Monitoring > Sensitive data masking.
4. Configure data masking by adding rules with a set of matchers that identify your sensitive data.

Environment configuration rules

The tenant scope is available in the settings menu.

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Sensitive data masking.
2. Configure data masking by adding rules with a set of matchers that identify your sensitive data.

Create rule

You can configure sensitive data masking on the host, host group or environment level.

1. Select Create new rule to start configuring your rule.

2. Name: The name to display for your configuration.

3. Search expression: A regular expression to match the string that you want to mask. Use the regular expression format.

4. Masking type: You can replace your data with a string or Secure Hash Algorithm 1 (SHA-1).

   • If you select replace with string, set Replacement to the string that is meant to replace your sensitive data.
   • If you select SHA-1, your data will be replaced by the 40-character hash string.

5. Select Add matcher to create a specific match for this rule and narrow down the scope for that rule. You can include multiple matchers in one rule. For example, the masking rule can be applied to logs from a specific container, namespace, or log source.

6. Select the matching attribute.

   Attribute	Description
   Container name	Matching is based on the name of the container.
   DT entity container group ID	Matching is based on the DT entity container group ID.
   K8s container name	Matching is based on the name of the Kubernetes container.
   K8s deployment name	Matching is based on the name of the Kubernetes deployment.
   K8s namespace name	Matching is based on the name of the Kubernetes namespace.
   Log source	Matching is based on a log path; wildcards are supported in form of an asterisk.
   Process group	Matching is based on the process group ID.
   Process technology	Matching is based on the technology name.

7. Select Add value and, from the Values list, select the detected log data items (log files or process groups that contain log data). Multiple values can be added to the selected operator. You can have one matcher that indicates log source and matches values /var/log/syslog and Windows Application Log.

8. Select Save changes.

Defined rules can be reordered, and they are executed in the order in which they appear on the Sensitive data masking page.

REST API

You can use the Settings API to manage your sensitive data masking configuration:

• View schema
• List stored configuration objects
• View single configuration object
• Create, edit, or remove configuration object

To check the current schema version for sensitive data masking configuration, list all available schemas and look for the builtin:logmonitoring.sensitive-data-masking-settings schema identifier. Sensitive data masking configuration objects are available for configuration on the following scopes:

• tenant—configuration object affects all hosts in a given environment.
• host_group—configuration object affects all hosts assigned to a given host group.
• host—configuration object affects only the given host.

To create a sensitive data masking configuration using the API

1. Create an access token with the Write settings (settings.write) and Read settings (settings.read) permissions.
2. Use the GET a schema endpoint to learn the JSON format required to post your configuration. The sensitive data masking configuration schema identifier (schemaId) is builtin:sensitive-data-masking-settings. Here is an example JSON payload with the sensitive data masking configuration:

jsoncopydownload
[
   {
      "schemaId":"builtin:logmonitoring.sensitive-data-masking-settings",
      "scope":"tenant",
      "value":{
         "config-item-title":"Added from REST API",
         "masking":{
            "expression":"run (\\d+?)",
            "type":"STRING",
            "replacement":"testing"
         },
         "matchers":[
            {
               "attribute":"log.source",
               "operator":"MATCHES",
               "values":[
                  "/var/log/syslog"
               ]
            }
         ]
      }
   }
]

SHA-1 examples

You can mask such data as your credit card or phone number, with or without specifying the capturing group.

Mask credit card number

In this example, you will configure a sensitive data masking rule that targets a credit card number in the following log record:

plaintextcopydownload
Username: John Doe, CreditCardNumber: 1234-1234-1234-1234

The rule is further narrowed to the c:\inetpub\logs\LogFiles\ex_*.log files in two process groups: IIS (PROCESS_GROUP-3D9D854163F8F07A) and IIS (PROCESS_GROUP-4A7B47FDB53137AE).

In the Dynatrace menu, go to Settings and select Log Monitoring > Sensitive data masking

1. Select Create new rule and provide the name for your configuration.
2. Provide a regular expression for the credit card number, such as CreditCardNumber: (\d{4}-\d{4}-\d{4}-\d{4}).
3. Select SHA-1 for Masking type.
4. Select Add matcher.
5. From the Attribute list select Process group.
6. In the Value field, type IIS, and select IIS (PROCESS_GROUP-3D9D854163F8F07A) from the suggestions list.
7. In the Value field, again type IIS, and select the second process group from the suggestions list: PROCESS_GROUP-4A7B47FDB53137AE.
8. Select Add matcher again.
9. Select the matching attribute Log Source.
10. Select Add value and type c:\inetpub\logs\LogFiles\ex_*.log.
11. Save changes.

Note: Only content found within a capturing group is masked, and it is transformed to the following:

plaintextcopydownload
Username: John Doe, CreditCardNumber: 7e938e089861f3975b38cff3a93cc3aa659f7779

Mask phone number

In this example, you will configure a sensitive data masking rule that targets all phone numbers in the following log record for all log files.

plaintextcopydownload
Username: John Doe, PhoneNumber: +48123010100

In the Dynatrace menu, go to Settings and select Log Monitoring > Sensitive data masking.

1. Select Create new rule and provide the name for your configuration.
2. Provide a regular expression for the phone number. For example, PhoneNumber: [0-9\-\+]{9,15}.
3. Select SHA-1 for Masking type.
4. Select Add matcher.
5. Save changes.

Note: The capturing group is not specified, so the full expression is treated as one capturing group and is masked so that it is transformed into the following in all log files:

plaintextcopydownload
Username: John Doe, 011897d555c81e88f286cbb74c59f4ad99ec2f8d

Advanced SHA-1 examples

In the examples below, you can see how various combinations of sensitive data can be masked. You can use the listed payload JSON files in the REST API, or enter the listed masking rules, matchers, Regex expressions, and attributes directly when creating your rules via Dynatrace web UI.

Mask credit card numbers and emails

To mask all credit card numbers and emails in your content, you need to create two separate rules, each with a different matcher:

jsoncopydownload
{
  "masking": {
    "expression": "(\\d{4}-\\d{4}-\\d{4}-\\d{4})",
    "type": "STRING",
    "replacement": "MaskedCreditCardNumber"
  },
  "matchers": [
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
  ],
  "enabled": true
}

Mask Apache logs

To mask logs that are written by Apache AND whose log filename is error.log, you can create one rule with two matchers:

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error.log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache OR whose log filename is error.log, you need to create two rules with one matcher each:

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error.log"
      ]
    }
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache and whose log filename starts with error AND ends with log, you need to have one rule with three matchers, each matcher having one value.

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error*"
      ]
    },
    {
      "attribute": "log.source",
      "values": [
        "*log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

To mask logs that are written by Apache and whose log filename starts with error OR ends with log, you need to have one rule with two matchers, one with the process group value, and the second one with two content values, /path/to/error* and *log:

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/error*", "*log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Mask Apache or MySQL logs

To mask logs that are written by Apache or MySQL, you need to have either two rules or one rule with one matcher that has two values.

The scenario with two rules:

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

The scenario with one rule with a matcher that has two values:

jsoncopydownload
{
  "masking": {
    "expression": "email: (.*),",
    "type": "SHA1"
  },
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
}

Regex examples

The common regex formats for sensitive data include:

Unsupported regular expressions

Data masking occurs within the entire expression or a capturing group. An expression has to match the regular expression engine syntax, and it cannot:

• Be part of more than one capturing group
• Contain the lookbehind zero-length assertion in a capturing group
• Contain the backreference zero-length assertion in a capturing group
• Contain greedy quantifiers (such as x?, x*, or x+) or possessive quantifiers (such as x?+, x*+, or xx++). Use lazy/reluctant qualifiers (such as x?? and x+?) instead.

FAQ

Sensitive data masking limits

Be aware of the following limitations to sensitive data masking:

• Sensitive data masking requires OneAgent version 1.243.
• The user interface in the preview release will change in the GA release.
• Sensitive data masking in Log Monitoring v1 cannot be migrated to the latest Log Monitoring.
• If the masking process takes too much time, the log file affected is blocked until the restart of OneAgent or any configuration change, and then you get the File not monitored - incorrect sensitive data masking rule message.