Cluster event notifications

You can manage the following notifications in a Dynatrace Managed cluster:

Insufficient disk space on a cluster node
Triggered when a disk partition on a cluster node has less disk space than required for a given storage type. In that case, you need to extend your disk or review cluster settings. Otherwise, data retention might be reduced.
Insufficient hardware on a cluster node
Triggered when a cluster node doesn't have enough CPU cores and RAM to meet recommended requirements. For details, see Managed hardware requirements.
Metrics storage exceeds supported size (4TiB)
Triggered when metrics storage for a cluster node is too high. In that case, you need to review your monitoring settings or add additional nodes to the cluster.
Transaction storage retention period truncation
Triggered when data retention has been automatically reduced to store new data. In that case, we recommend that you adjust the target retention time, review monitoring settings, or extend a disk.
Adaptive Load Reduction activity
Triggered when a cluster node is considered overloaded and unable to keep up with processing incoming requests. In that case, you need to review monitoring settings or increase CPU cores and RAM on the cluster node. For details, see Adaptive traffic management for Dynatrace Managed. To configure cluster event notifications settings

In order to interact with the Cluster API v2 and manage cluster event notifications, follow the procedure below:

Authentication

To generate a cluster token with the Write settings and Read settings scopes

Go to Settings > API tokens.
In the Cluster tokens section, select Generate token.
Enter a name for your token and define the Write settings and Read settings access scopes of your cluster API token.
Save and then Copy the token to a secure location.

Configure email notifications

To send your configuration as a JSON payload, use the POST an object endpoint with a cluster token that has the appropriate access scopes:

[
  {
    "schemaId": "builtin:cluster-events-notification-settings",
    "scope": "cluster",
    "value": {
                "insufficientDiskSpace": {
                    "sendEmail": true
                },
                "insufficientHardware": {
                    "sendEmail": true
                },
                "insufficientMetricStorage": {
                    "sendEmail": false
                },
                "transactionStorageTruncation": {
                    "sendEmail": false
                },
                "adaptiveLoadReductionActivated": {
                    "sendEmail": false
                }
            }
  }
]

Read configuration schema

To learn the JSON format required to post your configuration, use the GET a schema endpoint with a cluster token that has the appropriate access scopes. The configuration schema identifier (schemaId) is builtin:cluster-events-notification-settings.

{
    "dynatrace": "1",
    "schemaId": "builtin:cluster-events-notification-settings",
    "displayName": "Cluster events notification settings",
    "description": "Configuration of emails notifications for cluster events.",
    "documentation": "Email notifications are sent to recipients defined in Settings - Emails - Email notifications.",
    "version": "0",
    "multiObject": false,
    "maxObjects": 1,
    "allowedScopes": [
        "cluster"
    ],
    "enums": {},
    "types": {
        "SingleEventSettings": {
            "version": "0",
            "versionInfo": "",
            "displayName": "Single event settings",
            "summaryPattern": "",
            "description": "Settings for a single cluster event",
            "documentation": "",
            "properties": {
                "sendEmail": {
                    "displayName": "Send email",
                    "description": "",
                    "documentation": "",
                    "type": "boolean",
                    "nullable": false,
                    "maxObjects": 1,
                    "modificationPolicy": "DEFAULT",
                    "default": true
                }
            },
            "type": "object"
        }
    },
    "properties": {
        "insufficientDiskSpace": {
            "displayName": "Insufficient disk space on a cluster node",
            "description": "Triggered when a disk partition on a cluster node has less disk space than required for a given storage type. In that case, you need to extend your disk or review cluster settings. Otherwise, data retention might be reduced.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "insufficientHardware": {
            "displayName": "Insufficient hardware on a cluster node",
            "description": "Triggered when a cluster node doesn't have enough CPU cores and RAM to meet recommended requirements.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "insufficientMetricStorage": {
            "displayName": "Metrics storage exceeds supported size (4TiB)",
            "description": "Triggered when Metrics storage a cluster node is too high. In that case, you need to review your monitoring settings or add additional nodes to the cluster.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "transactionStorageTruncation": {
            "displayName": "Transaction storage retention period truncation",
            "description": "Triggered when data retention has been automatically reduced to store new data. In that case, it is recommended to adjust target retention time, review monitoring settings or extend a disk.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        },
        "adaptiveLoadReductionActivated": {
            "displayName": "Adaptive Load Reduction activity",
            "description": "Triggered when a cluster node has been considered as overloaded and not able to keep up with processing incoming requests. In that case, you need to review monitoring settings or increase CPU cores and RAM on the cluster node.",
            "documentation": "",
            "type": {
                "$ref": "#/types/SingleEventSettings"
            },
            "nullable": false,
            "maxObjects": 1,
            "modificationPolicy": "DEFAULT"
        }
    }
}

Read current configuration

To check the current configuration, use the GET objects endpoint (/api/cluster/v2/settings/objects?schemaIds=builtin:cluster-events-notification-settings&scopes=cluster) with a cluster token that has the appropriate access scopes.

If these settings have been previously changed, the items list contains a single object. Use the objectId from the list in making subsequent updates.

If the items list is empty, the default value is used (not visible in Dynatrace API):

{
  "value": {
            "insufficientDiskSpace": {
                "sendEmail": true
            },
            "insufficientHardware": {
                "sendEmail": true
            },
            "insufficientMetricStorage": {
                "sendEmail": true
            },
            "transactionStorageTruncation": {
                "sendEmail": false
            },
            "adaptiveLoadReductionActivated": {
                "sendEmail": false
            }
        }
}

Create notification settings object

To create a cluster event notification settings object, use the POST objects endpoint with a cluster token that has the appropriate access scopes. Use the ID of the newly created object (event notification settings) for later notification settings updates.

In this example, using a POST API call to the /api/cluster/v2/settings/objects endpoint and builtin:cluster-events-notification-settings schema, you create a notification settings object in your cluster settings:

[
  {
    "schemaId": "builtin:cluster-events-notification-settings",
    "scope": "cluster",
    "value": {
                "insufficientDiskSpace": {
                    "sendEmail": true
                },
                "insufficientHardware": {
                    "sendEmail": true
                },
                "insufficientMetricStorage": {
                    "sendEmail": false
                },
                "transactionStorageTruncation": {
                    "sendEmail": false
                },
                "adaptiveLoadReductionActivated": {
                    "sendEmail": false
                }
            }
  }
]

Update notification settings

There are two methods you can use to update cluster events notification settings once the notification settings object is created. In either case, make sure you have a cluster token that has the appropriate access scopes.

You can use the same POST method that you used to create the settings object (Create notification settings object). The schema doesn't allow duplicate settings objects, so if you attempt to create another settings object, you will overwrite the existing one.

You can modify an existing settings object by making a PUT API call to the /api/cluster/v2/settings/objects/<objectId> endpoint and providing the objectId obtained when creating the initial event notification settings object.

Make the PUT API call to the /api/cluster/v2/settings/objects/<objectId>

{
  "value": {
            "insufficientDiskSpace": {
                "sendEmail": true
            },
            "insufficientHardware": {
                "sendEmail": true
            },
            "insufficientMetricStorage": {
                "sendEmail": false
            },
            "transactionStorageTruncation": {
                "sendEmail": false
            },
            "adaptiveLoadReductionActivated": {
                "sendEmail": false
            }
        }
}

Delete notification settings

You can delete an existing notification settings object by making a DELETE API call to the /api/cluster/v2/settings/objects/<objectId> endpoint (with a cluster token that has the appropriate access scopes) and providing the objectId obtained when creating the initial event notification settings object. After the object is deleted, notification behavior falls back to the default: all notifications trigger an email to configured recipients.