Integrate Gremlin with Dynatrace Cloud Automation

The Gremlin chaos engineering platform equips teams with the tools, process, and customer success services that they need to reduce the number of high-severity outages, recover faster and more cost-effectively when inevitable issues do occur, and sustain over 99.9% uptime.

Integrating Dynatrace Cloud Automation with Gremlin enables both fine-grained control around and reporting against chaos engineering experiments. For example, as Gremlin executes the chaos engineering experiment, the webhook integration with Dynatrace allows you to report on any changes in system behavior recognized by the Dynatrace monitoring capabilities, making it easy to understand chaos engineering test results and address any uncovered issues.

To integrate Dynatrace Cloud Automation with Gremlin, follow the instructions below.

Prerequisites

Gremlin

Dynatrace

Gremlin

Create a Gremlin account.
Install the Gremlin agent on your targeted hosts.
Create a scenario.
Create a Gremlin API key.

Dynatrace

Create a Cloud Automation project with a shipyard file that has a custom task (for example, experiment) to trigger the Gremlin webhook subscription.
Add the Gremlin API key obtained above as a Cloud Automation secret.
How to add the API key as a Cloud Automation secret
1. On the Cloud Automation bridge, select your Cloud Automation project.
2. Select Settings, and then select Secrets.
3. Select Add Secret and enter the following values:
  - Name: Enter a name for the secret (for example, gremlin).
  - Scope: Select keptn-webhook-service.
  - Key/Value: Enter your values for the key/value pair (for example, key: gremlin-token, value: <your_Gremlin_API_key>).
4. Select Add secret.

Set up the Gremlin webhook subscription

Use case: Set up a webhook subscription that triggers a Gremlin experiment. The Cloud Automation sequence will not wait for Gremlin to send back a Cloud Automation sequence task finished event.

On the Cloud Automation bridge, select your Cloud Automation project.
Select Settings, and then select Integrations.
Select webhook-service from the list, and then select Add subscription.
Enter the following values:
- Task: Select the task associated with your shipyard file (in this case, experiment).
- Task suffix: Select triggered.
- Request method: Select POST.
- URL: Enter your URL endpoint (for example, https://api.gremlin.com/v1/scenarios/<scenarioID>/runs?teamId=<teamID>).
Select Add custom header and enter the following values:
- Name: Enter Authorization.
- Value: Enter {{.secret.<secret_name>.<key_name>}}, making sure to replace <secret_name> with the name of your secret and <key_name> with the key name you entered when creating the Cloud Automation secret (see Prerequisites). Example: {{.secret.gremlin.gremlin-token}}.

Enter a Custom payload.

Example:

json
{

"hypothesis": "Your hypothesis of what will happen when running this scenario."

}

Under Send finished event, select automatically.
Select Create subscription.

Trigger a sequence

Cloud Automation triggers a Gremlin experiment whenever the defined task in the sequence is triggered:

When the task associated with a Gremlin experiment in the sequence is executed, Cloud Automation triggers the corresponding webhook service.
The webhook service launches the Gremlin experiment, which is executed on the Gremlin side.

To trigger the sequence, you can send the event command in the Keptn CLI.

Create a file called triggered-event.json.

Example:

json
{
  "type": "sh.keptn.event.mystage.mysequence.triggered",
  "specversion":"1.0",
  "source":"manual-trigger",
  "data":{
      "project":"myproject",
      "stage":"mystage",
      "service":"myservice"
  }
}

Send the event.

bash
keptn send event --file triggered-event.json

After sending the event, you can

Monitor the sequence progress on the project’s Cloud Automation sequence page
Monitor the Gremlin experiment execution