Measure SLOs with Nobl9 and Dynatrace

Guest author Quan To, Director of Product, explains how to use Dynatrace data to create and manage SLOs using Nobl9.

Companies are evolving to become customer-centric by adopting service level objectives (SLOs) to understand customer needs better. Teams are using concepts from site reliability engineering to create SLO metrics that measure the impact to their customers and leverage error budgets to balance innovation and reliability.

It can be challenging to cut through the massive volume of telemetry data and quickly understand the impact on your business and customers. SLOs identify reliability goals based on expectations set by the users and business stakeholders. They give you visibility to report on reliability over any time window, based on real customer-impacting issues.

Nobl9 and Dynatrace simplify and accelerate SLO usage with an integrated approach that enables SLI collection, SLO calculation, and error budget alerting. The joint solution extends capabilities beyond the Dynatrace Software Intelligence Platform to maximize flexibility and help reduce operational risks and costs while enabling organizations to adopt SLOs faster.

Cutting through noise

Nobl9 integrates with Dynatrace to gather SLI metrics for your infrastructure and applications using real-time monitoring or synthetics. Nobl9 takes these SLI metrics and measures them against SLO targets to calculate error budget and burn rate. Operation teams can quickly understand the current state of an application using the near real-time dashboard and identify reliability trends with reporting. Nobl9 consolidates the telemetry from Dynatrace and other data sources, allowing you to focus on what matters the most to your users.

Embracing risk

Nobl9 establishes error budgets that can help organizations embrace risk by providing a framework for how much risk is acceptable. A company that embraces risk and releases software more often can deliver features faster to users.  If a company can release software quickly, it can react to new opportunities much more rapidly than its competitors. By embracing risk, organizations can innovate better.

Nobl9 calculates the error budget and error burn rate, visualizing your SLOs, adding labels, SLO reports, near real-time dashboard views, command line SLO configuration, and integration with multiple data sources and alert methods to be a central tool for all your SLO needs.

Nobl9 allows you to create SLOs on any of your metrics and define targets based on customer impact. Nobl9 also manages error budget calculations and triggers your alerting system when error budgets are burning too quickly.

Prerequisites to Get Started

Before you start creating SLOs, log into Dynatrace, navigate to Settings -> Integration -> Dynatrace API.  You’ll need the API Token and pass that to Nobl9.  Note – the token is only visible when you create it, so store it in a safe place.

Also, make sure the API v2 “Read metrics” scope is enabled.

At this point, we’ll also assume your monitoring data is flowing into Dynatrace.  Now, let’s head to app.nobl9.com and sign in.  If you haven’t signed up yet, you can request an evaluation of Nobl9 from nobl9.com.

Add Dynatrace as a Data Source

To set up your first SLO, you’ll need a few things.

1. Your Dynatrace URL.  When you log into Dynatrace, note the fully qualified domain in your browser.  It will look something like abc12345.live.dynatrace.com
2. The metric you want to measure, and will it be a threshold or a ratio metric?
   • A threshold metric is a single time series evaluated against a threshold
   • A ratio metric allows you to enter two time-series metrics to compare (good / total)
3. The query for the metric you want to measure.  You can use the Dynatrace Data Explorer to build your query and copy and paste the code.  In this example, I chose the DOM load time.

To start, let’s add Dynatrace as a data source to your Nobl9 account.  Navigate to Integrations and click the “+” button.  You’ll see a list of all the data sources supported by Nobl9.  Select Dynatrace.

Next, select Agent.  You will need to deploy a Nobl9 agent to collect the metrics and send them to Nobl9.

Now, you’ll see a form to add the Dynatrace URL and additional metadata:

For more information on the metadata, the In-app Help on the right of the form provides more details on each field.

Now click “Add Data Source.”

You’ll see the option to deploy the agent to your Kubernetes Cluster or deploy the agent via docker.  Follow the instructions depending on whether you want to use Kubernetes or Docker.  This page is also where you’ll be pasting in your Dynatrace API token.

After you deploy your Nobl9 agent, you can create your SLO.

Adding an SLO

Navigate to the Service Level Objectives menu and click the “+” icon.

The SLO Wizard has five steps.  The first is to select a service.  Choose an existing service to be the home for this SLO.  If you don’t have a service yet, you can navigate to the Services menu to create one, then return to the SLO wizard.

In step 2, select Dynatrace as your data source.  At this point, you can decide to use a threshold or a ratio metric.  This page is where you will paste in the query from Dynatrace’s Data Explorer.

Moving on to step 3 (Define Time Window), select whether you want a rolling or calendar-aligned time window and its parameters.

Next, in step 4, define the error budget and objectives. This page is where you would set your target, the experience name, and what you expect the values to be. Values exceeding the threshold would start burning your error budget.

You can define up to six SLO objectives per SLO. Each one would be a different target and experience you want to measure.

There’s one last step to setting up the SLO. You can add a name for your SLO and associate alert policies (you can pick an alert policy later). You can also add a link to a run book that could refer to the SLI details in Dynatrace. Alternatively, you can link to anything that will be useful when looking at SLO details, like a runbook or other documentation.

Now click “Create SLO.”

Your new SLO will appear in the Grid view.  It could take up to 20 minutes to start seeing data flow through as Nobl9 queries Dynatrace every minute and needs to build a history and draw the chart.

Once the data appears, you’ll start seeing charts looking something similar to this:

At this point, you’ve created your first SLO!

If you run into any issues along the way, check out our documentation on creating SLOs.

Summary

You can create as many SLOs as you need and monitor them, add alerts to them, view the health of your services on the Nobl9 dashboard, and generate reports on your SLOs.