How do I organize and manage large Dynatrace environments?

Dynatrace automatically discovers components in your environment thanks to the unique capabilities of Dynatrace OneAgent. Such components include all important processes, process groups, services and real user applications. Not only can these entities be auto-discovered, their underlying technologies can also be accurately identified (for example, Apache HTTP server, IBM WebSphere, cloud deployments like Cloud Foundry, OpenShift, Kubernetes, and much more).

As part of this auto-detection, Dynatrace discovers network topology as well. Once traffic is monitored, Dynatrace detects also load balancers and proxies. Moreover, Dynatrace supports 3rd party integrations, thus augmenting this data with topology and monitoring information from AWS, VMWare, Docker, Azure, OpenStack, and more.

All this information results in a real-time topological model (like a CMDB) of your environment that you can access via Smartscape. This model forms the basis of the Dynatrace Artificial Intelligence engine.

Managing such a huge amount of information, however, and organizing such large monitoring environments is a real challenge. To effectively cope with this challenge, Dynatrace uses tags and metadata.

Tags vs metadata

Dynatrace differentiates between tags and metadata.

Metadata properties are key/value pairs that are inherent to any monitored entity.

metadata are detected during startup or discovery of a monitored entity and immutable inside the product itself. Metadata is shown in the Properties section of any entity (see the image below).

metadata

In case of a process, all metadata are discovered during startup of the process and for the most part not refreshed until the process is restarted.

Metadata are considered part of the deployed application or deployed entity and are discovered and/or configured as part of that deployed application. They aren't changed in the product.

Tags in Dynatrace are labels or markers. They aren't information about an entity and don't belong to one specific entity. Tags are used to identify certain sets of entities.

tags

Tags are used to find/filter/designate sets of entities. They are configured within the product.

Dynatrace offers an efficient mechanism to automatically assign tags via rules that are based on metadata. Through this mechanism, tags can be dynamically assigned within Dynatrace, based on immutable metadata that come from the monitored system.

With this in mind, note that a Name or a Version is not a tag in Dynatrace. Rather, these are regarded as metadata. An example for tags would be one called Production and another called Staging. A deployment designation like blue/green is metadata but might often make sense as a tag—it's not intrinsic information but rather is used to look at either the blue or the green set of applications.

What are tags used for?

As mentioned above, a tag is a label or a marker attached to an entity. An entity can be anything from a host or process to a server-side service or AWS database. Tags involve a wide spectrum of usages, but in principle they are used to find certain types of entities or to filter views and actions for certain entities.

As tags have predominantly an organizational and management purpose within Dynatrace, they are useful for:

How do I define tags for my entities?

There are several ways to tag your entities. The best option for you depends on your environment and your needs. The easiest way is to use the UI to manually apply tags to your entities. However, the manual approach forces you to manually tag each new entity. This makes it hard to enforce a standard and is impractical in larger environments. Therefore, within larger environments, it's recommended that you use a combination of auto-tagging based on metadata and tags that are part of the monitored deployment. To better understand this, you need to take a closer look at metadata and deployments.

What kind of metadata do I get from my deployments?

Dynatrace not only discovers all processes running on a host, it also identifies their nature. As part of this, Dynatrace captures the most important domain-specific metadata. If Dynatrace discovers, for example, an Oracle WebLogic entity, the entity will also have metadata about the WebLogic Domain, the WebLogic Server name and the WebLogic Cluster. If Dynatrace identifies a Kubernetes environment, then every process will have metadata concerning its pod name and namespace. The same is true for VMWare and AWS hosts.

Dynatrace only knows about a subset of domain-specific information. This is why we also enable you to enrich this information with their own metadata as well as with additional metadata from other sources. In particular:

This metadata is shown in the properties of each entity within the Dynatrace UI and it's part of the data when you access these entities via the API. You can use this metadata to

Tags/labels defined in other systems

Dynatrace has several means of accepting tags from external systems. Dynatrace does this automatically for Kubernetes labels and AWS tags. You can define your own tags for the monitored system as system environment variables, via a Cloud Foundry service. You can also use the REST API to set tags.

Best way to tag entities

Begin by enriching the existing domain knowledge that's available to Dynatrace with metadata that you provide. This gives you the flexibility to define tags in Dynatrace via auto tagging based on metadata.

Tags in Dynatrace

  • Can be changed without changing the deployment itself.
  • Aren't dictated by an external system that you may not control.

Tags can have a key/value structure. For example, you can define a tag with key SecurityIsolationLevel that has a different value for different entities. Once a tag and its rules are defined, Dynatrace will apply the tags to new entities automatically.

You may have existing tags in 3rd-party systems that you rely on and that you want to ensure are the same across your organization in an automated fashion. Dynatrace recommends providing such tags via the different means at the time of deployment (Kubernetes annotations, Cloud Foundry tags, AWS tags, environment variables) or set up an integration to send them via the Dynatrace API. These tags will be “read only” in Dynatrace and therefore you won't be able to configure them via the UI.

Where should I use tags?

It's strongly recommended that you use tags instead of specific entities in any kind of chart, service or host list and alerting profiles. Basically, anywhere that you might want to refer to a set of entities, you should use tags instead of defining an indirect reference. This way ensures easier maintenance of your system configuration and integrations. Even in highly dynamic environments where new components are added and removed frequently, you can in this way configure for example your dashboard once. It also makes it far easier to transport configuration from one environment to another.

Whenever you create a dashboard, a chart, or an alerting profile, you should consider defining or using tags to refer to the related entities. This doesn't mean that you should define generic tags for everything upfront. Tags should really be use-case based. However, you can and should add or define metadata for anything that you think could become important as a basis for tags.

When shouldn’t I use tags?

Both Dynatrace tags and metadata are key/value concepts. As such, you can define, for example, a tag User and assign username as a value dynamically. You can do the same for email, owner and many other properties. However, you will rarely use such key/value pairs for the purposes described above. This would also clutter the Dynatrace UI. Tags are meant for organization, not for defining extra information for your entities. It is therefore important to consider whether something is really a tag (a label or marker) or simply metadata and additional information. When you're in doubt, we recommend that you metadata, as you can always use the Dynatrace tagging engine to create a tag based on metadata and assign the value dynamically.

What about Management zones?

Management zones are an important concept in Dynatrace and of high importance in large environments, as they can make large organizations work. Management zones are based on the same idea as automated tags, but designed for the explicit purpose of defining groups of entities that either belong together or have common security levels. A management zone can be

  • A demilitarized zone (DMZ).
  • The frontend for a line of business to facilitate easy access for the respective teams.
  • A production-first infrastructure layer, to facilitate access to infrastructure only for the respective teams.

Management zones can be used throughout Dynatrace to filter any view including Smartscape. Moreover, Management Zones enable you to define an additional layer of security and access restrictions. Thus, they serve management and security purposes. Management Zones may overlap with each other.

The best way to create management zones is to define rules based on your entities metadata (including custom metadata). However, if you want to maintain your management zones outside of Dynatrace, you can also base them on tags.

Problems, management zones, and tags

Problems are detected independently of management zones. This makes sense as a problem may involve entities from many zones. Only users that have access to these zones will be able to see the specific problem. Although they will be able to see the full problem, they will only be allowed however to access the entities belonging to the zones they are assigned to.

Problems can also be filtered by tags. This basically means that one can focus on problems related to entities with a specific tag. This is also how notification integration can leverage tags. One can define to only send problems that concern entities with a certain tag to a specific notification integration.

For more detailed information about tagging, see best practices and recommendations.