In a previous article, I demonstrated how to effectively and efficiently install the Dynatrace Application Monitoring solution using Ansible. In this post, I am going to explain how to achieve the same results using Chef with our official dynatrace cookbook available on GitHub and on the Chef Supermarket. In the following hands-on tutorial, we’ll also apply what we see as good practice on working with and extending our deployment automation blueprints to suit your needs.
Introduction to Chef
Chef supports a variety of use-cases in the enterprise and as such comes with a wide array of concepts and tools you’ll need to understand before you begin. First, I want to provide sufficient background for you to get our solution up and running in minutes — even if you are new to Chef.
Client-Server vs. Standalone
Chef can be run in only two modes: client-server or standalone. In the client-server mode (aka. agent-based), the (potentially highly-available) Chef server holds and distributes node configurations, policies and associated resources to all nodes under management, where they are enforced by periodically executing chef-client processes.
In standalone mode (in the absence of a Chef server), configurations, policies and resources must be made available on each node and are applied by executing chef-solo. The knife tool, especially in combination with knife-solo, greatly simplifies the bootstrapping and management of Chef nodes from a single provisioner machine, and can also handle cookbook executions via chef-solo on remote machines:
Concept #1: Cookbooks
A cookbook serves a defined scenario and contains everything (attributes, recipes, resources, etc.) that’s required to support this particular scenario. Examples are mysql, postgresql and roughly 2,500 more shared on the Chef Supermarket.
Concept #2: Recipes
A recipe defines everything that’s required to configure part of a system and must be stored in a cookbook. In order to become executed, a recipe has to be included in a run-list. Here’s an example of a simple recipe that installs a package nginx and then applies a set of actions to the corresponding service, given by node[‘nginx’][‘actions’], in file recipes/nginx.rb:
The node[‘nginx’][‘actions’] attribute has its default value specified in an attributes file that is given the recipe’s name in attributes/nginx.rb:
Concept #3: Run-lists
A run-list specifies a list of recipes to be executed in the defined order. The following example enforces the installation of the Dynatrace Server, and a Collector:
The Wrapper Cookbook Pattern
The Wrapper Cookbook is a popular pattern for changing the behaviour of an existing upstream cookbook without having to fork and make changes to it. I will apply this pattern throughout this tutorial and suggest you use it whenever you work with 3rd party cookbooks.
Here is how the pattern works: it enables you to decouple your specific business requirements — your company’s understanding of what comprises the installation and configuration of a particular piece of software — from the universality of a community-provided blueprint. By separating these aspects, you will not end up in the undesirable situation where you cannot update your 3rd-party cookbooks to a newer version anymore.
Automated Dynatrace Deployments
An installation of our Dynatrace Application Monitoring solution consists of an arbitrary number of Clients, multiple Collectors, and a Server. Performance data are stored in an external Performance Warehouse database. To learn more about the various components, please refer to our Dynatrace Community.
Now, without further ado, let’s dive into setting up Dynatrace using Chef. We’ll start by setting up a wrapper cookbook and integrate our dynatrace cookbook from the Chef Supermarket. Finally, we’ll use chef-solo to trigger the installation of a Dynatrace Collector, the Dynatrace Server and Postgres as a Performance Warehouse and tie them together — all done automatically through the wrapper cookbook we’ll create just now. Please make sure that you have a fairly recent version of the Chef Development Kit (chef-dk) installed.
#1: Create a Wrapper Cookbook
First, we want to create a so-called application repository to host our wrapper cookbook and all the other 3rd-party cookbooks our wrapper cookbook depends on. An application repository is a directory structure that supports multiple cookbooks to support an application. In this particular case, we’ll create a repository for a fictitious company AcmeCo to install the Dynatrace Application Monitoring solution. By convention, we’ll name the repository acmeco-dynatrace:
$ chef generate app acmeco-dynatrace
This command creates a directory structure that looks similar to what you see below. Note that the repository acmeco-dynatrace got pre-populated with a skeleton for our wrapper cookbook with the same name. For the course of this tutorial, this application repository will be our working directory.
acmeco-dynatrace |-- cookbooks | `-- acmeco-dynatrace | |-- Berksfile | |-- metadata.rb | `-- recipes `-- README.md
#2: Provide Wrapper Cookbook Metadata
We’ll now provide sufficient metadata to the wrapper cookbook by editing cookbooks/acmeco-dynatrace/metadata.rb and adapting the maintainer and maintainer_email entries to suit your needs. Note that the default license ‘all_rights’ means “Proprietary – All Rights Reserved” — well-suited for wrapper cookbooks which are typically for internal use only. Most importantly, we have to state that our wrapper cookbook depends on the dynatrace cookbook and the postgresql cookbook using the depends clause. More information on metadata.rb files can be found in the official documentation. Anyways, here’s what your file could look like:
#3: Resolve Wrapper Cookbook Dependencies
Cookbook dependencies are managed and resolved by Berkshelf. While the metadata.rb’s depends clause is merely informative, Berkshelf is capable of transitively resolving dependencies from various sources, such as the Chef Supermarket, GitHub and git into “The Berkshelf”, a local cookbook store typically located in ~/.berkshelf, or into any other location on your hard disk. Here is what a basic Berksfile which integrates dependencies from a metadata.rb file looks like:
Instead of resolving cookbook dependencies into the berkshelf store via the berks install command now, we want to have these dependencies resolved into our acme-dynatrace application repository as the single source of truth that we can keep in version control — a prerequisite for Continuous Delivery. However, this is not the only reason. Having our cookbooks available in a well-known location allows us to better integrate with tools such as chef-solo, vagrant and others. For this, we’ll assign all 3rd-party dependencies that our wrapper cookbook depends on into a vendor group of cookbooks by editing cookbooks/acmeco-dynatrace/Berksfile:
We now berks vendor these cookbooks into a directory vendor-cookbooks inside our application repository like so:
$ berks vendor vendor-cookbooks --berksfile=cookbooks/acmeco-dynatrace/Berksfile --only=vendor
acmeco-dynatrace |-- cookbooks | `-- acmeco-dynatrace |-- vendor-cookbooks | |-- apt | |-- dynatrace | `-- postgresql `-- README.md
#4: Configure the Wrapper Cookbook
The next logical step is to configure our wrapper cookbook to automatically install the Dynatrace Application Monitoring solution for our fictitious company AcmeCo. This involves the installation of a Performance Warehouse database for Dynatrace using Postgres, the installation of the Dynatrace Server plus license injection, as well as the installation of a Collector:
1. Install a Performance Warehouse Database for Dynatrace using Postgres
Create an attributes file in cookbooks/acmeco-dynatrace/attributes/server.rb and override the Performance Warehouse related attributes exposed by the dynatrace cookbook’s server.rb attributes file to suit your needs. By refining these attributes in the wrapper cookbook, we can effectively communicate their values to both Postgres and the Dynatrace Server from a single location:
Next, create the wrapper cookbook’s server recipe in cookbooks/acmeco-dynatrace/recipes/server.rb to describe the installation of a Postgres server, a database and an associated user given the attributes defined above (we’ll define and use shorthands definitions of the attributes for better readability):
2. Install the Dynatrace Server plus License
In order to install the Dynatrace Server and a license, we want to alter our wrapper cookbook’s server attributes file in cookbooks/acmeco-dynatrace/attributes/server.rb and add a location to a license file. According to the dynatrace cookbook’s server.rb attributes file we have two options here: we could either reference the license file from within our cookbook’s files directory or, alternatively, load it from a URL (typically a binary repository inside your company provided by JFrog’s Artifactory, Sonatype’s Nexus or a plain web server), as shown in the following example:
If you haven’t got a license yet, you can obtain a Dynatrace Free Trial license here. In order to install the Dynatrace Server and inject the license file, we have to include the following recipes in cookbooks/acmeco-dynatrace/recipes/server.rb:
3. Install a Dynatrace Collector
Installing a Dynatrace Collector couldn’t be any easier: we create a collector recipe for the wrapper cookbook in cookbooks/acmeco-dynatrace/recipes/collector.rb like so:
And that’s it! That’s all you need to set up the Dynatrace Application Monitoring solution in the enterprise and much more! By automatically provisioning a Performance Warehouse database and bonding it with Dynatrace, your installation is immediately able to preserve data for real-time data analytics. Wait a minute. Why aren’t there any installer artifacts? If you look at the dynatrace cookbook’s server.rb attributes file, for example, you’ll see that our scripts automatically obtain installers from downloads.dynatracesaas.com. A feature which we recently added, as described in my article Kick-Start Continuous Monitoring with Chef, Ansible and Puppet.
The Bigger Picture: Combined Results
Here’s a complete view on what we have achieved in cookbooks/acmeco-dynatrace/attributes/server.rb, cookbooks/acmeco-dynatrace/recipes/server.rb and cookbooks/acmeco-dynatrace/recipes/collector.rb, respectively:
#4: Execute the Wrapper Cookbook (using chef-solo)
Being a standalone tool, chef-solo knows nothing about where your cookbooks are located. This is the minimum configuration information that has to be provided to chef-solo, and is typically provided in a file named solo.rb. More information on solo.rb files can be found in the official documentation. Here’s what your solo.rb file could look for you if you had the acmeco-dynatrace wrapper cookbook checked out into /opt:
Finally, before we pull the trigger, we want to add the acmeco-dynatrace::server and acmeco-dynatrace::collector recipe to a run-list in run-list.json:
Now, we execute chef-solo within the acmeco-dynatrace wrapper cookbook and provide locations to solo.rb and run-list.json like so:
$ sudo chef-solo --config solo.rb --json-attributes run-list.json
As you can see from below screenshot, the automated installation of the entire suite takes less than 6 minutes on my machine, which is mostly determined by gathering dependencies from the Internet. The actual installation, wiring and the bringing up of services is actually done in slightly more than a minute. Do you want to give our integration a try? The acmeco-dynatrace wrapper cookbook is now available on GitHub!
After reading this tutorial you should have gained a fundamental understanding of Chef as well insights on installing our Dynatrace Application Monitoring solution in your enterprise using chef-solo. Even if you are running Chef in client-server mode, this walkthrough should have provided sufficient information to get you up and running with Dynatrace quickly.
Let me know if you have any questions and stay tuned for another part in this series when we will automatically inject the Dynatrace Agent into an application with Chef!