How to monitor ECS clusters for Docker easily

Update: Please also have a look on this “How-To Monitor Amazon ECS Guide” and this monitoring AWS Fargate blog post.

Amazon EC2 Container Service (ECS) is a container management service that enables running of applications in Docker containers on clusters of EC2 instances. In contrast to running Docker containers with AWS Elastic Beanstalk, the EC2 Container Service allows for fine-grained tuning of clusters with different kinds of EC2 instance types, load balancing, and availability zones. Continue reading and find out how Dynatrace helps you in monitoring ECS clusters.

The basics

ECS is built around a few concepts you should be familiar with:

  • Cluster – A set of container instances logically grouped together to perform tasks.
  • Container instance – An EC2 instance that has Docker and the ECS agent installed on it. Accordingly, a container instance isn’t a running Docker container but an EC2 instance.
  • Task definition – A description of one or more closely related container definitions that are deployed together on the same container instance.
  • Task – A task definition instance running containers on a container instance; you can have multiple tasks running the same task definition.
  • Container – A Docker container that is part of a task.

Scaling Docker in ECS

Horizontally scaling out a service or application in ECS means letting ECS start additional tasks from the respective task definition. This way new containers are launched on existing container instances of the cluster. If a cluster then runs out of available resources and runs at capacity, additional container instances need to be added to the ECS cluster to maintain service performance.

Monitoring highly dynamic environments

Monitoring highly dynamic environments where containers come and go and instances are added to or removed from clusters via autoscaling mechanisms can be a challenge. Your performance monitoring solution needs to scale along with the infrastructure being monitored. Your cloud and container monitoring solution must effortlessly include newly booted instances and started containers in its analysis and visualization mechanisms. This is especially true for Docker containers that are far more ephemeral than instances. They may only live for a matter of seconds and you can’t change configurations or scripts all the time.

Benefits of OneAgent auto-injection

This is where Dynatrace changes the cloud and container monitoring game for you. Monitoring Docker environments with Dynatrace doesn’t require that you touch your Docker images or run extra containers. You only need to install OneAgent on your EC2 instances to monitor all created containers automatically. In this way, all new containers created on EC2 instances via ECS scaling are automatically monitored by Dynatrace.

Install Dynatrace on EC2 instances

Amazon ECS requires EC2 instances with Docker and an installed ECS Agent. AWS provides a couple of ECS-optimized images for this. You can find ECS-optimized images here.

When launching a new EC2 instance from the EC2 management console, configure the EC2 instance using the appropriate ECS-optimized image for your needs. Don’t forget to provide the ECS_CLUSTER variable so the ECS Agent knows which cluster the instance will be associated with.

With this approach you can also configure the EC2 instance to install OneAgent upon startup. Open Advanced Details on the Step 3: Configure Instance Details page and type the following text into the User data field:

echo ECS_CLUSTER={ecs-cluster-name} >> /etc/ecs/ecs.config
curl -o /tmp/ https://{tenant}{token}
sh /tmp/

Be sure to replace the parameters for {ecs-cluster-name}, {tenant}, and {token}. You can find the {tenant} and {token} values in Dynatrace: From the menu, select Monitor another host. Click Linux and use the pattern-matching algorithm in your brain to do the rest.

Extending the cluster

Each time you want to extend your cluster with additional EC2 instances, click Launch More Like This on the EC2 dashboard of the AWS console. The new instance will automatically be added to the {ecs-cluster-name} with Dynatrace installed.

Launch new EC2 instance like this

That’s all there is to it! With this approach, new instances are automatically included in Dynatrace monitoring. Depending on the types of services you run in ECS you may also need to create a load balancer to send traffic to all container instances.

Visualize your cluster

Visualizing highly dynamic environments in ECS is straightforward with Dynatrace.

With Smartscape visualization (shown above) you can see all processes (running in containers) for a service we deployed to three ECS clusters. Dynatrace also shows the EC2 instances that host the Docker containers. This way you can easily check all clustered containers to see if they receive traffic from the Elastic Load Balancer and if the clusters are set up correctly.

Learn about Docker container monitoring

Have your own cluster of Docker containers running somewhere on IaaS clouds? If you haven’t already, Sign up for Dynatrace today to get started with application performance monitoring in your environment.