This eBook is inspired by the famous 6th century Chinese manuscript: The Art of War. Today, we will embark on a mission to continuously deliver assets to the operational battlegrounds safely, securely, and quickly.
Our strategy requires optimal monitoring of the frontlines, maximum situational awareness, and enhanced communication with the troops supporting asset deployment—from development and testing to operations.
While I know most of you are battle veterans and have fought many application development wars, I have conducted some reconnaissance over the years and have recommendations that will put you at strategic advantage to win today’s war.
You are fighting to deliver the absolute best services and features to your customers quickly, without requiring the war room scenario we all know too well. Continuously delivering the highest quality assets to the front lines—both quickly and reliably—is our mission. We WILL win this war!
Communicating orders via a world-class feedback and delivery loop
The elite DevOps team performs critical preparation drills on the islands of development necessary to research, build, and assemble high-quality assets and deploy them to the staging grounds. The drills provide each development troop member with clear and concise orders.
The critical preparation drills are:
For our mission to be successful, it’s imperative for troops to use the same tools, aligned by their functional purpose, across the end-to-end application delivery pipeline. The orders must be tracked in a system that prioritizes and provides full traceability. This enables officers to quantify areas of weakness early, identify accountability, and target training and/or rework.
Good tracking and release management systems provide line of sight to weaker troops and assets, if casualties are sustained in battle. Jira Agile, Testtrack, and Caliber are a few examples. With release management and tracking, each developer has more incentive to maintain high quality assets. Orders can then be tracked with confidence throughout each delivery sprint.
Our development arsenal: Advanced performance monitoring
Now it’s time to look at ways we can mitigate deployment risks. Anything we can do to avoid deploying assets plagued with issues will ultimately save time and money and reduce fatalities on the battlefield.
The earlier we find the issues in the application delivery pipeline, the better. One way to reduce risk is to enhance our developers’ IDEs such as Eclipse, Visual Studio or IntelliJ with advanced performance monitoring.
Advanced performance technologies, like Dynatrace Application Performance Monitoring, provide:
With advanced performance technology, we’ll pinpoint issues faster and reduce the need for numerous break points and line-by-line debugging. Developers will conduct end-to-end post mortem analysis on every local run because call stacks are recorded each time code is run. Not only does this prevent a risky check-in, but designers (i.e. solution engineers and architects) can perform visual architecture validations instead of having to do full code reviews.
Our unit testing arsenal: Advanced performance monitoring
At the next stage on islands of development—asset check-in and build on the CI server—there is another opportunity to use advanced performance monitoring technology.
Completion of the following steps will fully prepare our assets for the upcoming staging grounds. They will provide maximum situational awareness on the condition of the build and on how it will be impacted by future configuration changes.
CI servers, such as Jenkins or Bamboo, allow the development force to test and piece together the outbound assets necessary for deployment. During the CI build step, a series of unit testing scripts are executed to exercise as many critical methods as possible. Code coverage tools such as Clover, SonarQube, Team Test, or dotCover will determine critical areas of code to unit test.
Once unit test automation scripts are executed, the rich intelligence collected by advanced performance monitoring can be put to use. Performance technologies raise alerts or perform automated tasks such as issuing change requests back to the development troops when KPIs or SLAs are violated. Build-scripting tools like Maven, Ant, or MSBuild align and label performance data for each build, enabling actions to be auto-orchestrated.
Finally it’s time to automate the infrastructure configurations and deployments. Common tools like Chef, Puppet, and Ansible coordinate infrastructure changes across one or more machines and certify that all of the proper performance monitoring settings are configured within each targeted machine.
Determining battle readiness
Troops should now have deployed the build from the islands of development to the staging grounds.
We have outfitted our environments with advanced performance monitoring technology and prepared for testing to evaluate the battle readiness of the build.
In the staging grounds, each build must be exhaustively tested to uncover the issues—with performance, security, disaster recovery, UI, processes, and infrastructure—that can infiltrate a successful operation.
Risk analysis performed early in the islands of development (during the requirements gathering phase of each sprint) drives the testing approach.
Risks must be assessed based upon how each addition or change in the sprint will affect the overall operation. The risk level for each change will determine:
Automated testing as our supplemental guard
It can be challenging, time consuming, and expensive to construct automated tests. While they bolster our defenses, they also introduce new complexity because they need to be maintained and regularly updated.
We’ll use testing automation as a supplemental guard to our manual testing approach. Test automation reduces issue identification time, and enables troops to focus manual test cases on specific scenarios.
Automated tests require a solid feedback loop between operations and development. As a best practice, we’ll track and manage test cases in a test case manager like Zephyr, Silk Central, or TestRail that integrates with our requirements tracking tool. This will maintain traceability by linking test cases with requirements throughout the development lifecycle.
Our testing arsenal: Advanced performance monitoring
After we have established our testing strategy including use of automated tests, we are ready to select our testing weapons.
Advanced performance monitoring weapons are also a critical component of a testing arsenal in the staging grounds, lowering MTTR (mean time to resolution) and providing a major strategic advantage.
Performance monitoring services, like Dynatrace Synthetic Monitoring, offer comprehensive testing services via a global network of machines. These machines create a simulation emulating a vast number of end users, with a diverse set of desktop configurations and mobile devices located around the world. Traffic patterns can be increased to simulate high volumes of concurrent users to provide troops with accurate real world impacts on the build. This automated testing technique in combination with highly focused manual tests provides the optimal approach to issue identification.
Intelligence provided by advanced performance monitoring weapons:
Building lines of defense
The final step in the staging grounds is building our lines of defense. The lines of defense will cut the time it takes to promote changes into the operational battlegrounds and reduce the need for war rooms.
First line of defense: automatic alerts
Our first line of defense is automatic alerting, which includes:
Second line of defense: dashboard command center
A second line of defense is to construct an accurate picture of the build’s operations and make it available to every level of command. We will do this by creating consistent dashboards that offer visibility into the environment. Elite DevOps troops from the islands of development through to the operational battlegrounds should use the shared dashboards to view, communicate, and analyze the same metrics.
Third line of defense: drill-down intelligence
The third line of defense is the ability to isolate an issue down to a specific test case, metric, or even an end-to-end enterprise call stack. This intelligence must be packaged to send directly to development troops responsible for analysis or the change. This is what separates a modern battle strategy from an antiquated one. Implementing all three lines of defense will reduce MMTR and increase our troops’ confidence in moving the build successfully from the staging grounds to the operational battlegrounds.
Maximizing situational awareness
Everything up to this point has been preparation for the operational battlegrounds. This is where the world of users converges into our environment and often pushes assets to unexpected extremes.
Winning the war means happy customers and business commanders. We must have a comprehensive performance monitoring strategy for the operational battlegrounds with laser accurate detection and rapid resolution techniques. The same performance monitoring tools and metrics we used in development and testing—tools like Dynatrace should be deployed here.
Our virtual command center—the performance dashboard—maximizes situational awareness by collecting and managing intelligence on:
Standard dashboard views must be designed via user scenario before we start asset deployment to the battlegrounds, but ad hoc capabilities are also a necessity. These capabilities allow our troops to react quickly to gather intelligence in a changing environment.
Strategic intelligence to gather
Risks and issues board
Security details and risks
Winning the war
“You can win the battle but lose the war.” Operations can provide crucial intelligence on prioritizing the battles.
For example: what if Sargent Joe has been diligently working on two query issues over the last few days, but as it turns out the problem is affecting less than a quarter percent of the users?
When the entire engineering team works closely together informed by strategic business goals and metrics, we can prioritize sprints based upon business value, and the value of each deployed asset will be greatly increased.
Communication and issue resolution
Communication is the absolute key to winning the war. It’s one thing to simply have data visibility and quite another to get actionable information that can be shared with the entire engineering team. With common tools and metrics, troops can identify an issue on a dashboard and drill down for the root cause analysis, and the team member responsible can conduct problem diagnosis and resolution.
Common communication procedures can now be established when handling each class of issues, for example:
Intelligence for future missions
The Art of DevOps is a continuous delivery approach that accelerates time to market by integrating performance monitoring into the islands of development and staging grounds to eliminate the enemy of poor performance on the operational battlegrounds. It’s been fun sharing some of the insights I’ve collected over the years. I hope you’ve gained valuable intel during this series to aid you in fighting your battles more effectively and embracing diplomatic solutions between development and operations.
I am leaving you with some additional intel to review for your future missions. So, until next time, stay strong, push onwards and upwards.
DevOps tools we love
Here is a collection of tools that foster collaboration amongst Product Management, Development, IT Operations, and Technical Support teams to allow you to build more quality into your products, and support you in establishing better feedback loops.
Global DevOps Practice Lead, Dynatrace
Brett has twenty-five years of experience—from lead developer, product designer, solution architect, application manager to senior management—that has given him a unique 360°perspective that has earned the respect of customers and peers alike. Specializing in team dynamics and delivering complex mission critical projects under methodologies such as Agile, Lean and Waterfall, Brett brings a deep and broad experience of life cycle expertise, continuous delivery and DevOps into a practice at Dynatrace.