DevSecOps: Recent experiences in field of Federal & Government

Introduction

For Federal, State and Local agencies to take full advantage of the agility and responsiveness of a DevOps approach to the software lifecycle, Security must also play an integral role across lifecycle stages. Modern DevOps permits high velocity development cycles resulting in weekly, daily, or even hourly software releases. But outdated security practices pose a significant barrier even to the most efficient DevOps initiatives.

Overcoming the barriers presented by legacy security practices that are typically manually intensive and slow, requires a DevSecOps mindset where security is architected and planned from project conception and automated for speed and scale throughout where possible.

This is especially true when we consider the explosive growth of cloud and container environments, where containers are orchestrated and infrastructure is software defined, meaning even the simplest of environments move at speeds beyond manual control, and beyond the speed of legacy Security practices.

And this poses a significant risk. How to assess an environment’s security posture, risk and threat profile, and possible attack vectors, where perhaps you didn’t code the application or build the infrastructure, and where new code is deployed frequently, and the infrastructure is constantly changing?

In this blog we’re going to cover some of the recent, sometimes daily experiences I’ve encountered within Government and Federal agencies to demonstrate the role the Dynatrace platform plays to address these barriers and how agencies can leverage Dynatrace as an invaluable resource that contributes to DevSecOps success.

Quick recap of the Dynatrace platform

I don’t want to use too much space covering the Dynatrace platform in depth here, but I will provide a concise summary for contextual purposes. For a more detailed explanation, including why Dynatrace was named by Gartner as a Leader for the 10th consecutive time and received the highest scores in five of six Critical Capabilities use cases I recommend a quick review of the Dynatrace platform page and our Government solutions.

Fig. 2020 Gartner Magic Quadrant for APM and Gartner Critical Capabilities

Dynatrace is the leading Software Intelligence Platform, focused on web-scale cloud monitoring, delivering the richest, most complete data sets in the Application Performance Management market. By virtue of the incredible volume, quality, scope (we actually go far beyond just application monitoring) and granularity of the data the platform provides, our customers have at their fingertips unparalleled insights about their systems, users, and so much more. These insights are critical to ensuring proactive application monitoring and optimal system performance. But the benefits and insights delivered by the Dynatrace platform don’t have to stop there. Our powerful platform and the intelligence it provides can be leveraged in other areas as well – including part of our client’s cyber-security arsenal.

Challenge: Mapping environments in real-time to strengthen situational awareness

Earlier this year, well before COVID-19, when we could still travel freely (oh how I long for those days), I was at a defense industry conference focused on IT.  As attendees toured the conference halls, many people were drawn to the Dynatrace booth where they spotted images of our Smartscape entity maps that automatically display hundreds of nodes and their communication paths in real-time. Think of Smartscape as the visualization of ‘Observability’ across Applications, Services, Processes, Hosts, and Datacenters.

Fig. Smartscape showing real-time map of interconnected services

One group of service men and women who stopped by, attracted by the Smartscape, immediately asked “Can this help me in my cyber security mission?” Now, while Dynatrace is categorized as ‘Application Performance Management’ or ‘APM’, the reality is that the Dynatrace platform goes way beyond APM. This gave me the opportunity to pivot from purely APM talk, to dig into the topic of security. The reason for their question as it transpired, was the lack of anything that showed them an accurate and up-to-date map of the environments they were being asked to defend.

Most of my discussions begin with an explanation of how the Dynatrace platform works, including our OneAgent technology that automatically instruments and discovers application and infrastructure environments, and results in the entity map or ‘Smartscape’.  As I described how the Smartscape shows the relationships of host machines, processes, services, end users and their respective datacenter or enclaves, I saw them perk up.

Fig. Smartscape showing relationships between layers

As the conversation progressed I learned that this team, were responsible for cyber defensive activities, often working in environments where they didn’t have good situational awareness of the IT landscape, they often didn’t know exactly how the application and networks interconnected, what assets were most important, which services were accessible by internal and/or external sources, and therefore could not provide complete threat and impact assessments.

After understanding the Smartscape, they quickly saw how Dynatrace could provide the real-time situational awareness they needed to perform threat and impact assessments.

Outcomes

Utilizing the automatic dependency mapping functionality of the Dynatrace OneAgent, DevSecOps and SecOps teams gain real-time visibility into application and infrastructure architectures. This insight provides multiple benefits:

  • Identification of unknown infrastructure/services fulfilled by shadow IT, which is a huge concern for security professionals and often an easy attack vector for rogue actors.
  • When designing network segmentation programs that can help restrict lateral movement of bad actors across your infrastructure, understanding the design and flows of critical applications, whether on premise, in the cloud or containers is essential.
  • Understanding application relationships, communications flows, etc can help security teams reduce the time to isolate and contain threats. This visibility becomes even more important in containerized and microservices environments. As containerized environments grow and become more complex, attack vectors increase so visibility into and understanding the environment is critical.

Challenge: Monitoring processes for anomalous behavior

As we moved through the Smartscape, we started exploring data at the infrastructure level.  We looked at a hosts network devices, the flows between them and then at the process level details.  I continued to explain how Dynatrace was constantly monitoring at the process level and how our AI intelligently baselines performance, automatically detecting and alerting when anomalous behavior is identified.

This smart baselining capability drew great interest. “Could you tell me how this might help me understand if a system was compromised?” So, I gave the following scenario:

  • You are the Cyber Security resource attached to the DevOps team and the team just received a problem alert from Dynatrace, relating to user performance degradation localized to a single server, specifically caused by higher than normal CPU and network packets sent.
  • On further examination you discover an abnormal system process has initiated. Dynatrace knows it is not part of the system because the AI has detected an unusual increase in CPU resource consumption, and we see this process was not present the day before.
  • We also quickly check for system events and see there were no scheduled code deploys in the last week.
  • At this point the defensive security team gets involved, has the system shutdown and removed for forensic analysis.

Fig. Showing a list of key processes

Outcomes

Dynatrace has unparalleled capabilities to map, baseline and detect anomalous behavior in an application. This type of insight is invaluable when a system may have been compromised because it can reduce your time to detect.

Today, security teams often employ SIEMs for log analytics. These types of tools attempt to gather log data from as many sources as possible and then implement a rule based or machine learning approach to detect attacks. There are several problems with this approach, including frequent false positives, timeliness of detection, and the frequency of alert storms, all of which can ultimately result in alert fatigue and therefore a reduced security posture.

A much better approach is to get closer to problem, the compromised system, to understand system baselines. By understanding normal performance, false positives are significantly reduced. And by understanding the interconnectedness of the environment, alert storms are avoided as problems and root cause are identified along with what are just symptoms of the problem. This approach provides greater precision in detecting malicious behavior.

Along with the performance baseline analysis Dynatrace provides out of the box, there are several other fast and simple checks that you can perform to understand if a newly detected process is a threat.

  • From within the processes information screen you can get insight into the executable files name, path, command line parameters, etc as an initial check.

Fig. Showing system and process metadata

  • You can then analyze the processes incoming and outgoing network connections from process connections dashboard, to ensure the communication paths are what is expected.

Fig.  Auto generated process communication flows

  • This data could be fed into other forensics tools for deeper analysis. An example of this would be an integration with a Security Event and Incident Management (SEIM) platform to provide real-time asset and identity correlation, notable events, etc.  This type of integration could help eliminate the need to deploy other specialized agents to collect data.

Challenge: Mitigating unauthorized API calls?

In the final use case, I would like to share with you a conversation I had during a recent Proof of Concept (PoC) with one of the military branches.  PoCs are incredibly powerful as an exercise we do WITH our customers, rather than TO them. It helps customers experience the ease of install and automation rather than seeing it on a PowerPoint slide.

This PoC was no different, with the user being largely self-sufficient through the install and initial configuration of an air-gapped version of the Dynatrace platform. The team looking at the data quickly started to see their applications light up, bringing them instant insight into the true application topology, environmental dynamics down to the code level.

During one of our review sessions where we analyzed data that was being produced, one of the developers noticed an API call was being made to a system outside of the enclave where the application ran, presenting a major issue. The team rectified the problem immediately, but I was asked how Dynatrace can alert on external calls that are being made to the system to prevent this from happening again.

After a little brainstorming we developed a custom alert that focused on items tagged as being external to this system.

Fig.  Auto tracing of requests made to public networks

Outcomes

You probably have already guessed, but if not, Dynatrace makes analyzing and even alerting on API’s extremely easy.  After installing the Dynatrace OneAgent on the host(s) that serve the API(s) giving you full visibility into your incoming and outgoing calls. You could also use this with our out-of-the box log monitoring capability to quickly give insight into suspicious activity. Here are some key behaviors we might want to also monitor:

  • Multiple failed login attempts
  • Simultaneous API calls from known bad actor states
  • Uncommon API usage
  • DDOS attempts against your API’s

In conclusion

In this blog, we discussed just a few scenarios where the Dynatrace platform can bring together modern, agile and high-speed DevOps approaches with traditional security practices. Dynatrace automation enables DevSecOps capabilities to be realized, by automatically providing deep real-time, actionable insights and awareness into your applications and infrastructure, providing more complete situational awareness, while strengthening your security posture. In the future you will see even more innovation from Dynatrace in this space so please stay tuned.

Stay updated