Top Five Broken AA NPM Promises

You may have noticed – through experience, product literature, website content, or marketing emails – that virtually all AA NPM solutions tout benefits from capabilities embedded in such themes as “User Experience,” “Application Performance,” or “Business Impact,” – with enticing dashboards and lots of metrics and graphs to grab (and perhaps distract) attention. (I’ll be the first to admit I’m a sucker for modern, fast and beautiful dashboards.) In fact, at the recent Cisco Live show in San Diego, I counted at least a dozen vendors with signage using these terms. I’m also aware that many users of these solutions end up rather disappointed in the utility of these solutions, frustrated by the lack of new insights hinted by the terms. It’s not that marketing lied, it’s that the promises were filled with soaring but often misleading language.

In this blog, I’ll expose some of the more significant broken AA NPM promises, both implied and explicit.

Broken Promise #1: You can measure end-user experience

The implication: By measuring “response time” for end users, you will gain important insight into problems before they become pervasive, and can apply this context to infrastructure metrics to focus fault domain isolation and problem resolution efforts.

The reality: The definition of “end-user experience” is often diluted from its performance-centric meaning – which should be end-user response time, or “click to glass.” In fact, few AA NPM solutions measure real end-user response time, instead redefining the term to suit available measurements. Gartner may be a co-conspirator, referring obliquely to end-user experience as “The capture of data about how end-to-end application availability, latency and quality appear to the end user from a network perspective.” Given that stretched-thin definition, many AA NPM vendors simply group a few key network metrics to suggest the quality of the end-user’s experience – even though these metrics may have little or nothing to do with end-user response time. In a way, this approach supports the classic “defend the network” position; if the network is clean, it must be someone else’s problem.

Lacking an understanding of true end-user experience, IT faces at least two significant challenges:

  • You don’t know that there’s a problem until users complain
  • You are inundated with false alerts, chasing after problems that don’t exist or ignoring problems that warrant attention

An example: Application availability from a remote site is 100%, latency is normal at 30 milliseconds, and link quality (e.g., re-transmission rate) is perfect. Does that aaNPNmean the end user experience is good? What if latency changes from 30 milliseconds to 50 milliseconds? What impact does that have on end-user experience? (The answer is “It depends.” Some applications would exhibit minimal impact from such a change in latency, while others might slow dramatically.)

Summary: End-user experience provides the business-centric context for interpreting the importance and impact of the underlying device and application metrics; it’s the single metric sitting at the intersection between business and IT metrics. But it must be defined as end-user response time to deliver on its implied value.

Broken Promise #2: Application Awareness Enables Business Collaboration

The implication: Through application awareness (the AA part of AA NPM), IT will be able to communicate effectively with the business, align initiatives and prioritize activities in lock step with business goals.

The reality: I often wonder just what information provided by AA NPM solutions might be of interest to the business. (Actually, that’s a bit of a fib; I do wonder occasionally, but when I ask the question, it becomes clear that two of the key metrics of interest are application availability and application performance. No big surprise.) We know that, to be meaningful to the business, performance must be measured as business users experience it, not by internal metrics such as network utilization, session-layer response time, or round-trip delay; to a business owner, these are just as uninteresting as CPU utilization or heap size.

It is true that the application awareness inherent in most AA NPM solutions can be used to “protect the network” from rogue or heavy applications; this is fundamentally a defensive mechanism by which the network team can gather evidence to blame certain application traffic for disrupting the performance of other applications and users. In one sense, this is counter-productive to collaboration since it strains the relationship between IT and the business. Application awareness can also be useful for application-specific capacity planning; I’ll concede that there might be some collaborative value there – but it’s a small concession that doesn’t deliver on the promise.

Alignment with your business peers requires a common language and shared priorities. This doesn’t mean that business learns ITs language, but rather the opposite; IT must learn the language of the business. Understanding end-user experience as a key metric of business productivity provides the foundation for this communication. This perspective becomes the means to translate IT metrics into business metrics, enabling mutually understood goals and decisions. Lacking effective business collaboration, IT struggles to understand how to prioritize activities and investments that contribute most effectively to business goals.

Summary: Understanding end-user experience provides the foundation for effective business collaboration, enabling appropriate resource and financial investments.

Broken Promise #3: End-to-end Visibility Means End-through-end Visibility

The implication: End-to-end visibility suggests the ability to understand application performance through the entire application delivery chain – from the client, across the network and through firewalls and load balancers, and also through backend application and database tiers.

The reality: AA NPM solutions may provide network-level metrics at each tier, and often also include generic session-layer response time measurements. There might also be some ability to group servers together – graphically and/or logically – to form an end-to-end picture of the application’s architecture, reporting network performance at each tier. The solutions often apply heuristics to detect changes in response time at each tier; if a change exceeds a threshold, a fault is assumed and the offending tier is highlighted as a likely suspect. There are quite a few problems inherent in this approach; here are two of the most significant:

  • Without an understanding of end-user experience, how do you know you have a problem? When do you start to care about changes in performance reported by the solution?
  • What thresholds are used to trigger alerts or anomalies?
    An example. If first-tier session-layer response time changes from 50 milliseconds to 60 milliseconds, should you take action? If response time at the database tier changes from 4 milliseconds to 6 milliseconds, how does this impact end users?

Consider this response time report mockup:


This is end-to-end, in a crudely simplified sense; these are the servers involved in delivering the application to the client.

  • Can you tell whether the user is currently having a problem?
  • If the user is currently complaining about poor performance, can you tell which tier is at fault?

Summary: An understanding transaction-level performance at each tier of a complex application environment is critical to effective fault domain isolation, but it must be correlated to the measurement of end-user experience.

Broken Promise #4: Response Time Measurements are Actionable

The implication: By measuring response time and comparing this to baselines, IT can take informed actions to remediate problems as they occur, or prevent problems from affecting users.

The reality: While response time has always been integral to performance monitoring, its definition changes according to the discipline. Response time can refer to isolated singular request/response exchanges such as a memory fetch, a disk write, or a database query; these metrics are internal to the technology discipline. Response time can also be external, referring to much more complex exchanges such as loading a web page, requesting a document download, running a complex report. With today’s hyper-focus on the user, the ability to monitor response time is quite compelling – if response time is defined in the user’s terms. But not all response times are created equal. In fact most AA NPM vendors apply an internal network-centric definition, not an external user-centric definition.

Let’s take a quick look at 4 categories of response time you might expect to find in AA NPM solutions. (I’ve explored these in more detail in a webcast if you care to review.)

Round-trip time. Often derived from observing SYN/SYN/ACK exchanges used to set up TCP connections, this reports the round-trip delay between clients and servers. Similar to the ubiquitous ping, this measurement helps us understand the propagation delay across the network between two points, excluding server delay. Does it measure application response time? Of course not.

Session-layer response time. This is simple enough to calculate, and as such is ubiquitous in AA NPM solutions. Basically, it measures the delay between a request message and the corresponding reply message. By associating this with an application (via TCP port mapping or NBAR2, for example), this hints at an actionable application-level metric – but in reality, it is rarely even informative.

Hit-level response time. For a web page, a “hit” is a page component or element – such as a JavaScript file, an image, a stylesheet, or a java server page. (Many non-web applications behave similarly; I’m using web as a common example.) Fundamentally, a hit is equivalent to session-layer response time, with the notable difference being that it has a meaningful transaction label – such as a filename or a method call – enabling the measurements to be separated into unique hit-level categories. This adds important and valuable insight, but still (usually) falls well short of the goal of measuring end-user experience.

End-user response time. This is the true definition of response time as experienced by the user. This is the metric that is common to the business, the user, development and to IT. For web, this is a page load, while generically we can call it “click to glass.”

An example: Consider a web page comprised of a JSP, some JavaScripts, some style sheets, and a bunch of images. Let’s say there are 40 page components, meaning there will be 40 session-layer response time measurements for the page. Now let’s say they range from 1 millisecond (for static content) to 10 seconds (for the JSP). The simple application association (e.g., NBAR2 classification) inherent in session-layer response time means that these measurements are all included in the same reporting bucket. They may average 0.30 seconds – for a single page. Can you tell if the user is having a problem? Compound this with hundreds of users loading many pages, and you end up with statistical insignificance. Now let’s add labels to the page components to arrive at hit-level performance insight; no longer are the requests for images mixed with the long-running JSPs. But the performance of individual page components doesn’t directly correspond with the end-user’s experience, limiting the value to generally reactive interaction with development teams (see the next broken promise).

Summary: Response time measurements benefit from correlation to end-user experience; without this context, only the more catastrophic problems are clearly visible and actionable.

Broken Promise #5: AA NPM Facilitates Collaboration with Development Teams

The promise: Application awareness provides the foundation for effective collaboration (dare I say DevOps?) with development teams. For problem and incident management, the solution should be able to provide the diagnostic information to IT operations that allows the development team to:

  • Accept that there is a problem related to an understandable application/code function
  • Begin to investigate the application logic invoked by the transaction, along with underlying dependencies

The reality: This may sometimes be partially true, with a few important limitations. Some AA NPM solutions provide hit-level response time measurements (for some application types) that can be considered the lowest common denominator. But there remains a critical missing link, one that significantly impairs your ability to leverage this application transaction insight; if you haven’t guessed by now, that missing link is end-user experience.

Without measuring end-user experience, you won’t know users are having a problem until they call you (unless something catastrophic happens). And you will chase after problems that don’t affect users (because you’re monitoring hundreds, or thousands, of application components).

An example: Let’s say you’re monitoring a handful of web applications, with hit-level measurements for tier 1 (HTTP), tier 2 (SOAP) and tier 3 (SQL). How will you know when you have a problem? If the performance of the OpenSession JSP changes from 3 to 4 seconds, is that a problem that needs to be shared with the development team? What happens if JavaScript file download performance degrades from 1.0 to 1.5 seconds? Does the network team need to respond to this? What if the performance of a specific SQL query changes from 4 milliseconds to 6 milliseconds? How do you go about setting appropriate performance thresholds for thousands of application components?

The images below are load sequence charts that illustrate loading a simple web page; each bar in the Gantt chart illustrates the timing of a page component. (These screenshots could also represent an SAP transaction, or an Oracle Forms transaction.) The top image shows normal behavior, while the bottom image shows degraded end-user experience.



Hit-level response time measurements would show no problem; in fact, the long thread (this was a JSP) actually completes slightly faster in the slower page load. Without the context of end-user experience, even hit-level monitoring fails to deliver on this promise.


Promises hold value only if they deliver. It should be apparent that these five are all interrelated; they break primarily due to the lack of visibility into end-user experience. It’s this transaction-centric context that Dynatrace DC RUM uniquely provides for a comprehensive range of web and enterprise application protocols.

How does your experience support, or maybe contradict, these thoughts? Are there other promises you’ve heard that are similarly inflated? I’m quite interested to hear your comments.

Gary is a Subject Matter Expert in Network Performance Analytics at Dynatrace, responsible for DC RUM’s technical marketing programs. He is a co-inventor of multiple performance analysis features, and continues to champion the value of network performance analytics. He is the author of Network Application Performance Analysis (WalrusInk, 2014).