Defending Network Performance with Packet Level Detail

My favorite war room accusation is: “It’s always the network at fault!” Whether you’re the one taking the blame or the one pointing the finger likely has everything to do with which seat you occupy in that war room. I suppose that comes with the territory, because at the same time there seems to be a consensus that the network is vital to understanding that holistic “Big Picture” on how application delivery is experienced by your end users. I suppose we’ll have to live with those accusations, but we don’t have to take the blame. Here’s how you can defend yourself.

The view from the network provides both context and accuracy relative to performance and availability problems in your environment. By collecting a combination of network based performance information and data obtained from points in the application delivery chain will provide you with the fastest path to the root cause of an issue. This collective view coupled with fault domain isolations analytics, will point you to the areas to focus on. Some application problems require a deeper view that might look into the application logic and execution, while others with a network focus, require that same level of detail but at a packet level. Knowing how and when to determine this quickly is the key to minimizing the impact.

Conventional wisdom will tell you that back in the “olden days” this was easy because there were certain inalienable truths:

  • Networks were slower
  • Traffic volumes were lower
  • Applications were less complex.

If only that was the case today!

Packet Level Data is the Key

Packet level data is often sought out as the source of the truth, however, in practice this is often believed to be unobtainable. The best available approach, it would seem, is to summarize that data. But as network engineers we know that summarizing data is not going to give us what we need. As by either summarizing or selectively sampling at this level you’re very likely to either miss the key packet or at best blend contributing factors; “the packets never lie.” And that’s exactly what’s needed to obtain definitive proof.

So, if we believe that the answer lies in that packet level data, then our objective should really be to provide that packet level data, even when faced with the growing complexities and obstacles that impede the traditional approaches.

The first issue is one simply of load. The increase in interface speeds coupled with the sheer amount of data in flight these days, means that a simply sitting packet collector into a promiscuous mode on the network will at best present you with extremely large amounts of data. Then, assuming you can collect everything, it requires significant effort and time to even locate the conversation and find the packet conditions of interest.

The trick to overcoming large quantities of data is to capture packets specifically in the context of the conversation (e.g. source and destination) that you’re investigating.

Using a “high-level” report view allows you to “frame” the traffic of interest.
Using a “high-level” report view allows you to “frame” the traffic of interest.

Right Packet at the Right place at the Right Time

The next issue is collecting the packets at the right points in the transmission path. After all, one of the reasons you might be looking at the packet level is to understand the in flight packet and traffic behavior.

Multi-point simultaneous capture allows you to collect the packets at different points in their journey.
Multi-point simultaneous capture allows you to collect the packets at different points in their journey.

Finally, you need to have that packet level data available when you need it.  Whilst we all strive to operate in a proactive manner the truth is there will inevitably be problems that are reported retrospectively, and the evidence required to resolve can lie anywhere from minutes to, in some cases, days ago.  If the problem you encountered occurred sometime in the past, it doesn’t diminish the argument that the packets provide a source of truth. So you also need to be able to apply that same search criteria and retrieve those packets retrospectively.


 If you need the packets to solve your issue, you not only need real-time performance monitoring, but also convenient and efficient access to historical packet level so that when a subtle or intermittent problem occurs, you can quickly go back in time to discover the root cause.

Understanding Packet Level Data

Collecting the correct and relevant packets is key to this process. But what good is a series of packets if your next stage of analysis is complicated and not intuitive? We’re looking at the packet level data because we need that level of detail in order to definitively solve the issues. It doesn’t matter how pretty or aesthetically pleasing your reports are if the views presented are confusing or vague. But if you can, provide transaction visibility between network and application level user functions. You will be able to not only create a mutually acceptable cross domain workflow, but also provide data in a common format, coupled with intelligent, intuitive analysis, to investigate application performance, and application behavior, down to packet level diagnostics. This ultimately provides you with the evidence you need to communicate with your cross department colleagues that, on this occasion, it wasn’t actually the network at fault and here’s why.

It’s important to show the packet level data at a transactional level with contextual details and data.
It’s important to show the packet level data at a transactional level with contextual details and data.

Mike is a Senior Product Manager for Network Performance Monitoring at Dynatrace. In his 30+ years in the industry he has worked closely with the majority of infrastructure vendors in the area of Application profiling, network performance, and management. In addition, Mike has authored two books, Managing Distributed Applications: Trouble shooting in a heterogeneous environment (Prentice Hall 2000) and Optimising Applications on Cisco Networks (Cisco Press 2004).