Keynote offers a version of its Web Performance Application Perspective and Transaction Perspective services as private agents—turn-key, managed appliances that can be placed anywhere within a customer's network or web application environment. Private agents allows a customer to test and measure web performance from within their own network environment and compare performance from different parts of that environment, such as geographically distributed data centers, branch offices, retail store locations, etc.
Another key feature of private agents is that customers are able to provision as many measurements as a given private agent can support, in any combination the customer desires. This can include very high-frequency measurements (e.g., at one-minute intervals), a large number of single- or multi-page measurements (i.e., 1,000+ single-page, 100+ ten-page, etc.), and large-file downloads (e.g., 100 MB files), all at the same cost to the customer. Given the infinite combinations and variables that can make up a given private agent's measurement profile, it can be difficult to understand and calculate private agent measurement capacity.
Those considerations include:
- Number of elements (which varies for each measurement)
- Measurements timeout setting
- Frequency of measurements
- Bandwidth of network connections, especially if you're measuring an intranet site (when a website times out or fails, the data generated increases considerably)
- Whether WCD is configured
- ApP (Darcom 2 appliance), TxP (Darcom 2 appliance) or TxP (Mazda appliance)
- Maximum simultaneous measurements configured (restricted by maximum number of threads count—App has 20 and TxP HF 8)
Method 1: rough guidelines
If you don't have time or the background information on the types of measurements that the prospect/customer has, then you can use the following conservative guidelines.
- ApP: "Up to 5,000 page views per hour"
- TxP "Up to 300 page views per hour"
- TxP HF with 8x measurement threads: "Up to 2,400 page views per hour"
These estimates allow for the customer having WCD enabled.
These numbers are rough estimates, so you should never guarantee these capacities. If you need more accurate assessment of agent capacity, please follow Method 2 below.
Method 2: measurement minutes
To gain the most accurate understanding of the capacity of your private agent, consider the factors detailed in the following sections:
- Available Measurement Minutes
- Setting Timeout Thresholds
- Available Bandwidth
- Simultaneous Measurement
It is important to understand that measurement capacity is highly dependent upon the following factors:
- Local network environment where the private agent is located
- Performance of the web applications being measured
- Type and complexity of measurement scripts themselves
- Number of pages and frequency of measurements
- Bandwidth available to the private agent
- Size of files being downloaded
- Number of simultaneous measurement "threads" a given private agent is configured to support.
Each one of these areas can affect the other and therefore there is a high interdependence between them.
Available measurement minutes
Given this complexity and the need to provide for a simple way to calculate and communicate private agent measurement capacity, the best measure of capacity is Available Measurement Minutes per Hour (AMMH). To calculate AMMH, it is important to understand the different time elements involved in a given measurement.
Each measurement data point has an
agent_delta_msec metric reported as part of its measurement. This measurement time in milliseconds is the total time the agent took to fully execute, download, parse, calculate, and "package" a given measurement into a data point that is then sent to Keynote for insertion into the backend database(s). Using
agent_delta_msec with the concept of Available Measurement Minutes provides the best means of calculating a private agent's capacity.
- TxP private agents have a capacity of 54 measurement minutes per hour.
- TxP HF private agents have a capacity of 432 measurement minutes per hour with up to 8 simultaneous measurements.
- ApP private agents have a capacity of 1080 measurement minutes per hour with up to 20 simultaneous measurements.
- The maximum measurement time is determined by the configured timeout.
- Your capacity is also determined by the network connection speeds, which becomes a large factor when running multiple simultaneous measurements.
Setting timeout thresholds
To size how many minutes an individual measurement can take, you need to know the timeout setting. Then, when there is an outage on the website being measured, the worst case scenario is the measurement's timeout. The default timeouts are as follows:
|Measurement||Default Timeout (in Seconds)|
To maximize private agent capacity, we recommend that you:
- Set the measurement timeouts to the most accurate number of seconds that you think your users will wait before abandoning your web page.
- As it is highly unusual for all transaction measurements on a private agent to fail and timeout simultaneously, calculate the percentage of the private agent measurements that time out and the average response time for the rest of the measurements.
The amount of bandwidth available to a private agent can affect its measurement capacity. If the amount of available bandwidth is less than the bandwidth requested by the agent, all measurements executed during this period of time will inevitably "fill up" the network bandwidth, causing reports of slower performance times. Since slower performance times cause in an increase in the
agent_delta_msec reported, these slower measurements can negatively impact capacity. Here are the bandwidth items to consider:
To manage bandwidth consumption by a private agent, there are a number of factors to consider: Physical connection to the Private Agent (i.e. Fast Ethernet, Gigabit Ethernet, etc.), available throughput between the private agent and source of the content, and the number of simultaneous measurements executed by a private agent. It is best to install a private agent on a full-duplex, 100 Mbps Fast Ethernet network. Private agents support Gigabit Ethernet (copper) as well. The faster the connection to the agent, the lower the chance of any network resource contention issues.
Many factors can impact the available throughput between a private agent and a distant web server/application/etc. It is safe to assume that the closer a private agent is to the source of the content it is downloading, the greater the available bandwidth will be for that given measurement. For example, if a private agent and web server are part of the same 100 Mbps subnet with little to low traffic load, theoretical throughput could reach 100 Mbps. However, this is rarely realized in practice due to operating system overhead, etc. If the web server were, say ten network hops away with a 1.54 Mbps T-1 in the middle, then the maximum available throughput could be no greater than 1.54 Mbps and likely much lower because different network hops in between could further reduce this number.
Application Perspective (ApP) and Transaction Perspective High-Frequency (TxP HF) support the ability to execute multiple simultaneous transaction scripts/measurements called measurement threads. This allows for more measurements to be supported within a given time frame (e.g., one hour) than if only one measurement could be executed. However, like anything, there is a tradeoff.
As you increase the number of measurement threads, the impact on system resources goes up at an exponential rate. Each measurement thread is handled by a playback engine. Each playback engine requires some amount of memory and CPU to be allocated to it during a given measurement. As more playback engines compete for the same finite resources, contention can occur and affect measurement results, e.g., produce invalid and/or slower results. Keynote's ApP and TxP HF agents have default measurement thread values of 20 and 8, respectively. These should never be increased, and if resource contention is a concern, should be decreased.
In addition to system resource contention, as you increase the number of measurement threads, the peak bandwidth needed by a private aagent also increases. This is because more measurements are downloading more content at the same time. How much bandwidth is required is very dependent upon the type and size of files and measurement being conducted, as well as frequency.
This article is intended to provide private agent users a much more accurate sizing estimate than the previous estimated pages per hour ceiling system. The only challenge of this new sizing system is that it requires accurate estimations, calculations, and a special focus on the perceived bottleneck(s). Keynote Solution Consultants and KDS Support Engineers are best qualified to use this information to size private agents. If there are remaining questions and/or input on private agent capacity sizing, please email the following mailing lists:
"Private Agent Escalation List"