Two weeks ago I wrote about how the world’s largest exporter of dairy products uses SAP to support their huge delivery chain of dairy products.. We recounted how Fonterra used an Application Performance Monitoring (APM) tool to discover that additional training for employees was necessary to speed up certain critical SAP transactions: some employees weren’t optimizing attributes passed to the reports based on custom T-Codes.

Apparently Milk Churns can block Wi-Fi

Unfortunately, additional SAP training cannot solve all problems. Separately and unrelated to the training issues, Fonterra started to notice that end-to-end SAP transactions were much slower than usual; especially during the summer months. Correlating long-term network performance data, such as loss rate or RTT measures, with end-to-end SAP transaction response times revealed the actual network problem caused by something nobody thought about.

Managing a Dairy Warehouse with SAP

One of the reasons Fonterra invested in SAP was to keep track of milk delivered to their subsidiaries across the world. Each time milk can enters or leaves the warehouse it is scanned using a dedicated handheld device (VMU). These devices use wireless networks (WLAN) to connect to the dedicated Telnet servers, which in turn translate the scanning process into communication with the actual SAP servers (see Fig. 1).

Fonterra Scanning Diagram

Figure 1. Mobile devices (VMUs) use WLAN to communicate with the Telnet Servers 

This process is critical to the whole end-to-end efficiency of the dairy production, as it provides critical information on milk sources, amounts received, processed and shipped, and also provides mandatory data required to comply with food health regulations. No doubt the performance of this IT service is critical to the whole enterprise efficiency.

How Network Issues Affected SAP Performance

A few months ago, employees using those VMUs were complaining that it was taking longer and longer to get the scanned data delivered through to SAP.

Operations Sap Ecc

Figure 2. There are no significant application performance problems with T-Codes consisting on the SAP transaction (Bin to Bin Movement), which users complained on 

The Operations team analyzed the performance of the SAP application. They could not confirm any significant performance problems with the “Bin to Bin Movement” transaction; neither in general (see Fig. 2) nor for any major users (see Fig. 3).

All Users Sap

Figure 3. Looking from the SAP server performance there are not significant performance problems that affect users 

The Application Health Status report (see Fig. 4) confirmed that it was not an application problem as none of tiers were affected. However the report indicated that the problem might relate to potential network issues.

Poor Network Telnet Proxy

Figure 4. Poor network performance with high loss rate affects the overall user experience during the scanning process 

The Operations team consulted the network metric charts report (see Fig. 5), which confirmed network problems, especially with high client loss rate and increased RTT.

Poor Network Metric Charts

Figure 5. The process of sending data using VMUs suffers from increased RTT and high loss rate 

Since the traffic was rather low, network congestion was not to be blamed. High loss rate and RTT measures indicated that in fact it was some network problem that affected the user experience of the scanning process.

Why Did Network Quality Worsen in Summer Months?

Further analysis showed that the network performance between handheld devices and the Telnet server changed over time. The Operations team charted the loss rate trends across the whole year (see Fig. 6). They noticed that the loss rate was much higher in summer months, i.e., December and January in Southern Hemisphere.

What could introduce such a trend in network performance degradation?

The Operations team added session count metric to the report (see Fig. 6), which reflected the number of VMU scanning operations. They noticed that the session count is also increased in summer months.

Loss Rate Over Year

Figure 6. Loss rate for the Telnet software services over the year indicates some correlation to scanning activity which changes over the seasons 

Fonterra figured that during the summer season, milk production is increased which resulted in more milk cans being stored in the warehouses. Apparently the large number of milk cans can block the wireless signal, which increases loss rate. To solve the problem Fonterra began to upgrade the Wi-Fi Access Points in their warehouses.

Conclusions

Fonterra, as the largest global exporter of dairy products, required reliable solution to keep track of its goods entering and leaving warehouses across the world. Therefore it decided to monitor its SAP infrastructure with Data Center Real User Monitoring (DC RUM), part of the Dynatrace APM suite.

This story shows that looking at SAP server performance alone will not tell what is causing the performance problems experienced by the users scanning the milk cans. The reports showed that SAP was not at fault and the SAP basis team could prove it.

By monitoring network in context of transactions the Operations team could see an increase in loss ratewhich was flagged by DC RUM reports as a network issue.

However, only thanks to the long-term performance management database in DC RUM and flexible on-demand reporting the team could correlate loss rate to the scanning activity (shown with user sessions, see Fig. 6) and show that both have periodic changes over the year. In the milk industry, the seasonal changes in the business activity can attributed to changes in milk production. More milk in summer months leads to more milk cans in the warehouses, which eventually affected the wireless network reception.

Fonterra plans to change currently used generic handheld devices, which communicate with SAP servers though the telnet proxy, with dedicated SAP mobile devices. To communicate with the SAP server these devices use SAP RFC protocol, which is “understood” by DC RUM network probe; it will effectively improved end-to-end visibility in the milk cans scanning process.


(This blog post is based on materials contributed by Matt Lewis and Krzysztof Ziemianowicz based on original customer story. Some screens presented are customized while delivering the same value as out of the box reports.)