In my previous posts I wrote about how important it is to have end-to-end visibility into SAP in order to avoid serious problems in our delivery chain or to discover that application performance degradation was caused by malfunctioning hardware.
One of our New Zealand customers, Fonterra, who is the world’s largest exporter of dairy products, uses SAP to support its delivery chain of dairy products made from 22 billion liters of milk collected each year.
In this article we show how Fonterra uses a new generation of APM tools which provide both user and transactional insight, as well as complete end-to-end coverage to monitor its SAP infrastructure. Monitoring SAP infrastructure lead Fonterra to quite surprising discoveries that some performance problems could be caused by insufficient SAP training or that milk churns block Wi-Fi signal in Fonterra warehouses; we will discuss the latter story in my upcoming blog post.
Tracking Milk from Cow to Cheese
Fonterra uses SAP to manage its operations, enterprise resources and controlling. Fonterra sites span across the whole world: including New Zealand, Australia, China, the Middle East, the US, Mexico, and Brazil (see Fig. 1).
Although dairy production is typically local to the markets, the management of such a large enterprise requires centralized resource planning and financial controlling, using company-wide best practices that enable the company to deliver high levels of productivity while staying ahead of the competition. This is where SAP ERP plays pivotal role and where global access to the central SAP systems is critical to the whole company operations.
With such a critical, diverse and global SAP implementation, Fonterra decided to use an APM solution to assure uninterrupted access and a high quality end-user experience with the SAP ERP, worldwide. Fonterra extended the out-of-box reporting with a number of custom built reports that enabled focused views into specific SAP performance use cases relevant to Fonterra.
How Mexico is Different from São Paulo?
One of the first problems discovered by the operations team was that certain SAP transactions at the subsidiary in Mexico were much slower than other offices. Both offices in Latin America, i.e., in Mexico and São Paulo, unlike the subsidiaries in Tokyo or Dubai, are not connected directly to the SAP servers in New Zealand. Instead the traffic is routed through the office in the USA. However, only the office in Mexico was experiencing performance problems.
The operations team from Fonterra compared application and network performance between subsidiaries (see Figure 2). As expected, network time, network performance and client RTT metrics show that the quality of the network in LATAM sites is worse compared to other offices, especially the main office (Fonterra Centre). Still, the overall application performance (reported by operation time and server time metrics) does not seem to be affected by the geographical location of the branch office.
This report confirms that there is additional network latency from Mexico and São Paulo (see Figure 2) but it does not explain why employees from Mexico report poor performance while those from São Paulo do not? This required a deeper inspection into the transaction times across the four remote offices: São Paulo, Mexico, Dubai and Tokyo. All transactions had comparable operation times except for Sales Order Status Report. This transaction was more than three times slower in Mexico than in other locations, including São Paulo (see Figure 3).
Lesson Learned: Monitor and Optimize T-Code Attributes for Performance
The question remained as to what was the key difference between Mexico and São Paulo? In order to answer this question the team looked even deeper into the specific SAP transaction code (T-Code) and the specific attributes that were used by users from the various locations. This revealed that users from Mexico were not using the specific attributes that were optimized for the specific custom T-Code.
In this case the problem was not related to the SAP application or the infrastructure, but rather a training issue. It turned out that employees from Mexico, who had not yet received adequate SAP training, did not know how to optimize attributes passed to the reports based on custom T-Codes.
To avoid similar performance problems in the future, Fonterra started to monitor performance of reports per employee (see Figure 4) to determine who else might need additional training to improve their skills in using SAP.
Fonterra required a reliable solution to keep track of their goods entering and leaving warehouses across the world. Therefore it decided to monitor its SAP infrastructure with Data Center Real User Monitoring (DC RUM), part of the Dynatrace APM suite.
As we showed, the Operations team managed to pinpoint the performance problem with certain SAP reports to incomplete training of some employees. This story shows that looking only at the aggregatedResponse Time and network performance metrics alone is not enough to demystify such problems. The team leveraged detailed application performance monitoring at transaction and T-code level, which helped pinpoint the root cause of the issue experienced by specific users in Mexico.
With the 12.2 release of DC RUM, the Operations team can define up to 6 T-Code attributes which will be reported additionally along with performance metrics for SAP transactions. With this additional information Fonterra operations team will be able to more easily find and further improve non-optimally designed SAP reports.
(This blog post is based on materials contributed by Matt Lewis and Krzysztof Ziemianowicz based on original customer story. Some screens presented are customized while delivering the same value as out of the box reports.)