On November 30th 2015 IBM released IBM Commerce v8.o. It’s been six years since IBM released a major version of IBM Commerce. While there have been many feature and fix packs released over the last 6 years, the idea of deploying a new major version of application software is one that can give the digital business owners and IT roles many restless nights.
Application upgrades can be a nerve-wracking experience for any organization, whether you’re a Continuous Integration/Continuous Delivery shop or, especially, still living in the waterfall world. When that system is responsible for generating the bulk of your revenue, it can be especially stressful. Questions arise: Did the developers incorporate the new features correctly? Did the testers cover all the new scenarios properly? What didn’t we think of? Will that little mysterious blip we saw in the dev/test environment blow up in production? Managing the risk is the name of the game. With the latest IBM Commerce FastPack from Dynatrace, you can manage that risk with confidence. The FastPack comes with several dashboards showing key technical and business metrics – detailing whether the deployment was a success or not:
With our years of software analysis experience and close collaboration with IBM engineering we hope you find these metrics useful. You should also adbide by the following 3 rules when upgrading your application software. In Rule 3 below, I highlight a significant performance degradation due to a security setting that was enabled for our specific environment, which stress the importance of maintaining consistent settings across environments.
Rule #1: Keep it simple.
This seems obvious, but then again, it also seems obvious not to leave your cell phone or loose change visible in your car overnight. Upgrades come with their own feature sets.
IBM Commerce 8 includes new features based around customer support, customer & business user experience and Management Center, in addition to updating the underyling software components like the application server and database. Deploying and testing these new features while also ensuring that existing functionality still operates in parity with WebSphere Commerce 7 (now IBM Commerce) provides more than enough work and risk. But, if you’ve worked in technology long enough, I’m sure you’ve encountered the situation where somebody, from either product or marketing, starts making a lot of noise to get some of their wishlist items added to the release. Don’t do it. This is not the time. It will only add complexity and risk. If a difficult issue arises, you’ll now be trying to figure out if it’s due to something in the upgrade or something from the wishlist.
Rule #2: If it quacks like a duck, make sure it actually still is a duck.
This rule is tempting to ignore, especially if you’re adhering to Rule #1. When deploying an upgrade where new features from the upgrade are involved, the tendency is to concentrate most efforts on thoroughly testing these new features. Regression testing is typically regulated to a few high level metrics, like CPU utilization, response time and placing a checkmark in the box to say it still works. Because it’s nearly impossible to create real world scenarios in a testing environment, it’s important to understand how code execution has changed in a new release and what impact that change may have. Imagine that the Checkout transaction passes a response time performance test and takes about the same amount of time as before the upgrade, but what if there were 3 more database queries added to the checkout function with the upgrade? The increased calls to the database could wreak havoc during a sale. We call this an architectural regression!
By monitoring deltas in key metrics like DB calls, response times, CPU usage of key transactions, exceptions, etc., instead of just thinking everything “feels about right”, you can have a comprehensive understanding of how the application utilizes resources and the confidence that there are no hidden gremlins in the system waiting to be fed after midnight. By using the proper tools to compare tests before/after the upgrade, you can gain this insight without taking extra time in the testing cycle, or worse, finding problems once the new store goes live. For more information on how to track meaningful performance KPI’s through the lifecycle, check out these two blog posts by my colleague Andreas Grabner:
- Key Performance Metrics For Load Tests Beyond Response Time- Part I
- Key Performance Metrics For Load Tests Beyond Response Time- Part II
Rule #3: Understand and address degradations quickly.
I’ll illustrate this rule with the experiment I ran using the WebSphere Commerce 7 (now IBM Commerce) Aurora Store, the WebSphere Commerce 8 Aurora Store, the Dynatrace IBM Commerce FastPack, and Dynatrace Load. The Aurora Store is a template store provided by IBM with IBM Commerce. With Dynatrace AppMon (Application Monitoring) and the IBM Commerce FastPack installed on both Aurora stores, I generated load against both instances via Dynatrace Load. Upon completion, I loaded the two recorded Dynatrace AppMon sessions into the Regression Test Comparison dashboard included in the FastPack. When I first saw the results, I was ready to fail IBM Commerce 8. However, this brings us back to part 1 of Rule #3 – Understand the degradations.
The regression dashboard compares the test sessions from both Aurora instances and shows the differences: red is bad, green is good. Looking at the image above, we see a lot of red. What’s worse is that a major component of the degradation is the IBM Command Bod API. My first thoughts were about how I was going to be a hero to the IBM teams and help them fix a potentially fatal flaw in the new release of IBM Commerce 8. Instead, I found that this was the all too common problem of misconfigured test environments. While trying to understand the problem, I discovered my WebSphere Commerce 7 (now IBM Commerce) Aurora Store had a different WAS security configuration than my IBM Commerce 8 Aurora Store. Putting this in terms of a real world situation, when the new environment was built out, somebody changed a configuration setting that introduced latency. Misconfigured test environments are all too common and often are hard to detect, causing massive delays in releases. Let’s take a look at how I came to understand the problem.
First, I switched the comparison view to look at Page Class performance. This compares page types as identified in the FastPack by the CmdImpl method invoked. TopCategoriesDisplayCmdImpl, the store home page, was impacted the greatest, so I decided to investigate in this reference as the home page is often the most critical page for a commerce site. A slow home page first impression can drive users away from your site.
We can see above an addition of about half a second on average, or a 171% degradation. Diving deeper down into the PurePath, we see the BusinessObjectDocument method running much slower. In the IBM Commerce 8 Aurora store, calls are being made to the IBM NTRegistryImpl Security API. These calls are responsible for the degradation.
Under normal circumstances, as a performance tester, I might put on my superhero cape and declare this new release a failure. However, we’re not finished understanding the problem. Part of what makes an organization great is communication. It’s the foundation of the DevOps culture. Further, DevOps doesn’t just mean the development and operations teams keep an open line of communication. It means all teams keep an open line of communication. So, instead of donning my cape, my next step would be to approach the development team and IBM Commerce 8 (WebSphere) architects with the findings to understand why these security calls are here, which is precisely what I did.
In speaking to my counterparts at IBM, I learned that this is not any new function of IBM Commerce 8. Instead, it’s a security feature that was enabled in the latest version of the Aurora Store specifically for the Softlayer test/development template for technology partners such as Dynatrace, as it provides added security in the partner ecosystem. In short, configurations across my testing environments were not in sync. Administrative Security was off in my AuroraStore 7 environment but on in AuroraStore 8. Part 2 of Rule #3 – Addressing the degradation – is simple: configure security the same across the environments.
Configuration differences between test environments are a common problem. Deployment automation is something that helps quite a lot in this area, but it’s a practice not enough organizations have implemented. In this case, by leveraging Dynatrace Application Monitor, we were able to fully understand the nature of the latency, which led us directly to the cause, which was quickly addressed. By reconciling the security settings, I was able to confirm there were no other degradations originating from the new IBM Commerce 8 package. We avoided war rooms altogether. Most importantly, though, if this were to occur in your upgrade efforts, all the time that would typically be spent chasing down the cause of this issue can be better utilized in testing and deploying all the great new features in IBM Commerce 8.
Take advantage of the Dynatrace 30 Day Free Trial and Dynatrace IBM Commerce FastPack to see a wealth of information including your end-user experience, business transactions, key page, process, and database performance. Be sure to check our other IBM FastPacks like the IBM MQ Channel Monitoring Plugin.