We all know that slow performance – and service disruption even more – affects our business services and eventually our revenue. At the same time we say that major parts of companies are not willing to invest in performance. In this post I will discuss why we find ourselves in this paradox and how to escape it.
Applications fail and management does not care
dynaTrace recently conducted a study on performance management in large and small companies. The quick facts paint a horrible picture. 6o percent of the companies admit that they do not have any performance management processes installed or what they have is ineffective. Half of the companies who answered that they have performance management processes admitted that they are doing it only in a reactive way when problems occur. One third of all companies said that management is not supporting performance management properly. From this data we can obviously conclude that performance management is not a primary interest in most companies.
At the same time companies admitted that performance problems are impacting their business. and 10 percent responded that they spend up to half of their development time troubleshooting problems. Most companies also said that they find about 50 percent of application problems in production – when those problems have impacted end users and are most expensive to resolve.
While this is already scary enough, two thirds of all respondents said that they are convinced that shorter release cycles, more complex architectures and other factors will make the situation even worse.
Does this all make sense?
When I was reading the study the first time I was really wondering why most companies are in this strange situation. They know that they have serious problems but they don’t do anything about it. However this first impression is wrong. There are people who care about the problems and want to do something. These are the technical guys who work longer hours to keep everything running. The problem – as the study also showed – is that management is not supporting more efficient performance management activities.
In some cases these problems are not visible to management and in others they expect that their technical staff will fix the problems. They do not necessarily see the need for major changes or investments.
The big problem here is communication. Technicians and business people think differently. This causes a lot of misunderstandings. The time when they seem to agree the most is when companies have a production problem. However their agreement is mostly a coincidence.
Let’s assume we have a production outage. Both management and technical staff will do everything to get systems up and running again. So both are interested in stable software – right? Actually the answer is no. Management is interested avoiding the business disruption caused by not being able to sell online or provide your online service. They see the revenue and reputation loss they have due to this service outage and marketing numbers can easily convert this into money. At the same time the technical guys want to fix that memory leak that keeps the servers constantly crashing. Development teams are analyzing runtime data and dumps to find a solution. So management is thinking about lost revenue while IT talks about references that are not cleared leading to memory leaks.
So management and IT are definitely speaking two different languages. This is the reason why most performance management projects initiated by IT do not have success. IT is requesting resources and money for problems management does not understand. Let’s look at the following fictional conversation between management and IT
IT: We need to invest in xyz to reduce memory footprint and data transfer volume.
Manager: How much does this cost?
IT: It costs a gazillion of dollars and one employee doing the work.
Manager: That is expensive
IT: Yes ….it is.
Manager: So what’s the benefit?
IT. Our application will get faster and better (Didn’t he listen to me?)
Manager: OK (Didn’t he understand my question?)
So our users have problems?
IT: No, we keep the system running, but …
Manager: So everything is fine – right?
IT: Well, yes but …
Manager: Ok, thanks.
Perhaps this is a bit overstated, but this conversation makes the point clear. IT had a real issue to discuss but they weren’t able to communicate it to management. The manager had no reason to support their request. For those of you who think they never understand management here is the two-liner on management thinking
Management has to ensure company success. Success means profit. In order to create profit you have to earn more money than you spend. Management is steering the company in this direction.
So here is how this conversation should have gone:
IT: We have a serious issue with our eCommerce site. User searches are taking longer and longer due to some complex application problems. We already see searches and buying transactions declining. Development has already shifted resources to fix that problem – which however delays the development of the secret cool new feature.
Manager: I see … (Marketing also told him about the decrease in sales. Secret cool feature must be out by the end of next month. However, if we cannot process sales then the feature does not bring any more revenue)
Manager: (This means we are losing four gazillions of money)
IT: The problem is we do not have the means to resolve the problem immediately. We have to try …
Manager: I don’t want you to try. I want you to fix. What is your proposal?
IT: Well, if we invest in proper tooling, we will get all information we need in a day and in one week we will be able to go live with a fixed version.
Manager: How much?
IT: A gazillion dollars
Manager (That’s a fourth of the projected loss)
Manager: OK and keep me in the loop
So our IT guy this time made it. He got the resources he needed. He also has management support for his activities. Why? Because he was speaking to management in a way that they could understand what he wants. Management got the information they needed to take a decision.
Some of you might now think this example is fictional. Honestly it is not. I have been working with many Fortune 500 companies over the years on performance management practices. The short story is, those who were able to relate performance management to business and make it part of their IT strategy were successful. Those who did not get to that point are still living in firefighting mode. They might have invested in tools, but they never got the ROI they could get.
After the problem
The situation discussed above was caused by a problem. Very often things revert to how they were before the problem. The challenge is now to get management support for becoming more proactive. The problem with pro-activeness is that you avoid problems which never then exist. That makes it tough to make a business case. Here are the top mistakes people make in becoming more proactive:
- Tell management they need more money.
- Tell what they could do then.
- Tell which additional things people can do.
- Talk about the future and forget about the now.
- Throw technical terms and features at management until they get headache.
- Try to convince when they should provide decision support
- They have no plan
- Use a lot of “could would should” sentences.
So what happens is that you tell management that you need money for technical gimmicks to do additional work – you and others. You are talking about a future situation which they might a) think they have already reached or b) is the ideal you never reach under real world constraints. Finally your goals look so ambitious that management is not convinced that you will ever achieve them. Even if they still want to support you they don’t have any metrics to take a decision. In particular they do not understand the business benefits. And your “would should” sentences were not creating a lot of trust either. At the end you walk out of the room thinking that management never gets the point.
How to get the point across
The important point is that management is willing to support you if you bring value to the business. It is up to you to provide meaningful data – all facts and figures that matter to the business – as well as a plan they can use as a basis for decisions. Very often talking to management is seen as the first step while it actually is the last step in the process. So you better do your homework first.
Here is how we at dynaTrace work with customers to get the optimal performance management solution for them. Do not forget it is about what they need – not what you want them to have.
First I start with an analysis of the current situation. I need to understand what their current problems are and what their impact is on the business – never communicate a technical problem without knowing the business impact. Then I look at the overall situation. This includes responsibilities, processes, efforts etc. When I have collected all that data I condense it into a single-page SWOT (strengths, weaknesses, opportunities, threats) matrix. This is my basis for talking to management.
Then I develop a plan how to cover weaknesses and threats. I assign importance and immediacy to all action points and the achieved benefits. This is my basis for talking to management.
In the actual discussion I can then first tell them what the current situation looks like and what risks are involved. Then I discuss with management how they see the risks and then define the measures to cover them.
So actually I spend quite some time before talking to management to develop a concise plan how my proposed improvements will support the company’s business. While this does not always ensure that you are successful, it at least means that you have done the best you can do.
How to get business level information
The tricky part is how to get business-relevant information. Honestly your system and application health monitoring tools won’t do the job. They are not designed to provide these metrics. They give you the technical details how to fix problems – not the meaningful data business needs.
What you really need is a means to provide higher-level information. In its simplest form, you can make sense out of a URL. However – especially in more complex systems – you need more information about the actual user transactions which are collected at code level. At dynaTrace we refer to this activity as Business Transaction Management (BTM). BTM is different from Business Activity Management in that it has a clear focus on optimizing applications for business purposes while BAM systems monitor their performance.
BTM relates business metrics like customer information, trading volume of a transaction etc. to performance metrics like load or response times. Having these metrics you can run analysis like how response time affect searches or trading volume or what is the abortion threshold for certain transactions. BTM however goes further as it also provides resolution information like how to make certain transactions faster by changing code. This information can then be taken as a basis for decision – like “is the likely increase in searches worth the 200k development effort?”
As I said these metrics are often hard to get with conventional monitoring. Some performance management solutions can provide them. However at this point you might not have one, so you likely will have to do it the hard way by collecting all the information manually and making sense out of it. This might be a huge effort – but remember you want management to invest a lot of money.
What management and IT want is not as different as it often seems. However their level of abstraction is totally different; and so is the data. Management needs meaningful data from transactions on revenue, the experience of end users, or the performance of key transactions in order to understand how sub-optimal performance impacts the business. IT needs actionable data at code-level from individual transactions that enables them to quickly isolate the issue causing the sub-optimal performance. Only with such data in both sets of hands, will you be able to get management to begin to consider application performance a business priority.
If you are interested in learning more about Business Transaction Management visit the dynaTrace website and also learn how it helped IntraLinks to optimize their business.
If you want to read more articles like this visit dynaTrace 2010 Application Performance Almanac