Through my Share Your PurePath program I can confirm that many software companies are moving towards a more service-oriented approach. Whether you just call them services – or Micro-services doesn’t really matter. If you want to get a quick overview of the concepts I really encourage you to read Benjamin Wootton’s blog and the comments on “Microservices – Not A Free Lunch!”
In the data that has been shared with me over the last couple of months I see the same trend, but also some interesting approaches to convert monolithic applications into a service-oriented architecture where services are actually not hosted in an external container but are artificially created within a container. The most recent example I’ve seen confirms what Benjamin writes in his blog: Microservices add a new dimension of complexity, and we repeatedly see familiar architectural problem patterns (1+N Query) having huge scalability and performance impacts when applied to service calls!
The following screenshot is what I’ve sent back to the Dynatrace Free Trial user that shared his PurePaths with me, including some bullet points on the problems I could identify in their re-architectured application:
I keep teaching my Free Trial Users how to read that Transaction Flow and the underlying PurePaths to identify problematic architectural implementations like the one shown above. The overall summary that I gave for the 3 PurePaths they shared with me was:
Highly Complex Multi-Threaded Architecture
- Making 20 Internal (within the same JVM) Web Service Calls for Login in the same user
- Executing a Total of 40! Internal Web Service calls against 3 service interfaces
- Executing 25k! SQL Calls in Total on 34 different Database Connections!
- Binding a lot of threads due to combination of sync & async executions
- Lots of Database Exceptions
If you want me to do a quick check on your application feel free to Share Your PurePath with me and I’ll be happy to send you a free architectural and performance review. But you should also understand the problem patterns I watch out for and how you can do the same.
Problem Pattern: Expensive Multi-Threading
Services are often called asynchronously – which makes a lot of sense considering that they should be independent from each other working on a specific task. Applications that have not been built from ground-up with this concept in mind, but rather being “migrated” from monolithic to service-oriented, show several strange tendencies:
- Excessive parallel execution of code in background threads
- Synchronization of all threads to “maintain transactional consistency”
The following screenshot shows the PurePath for the Transaction Flow I showed earlier. I always follow the flow and try to work out how these services are invoked by focusing on the involved threads (Thread Column), whether services and threads are called synchronously or asynchronously (Link Node), and where time is spent in synchronization and wait (Breakdown Column):
Key Metrics: # of Threads, Time Spent in CPU, Sync and Wait
If you analyze your PurePaths (or whatever other technology you use to analyze your application and service flows) I suggest to always look at the
- Total # of Threads involved to execute a task
- Execution Time of a Thread
- Time spent in Synchronization and Waiting for Threads
Looking at these metrics allows you to identify architectural issues without going through every single detail of your execution path. They also allow you to plan proper sizing of your Thread Pools.
Problem Pattern: Duplicating Resource Access
In his blog Benjamin had a section called “Duplication of Efforts” where he walks through different scenarios of providing a core functionality to multiple consumers. Either offer it as a service that everyone can call, duplicate code, or a shared library. In the example of this blog we can see how the same functionality is called from multiple consumers. The effect of this however resulted in 20 different calls to the same login service with the same username where each service itself had to query the database to validate that user account.
Let’s have a look at the same PurePath as before but now focus on how many service calls we actually have and how often we access shared resources such as a database.
Dynatrace also provides specialized views that give you this information in a single table. Following screenshot shows the Database and Web Service Dashlet that shows a nice overview and confirms the bad architectural pattern:
Key Metrics: # of “same” Service Calls, # of “same” SQL Queries
If you analyze the requests I suggest, you get to answer the following questions;
- Total # of SQL Queries executed to finish that task
- Do we execute the same SQL Query more than once?
- How many Service Calls to fulfill a task? Any duplicates?
Looking at these metrics allows you to identify classical N+1 Query Problem Pattern which can now also be applied to service calls and not just database roundtrips. Especially when extracting functionality into a service or “reusing” code it is possible that you duplicate efforts. Make sure you do it in a smart way without huge impact on your infrastructure or your data storage.
Have you checked your Service-Oriented-Architecture?
I hope this gave you some ideas on how to validate your architecture. Whether you use Dynatrace or other tools is up to you. The benefit I see in Dynatrace is that through its Native Agent approach (see Technical Pros/Cons Blog from Chief Architect Christian Schwarzbauer) it can trace every transaction across thread and runtime boundaries and can capture the impact of Wait, Sync, I/O and even Garbage Collection Time. This is a unique but from my perspective mandatory capability.
If you want to raise this to the next level you can integrate this type of metric collection into your Continuous Integration. This allows you to identify problems early on by baselining these metrics across builds and test runs, and lets them act as a quality gate to stop builds that just introduced an architectural regression. Find more on that in my blog series around Key Metrics for Continuous Delivery.
As a reminder: Test this out on your own using Dynatrace Free Trial and feel free to Share Your PurePaths. Dynatrace Free Trial is also yours for free forever after 30 days as we allow you to use it on your workstation with our Personal License offering.