There’s no shortage of guides and blog posts available to provide you with best practices in architecting microservices. While all this information is helpful, what doesn’t seem to be available in such a great number are hands-on guidelines regarding how microservices can be scaled. Following a little research and sifting through lots of theoretical discussion, here is how load-balancing microservices is done in practice by the big players.
Living on the edge
When a web application frontend client communicates with a microservices-based backend server, does the frontend need to know about all the microservice instances that are available to it? For example, does a client really need to be aware of all five services that deliver web page data? The answer is a resounding NO!
Sudhir Tonse, who worked with Netflix previously and now works for Uber, talks about the concept of edge services in his talk on Scalable Microservices at Netflix. An edge service serves as a gateway to a microservices infrastructure. So, in regards to the question of which microservices a frontend client needs to know about, following Sudhir’s approach, each client only communicates directly with just a single edge service. There can be one dedicated edge service per client. For example, Netflix serves more than a thousand device types—and each device type has its own dedicated edge service that serves as its single entry point.
Big players like Netflix and Riot Games, both of which run on Amazon AWS, utilize Elastic Load Balancers (ELB) to ensure that their edge services are available at all times.
Beyond the edge service
Each incoming request is analyzed. Multiple fan-out requests are then issued to the microservices that form the ecosystem. A single inbound request results in an average of about ten fan-out requests. The nearly two-billion requests that Netflix receives each day result in roughly 20 billion internal API calls.
How does Netflix ensure that its microservices can handle such load and remain available 24/7? Again, load-balancing is the solution. But this time, it’s not by means of ELBs. With 500 different microservices, you’d need to configure about 500 ELBs! This is why Netflix’s tools come with built-in load-balancing capabilities. Netflix has created numerous libraries and tools that integrate easily with one another. By integrating the required libraries directly into each microservice, it is able to register itself with all managing services.
Fear not the edge service
With edge services being so important, you’re screwed if your edge service fails, right? Actually, no. For one, edge services absolutely must be load-balanced. This means that your visitors likely won’t even notice edge service outages. And besides, what’s the alternative? With monolithic-application environments, each service is like an edge service, so an outage of any central service—in the absence of a load-balancer—implies total outage, right?
Nonetheless, it’s true that edge services are amongst the most delicate of services and therefore do require special attention.
The takeaway
You should seriously consider running edge services to handle your inbound traffic. And definitely load-balance your edge services with whatever mechanism is provided by your cloud provider. All internal traffic should be handled by your own tools as this allows you to run your environment with minimal configuration overhead. So, ultimately, the most important tool required for effective scaling in microservices is, not surprisingly, load balancing.
An easy way to analyze your load characteristsics
Dynatrace visualizes and analyzes your load, providing all the necessary information for making decisions about load balancing. We offer a free 30-days trial, try it!
Looking for answers?
Start a new discussion or ask for help in our Q&A forum.
Go to forum