What is a message queue? How an observability platform eases message queue monitoring

What is a message queue?

A message queue is a form of middleware used in software development to enable communications between services, programs, and dissimilar components, such as operating systems and communication protocols. A message queue enables the smooth flow of information to make complex systems work.

The concept is like text messaging — a feature most mobile phone users understand. If you need to send a message, you can call the person. However, that assumes he or she is available and has time to talk. For nonurgent messages, texting is a more efficient approach. The message goes into a queue on the recipient’s phone, and he or she responds when possible.

In a distributed processing environment, message queuing is similar, although the speed and volume of messages are much greater. Queued messages are typically small and specific. They may be requests, replies, error messages, or information needed for logging or tracing. A producer creates the message, and a consumer processes it.

Consumers store messages in a queue — usually in a buffer or on a storage medium — until they can process and delete them. A given consumer only processes each message once. Teams can use message queues to store tasks during periods of heavy processing for later attention. Additionally, a message queue can smooth out spiky workloads by enabling the producers and consumers to work at a consistent pace without losing data.

What is the Advanced Message Queuing Protocol?

The Advanced Message Queuing Protocol (AMQP) is an open standard for passing messages between applications and organizations. It provides a consistent platform that integrates with a variety of message queuing types to enable consistent performance regardless of the platform.

Two styles of message queuing

Message queuing has been used for many years, most prominently in email. Without it, sending an email over a long distance would require the immediate availability of every node on the routing network to forward each message. Using queuing, IT teams store messages in intermediate nodes until they can forward them. This enables email message processing in a quick and reliable way, even during periods of heavy network congestion.

A queue manager is the control-plane component of a message queuing system. It specifies such details as which messages receive priority, the necessary types of responses, and where to store and process messages. A complex system may have many message queues, with the queue manager ensuring they all work properly.

There are two basic styles of message queuing:

  • Point-to-point messaging occurs between two discrete entities that share responsibility for a given task, such as a controller on a factory floor and a discrete piece of machinery. The producer application needs to know something about the consumer, such as where to send the message and the expected actions. This approach is common in transaction-intensive environments in which teams perform the same tasks repetitively.
  • Publish-and-subscribe messaging decouples the provider of the information from the consumer. Neither requires information about the other. And messages are sent to all subscribers at the same time — usually without needing a response. RSS feeds are an example of a widely used publish-and-subscribe mechanism. A user visiting a website that supports RSS can cut and paste a URL into a reader or compatible email client. Thereafter, the user is notified of all updates to the site. No action is required by the consumer or expected by the publisher.

How message queuing benefits microservices

Microservices are an increasingly popular way to build software because of their speed and flexibility compared with traditional monolithic approaches. Applications built with microservices are broken into discrete modules so development teams can work on them in parallel.

Services communicate using application programming interfaces, which means there is no need to write them in specific programming languages or frameworks. Developers can easily modify applications by adding or swapping out microservices, and testing requirements are reduced because microservices are isolated and often pre-tested. A failure in one service doesn’t trigger unexpected problems elsewhere in the application.

Message queues coordinate microservices in a distributed application. Because queues are persistent, they reduce the risk of errors if parts of the system go offline. A queue manager’s rules always determine how to process messages. As a result of persistent queues, a system benefits from improved performance, reliability, and scalability. It isn’t surprising that 99% of large organizations either use microservices now or plan to in the future.

Serverless platforms provision microservices as needed and shut them down immediately thereafter, allowing applications to be highly flexible, inexpensive to operate, and customizable. In this scenario, message queues coordinate large numbers of microservices, which operate autonomously without the need to provision virtual machines or allocate hardware resources.

Message queue software options to consider

There are several message queuing software options available for use today, including the following:

  • Apache Kafka is an open source, distributed platform optimized for streaming information.
  • RabbitMQ is a flexible, lightweight, and easy-to-deploy platform, and it supports many messaging protocols.
  • IBM MQ was originally built for high-volume applications and is valued for its messaging reliability.
  • TIBCO Enterprise Message Service features compatibility with software written in Java.

However, this profusion of options can create problems, particularly when two organizations want to exchange messages with each other, but each uses a different queuing solution.

Facing the challenges of message queue observability

For all its virtues, message queuing presents some challenges from an observability perspective. Queuing doesn’t process messages in a well-defined sequence. Therefore, detecting and fixing anomalies can be difficult, and the queue itself can be a black box.

Additionally, it can be challenging to determine why messages are delayed or fail to reach a user. The problem could be in the database, the HTTP connection, the configuration of the message, or an outage on the sending or receiving end. Capturing copies of all messages for forensic purposes consumes system resources and requires time-consuming manual effort.

The importance of an observability platform approach

Observability platforms address the challenge of message queue monitoring by capturing and analyzing queue data. The huge volume of topics entering and exiting message queues, particularly in microservices environments, can be extremely challenging for many traditional observability solutions.

The Dynatrace Software Intelligence Platform supports a broad range of queuing platforms, including Apache KafkaRabbitMQ, and IBM MQ. It automatically discovers and monitors message queues and uses AI to instantly identify issues and their impact on applications and user experience. As a result, the platform scales to meet the requirements of the largest environments and dramatically improves application performance and availability.

To learn more about automatic and intelligent observability, register for our  on-demand webinar today.

The Author

Jay leads product marketing for Dynatrace’s Applications and Microservices and Digital Experience portfolios. He has held a range of marketing roles in both large and small companies and has also worked as a quota carrying sales rep and financial analyst. In his free time, he enjoys running, digital photography and automating his home with open source technologies and custom Python applications.

Looking for answers?


Start a new discussion or ask for help in our Q&A forum.

Go to forum