IBM MQ

On this new version of this popular extension, collect performance metrics from your queue managers running locally on your OneAgent host or remotely from your ActiveGates. Capture metadata and new metrics on this version.

Metrics

Queue manager

• Availability: Percentage of time queue manager was reachable and available to this extension. While a queue manager sometimes may be up and running, availability is within the context of this extension.
• Uptime: Time in seconds since last time queue manager started
• Connections: Number of active connections to queue manager
• Queues count: Number of queues being monitored by the extension
• DLQ depth: If DLQ is enabled, number of messages in dead letter queue
• Active channels: Number of channels in an active state
• Log utilization (Distributed only): Percentage estimate of how well the queue manager workload is contained within the primary log space
• Log in use (Distributed only): Percentage of the primary log space in use for restart recovery
• Log archive size (Distributed only): Amount of space occupied, in megabytes, by log extents no longer required for restart or media recovery, but waiting to be archived.
• Log reuse size (Distributed only): Amount of space occupied, in megabytes, by log extents available to be reused.
• Total log data sets (z/OS only): The total number of active log data sets.
• Full log data sets (z/OS only): The total number of full active log data sets that have not yet been archived.

Queues

• Depth: Number of messages in queue
• Depth percentage: Percentage of messages filling up queue
• Inhibit GET: Whether queue inhibits MQGET
• Inhibit PUT: Whether queue inhibits MQPUT
• Open input handles: Number of handles opened by applications for input
• Open output handles: Number of handles opened by applications for output
• Uncommitted messages: Number of uncommitted changes (puts and gets) pending for the queue

In order to retrieve the following metrics, queue monitoring must be enabled on queues. Any monitoring resolution will start collecting these statistics.

• Oldest message: Age in seconds of the oldest message on the queue.
• Last GET: Time at which the last message was successfully read from the queue.
• Last PUT: Time at which the last message was successfully put in queue.
• Short time indicator: Amount of time that a message spent on the queue over a short period.
• Long time indicator: Amount of time that a message spent on the queue over a long period.

In order to retrieve the following metrics, statistics on queues are reset. To reset them, CHG permission must be granted to queues

• Enqueue count: Number of messages enqueued. This count includes messages that have been put to the queue, but have not yet been committed.
• Dequeue count: Number of messages dequeued. This count includes messages that have been successfully retrieved (with a nonbrowse MQGET) from the queue, even though the MQGET has not yet been committed.
• Enqueue rate: Number of messages enqueued per second.
• Dequeue rate: Number of messages dequeued per second.

Channels:

• Messages: Number of messages sent or received, or number of MQI calls handled.
• Bytes sent: Number of bytes sent
• Bytes received: Number of bytes received
• Buffers sent: Number of buffers sent.
• Buffers received: Number of buffers received.
• Last message sent: Time when last message was sent, or MQI call was handled
• In-doubt: Whether the channel is currently in doubt. 0 = No. 1 = Yes
• In-doubt messages: Number of messages in-doubt
• Current shared conversations: Number of conversations currently active on this channel instance. This parameter is returned only for TCP/IP server-connection channels.
• Channel instances: Number of instances of this channel. This metric is only available to server-connection channels.
• Batches: Number of completed batches
• Network time short: Amount of time, in microseconds, to send an end of batch request to the remote end of the channel and receive a response over a short period.
• Network time long: Amount of time, in microseconds, to send an end of batch request to the remote end of the channel and receive a response over a longer period.
• XMIT queue time short: Time, in microseconds, that messages remained on the transmission queue before being retrieved over a short period.
• XMIT queue time long: Time, in microseconds, that messages remained on the transmission queue before being retrieved over a longer period.
• Channel status: Used to round up channel status counts for alerting and charting purposes
• Long retries left: Number of long retry attempts remaining
• Short retries left: Number of short retry attempts remaining

Listeners

• Availability: Percentage this listener was running and available.

Topics

• Publish count: The number of applications currently publishing to the topic.
• Subcription count: The number of subscribers for this topic string, including durable subscribers who are not currently connected.
• Inhibit publish: Whether publishes are not allowed
• Inhibit subscription: Whether subscriptions are not allowed

Usage (z/OS only)

• Total pages: The total number of 4 KB pages in the page set.
• Expanded count: The number of times the page set has been dynamically expanded since restart.
• Unused pages: The number of pages that are not used (that is, available page sets).
• Persistent data pages: The number of pages holding persistent data.
• Non-persistent data pages: The number of pages holding nonpersistent data.
• Free pool buffers: Number of free buffers in buffer pool.
• Percent free pool buffers: Number of free buffers as a percentage of all buffers in the buffer pool.
• Total pool buffers: The number of buffers defined for specified buffer pool.

Requirements:

OneAgent deployment: (For ActiveGate deployment, continue below)

• OneAgent for Windows or Linux only.

• OneAgent must be running and monitoring queue manager processes.

• Dynatrace version 1.269+

• IBM MQ 9.1+. IBM dropped support to 9.0.x in 2021.

• IBM MQ Command server must be running on queue managers

• MQ Libraries must be in PATH on Windows or LD_LIBRARY_PATH on Linux. User running extension must be able to find these libraries.

  Additional requirements for Linux systems:

• User running the OneAgent (dtuser by default) must also have read access to the queue manager directories /var/mqm/qmgrs/<queue_manager> and read/write to log directories /var/mqm/log so that it can bind properly.

• Ensure MQ libraries are accessible by dtuser or whichever user your OneAgent service is running under. Default location for these libraries is /opt/mqm/lib64

• By default, IBM MQ installs an ldconfig file called /etc/ld.so.conf.d/mqm.conf that adds /usr/lib64 to the LD_LIBRARY_PATH. Make sure IBM MQ library symbolic links to /opt/mqm/lib64 are found in that path. If there is no reference in /usr/lib64 to /opt/mqm/lib64 libraries, you may have to create your own .conf file and place it in /etc/ld.so.conf.d/.

• Using ldconfig will allow the MQ libraries to be loaded by default. To do this you can do the following:

  • Create a .conf file in /etc/ld.so.conf.d/ with any name. Example: ibmmq_ext.conf
  • Inside that file, enter the location for the IBM MQ libraries: /opt/mqm/lib64
  • Save the file
  • Reload ldconfig by running command: ldconfig

ActiveGate deployment:

• ActiveGates on Windows or Linux
• Dynatrace version 1.269+
• IBM MQ 9.1+. IBM dropped support to 9.0.x since 2021.
• IBM MQ Command server must be running on queue managers.
• IBM MQ running on AIX, z/OS, Linux, Windows, zLinux, MQ Appliance are supported. While it may work on other OS, they may not be officially supported.
• Server-connection channel to communicate with each queue manager. This is require for any Client connection to IBM MQ.
• Firewall from ActiveGates to queue managers must be open. Queue manager could run on any port so ensure those ports are open from your ActiveGates. Consult with your MQ administrator to know those ports.

Authentication:

IBM MQ has several ways to authenticate client and local connections. This is determined by the administrator of your MQ environment according to their own security policies.

User authentication depends on Channel Authentication (CHLAUTH) configuration. If disabled or ADOPTCTX is disabled, IBM MQ will attempt to authenticate the user running the extension process, not the user passed in the endpoint configuration.

When ADOPTCTX is enabled and CHLAUTH is enabled, the user and password combination entered in the endpoint is passed in a MQCSP block that IBM MQ will attempt to authenticate.

• Please take this into consideration when authenticating users or if you receive unauthorized errors.

User permissions to MQ objects (Distributed)

• CHG permission is only required if collecting Enqueue/Dequeue metrics. Enqueue/Dequeue metrics can only be obtained after issuing RESET_Q_STATISTICS command, therefore needing CHG permissions. If you cannot provide this permission, make sure you deselect those metrics on Feature Sets page of your monitoring configuration.

User permissions to MQ objects (z/OS)

Note:

• CONTROL permission is only required if collecting Enqueue/Dequeue metrics.

Dynatrace configuration

Under Extensions on the left menu, select IBM MQ.

• Add a monitoring configuration.

OneAgent deployment, also referred as "Local", can run on a specific host, host groups, hosts in a management zone or tagged hosts. ActiveGate deployment, also referred as "Remote", runs on ActiveGate groups. Groups can one or multiple ActiveGates for more reliability.

For Local (OneAgent) deployment:

• Select which hosts will run this extension

  • Log level: Enable DEBUG only when troubleshooting or support makes that recommendation.

  • Add a Queue Manager: Any queue managers matching its filter will use this configuration. You may create one separate configuration per queue manager on host, or one configuration for all queue managers on that host.

  • Queue manager filter: A name matching this filter will be matched against running processes on this host. If a name matches this filter, all settings on this configuration endpoint will apply to it.

  • Use credential vault: This will allow you to select a username/password credential stored on Dynatrace's Credential Vault. Using this feature will ignore any username/password entered manually and instead will only use the credential stored in the vault.

  • User: Optional but recommended to add a user. Otherwise, if left blank, IBM MQ will try to authenticate user running process, dtuser by default.

  • Password: Password to user above.

  • Objects to monitor: Two methods of adding objects.

    • Using new wizard mode, just click buttons to enter rules. If there is an object you absolutely do not want to monitor then remove that rule.
    • Advanced mode, add filters in a comma-separated fashion under each tab. Hover your mouse over the (i) icon next to each object type for rules and examples.
    • Note: Rules are evaluated in sequential order. This way you can include or exclude objects from the previous rule result. For example: *, -SYSTEM.*, SYSTEM.CLUSTER* would include all queues, then remove those that start with SYSTEM. then include those that start with SYSTEM.CLUSTER. Pay close attention to the order you enter your rules, whether in Wizard or Advanced mode.

  • Send log event on Dead letter queue: Enable this if you would like any DLQ on any queue manager matching this configuration to trigger a log event indicating that the number of messages in DLQ exceeded threshold. You may create an alert based on that log event

  • Send log event on high queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth threshold is exceeded. This threshold is configured on IBM MQ.

  • Send log event on low queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth lower than threshold. This threshold is configured on IBM MQ.

  • Retrieve topology for improved transaction tracing: Enabling this setting will collect mappings for alias queues, remote queues, and cluster queues. This mapping is then sent to Dynatrace to complement mapping information for improved transaction tracing. You may only need to collect this once unless your mappings on IBM MQ change frequently. When enabled, enter a Dynatrace token to communicate with the API with these permission: Read settings, Write settings.

  • Rename Queue managers: Enabling this setting will allow you to add an alias to queue managers matching your queue manager filter (above). The real queue manager name will still be included in the name. This is only to differentiate queue managers with similar or same names on Dynatrace. This will not rename any queue manager on IBM MQ.

  • Inquire metrics in bulk: Enabling this setting will reduce the number of requests to IBM MQ and collect metrics for queues and channels in less requests thus reducing time and potential CPU overhead. However, this setting will require granting permissions to all queues and all channels, even SYSTEM objects too. This setting is not suitable for large environments with a large amount of queues or channels.

  • Lastly, select which feature sets (metric groups) you would like this configuration to collect. Selecting too few will not give you the proper visibility and you may miss out on important performance data, as well as key indicators that can be correlated between different objects.

For Remote (ActiveGate) deployment:

• Select which ActiveGate group will run this extension

  • Log level: Enable DEBUG only when troubleshooting or support makes that recommendation.

  • Task bucket size: By default, all endpoints defined in a monitoring configuration create a task. Use this setting to create separate tasks for groups of endpoints defined in a monitoring configuration. This will allow you to optimize resource usage on your ActiveGate and potentially distribute execution across other ActiveGates in your group.

  • Add a Queue Manager: Click on this button for each queue manager you want to connect. Multiple queue managers per ActiveGate group is recommended.

  • Comma-separated hosts: Most queue managers run on a single host but there are times you have multi-instance queue managers that run across multiple hosts. Enter that single host and port here, or the comma-separated list of host and ports for that one queue manager.

  • Queue manager name: Enter the name of the single queue manager you are connecting to. This is required.

  • Server-connection channel: Any client application, such as this extension, that connects to IBM MQ must talk to the queue manager via a channel. Enter the name of that Server-connection channel here.

  • Use credential vault: This will allow you to select a username/password credential stored on Dynatrace's Credential Vault. Using this feature will ignore any username/password entered manually and instead will only use the credential stored in the vault.

  • Username: Optional but recommended to add a user. A user is always passed, but depending on the queue manager configuration on IBM MQ, it can be ignored or this user in this field can be authenticated. It is best to always enter a user.

  • Password: Password to user above.

  • Model queue: Optional. If not set, it will use the SYSTEM default model queue for command requests and responses.

  • Reply prefix: If you entered a model queue, you must enter a prefix for the reply queue.

  • Objects to monitor: Two methods of adding objects.

    • Using new wizard mode, just click buttons to enter rules. If there is an object you absolutely do not want to monitor then remove that rule.
    • Advanced mode, add filters in a comma-separated fashion under each tab. Hover your mouse over the (i) icon next to each object type for rules and examples.
    • Note: Rules are evaluated in sequential order. This way you can include or exclude objects from the previous rule result. For example: *, -SYSTEM.*, SYSTEM.CLUSTER* would include all queues, then remove those that start with SYSTEM. then include those that start with SYSTEM.CLUSTER. Pay close attention to the order you enter your rules, whether in Wizard or Advanced mode.

  • Send log event on Dead letter queue: Enable this if you would like any DLQ on any queue manager matching this configuration to trigger a log event indicating that the number of messages in DLQ exceeded threshold. You may create an alert based on that log event

  • Send log event on high queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth threshold is exceeded. This threshold is configured on IBM MQ.

  • Send log event on low queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth lower than threshold. This threshold is configured on IBM MQ.

  • Retrieve topology for improved transaction tracing: Enabling this setting will collect mappings for alias queues, remote queues, and cluster queues. This mapping is then sent to Dynatrace to complement mapping information for improved transaction tracing. You may only need to collect this once unless your mappings on IBM MQ change frequently. When enabled, enter a Dynatrace token to communicate with the API with these permission: Read settings, Write settings.

  • Rename Queue manager: Enabling this setting will allow you to add an alias to the queue manager defined above. The real queue manager name will still be included in the name. This is only to differentiate queue managers with similar or same names on Dynatrace. This will not rename any queue manager on IBM MQ.

• Inquire metrics in bulk: Enabling this setting will reduce the number of requests to IBM MQ and collect metrics for queues and channels in less requests thus reducing time and potential CPU overhead. However, this setting will require granting permissions to all queues and all channels, even SYSTEM objects too. This setting is not suitable for large environments with a large amount of queues or channels.
• Disable auto-alerting: Automatic problem creation to failures will be disabled. Errors will still be logged as events but will not trigger problems. If you wish to create problems then you would have to create rules to trigger them from log events.
• Lastly, select which feature sets (metric groups) you would like this configuration to collect. Selecting too few will not give you the proper visibility and you may miss out on important performance data, as well as key indicators that can be correlated between different objects.

• How many DDUs will it consume per year?

The following is an approximation. Metrics change depending on type of objects. Some metrics are not available to all channels, some metrics are not available to all queues.

Assuming you get all metrics for all objects: ((10) + (Local queues * 16) + (Alias + Remote queues) * 2 + (Channels * 18) + (Listeners) + (Topics * 4)) * Queue managers * 525.6 = DDUs per year

For example:

1      Queue manager
400    Local queues
20     Alias and remote queues
50     Channels
1      Listener
50     Topics

( 10 + (400 * 16) + (20 * 2) + (50 * 18) + (1 * 1) + (50 * 4) ) * 1 * 525.6 =
(10 + 6400 + 40 + 900 + 1 + 200)  * 525.6 = approximately 3.9M DDUs per year

• I am not seeing all queue metrics

Some metrics depend on the type of queues. Other metrics depend on your queue manager configuration and if your user has enough permissions to collect them.

• Are you missing Enqueue and Dequeue? Make sure your user has CHG permission on queues to reset statistics on queues. Only then you can receive Enqueue and Dequeue metrics.

• Are you missing Oldest message and Last get/put? Make sure Monitoring statistics is enabled on your queue manager for all queues.

• Are you looking at Alias or Remote queues? Only Locally defined queues have status metrics. Alias and Remote only have Inhibit GET/PUT metrics.

• I keep getting Unauthorized errors but I know my user is correct

Check the AMQERR01.LOG on the queue manager itself. This log file will tell you what it tried to authenticate and what permissions you are missing. If you are unable to connect to your queue manager, make sure that either the user running the extension process has permissions to the queue manager, or ADOPTCTX and CHLAUTH is enabled so that it can authenticate the user and password combination passed in the MQCSP block.

• I get a pymqe error saying libmqm_r.so not found

This means that the IBM MQ libraries were not found in the unix PATH. Make sure you follow the ldconfig step above under Local deployment

• Other common errors and their meanings

MQRC_HOST_NOT_AVAILABLE: Queue manager is not reachable from your ActiveGate. This usually means a firewall rule is missing. Because a queue manager can run on a custom port, you must make sure that host:port is reachable from the ActiveGate running the extension.

MQRC_NO_MSG_AVAILABLE: It took too long to get a response back from your queue manager and the response was blank. This can happen if the MAX LENGTH size of your reply queue is not enough to fit the response payload. You can try increasing that value. It may also be caused because of a timeout or the queue manager did not respond in time.

MQRC_CONNECTION_BROKEN: The connection to the queue manager is severed either due to a network interruption or the queue manager closed it. Make sure your network performance is optimal. Also, if you are using SSL, make sure that you are not using an older TLS cipher that requires manual reset of TLS keys such as TLS 1.2 AES-GCM ciphers. Switch to a TLS 1.3 Cipher that has automatic TLS keys reset.