IBM MQ

With the Extensions Framework 2.0 release of the IBM MQ extension, collect performance metrics from your queue managers running locally on your OneAgent host or remotely from your ActiveGates.

New metadata and metrics are included in this version.

• Monitor IBM MQ infrastructure, including Queue Managers, Queues, Channels, Topics, and listeners.
• Get alerted if a Queue Manager recently has started, is not available, has retrying channels, and more.
• Monitor your IBM MQ by deploying locally on a OneAgent monitored-host, or remotely using an ActiveGate.

Requirements:

OneAgent deployment: (For ActiveGate deployment, continue below)

• OneAgent for Windows or Linux only.

• OneAgent must be running and monitoring queue manager processes.

• Dynatrace version 1.269+

• IBM MQ 9.1+. IBM dropped support to 9.0.x in 2021.

• IBM MQ Command server must be running on queue managers

• MQ Libraries must be in PATH on Windows or LD_LIBRARY_PATH on Linux. User running extension must be able to find these libraries.

  Additional requirements for Linux systems:

• User running the OneAgent (dtuser by default) must also have read access to the queue manager directories /var/mqm/qmgrs/<queue_manager> and read/write to log directories /var/mqm/log so that it can bind properly.

• Ensure MQ libraries are accessible by dtuser or whichever user your OneAgent service is running under. Default location for these libraries is /opt/mqm/lib64

• By default, IBM MQ installs an ldconfig file called /etc/ld.so.conf.d/mqm.conf that adds /usr/lib64 to the LD_LIBRARY_PATH. Make sure IBM MQ library symbolic links to /opt/mqm/lib64 are found in that path. If there is no reference in /usr/lib64 to /opt/mqm/lib64 libraries, you may have to create your own .conf file and place it in /etc/ld.so.conf.d/.

• Using ldconfig will allow the MQ libraries to be loaded by default. To do this you can do the following:

  • Create a .conf file in /etc/ld.so.conf.d/ with any name. Example: ibmmq_ext.conf
  • Inside that file, enter the location for the IBM MQ libraries: /opt/mqm/lib64
  • Save the file
  • Reload ldconfig by running command: ldconfig

ActiveGate deployment:

• ActiveGates on Windows or Linux
• Dynatrace version 1.269+
• IBM MQ 9.1+. IBM dropped support to 9.0.x since 2021.
• IBM MQ Command server must be running on queue managers.
• IBM MQ running on AIX, z/OS, Linux, Windows, zLinux, MQ Appliance are supported. While it may work on other OS, they may not be officially supported.
• Server-connection channel to communicate with each queue manager. This is require for any Client connection to IBM MQ.
• Firewall from ActiveGates to queue managers must be open. Queue manager could run on any port so ensure those ports are open from your ActiveGates. Consult with your MQ administrator to know those ports.

Authentication:

IBM MQ has several ways to authenticate client and local connections. This is determined by the administrator of your MQ environment according to their own security policies.

User authentication depends on Channel Authentication (CHLAUTH) configuration. If disabled or ADOPTCTX is disabled, IBM MQ will attempt to authenticate the user running the extension process, not the user passed in the endpoint configuration.

When ADOPTCTX is enabled and CHLAUTH is enabled, the user and password combination entered in the endpoint is passed in a MQCSP block that IBM MQ will attempt to authenticate.

• Please take this into consideration when authenticating users or if you receive unauthorized errors.

User permissions to MQ objects (Distributed)

• CHG permission is only required if collecting Enqueue/Dequeue metrics. Enqueue/Dequeue metrics can only be obtained after issuing RESET_Q_STATISTICS command, therefore needing CHG permissions. If you cannot provide this permission, make sure you deselect those metrics on Feature Sets page of your monitoring configuration.

User permissions to MQ objects (z/OS)

Note: CONTROL permission is only required if collecting Enqueue/Dequeue metrics.

Collecting Enqueue/Dequeue metrics:

In order to retrieve the following metrics, statistics on queues are reset. To reset them, CHG permission must be granted to queues

• Enqueue count: Number of messages enqueued. This count includes messages that have been put to the queue, but have not yet been committed.
• Dequeue count: Number of messages dequeued. This count includes messages that have been successfully retrieved (with a nonbrowse MQGET) from the queue, even though the MQGET has not yet been committed.
• Enqueue rate: Number of messages enqueued per second.
• Dequeue rate: Number of messages dequeued per second.

Collecting Queue Status metrics:

In order to retrieve the following metrics, queue monitoring must be enabled on queues. Any monitoring resolution will start collecting these statistics.

• Oldest message: Age in seconds of the oldest message on the queue.
• Last GET: Time at which the last message was successfully read from the queue.
• Last PUT: Time at which the last message was successfully put in queue.
• Short time indicator: Amount of time that a message spent on the queue over a short period.
• Long time indicator: Amount of time that a message spent on the queue over a long period.

Dynatrace configuration

Under Extensions on the left menu, select IBM MQ.

• Add a monitoring configuration.

OneAgent deployment, also referred as "Local", can run on a specific host, host groups, hosts in a management zone or tagged hosts. ActiveGate deployment, also referred as "Remote", runs on ActiveGate groups. Groups can one or multiple ActiveGates for more reliability.

For Local (OneAgent) deployment:

• Select which hosts will run this extension

  • Log level: Enable DEBUG only when troubleshooting or support makes that recommendation.

  • Add a Queue Manager: Any queue managers matching its filter will use this configuration. You may create one separate configuration per queue manager on host, or one configuration for all queue managers on that host.

  • Queue manager filter: A name matching this filter will be matched against running processes on this host. If a name matches this filter, all settings on this configuration endpoint will apply to it.

  • Use credential vault: This will allow you to select a username/password credential stored on Dynatrace's Credential Vault. Using this feature will ignore any username/password entered manually and instead will only use the credential stored in the vault.

  • User: Optional but recommended to add a user. Otherwise, if left blank, IBM MQ will try to authenticate user running process, dtuser by default.

  • Password: Password to user above.

  • Objects to monitor: Two methods of adding objects.

    • Using new wizard mode, just click buttons to enter rules. If there is an object you absolutely do not want to monitor then remove that rule.
    • Advanced mode, add filters in a comma-separated fashion under each tab. Hover your mouse over the (i) icon next to each object type for rules and examples.
    • Note: Rules are evaluated in sequential order. This way you can include or exclude objects from the previous rule result. For example: *, -SYSTEM.*, SYSTEM.CLUSTER* would include all queues, then remove those that start with SYSTEM. then include those that start with SYSTEM.CLUSTER. Pay close attention to the order you enter your rules, whether in Wizard or Advanced mode.

  • Send log event on Dead letter queue: Enable this if you would like any DLQ on any queue manager matching this configuration to trigger a log event indicating that the number of messages in DLQ exceeded threshold. You may create an alert based on that log event

  • Send log event on high queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth threshold is exceeded. This threshold is configured on IBM MQ.

  • Send log event on low queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth lower than threshold. This threshold is configured on IBM MQ.

  • Retrieve topology for improved transaction tracing: Enabling this setting will collect mappings for alias queues, remote queues, and cluster queues. This mapping is then sent to Dynatrace to complement mapping information for improved transaction tracing. You may only need to collect this once unless your mappings on IBM MQ change frequently. When enabled, enter a Dynatrace token to communicate with the API with these permission: Read settings, Write settings.

  • Rename Queue managers: Enabling this setting will allow you to add an alias to queue managers matching your queue manager filter (above). The real queue manager name will still be included in the name. This is only to differentiate queue managers with similar or same names on Dynatrace. This will not rename any queue manager on IBM MQ.

  • Inquire metrics in bulk: Enabling this setting will reduce the number of requests to IBM MQ and collect metrics for queues and channels in less requests thus reducing time and potential CPU overhead. However, this setting will require granting permissions to all queues and all channels, even SYSTEM objects too. This setting is not suitable for large environments with a large amount of queues or channels.

  • Lastly, select which feature sets (metric groups) you would like this configuration to collect. Selecting too few will not give you the proper visibility and you may miss out on important performance data, as well as key indicators that can be correlated between different objects.

For Remote (ActiveGate) deployment:

• Select which ActiveGate group will run this extension

  • Log level: Enable DEBUG only when troubleshooting or support makes that recommendation.

  • Task bucket size: By default, all endpoints defined in a monitoring configuration create a task. Use this setting to create separate tasks for groups of endpoints defined in a monitoring configuration. This will allow you to optimize resource usage on your ActiveGate and potentially distribute execution across other ActiveGates in your group.

  • Add a Queue Manager: Click on this button for each queue manager you want to connect. Multiple queue managers per ActiveGate group is recommended.

  • Comma-separated hosts: Most queue managers run on a single host but there are times you have multi-instance queue managers that run across multiple hosts. Enter that single host and port here, or the comma-separated list of host and ports for that one queue manager.

  • Queue manager name: Enter the name of the single queue manager you are connecting to. This is required.

  • Server-connection channel: Any client application, such as this extension, that connects to IBM MQ must talk to the queue manager via a channel. Enter the name of that Server-connection channel here.

  • Use credential vault: This will allow you to select a username/password credential stored on Dynatrace's Credential Vault. Using this feature will ignore any username/password entered manually and instead will only use the credential stored in the vault.

  • Username: Optional but recommended to add a user. A user is always passed, but depending on the queue manager configuration on IBM MQ, it can be ignored or this user in this field can be authenticated. It is best to always enter a user.

  • Password: Password to user above.

  • Use SSL: Enable this to establish TLS communication from the extension (client) to the MQ server. One-way authentication or Two-way authentication is supported and this is all handled by the IBM MQ server. The extension always presents the certificate and it is up to IBM MQ whether to accept it or not.

  • Model queue: Optional. If not set, it will use the SYSTEM default model queue for command requests and responses.

  • Reply prefix: If you entered a model queue, you must enter a prefix for the reply queue.

  • Objects to monitor: Two methods of adding objects.

    • Using new wizard mode, just click buttons to enter rules. If there is an object you absolutely do not want to monitor then remove that rule.
    • Advanced mode, add filters in a comma-separated fashion under each tab. Hover your mouse over the (i) icon next to each object type for rules and examples.
    • Note: Rules are evaluated in sequential order. This way you can include or exclude objects from the previous rule result. For example: *, -SYSTEM.*, SYSTEM.CLUSTER* would include all queues, then remove those that start with SYSTEM. then include those that start with SYSTEM.CLUSTER. Pay close attention to the order you enter your rules, whether in Wizard or Advanced mode.

  • Send log event on Dead letter queue: Enable this if you would like any DLQ on any queue manager matching this configuration to trigger a log event indicating that the number of messages in DLQ exceeded threshold. You may create an alert based on that log event

  • Send log event on high queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth threshold is exceeded. This threshold is configured on IBM MQ.

  • Send log event on low queue depth from queue manager: Enable this if you would like any matching queue on any queue manager in this configuration to trigger a log event when its configured depth lower than threshold. This threshold is configured on IBM MQ.

  • Retrieve topology for improved transaction tracing: Enabling this setting will collect mappings for alias queues, remote queues, and cluster queues. This mapping is then sent to Dynatrace to complement mapping information for improved transaction tracing. You may only need to collect this once unless your mappings on IBM MQ change frequently. When enabled, enter a Dynatrace token to communicate with the API with these permission: Read settings, Write settings.

  • Rename Queue manager: Enabling this setting will allow you to add an alias to the queue manager defined above. The real queue manager name will still be included in the name. This is only to differentiate queue managers with similar or same names on Dynatrace. This will not rename any queue manager on IBM MQ.

• Inquire metrics in bulk: Enabling this setting will reduce the number of requests to IBM MQ and collect metrics for queues and channels in less requests thus reducing time and potential CPU overhead. However, this setting will require granting permissions to all queues and all channels, even SYSTEM objects too. This setting is not suitable for large environments with a large amount of queues or channels.
• Disable auto-alerting: Automatic problem creation to failures will be disabled. Errors will still be logged as events but will not trigger problems. If you wish to create problems then you would have to create rules to trigger them from log events.
• Lastly, select which feature sets (metric groups) you would like this configuration to collect. Selecting too few will not give you the proper visibility and you may miss out on important performance data, as well as key indicators that can be correlated between different objects.

Reference the Get Started section for more details on compatibility information.

• How many DDUs will it consume per year?

The following is an approximation. Metrics change depending on type of objects. Some metrics are not available to all channels, some metrics are not available to all queues.

Assuming you get all metrics for all objects: ((10) + (Local queues * 16) + (Alias + Remote queues) * 2 + (Channels * 18) + (Listeners) + (Topics * 4)) * Queue managers * 525.6 = DDUs per year

For example:

1      Queue manager
400    Local queues
20     Alias and remote queues
50     Channels
1      Listener
50     Topics

( 10 + (400 * 16) + (20 * 2) + (50 * 18) + (1 * 1) + (50 * 4) ) * 1 * 525.6 =
(10 + 6400 + 40 + 900 + 1 + 200)  * 525.6 = approximately 3.9M DDUs per year

• I am not seeing all queue metrics

Some metrics depend on the type of queues. Other metrics depend on your queue manager configuration and if your user has enough permissions to collect them.

• Are you missing Enqueue and Dequeue? Make sure your user has CHG permission on queues to reset statistics on queues. Only then you can receive Enqueue and Dequeue metrics.

• Are you missing Oldest message and Last get/put? Make sure Monitoring statistics is enabled on your queue manager for all queues.

• Are you looking at Alias or Remote queues? Only Locally defined queues have status metrics. Alias and Remote only have Inhibit GET/PUT metrics.

• I keep getting Unauthorized errors but I know my user is correct

Check the AMQERR01.LOG on the queue manager itself. This log file will tell you what it tried to authenticate and what permissions you are missing. If you are unable to connect to your queue manager, make sure that either the user running the extension process has permissions to the queue manager, or ADOPTCTX and CHLAUTH is enabled so that it can authenticate the user and password combination passed in the MQCSP block.

• I get a pymqe error saying libmqm_r.so not found

This means that the IBM MQ libraries were not found in the unix PATH. Make sure you follow the ldconfig step above under Local deployment

• Other common errors and their meanings

MQRC_HOST_NOT_AVAILABLE: Queue manager is not reachable from your ActiveGate. This usually means a firewall rule is missing. Because a queue manager can run on a custom port, you must make sure that host:port is reachable from the ActiveGate running the extension.

MQRC_NO_MSG_AVAILABLE: It took too long to get a response back from your queue manager and the response was blank. This can happen if the MAX LENGTH size of your reply queue is not enough to fit the response payload. You can try increasing that value. It may also be caused because of a timeout or the queue manager did not respond in time.

MQRC_CONNECTION_BROKEN: The connection to the queue manager is severed either due to a network interruption or the queue manager closed it. Make sure your network performance is optimal. Also, if you are using SSL, make sure that you are not using an older TLS cipher that requires manual reset of TLS keys such as TLS 1.2 AES-GCM ciphers. Switch to a TLS 1.3 Cipher that has automatic TLS keys reset.