IBM MQ

This extension isn't required for tracing IBM MQ messages. Dynatrace automatically detects and monitors queues and messages in your applications. For details, see the blog post End-to-end tracing of messages across IBM MQ.

While this extension automatically discovers aliases—and remote and cluster queues—you can configure the same mapping using the Dynatrace API. For details, see the blog post Breaking down barriers IBM MQ tracing across queue managers and queue clusters.

Prerequisites

  • IBM MQ 8.0+ for Windows/Linux/AIX/MQ Appliance. z/OS and iSeries are not supported.
  • An Environment ActiveGate (version 1.155+) that has the ActiveGate plugin module installed and isn't used for synthetic or mainframe monitoring
    • 1 environment ActiveGate can typically support 30-50 IBM MQ queue managers
  • IBM MQ client installed on Environment ActiveGate
  • A server-connection channel on the queue manager for communication with the plugin
  • if using SSL:
    • Two key repositories must exist. One on the ActiveGate where the MQ Client resides and the other on the MQ Server. Each repository must have each other’s public key certificate. The certificate label of the MQ Server must be ibmwebspheremq<queue_manager> (all lower case). The certificate label of the MQ Client on the ActiveGate must be ibmwebspheremq<username> (all lower case). The username is the user in which the Remote Plugin process runs.
    • GSKit

Interested in monitoring IBM MQ with Dynatrace?

The quickest way to get started is by contacting a Dynatrace ONE product specialist. Just click the chat button in the upper-right corner of the Dynatrace menu bar.

Environment ActiveGate installation

ActiveGate plugin module installation

ActiveGate version 1.175+ has the plugin module installed by default.

IBM MQ Client Configuration

See the current version here. Alternatively, you can search the IBM website for the MQ client version download that matches your environment.

  • For Windows, install the MQ client as specified.
  • Note that for Linux-based ActiveGates, you only need to install MQSeriesRuntime.rpm, MQSeriesClient.rpm and MQSeriesGSKit.rpm.
    1. Add the MQ client lib64 libraries to LD PATH by creating a new file in /etc/ld.so.conf.d/. The content of this file is only the path to the lib64 libraries, /opt/mqm/lib64 by default.
    2. Save and restart the LD configuration with sudo ldconfig.

IBM MQ connectivity to ActiveGate

In order to work, the IBM MQ plugin extension must connect to the MQ server passing a user and password. The user:password pair from the Dynatrace UI is passed in an MQCSP structure block. However, depending how the MQ server is configured, it may ignore the MQCSP block. Instead, the MQ server will authenticate the “UserID” passed which is the user running the Remote Plugin Module process. If this is the case, that user must exist on the MQ server and must have the proper permissions to access Channels and Queues. You are also able to connect without a user or password but your Queue Manager must allow this, and have properly configured channels (IDPWOS, IDPWLDAP, etc.). The user must have at least the following permissions: connect,display,browse,put,inquire and change(only required if requesting Enqueue/Dequeue counts)

Extension installation

  1. Obtain the install file (custom.remote.python.ibmmq.zip). Don't rename the file.

  2. Unzip custom.remote.python.ibmmq.zip to the plugin_deployment directory of your ActiveGate host.

  3. If the resulting directory structure isn't .\plugin_deployment\custom.remote.python.ibmmq\, please make the neccessary changes.

  4. Restart the Dynatrace Remote Plugin Module service.

    • On Linux, restart the service using the following commands with admin rights:
      • systemctl restart remotepluginmodule.service
    • On Windows, run these two commands in a Command Prompt launched as Admin:
      • sc stop "Dynatrace Remote Plugin Module"
      • sc start "Dynatrace Remote Plugin Module"
  5. Return to the Dynatrace web UI. Click Settings, the Add new technology monitoring button, and finally the Add ActiveGate plugin button.

  6. Click the Upload plugin button and upload custom.remote.python.ibmmq.zip.

  7. Enter the following information to connect to your IBM MQ Queue Manager:

    • Endpoint name: Type a meaningful endpoint name.
    • User: User to authenticate against MQ Server. If blank user, make sure Queue Manager and Server-connection channel are configured to allow this.
    • Password: The user password.
    • Comma-separated Queue Manager host(s):port(s): This is a connection name list so you may enter more than one host and IP address (with ports) for one queue manager only. Example: 192.168.55.180(1414), 192.168.55.181(1414), 192.168.55.182:1415, 192.168.55.183:1414
    • Server-connection channel: Channel for the plugin to communicate with the queue manager
    • Single queue manager: : Name of queue manager to collect data from. Only one queue manager per end-point. You may add other end-points to other queue managers.
    • Channels: List of channels to collect data for. Use of wildcards is allowed, and use of exclusions denoted by “-“. i.e.: “abc*” will get all channels that start with “abc”. “abc*, -mq.chan*” will get all channels that start with “abc” and exclude all that start with “mq.chan”.
    • Comma seperated queues: List of queues to collect data for separated by commas. Use of wildcards * and exclusions denoted by “-“ is allowed. i.e.: “abc*” will get all queues that start with “abc”. “abc*, -amq*” will get all channels that start with “abc” and exclude all that start with “amq”.
    • Listener channels: Leave empty for none, * for all
    • Path to key repository: If using SSL, you must provide the path on the ActiveGate where the SSL key repository is located. The path must end in the key name minus the extension (e.g. D:/Some/path/SSL/key). If no SSL, leave empty
      • Two key repositories must exist - one on the ActiveGate where the MQ Client resides, and the other on the IBM MQ server.
      • Each repository must have the other’s public key certificate.
      • The certificate label of the MQ Server must be ibmwebspheremq<queue_manager>(all lowercase).
      • The certificate label of the MQ Client on the ActiveGate must be ibmwebspheremq<username> (all lowercase).
      • The username is the account through which the Remote Plugin process runs the Remote Plugin process.
    • Cipher spec: Cipher specification of encryption used in the channel communication. Must match the Cipher specification of the Server-Connection channel configuration.
    • Exclude SYSTEM: Check this box if you want to automatically exclude all queues and channels that start with “SYSTEM.”
    • Run Reset Statistics: This will issue the RESET_STATS command to queues in order to collect Enqueue/Dequeue count metrics. This requires CHG permissions on queues.
    • Retrieve topology for improved transaction tracing: This checkbox enables a flag to use Dynatrace Configuration API to feed the MQ topology for local, alias, remote and cluster queues. This way, application transaction tracing will have improved visibility to MQ calls.
    • Cluster environment or tenant: Your current cluster environment or tenant URL.
    • API Token: A Dynatrace API token with “Write configuration” and “Accept problem and event feed, metrics and topology”.
    • Name of group: If the device is part of a cluster, enter the name here to group the devices in the GUI. This will group your devices in your Technologies view.
    • ActiveGate: Choose the ActiveGate where the plugin resides which will poll MQ Server every minute

    IBM MQ

Installation troubleshooting

  • Error (libmqic_r.so: cannot open shared object file: No such file or directory)
    • See the IBM MQ client installation section for details. This error is likely caused by the IBM MQ client not being installed on the ActiveGate host, or the path hasn't been updated to point to the libmqic_r.so.
  • Update failed - Authorization error Connection failed. Unauthorized user <username>. FAILED: MQRC_NOT_AUTHORIZED
    • Either the provided user wasn't added to the IBM MQ environment or the password is incorrect. Note that the password isn't applied automatically during updates; you need to manually add the password again.
  • Troubleshooting ActiveGate plugin installation issues

SSL configuration

Self-signed certificates

  1. Your keystore on your MQ server must have a certificate with the proper label. Your keystore can have any name, but the certificate label is what is important. It will not accept any other format. Format is: ibmwebspheremq<queue_manager_lower_case> where <queue_manager_lower_case> is the name of your queue manager. So if your queue manager is QM_ORANGE, then the certificate label must be ibmwebspheremqqm_orange

    • You can create a new keystore on your MQ server with the following command: /opt/mqm/bin/runmqakm -keydb -create -db /var/mqm/qmgrs/QM_ORANGE/ssl/qm_orange.kdb -pw changeit -type cms -stash

    • Then you can create the certificate into the keystore with the following command: /opt/mqm/bin/runmqakm -cert -create -db /var/mqm/qmgrs/QM_ORANGE/ssl/qm_orange.kdb -pw changeit -label ibmwebspheremqqm_orange -size 2048 -sigalg SHA512withRSA -san_dnsname qm_orange.dynatrace.com -dn "CN=qm_orange.dynatrace.com,OU=Extensions,O='Dynatrace',L='Detroit',ST=Michigan,C=US" -expire 1825

    • Note: The filename can be anything, and the location of the keystore is your preference. The example below uses the default location which reflects the same location in the Queue Manager properties page.

  2. Your keystore on you client side (where ActiveGate is, where your MQ client is installed) must have a certificate with the proper label. Your client keystore can have any name but the certificate label is what is important. It won't accept any other format. Format is: ibmwebspheremq<user_running_plugin_process> where <user_running_plugin_process> is the operating system user that is running the Dynatrace Remote Plugin process/service (not the ActiveGate). If it’s Windows, then it's likely that it’s using “Local Service” as the account that runs it and you should change it to a real user (service account, for example). So if the user is svcdyn then the certificate label on the client must be ibmwebspheremqsvcdyn. SSL Create the client certificate in the client keystore: SSL Username running the process must match username in the client certificate: SSL SSL

  3. Both keystores on the MQ server and the MQ client (where ActiveGate is) have to be in CMS format (not JKS or PK12) and must contain .kdb, .rdb, .sth files at least. STH is the stashed password so make sure to stash the password when you create the keystore.

    MQ server: SSL ActiveGate: SSL

  4. Client keystore must have the public key of the MQ Server certificate imported. So export the certificate ibmwebspheremq<queue_manager> from the MQ server keystore and import it into the client keystore.

    You can export the certificate from the MQ server keystore by using the following command: /opt/mqm/bin/runmqakm -cert -extract -db /var/mqm/qmgrs/QM_ORANGE/ssl/qm_orange.kdb -pw changeit -label ibmwebspheremqqm_orange -target /var/mqm/qmgrs/QM_ORANGE/ssl/ibmwebspheremqqm_orange.arm -format ascii

    Copy the extracted certificate file to the ActiveGate, then import the extracted certificate into the ibmmqkeystore.kdb keystore on the ActiveGate: SSL

  5. MQ server keystore must have the public key of the client certificate imported. So export the certificate ibmwebspheremq<username> from the client keystore and import it into the MQ server keystore.

    Export the certificate from the ActiveGate keystore: SSL

    Copy the extracted client certificate to the MQ server then import it using the following command:

    /opt/mqm/bin/runmqakm -cert -add -db /var/mqm/qmgrs/QM_ORANGE/ssl/qm_orange.kdb -pw changeit -label ibmwebspheremq**diego** -file /var/mqm/qmgrs/QM_ORANGE/ssl/client_cert.arm -format ascii

  6. Now your MQ server should have a keystore with its own certificate and the client certificate, and your ActiveGate should have its own keystore with its own certificate and the MQ server certificate. You can list the certificates in MQ server keystore with the following command: /opt/mqm/bin/runmqakm -cert -list -db /var/mqm/qmgrs/QM_ORANGE/ssl/qm_orange.kdb -pw changeit

    SSL

    ActiveGate keystore showing imported certificate from MQ server: SSL

  7. Make sure that your server-connection channel has the proper cipher spec selected and you select the exact same spec in the plugin configuration UI. SSL

  8. If you're using a Peer Name (PNRP) on the server-connection channel (Distinguished Name), then make sure that the Distinguished Name exists in the CLIENT certificate (the one labeled ibmwebspheremq<user>)

  9. The path to the repository field on the plugin UI is the path to the client keystore, including the keystore name without the extension. So if the keystore is D:\ssl\clientkeystore.kdb then the path to repository field must say: D:\ssl\clientkeystore

### CA-signed certificates

  1. Your MQ server and MQ client (ActiveGate) need to have separate CMS keystores. They must be CMS, not JKS or PK12 or other. The name of your keystores doesn't matter.

  2. Your MQ server keystore must have your CA root certificate imported as a “Signer Certificate”. SSL

  3. You must create a Certificate Signer Request (CSR) from your MQ server keystore because it writes a record in the .rdb file of your keystore. You can't create a CSR from elsewhere. SSL

  4. Create a signed certificate using your Certificate Authority and your Signer Request. For OpenSSL, you could execute the following command: openssl x509 -req -in certcsr.arm -CA myCARoot.crt -CAkey myCARoot.key -CAcreateserial -out mysignedcert.crt -days 500 -sha256

  5. Import/Receive your signed certificate into your keystore in Personal Certificates. Make sure your certificate has the label (alias) ibmwebspheremq<queue_manager> where <queue_manager> is your queue manager name all in lower case. SSL

  6. Repeat steps 2-4 for your MQ client (ActiveGate) keystore.

  7. Import/Receive your signed certificate into your keystore in “Personal Certificates”. Make sure your certificate has the label (alias) ibmwebspheremq<username> where <username> is the user that runs your Remote Plugin Module process (not your ActiveGate process). SSL

  8. When done, place all files for MQ server in the proper location and configure the Queue Manager to look for the keystore in that location. SSL

  9. Configure your SSL server-connection channel to use the right cipher spec, they must match on both sides of the communication pipeline.

  10. For the MQ client, place the files in your location of choice and in the configuration UI of the plugin, enter the path to the keystore WITHOUT the .kdb extension. So just type /path/to/SSL/keystore/filename in the SSL Repository field. Also, make sure that the proper cipher spec is selected. SSL

  11. Remember to refresh your SSL config on your Queue Manager to pick up the new SSL changes to your Queue Manager.

Metrics

Queue Manager

  • Availability %
  • Connections
  • Active channels

Channel (split by channel)

  • Availability: Whether the channel is running or not.
  • Messages: Number of messages sent and received, including MQI calls for all channel instances.
  • Bytes Sent/Received: Number of bytes sent and received for all channel instances.
  • Buffers Sent/Received: Number of buffers sent and received for all channel instances.
  • Last Message Date/Time: Time last message was sent.
  • Current Sharing Conversations: Number of conversations being shared for this channel.

Queues (split by queue)

  • Queue Depth: Current queue depth value
  • Percent queue depth: Calculated using the Current Queue Depth over Max Queue Depth value.
  • Inhibit Get/Put: Property value denoting whether PUT and GET are allowed
  • Open Input/Output Count: Open input and output handles. IPPROCS/OPPROCS
  • Oldest Message Age: Time in seconds of oldest message.
  • Uncommitted Messages: Number of messages that have not been committed.
  • Enqueue/Dequeue Rate: Number of messages in queue that were PUT or RETRIEVED and not committed per second.
  • Time Indicator: Time messages stay in queue
  • Last get/Last put: Time in milliseconds when MQGET and MQPUT commands were executed on this queue.

Listener (split by listener)

  • Availability %