This page describes various techniques for troubleshooting and for collecting the necessary data for Customer Support.
Component set up verification
The following sections describe how to identify set up issues.
The group name may be different, as well as the two-character suffix representing the CICS release of the code module (for example, CTS52 uses 69).
From the CICS Region, look for these messages to validate the CICS resources have been defined:
CICSAPPL Install for group XXXX has completed successfully. CICSAPPL OWNER CSSY Resource definition for ZDTAGT72 has been added. CICSAPPL OWNER CSSY Resource definition for ZDTDC2 has been added. CICSAPPL OWNER CSSY Resource definition for ZDTDC2A has been added. CICSAPPL OWNER CSSY Resource definition for ZDTPLT has been added. CICSAPPL OWNER CSSY Resource definition for ZDTPLTSD has been added. CICSAPPL OWNER CSSY Resource definition for ZDTSOAPH has been added. CICSAPPL OWNER CSSY TRANSACTION definition entry for DTAX has been added.
The CICS code module doesn't log its messages to the JESMSGLG DD.
Messages are logged to the zRemote log, accessible from within Dynatrace, in the same manner as all the other code module log files. Look for log entries similar to the following:
2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registering a pgi for the job: HVBAC021, host=10.30.220.41, groupId= f39f4801966aa7c7, pgir.groupInstanceID= fad6dee63cfd1522, hostID= 95c0bb0371704b8c, nodeID= fad6dee63cfd1522, groupName=HVBAC021, hostGroup=, processGroupType= 28 2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registered SubAgent[C021,51,32aa8d038887d1c9] with zDC[Z021,52], rc=true 2019-05-09 20:19:11.789 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021], subagentid[32aa8d038887d1c9] snaId[NETD .HVBAC021], CICS release 54 was successfully registered with zdc using protocol version=7.2.0, allocator=pooled. 2019-05-09 20:19:13.789 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[Z021], jobName[AFVBZ021] - ZDC955I - Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021. 2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP008I - ZDTP008I ZDTAGT71. 2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP020I - ZDTP020I Active Sensors: MQ DB2 DB2R SOAP CTG DB2Fetch DLI DLIR HTTP .
Verify that the zDC has started the correct version, initialized successfully and has started/attached to the zLocal (messages in zDC SYSPRINT DD).
ZDC000I INITIALIZATION STARTED FOR zDC VER 07.02.00 ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.00 ZDC053I LPAR NAME IBMSYS1 CVTSNAME S0W1 . . . . ZDC955L Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021 ZDC958L Dynatrace INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z021 ZDC993I Opn1RFD:0008 /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log ZDC955I Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021
If the following error message appears in the zRemote log file:
JJJJ-MM-DD HH:MM:SS [37ab2e00] severe [native] Exception while connecting to Collector, info:<Could not register Agent with Collector as the Collector declined service>
Make the appropriate adjustments to the affected components.
If the following message appears in the zDC job log (your subsystem name may be different than MEPC):
11:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE
Follow the steps outlined in the Emergency ZDC Termination section.
OneAgent status information and OneAgent log files for connected OneAgents are available through the Dynatrace UI, but this process can help to diagnose code module connection failures.
The standard code module logfile contains the standard code module messages relating to startup, logging, versioning, and connectivity to the Server/Collector. The zDC SYSPRINT contains the
stdout for the zLocal, and this contains the location of the zLocal log file.
If the SYSPRINT does not contain this information, instruct the zDC to display it using the following command:
Modify zdc,dt1 stdo info (native) Log level console ........... info info (native) Log level file .............. info info (native) Log file .................... <dtInstallFolder>/log /dt_<agent_name>\_<process_id>.log
Once that log file is located, it is accessible by the standard techniques. Use OMVS, ssh/telnet, or view it in the Dynatrace UI like a regular OneAgent log file.
To verify that all the channels are connected properly, look for the following items in the OneAgent log file:
info [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake. info [native] dynaTrace z Remote Agent client handshake performed. info [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb]. info [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake info [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.
Also, look for the following records in the corresponding zRemote log file (the values inside the bracket should reflect your live data).
info [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb]. info [native] Data client socket listener thread started info [native] ASID, smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1 .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.0 info [native] zDC[Z208] SMO is initialized with size=10M. info [native] Registering the zdc
- Log in to CICS.
- Execute the
DTAXtransaction with a parameter of 'PING'.
- Verify that the PING traveled from CICS to zDC to zRemote and back again from zRemote to zDC.
The output of the DTAX Transaction looks like this:
The log message in the OneAgent log file looks like this:
info [native] ASID, smfID[CW06], sysid[C208], jobName[H06AC208] Ping data=CA72106A
zLocal could not be started
Check the job log of the zDC for any of the following error messages.
These do not appear in the zLocal log file, since the zLocal could not be started.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied ZDC997D Notice: unknown modid, reason text may be incorrect ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort
Correct the permission on the zLocal folder and binaries, so they are at least readable and executable.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied ZDC997D Description: SAF CKACC returned error. ZDC996E @Abrtz/OS Unix agent attach failed,Abort
If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC member exists ZDC997D JRFileNotThere: The requested file does not exist ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort ZDC975E z/OS Unix agent attach failed,Abort
Correct the DTAGTCMD parameter in the SYSIN DD of the zDC to point to the correct zLocal binary location.
System under diagnosis (CICS region) could not connect to the zDC
Check the job log of the affected CICS regions for the following message, where
yyyy is the subsystem ID of the zDC that the CICS region is trying to connect to. It may be blank if the CICS region is trying to connect to the default subsystem that is configured with the DEFAULT(YES) parameter. We recommend to simply search for the error message code.
ZDTP002W zDC yyyy unavailable or CICS exits disabled
Verify that the zDC with that subsystem ID is started. If so, then try to issue a DTAX Enable command to re-enable the connections.
Expected queues and/or transactions (including SOAP and CTG transactions) are not being monitored
Verify that the CICS code module is configured correctly, and that the monitored CICS region's configuration matches that of the CICS Sensor in the system profile.
In the Dynatrace UI select Settings then select Server-side service monitoring. Now under Server-side service monitoring select Deep monitoring. Under Deep Monitoring select New OneAgent features. Hit the right tab arrow (>) until you see the feature(s) you want.
Disabled IMS code module
If an ABEND occurs in the IMS code module code, the recovery process produces ABEND diagnostics if possible, then disables the IMS code module. The IMS system continue to function. When this occurs a series of WTO messages are written to the system log for the IMS Control Region and/or IMS dependent region. A sample normal message set follows:
ZDTI032W Recovery routine entered. ZDTI036W ZDTIII14 BDCSBK00 20170713 10.15 VER 07.02.00 ABEND at offset 001DB6. ZDTI033W Successful ABEND recovery, agent disabled.
Different or additional messages may be issued if abnormal conditions are encountered by the recovery process, such as when dynamic storage cannot be obtained, retry is not permitted, or no SDWA was passed. All of the messages related to the ABEND recovery process are documented in the z/OS code module messages section.
A Software (SFT) Error Record further describing the ABEND is usually written to the z/OS system SYS1.LOGREC data set. You should run the z/OS EREP utility program to print the Software (SFT) Error Record associated with the ABEND.
Optionally, an SVC dump may be taken during recovery, depending on the ABEND recovery option specified or defaulted to when the IMS code module was injected. The default action is to not capture an SVC dump when ABEND recovery is driven. This option can be specified as a parameter when the IMS code module is initially injected, or specified in conjunction with the Modify or Enable function parameters to toggle dump capture during recovery on or off for a previously injected IMS code module. See Deploy OneAgent on IMS for more details.
When the IMS code module becomes disabled as a result of the ABEND recovery process, it remains disabled until explicitly re-enabled using the IMS code module injection program contact support for a resolution.
To ensure proper migration:
- Do not connect newer CICS and IMS code modules of OneAgent to older z/DCs.
Dynatrace ActiveGate always supports OneAgents of earlier versions. with older other components should normally be fine no matter if code modules are bootstrapped or not.
Agents started by the bootstrap part update on application restart.