Troubleshoot code module installation on z/OS
This page describes various techniques for troubleshooting and for collecting the necessary data for Dynatrace ONE.
Component set up verification
The following sections describe how to identify setup issues.
The group name might be different, as well as the two-character suffix representing the CICS release of the code module (for example, CTS52 uses 69).
From the CICS Region, look for these messages to validate the CICS code module resources have been defined:
CICSAPPL Install for group XXXX has completed successfully. CICSAPPL OWNER CSSY Resource definition for ZDTAGT72 has been added. CICSAPPL OWNER CSSY Resource definition for ZDTDC2 has been added. CICSAPPL OWNER CSSY Resource definition for ZDTDC2A has been added. CICSAPPL OWNER CSSY Resource definition for ZDTPLT has been added. CICSAPPL OWNER CSSY Resource definition for ZDTPLTSD has been added. CICSAPPL OWNER CSSY Resource definition for ZDTSOAPH has been added. CICSAPPL OWNER CSSY TRANSACTION definition entry for DTAX has been added.
The CICS code module logs initialization messages to the zRemote log.
The zRemote log can be accessed from within Dynatrace, in the same manner as all the other code module log files. Look for log entries similar to the following:
2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registering a pgi for the job: HVBAC021, host=10.30.220.41, groupId= f39f4801966aa7c7, pgir.groupInstanceID= fad6dee63cfd1522, hostID= 95c0bb0371704b8c, nodeID= fad6dee63cfd1522, groupName=HVBAC021, hostGroup=, processGroupType= 28 2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registered SubAgent[C021,51,32aa8d038887d1c9] with zDC[Z021,52], rc=true 2019-05-09 20:19:11.789 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021], subagentid[32aa8d038887d1c9] snaId[NETD .HVBAC021], CICS release 54 was successfully registered with zdc using protocol version=7.2.0, allocator=pooled. 2019-05-09 20:19:13.789 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[Z021], jobName[AFVBZ021] - ZDC955I - Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021. 2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP008I - ZDTP008I ZDTAGT71. 2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID, smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP020I - ZDTP020I Active Sensors: MQ DB2 DB2R SOAP CTG DB2Fetch DLI DLIR HTTP .
Verify that the zDC has started the correct version, initialized successfully and has started/attached to the zLocal (messages in zDC SYSPRINT DD).
ZDC000I INITIALIZATION STARTED FOR zDC VER 1.195.00 ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.00 ZDC053I LPAR NAME IBMSYS1 CVTSNAME S0W1 . . . . ZDC955L Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021 ZDC958L Dynatrace INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z021 ZDC993I Opn1RFD:0008 /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log ZDC955I Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021
If the following message appears in the zDC job log (your subsystem name might be different than MEPC), refer to the steps outlined in the Emergency zDC Termination section to stop the zDC. Restart the zDC with a different
11:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE
Follow the steps outlined in the Emergency zDC Termination section.
If the following messages (or something similar) appear in the zDC job log:
H408I USER(xxxxxxxx) GROUP(xxxx) NAME(STARTED TASK ) BPX.FILEATTR.PROGCTL CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
And the associated messages appear in the zLocal logfile:
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Server requests us to use Agent dTMajor.dTMinorVersion.0.dTBuild with a hash of 0317af199c1ab1a03dda2cee90c2ea61 JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Requesting Agent library from Server JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Error setting Agent library program controlled: EDC5139I Operation not permitted. JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Loading Agent /dt/dynatrace-<dTMajor.dTMinorVersion.0>/agent/downloads/<dTMajor.dTMinorVersion.0.dTBuild> /native/zos-s390-64/libdtzagent.so
These can be safely ignored.
After the initial download of the zLocal, the operating system tries to set a flag for the downloaded library that is needed in certain circumstances. The flag is not set during the second run because the library will not be re-downloaded if it already exists.
The zLocal writes the log messages to the zDC SYSPRINT DD in addition to
/u/dt/log in the z/OS Unix environment. The zLocal log contains information relating to zLocal startup, versioning and connectivity to the zRemote.
To verify that all the channels in zLocal are connected to the zRemote, look for the following messages in the SYSPRINT:
info [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake. info [native] dynaTrace z Remote Agent client handshake performed. info [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb]. info [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake info [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.
Use the command below to instruct the zDC to display the zLocal status and to locate the log file paths of zLocal and zRemote.
Modify <zDCJobName>,dt1 stdo
Once the zLocal log file is located, it can be accessed using the standard techniques. Use OMVS, ssh/telnet, or view it in the Dynatrace UI like a regular OneAgent log file.
Also, look for the following messages in the corresponding zRemote log file (the values inside the bracket should reflect your live data).
info [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb]. info [native] Data client socket listener thread started info [native] ASID, smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1 .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.0 info [native] zDC[Z208] SMO is initialized with size=10M. info [native] Registering the zdc
- Login to CICS.
- Execute the DTAX transaction with the parameter
- Verify that the
PINGtraveled from CICS to zDC to zRemote and returned.
The output of the DTAX Transaction using
PING looks like this:
The log message in zRemote log file looks like this:
info [native] ASID, smfID[CW06], sysid[C208], jobName[H06AC208] Ping data=CA72106A
zLocal could not be started
Check the job log of the zDC for any of the following error messages.
These do not appear in the zLocal log file, since the zLocal could not be started.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied ZDC997D Notice: unknown modid, reason text might be incorrect ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort
Correct the permission on the zLocal folder and binaries, so they are at least readable and executable.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied ZDC997D Description: SAF CKACC returned error. ZDC996E @Abrtz/OS Unix agent attach failed,Abort
If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC member exists ZDC997D JRFileNotThere: The requested file does not exist ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort ZDC975E z/OS Unix agent attach failed,Abort
Correct the DTAGTCMD parameter in the SYSIN DD of the zDC to point to the correct zLocal binary location.
System under diagnosis (CICS region) could not connect to the zDC
Check the job log of the affected CICS regions for the following message, where
yyyy is the subsystem ID of the zDC that the CICS region is trying to connect to. It might be blank if the CICS region is trying to connect to the default subsystem that is configured with the DEFAULT(YES) parameter. We recommend to simply search for the error message code.
ZDTP004W zDC yyyy unavailable
Verify that the zDC with that subsystem ID is started. If so, try to issue the DTAX transaction command
ENABLE to re-enable the connections.
Disabled IMS code module
If an ABEND occurs in the IMS code module, the recovery process produces ABEND diagnostics if possible, and then it disables the IMS code module. The IMS system continues to function. When this occurs, a series of WTO messages are written to the system log for the IMS control region and/or IMS dependent region. A sample normal message set follows:
ZDTI032W Recovery routine entered. ZDTI036W ZDTIII14 BDCSBK00 20170713 10.15 VER 07.02.00 ABEND at offset 001DB6. ZDTI033W Successful ABEND recovery, agent disabled.
Different or additional messages might be issued if abnormal conditions are encountered by the recovery process, such as when dynamic storage cannot be obtained, retry is not permitted, or no SDWA was passed. All of the messages related to the ABEND recovery process are documented in the z/OS code module messages section.
A Software (SFT) Error Record further describing the ABEND is usually written to the z/OS system SYS1.LOGREC data set. You should run the z/OS EREP utility program to print the Software (SFT) Error Record associated with the ABEND.
Optionally, an SVC dump might be taken during recovery, depending on the ABEND recovery option specified or defaulted to when the IMS code module was injected. The default action is to not capture an SVC dump when ABEND recovery is driven. This option can be specified as a parameter when the IMS code module is initially injected, or specified in conjunction with the Modify or Enable function parameters to toggle dump capture during recovery on or off for a previously injected IMS code module. For details, see Install the IMS code module.
When the IMS code module is disabled as a result of the ABEND recovery process, it remains disabled until explicitly re-enabled using the IMS code module injection program. Please contact a Dynatrace ONE product specialist by selecting the chat button in the upper-right corner of the Dynatrace menu bar.
Ensure CICS and IMS code module version is less than or equal to zRemote version. Do not connect newer CICS and IMS code modules to older zRemote modules. Following is a sample message in the zRemote log when an IMS code module version is incompatible with the zRemote version.
severe [native] IMS14CR1[asid = 108] is trying to initialize with an invalid protocol version number : x.xxx.xx