Install the zDC subsystem
The z/OS Data Collection (zDC) subsystem interacts with the CICS, IMS, and z/OS Java code modules via shared memory object (SMO) on a LPAR. The zDC subsystem maintains this SMO, and the code modules write their monitoring data to it.
The zLocal (
libdtzagent.so), hosted in the z/OS Unix System Services (USS) environment, runs as part of the zDC. It manages the TCP/IP socket connection to the zRemote module, reads monitoring data from SMO, and transfers these data to the zRemote.
The bootstrapper (
dtzagent), hosted in the z/OS USS environment, runs as part of the zDC. It manages the update process of the zLocal.
When the zDC initializes (system startup or manual start), it starts the bootstrapper, and the bootstrapper starts the zLocal. When the zDC terminates, the zLocal and the bootstrapper are stopped.
The zDC (zLocal) uses TCP/IP socket connections, so it may be necessary to update your security system's access rules if they are configured to deny TCP/IP access by default.
The zDC doesn't limit which user IDs can enter operator commands.
You can control permissions assigned to files created by the zLocal and bootstrapper by setting
DT_UMASK. The default is
The zDC runs as an authorized z/OS process (typically as a system task). This means the programs must reside in an authorized library. It is intended to start automatically as a started task at a system IPL. This gives the zDC continuous availability to collect monitoring data from code modules.
The zDC can also be run as a batch job. The service class of the zDC must be high enough so that it is always available for keep-alive messages from the zRemote. The zDC should have a priority equal to or higher than the monitored CICS and IMS regions.
A zDC subsystem must be installed on each LPAR you want to monitor.
Download z/OS product datasets. Make a note of the high-level qualifier used in the download procedure.
Create the RACF user ID for the zDC process. This ID must have a z/OS USS segment. The recommended home directory name is
If a home directory other than
/u/dtis set, change in step 8 the path to the bootstrapper, which defaults to
/u/dt/agent/lib64/dtzagent. The path names below the home directory can't be changed.
The home directory or the high-level qualifier that replaces it must be writeable by the zDC process.
- It writes the zLocal binary to
m.buildis the subversion of the release.
- It writes the zDC and zLocal logs to
If write fails, the the zDC processes uses the directory
If the zDC is run as a started task, the ID must be capable of representing the zDC started task and have access to the USS path above.
- It writes the zLocal binary to
Change the dataset high-level qualifier represented by <hlq> in member
SZDTSAMPto the value you used in the Download z/OS product datasets procedure.
COPYAGNTjob to create the required z/OS USS subdirectories under the home directory of the zDC started task.
SZDTSAMPand set it to be executable.
If you're using the default home directory of
/u/dt, the resulting path is
If a home directory other than
/u/dtis used, the path in this job must be changed accordingly. File and directory names are case-sensitive. The path names below the home directory can't be changed.
COPYAGNTjob should run under the user ID that was created for the zDC process, so that user ID owns the
dtzagentbinary. If this is inconvenient, you can also use the
chgrpcommands to reset the owning user and group for the
dtzagentbinary after running
<hlq>is the high-level qualifier value you used in the Download z/OS product datasets procedure.
For example, create a member named
APF FORMAT(DYNAMIC) APF ADD DSNAME(<hlq>.SZDTAUTH) VOLUME(XXXXXX)
Then issue the console command:plaintext
ZDCMEPCsample zDC started task PROC from
SZDTSAMPto a system PROCLIB used for started tasks.
The default name for the PROC is
MEPC. Customize it as necessary for local standards. If a different subsystem name is chosen, change the
SUBSYSTEM_ID()parameter in the SYSIN parameters to match.plaintext
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN) SUBSYSTEM_ID(MEPC)
The PROC and the two JCL SYSIN members
ZDCMEPCAthat it uses contain dataset names that must be edited to replace
<hlq>.with the appropriate high-level qualifiers.
DTAGTCMD()parameter in the SYSIN parameter to reflect your definitions.plaintext
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN) DTAGTCMD(/u/dt/agent/lib64/dtzagent nobootstrap=false zremoteagent=<ipaddress>[:port])
/u/dt/agent/lib64/dtzagentsets the path to the bootstrapper. The default home directory name is
/u/dt. If you have set a different home directory in step 2, change it accordingly.
nobootstrap=falseenables the bootstrapper to automatically update the zLocal once a newer version is available. Per default, the zLocal receives automatic updates. Set
trueto disable automatic updates and manually update the zLocal.
zremoteagent=<ipaddress>[:port]set the IP address and port of a installed zRemote module. The IP address is required, the port is optional and defaults to 8898.
optional Add a command to your system startup to automatically start the zDC subsystem at IPL.
SZDTSAMPto verify the zDC installation and the zRemote connectivity.
Good test for the zRemote connectivity returns
Connection timeout, as the zRemote does not support FTP protocol. If a zRemote connectivity problem exist, it is indicated by
EDC8128I Connection refused. This implies that the zRemote is not listening on the expected port.
All values that tailor the execution of the zDC reside in the dataset pointed to by
DDNAME SYSIN. The sample started task PROC
ZDCMEPC refers to the JCL SYSIN member
SZDTSAMP as shown below:
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
The dataset must contain 80 character records, blocked to a multiple of 80 bytes.
When you make any change to the SYSIN dataset, you must restart the zDC process for the change to take effect.
Positions 1 through 71 of each statement can contain parameter values. Positions 72 through 80 are ignored and can optionally contain a sequence number.
Each parameter uses
KEYWORD(value) format. Each keyword and the requirements for the values associated with it are documented in the JCL SYSIN member
If a parameter value spans multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.
If you want to use a comment, specify it by placing an asterisk in position 1 of a statement. The entire statement is considered part of the comment.
A comment can be enclosed between a slash-asterisk and an asterisk-slash.
/* This is a comment. */
Multiple lines can be contained within comments of this type.
Sets the path to the bootstrapper
Defines the name of the zDC subsystem.
Sets the log level of the zDC. Log level values range from
Sets the maximum amount of storage (MB) for messages that can be queued in the zDC shared memory object while awaiting writing to the z/OS code modules. The default of 1 MB should be adequate in most cases when transaction buffering is enabled. When transaction buffering is disabled (not recommended), set the SMO size to 10 MB if very high volumes of transactions are traced.
Changes the z/OS USS current directory where the bootstrapper
Overrides the default number and size of transaction buffers. Transaction buffers offer better performance for CICS and IMS code modules by placing event messages related to each distributed trace into dedicated buffers. Rather than send individual event messages for each distributed trace as they occur, they are blocked into one or more buffers and sent together. Additionally, the buffer of related messages is processed more efficiently by the zRemote.
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN) DTLOGLEVEL(3) MODIFY <jobName>,DT1 LOG=1
View log message output in the zDC job spool. The job spool helps determine errors that may occur during the zDC startup or when connecting with the zRemote. Once the zDC successfully connects to the zRemote, error messages from the zDC subsystem and the CICS and IMS code modules are routed to the zRemote.
The log level of the zLocal is set with the
DTAGTCMD(loglevel=INFO) parameter in the SYSIN parameters. You can change this log level dynamically by issuing a z/OS MODIFY command. The
DTAGTCMD(name=zlocal_<lpar>) defines the name of the zLocal for log files. The name should reflect that this is a zLocal. It could include the SMF ID of the LPAR that it services.
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN) DTAGTCMD(loglevel=INFO name=zlocal_<lpar>) MODIFY <jobname>,DT1 ZLALOGLEVEL=FINE
There are two sets of logs created for the zLocal. Both are created in the OMVS file system. One set is a temporary set of logs only valid for the current execution of the zDC, and the location of these logs defaults to the home directory of the zDC user ID. You can override the location by using the
DTCHDIR() parameter in the SYSIN parameters.
The zLocal also creates a standard set of logs, one for the bootstrapper, and one for the zLocal itself. These logs are in the standard Dynatrace log locations on the OMVS file system.
Update and maintenance
To update the zDC to a newer version, see the update and maintenance procedure in Download z/OS product datasets.
By default, the zLocal receives automatic updates if
DTAGTCMD(nobootstrap=false) is set in the SYSIN parameters. At every start of the zDC, the bootstrapper queries the zRemote module for available updates. The latest zLocal is installed as part of the zRemote installation. If an update is available, the zLocal will be automatically downloaded from the zRemote and updated.
info (native) The bootstrap channel connected successfully, requesting version: v.r.m.bbbb info (native) Interprocess lock acquired for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so info (native) Fetching agent binary succeeded for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so
Manually update the zLocal
- Don't use the manual update process for the zLocal when connecting the first time to your Dynatrace environment because the zDC won't start successfully.
- The zRemote module must be running when starting the zDC for the first time to avoid a
Suppose you don't want to receive automatic updates for the zLocal, set
DTAGTCMD(nobootstrap=true) in the SYSIN parameters. For the manual update process, the zLocal binary
libdtzagent.so must be present on the LPAR in the
To accomplish this
- Run the bootstrapper
dtzagentat least once in either a production LPAR or non-production LPAR.
- Copy the zLocal binary to the target LPAR. In the
SZDTSAMPdataset, the member
OCOPYAGTis a job that copies the zLocal binary
- Complete the instructions to start the zDC and connect it to your Dynatrace environment.
- If the zDC successfully connects to Dynatrace,
libdtzagent.sois copied to the
mis the release number
bbbbis the build number
info (native) Configured to not download module - loading local module info (native) Start loading local agent binary info (native) Successfully loaded agent binary /u/dt/agent/lib64/libdtzagent.so
Verify that the zDC has started the correct version, initialized successfully, and has started the zLocal (messages in zDC SYSPRINT DD).
ZDC000I INITIALIZATION STARTED FOR zDC VER 1.195.00 ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.00 ZDC053I LPAR NAME IBMSYS1 CVTSNAME S0W1 . . . . ZDC955L Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021 ZDC958L Dynatrace INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z021 ZDC993I Opn1RFD:0008 /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log ZDC955I Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021
If the following message appears in the zDC job log (your subsystem name might be different than
MEPC), restart the zDC it with a different
SUBSYSTEM_ID in the SYSIN parameters.
11:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE
To restart the zDC, follow the steps in Terminate the zDC.
If the following messages (or something similar) appear in the zDC job log:
H408I USER(xxxxxxxx) GROUP(xxxx) NAME(STARTED TASK ) BPX.FILEATTR.PROGCTL CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
And the associated messages appear in the zLocal log file:
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Server requests us to use Agent dTMajor.dTMinorVersion.0.dTBuild with a hash of 0317af199c1ab1a03dda2cee90c2ea61 JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Requesting Agent library from Server JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Error setting Agent library program controlled: EDC5139I Operation not permitted. JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Loading Agent /dt/dynatrace-<dTMajor.dTMinorVersion.0>/agent/downloads/<dTMajor.dTMinorVersion.0.dTBuild> /native/zos-s390-64/libdtzagent.so
These can be safely ignored.
After the initial download of the zLocal, the operating system tries to set a flag for the downloaded library that is needed in certain circumstances. The flag is not set during the second run because the library will not be downloaded anymore if it already exists.
The zLocal writes the log messages to the zDC SYSPRINT DD in addition to
/u/dt/log in the z/OS USS environment. The zLocal log contains information relating to zLocal startup, versioning, and connectivity to the zRemote.
To verify that all the channels in zLocal are connected to the zRemote, look for the following messages in the SYSPRINT:
info [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake. info [native] dynaTrace z Remote Agent client handshake performed. info [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb]. info [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake info [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.
Use the z/OS MODIFY command below to instruct the zDC to display the zLocal status and to locate the log file paths of zLocal and zRemote.
MODIFY <zdc-jobname>,DT1 STDO
Once the zLocal log file is located, it can be accessed using the standard techniques. Use OMVS, ssh/telnet, or view it in the Dynatrace web UI like a regular OneAgent log file.
Also, look for the following messages in the corresponding zRemote log file (the values inside the bracket should reflect your live data).
info [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb]. info [native] Data client socket listener thread started info [native] ASID, smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1 .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.0 info [native] zDC[Z208] SMO is initialized with size=10M. info [native] Registering the zdc
Check the job log of the zDC for any of the following error messages.
These do not appear in the zLocal log file, since the zLocal could not be started.
Invalid permission on the zLocal folder or libraries:
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied ZDC997D Notice: unknown modid, reason text might be incorrect ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort
Correct the permission on the zLocal folder and binaries, so they are at least readable and executable.
TCP/IP access denied:
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied ZDC997D Description: SAF CKACC returned error. ZDC996E @Abrtz/OS Unix agent attach failed,Abort
If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.
Invalid location specified for the zLocal:
ZDC996E Atx5!DTAGTCMD attach failed ZDC997D RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC member exists ZDC997D JRFileNotThere: The requested file does not exist ZDC992I IniA!z/OS Unix agent attach failed ZDC996E @Abrtz/OS Unix agent attach failed,Abort ZDC975E z/OS Unix agent attach failed,Abort
DTAGTCMD parameter in the SYSIN parameters to point to the correct zLocal binary location.
Browse the install library
<hlq> is the high-level qualifier of the product dataset, to find the
ZDCDTAGT load module for the zLocal.
In the header example below, you can see
1.nnn.00, which indicates this is the version of the load module without any maintenance. When service is applied, the field contains a subversion number like
1.nnn.01. The dates vary over time.
ZDCDTAGT 00000000 YYYYMMDD HH.MM VER 1.nnn.00 COPYRIGHT (C)...
ZDCDTAGT load module can produce diagnostic messages in the following format:
n is the log level and
i is the severity.
zDC user abends
A zDC may terminate with a user abend due to internal errors and initialization failures. All user abends, except for
U106, create a dump file. See the list of possible user abends below.
An internal error occurred due to zDC initialization failure.
zDC initialization failed and is unable to display the failure message.
ECSA storage is not available for the zDC.
A problem occurred during a RETRY operation from a recovery exit routine.
A queue task ended abnormally. This abend is accompanied by a
A queue task ended abnormally while shutdown was NOT in progress.
A fatal error occurred in TCPSP routine.
This abend might occur for various internal error reasons. It is accompanied by one of the following zDC error messages in the SYSLOG:
An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following error messages in the SYSLOG:
The zDC is unable to open SYSPRINT or SYSPRIN3 DD for displaying log messages.
An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following messages in the SYSLOG: