Install the zDC subsystem

The z/OS Data Collection (zDC) subsystem interacts with the CICS, IMS, and z/OS Java modules via shared memory object (SMO) on a LPAR. The zDC subsystem maintains this SMO, and the modules write their monitoring data to it.

The zLocal (libdtzagent.so), hosted in the z/OS Unix System Services (USS) environment, runs as part of the zDC. It manages the TCP/IP socket connection to the zRemote module, reads monitoring data from SMO, and transfers these data to the zRemote.

The bootstrapper (dtzagent), hosted in the z/OS USS environment, runs as part of the zDC. It manages the update process of the zLocal.

When the zDC initializes (system startup or manual start), it starts the bootstrapper, and the bootstrapper starts the zLocal. When the zDC terminates, the zLocal and the bootstrapper are stopped.

System security

The zDC (zLocal) uses TCP/IP socket connections, so it may be necessary to update your security system's access rules if they are configured to deny TCP/IP access by default.

The zDC doesn't limit which user IDs can enter operator commands.

You can control permissions assigned to files created by the zLocal and bootstrapper by setting DT_UMASK. The default is umask(022).

Installation

The zDC runs as an authorized z/OS process (typically as a system task). This means the programs must reside in an authorized library. It is intended to start automatically as a started task at a system IPL. This gives the zDC continuous availability to collect monitoring data from modules.

The zDC can also be run as a batch job. The service class of the zDC must be high enough so that it is always available for keep-alive messages from the zRemote. The zDC should have a priority equal to or higher than the monitored CICS and IMS regions.

A zDC subsystem must be installed on each LPAR you want to monitor.

1. Download z/OS product datasets. Make a note of the high-level qualifier used in the download procedure.

2. Create the RACF user ID for the zDC process. This ID must have a z/OS USS segment. The recommended home directory name is /u/dt.

   If a home directory other than /u/dt is set, change in step 8 the path to the bootstrapper, which defaults to /u/dt/agent/lib64/dtzagent. The path names below the home directory can't be changed.

   The home directory or the high-level qualifier that replaces it must be writeable by the zDC process.

   • It writes the zLocal binary to /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so, where a.b.c.d is the version of the zLocal (i.e. 1.0.1.0).
   • It writes the zDC and zLocal logs to /u/dt/log/dtxxx.log.

   If write fails, the zDC processes use the directory /tmp/dynaTrace.

   If the zDC is run as a started task, the ID must be capable of representing the zDC started task and have access to the USS path above.

3. Change the dataset high-level qualifier represented by <hlq> in member COPYAGNT of SZDTSAMP to the value you used in the Download z/OS product datasets procedure.

4. Run the COPYAGNT job to create the required z/OS USS subdirectories under the home directory of the zDC started task.

5. Copy the dtzagent binary from SZDTSAMP and set it to be executable.

   If you're using the default home directory of /u/dt, the resulting path is /u/dt/agent/lib64/dtzagent.

   If a home directory other than /u/dt is used, the path in this job must be changed accordingly. File and directory names are case-sensitive. The path names below the home directory can't be changed.

   The COPYAGNT job should run under the user ID that was created for the zDC process, so that user ID owns the dtzagent binary. If this is inconvenient, you can also use the chown and chgrp commands to reset the owning user and group for the dtzagent binary after running COPYAGNT.

6. Authorize <hlq>.SZDTAUTH where <hlq> is the high-level qualifier value you used in the Download z/OS product datasets procedure.

   For example, create a member named SYS1.PARMLIB(PROGDT) containing:

   plaintextcopydownload
   APF FORMAT(DYNAMIC)
   APF ADD DSNAME(<hlq>.SZDTAUTH) VOLUME(XXXXXX)

   Then issue the console command:

   plaintextcopydownload
   SET PROG=DT

7. Copy the ZDCMEPC sample zDC started task PROC from SZDTSAMP to a system PROCLIB used for started tasks.

   The default name for the PROC is MEPC. Customize it as necessary for local standards. If a different subsystem name is chosen, change the SUBSYSTEM_ID() parameter in the SYSIN parameters to match.

   plaintextcopydownload
   //SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
   
   SUBSYSTEM_ID(MEPC)

   The PROC and the two JCL SYSIN members ZDCSYSIN and ZDCMEPCA that it uses contain dataset names that must be edited to replace <hlq>. with the appropriate high-level qualifiers.

8. Update the DTAGTCMD() parameter in the SYSIN parameter to reflect your definitions.

   plaintextcopydownload
   //SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
   
   DTAGTCMD(/u/dt/agent/lib64/dtzagent
   nobootstrap=false
   zremoteagent=<ipaddress>[:port])

   • /u/dt/agent/lib64/dtzagent sets the path to the bootstrapper. The default home directory name is /u/dt. If you have set a different home directory in step 2, change it accordingly.

   • nobootstrap=false enables the bootstrapper to automatically update the zLocal once a newer version is available. Per default, the zLocal receives automatic updates. Set true to disable automatic updates and manually update the zLocal.

   • zremoteagent=<ipaddress>[:port] set the IP address and port of a installed zRemote module. The IP address is required, the port is optional and defaults to 8898.

9. optional Add a command to your system startup to automatically start the zDC subsystem at IPL.

10. Use the ZDCIVP PROC in SZDTSAMP to verify the zDC installation and the zRemote connectivity.

    Good test for the zRemote connectivity returns Connection timeout, as the zRemote does not support FTP protocol. If a zRemote connectivity problem exist, it is indicated by EDC8128I Connection refused. This implies that the zRemote is not listening on the expected port.

SYSIN dataset

All values that tailor the execution of the zDC reside in the dataset pointed to by DDNAME SYSIN. The sample started task PROC ZDCMEPC refers to the JCL SYSIN member ZDCSYSIN of SZDTSAMP as shown below:

plaintextcopydownload
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)

The dataset must contain 80 character records, blocked to a multiple of 80 bytes.

SYSIN parameters

Positions 1 through 71 of each statement can contain parameter values. Positions 72 through 80 are ignored and can optionally contain a sequence number.

Each parameter uses KEYWORD(value) format. Each keyword and the requirements for the values associated with it are documented in the JCL SYSIN member ZDCSYSIN of SZDTSAMP.

If a parameter value spans multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.

plaintextcopydownload
/* This is a comment. */

Logging

The log level of the zDC is set with the DTLOGLEVEL() parameter in the SYSIN parameters. You can change this log level dynamically by issuing a z/OS MODIFY command.

plaintextcopydownload
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
DTLOGLEVEL(3)

MODIFY <jobName>,DT1 LOG=1

View log message output in the zDC job spool. The job spool helps determine errors that may occur during the zDC startup or when connecting with the zRemote. Once the zDC successfully connects to the zRemote, error messages from the zDC subsystem and the CICS and IMS modules are routed to the zRemote.

The log level of the zLocal is set with the DTAGTCMD(loglevel=INFO) parameter in the SYSIN parameters. You can change this log level dynamically by issuing a z/OS MODIFY command. The DTAGTCMD(name=zlocal_<lpar>) defines the name of the zLocal for log files. The name should reflect that this is a zLocal. It could include the SMF ID of the LPAR that it services.

plaintextcopydownload
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
DTAGTCMD(loglevel=INFO name=zlocal_<lpar>)

MODIFY <jobname>,DT1 ZLALOGLEVEL=FINE

There are two sets of logs created for the zLocal. Both are created in the OMVS file system. One set is a temporary set of logs only valid for the current execution of the zDC, and the location of these logs defaults to the home directory of the zDC user ID. You can override the location by using the DTCHDIR() parameter in the SYSIN parameters.

The zLocal also creates a standard set of logs, one for the bootstrapper, and one for the zLocal itself. These logs are in the standard Dynatrace log locations on the OMVS file system.

Update and maintenance

To update the zDC to a newer version, see the update and maintenance procedure in Download z/OS product datasets.

zLocal

By default, the zLocal receives automatic updates if DTAGTCMD(nobootstrap=false) is set in the SYSIN parameters. At every start of the zDC, the bootstrapper queries the zRemote module for available updates. The latest zLocal is installed as part of the zRemote installation. If an update is available, the zLocal will be automatically downloaded from the zRemote and updated.

plaintextcopydownload
info    (native) The bootstrap channel connected successfully, requesting version: a.b.c.d
info    (native) Interprocess lock acquired for /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so
info    (native) Fetching agent binary succeeded for /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so

Manually update the zLocal

Suppose you don't want to receive automatic updates for the zLocal, set DTAGTCMD(nobootstrap=true) in the SYSIN parameters. For the manual update process, the zLocal binary libdtzagent.so must be present on the LPAR in the /u/dt/agent/lib64 directory.

To accomplish this

1. Run the bootstrapper dtzagent at least once in either a production LPAR or non-production LPAR.
2. Copy the zLocal binary to the target LPAR. In the SZDTSAMP dataset, the member OCOPYAGT is a job that copies the zLocal binary libdtzagent.so to the /u/dt/agent/lib64 directory.
3. Complete the instructions to start the zDC and connect it to your Dynatrace environment.
4. If the zDC successfully connects to Dynatrace, libdtzagent.so is copied to the /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so directory, where a.b.c.d is the version of the zLocal (i.e. 1.0.1.0)

plaintextcopydownload
info    (native) Configured to not download module - loading local module
info    (native) Start loading local agent binary
info    (native) Successfully loaded agent binary /u/dt/agent/lib64/libdtzagent.so

Troubleshooting

plaintextcopydownload
ZDC000I INITIALIZATION STARTED FOR zDC  VER 1.195.00
ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.00
ZDC053I LPAR NAME IBMSYS1    CVTSNAME S0W1
.
.
.
.
ZDC955L Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021
ZDC958L Dynatrace INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z021
ZDC993I Opn1RFD:0008  /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log
ZDC955I Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021

plaintextcopydownload
11:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE

plaintextcopydownload
H408I USER(xxxxxxxx) GROUP(xxxx) NAME(STARTED TASK )
BPX.FILEATTR.PROGCTL CL(FACILITY)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

plaintextcopydownload
JJJJ-MM-DD HH:MM:SS  3f8bbe02¨ info  native¨ Server requests us to use Agent
dTMajor.dTMinorVersion.0.dTBuild with a hash of 0317af199c1ab1a03dda2cee90c2ea61
JJJJ-MM-DD HH:MM:SS  3f8bbe02¨ info  native¨ Requesting Agent library from Server
JJJJ-MM-DD HH:MM:SS  3f8bbe02¨ info  native¨ Error setting Agent library program
controlled: EDC5139I Operation not permitted.
JJJJ-MM-DD HH:MM:SS  3f8bbe02¨ info  native¨ Loading Agent
/dt/dynatrace-<dTMajor.dTMinorVersion.0>/agent/downloads/<dTMajor.dTMinorVersion.0.dTBuild>
/native/zos-s390-64/libdtzagent.so

plaintextcopydownload
info    [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake.
info    [native] dynaTrace z Remote Agent client handshake performed.
info    [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb].
info    [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake
info    [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.

plaintextcopydownload
MODIFY <zdc-jobname>,DT1 STDO

plaintextcopydownload
info    [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb].
info    [native] Data client socket listener thread started
info    [native] ASID[48], smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1    .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.0
info    [native] zDC[Z208] SMO is initialized with size=10M.
info    [native] Registering the zdc[48]

plaintextcopydownload
ZDC996E      Atx5!DTAGTCMD attach failed
ZDC997D      RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied
ZDC997D      Notice: unknown modid, reason text might be incorrect
ZDC992I      IniA!z/OS Unix agent attach failed
ZDC996E      @Abrtz/OS Unix agent attach failed,Abort

plaintextcopydownload
ZDC996E      Atx5!DTAGTCMD attach failed
ZDC997D      RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied
ZDC997D      Description: SAF CKACC returned error.
ZDC996E      @Abrtz/OS Unix agent attach failed,Abort

plaintextcopydownload
ZDC996E      Atx5!DTAGTCMD attach failed
ZDC997D      RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC
             member exists
ZDC997D      JRFileNotThere: The requested file does not exist
ZDC992I      IniA!z/OS Unix agent attach failed
ZDC996E      @Abrtz/OS Unix agent attach failed,Abort
ZDC975E      z/OS Unix agent attach failed,Abort

plaintextcopydownload
ZDCDTAGT 00000000 YYYYMMDD HH.MM VER 1.nnn.00 COPYRIGHT (C)...

plaintextcopydownload
ZDC99<n><i>

zDC user abends

A zDC may terminate with a user abend due to internal errors and initialization failures. All user abends, except for U103 and U106, create a dump file. See the list of possible user abends below.