The z/OS Data Collection subsystem (zDC) controls communication between the zLocal and CICS and IMS code modules of OneAgent. You can enter Operator start, stop, and modify commands from a system console to control the zDC and the lifecycle of the zLocal.
Installing the zDC
The zDC code module of OneAgent runs as an authorized z/OS process (typically as a system task). This means the programs must reside in an authorized library. It is intended to start automatically as a started task (STC) at a system IPL. This gives the zDC continuously availability to collect PurePath data from CICS and IMS code modules of OneAgent and forward it to Dynatrace.
The zDC can also be run as a batch job. The service class of the zDC must be high enough so that it is always available for keep-alive messages from the zRemote. The zDC should have a priority equal to or higher than the CICS application regions and the IMS dependent regions.
Download the OneAgent to the mainframe. Make a note of the high-level qualifier used in the download procedure. The sample high-level qualifier of
DT.DYNA is used in this procedure. Substitute your value as needed.
Create the RACF user ID for the zDC process.
This ID must have a z/OS Unix segment. The recommended home directory name is
/u/dt. If a directory other than
/u/dt is used, update the DTAGTCMD argument in the zDC parameter file (the SYSIN DD of the proc) to specify the correct path to the
dtzagent binary, which defaults to
/u/dt/agent/lib64/dtzagent. Only the
/u/dt/ part can be changed, because the path to the OneAgent binary must remain
agent/lib64/dtzagent , regardless of the high-level qualifier used for this directory path. The
/u/dt directory or the high-level qualifier that replaces it must be writeable by the zDC process. This is used for writing to /u/dt/agent/download/appmon/native/v.r.m.build to store the zLocal and to store the zDC and zLocal logs in /u/dt/log/dtxxx.log. If read/write fails folder /tmp/dynaTrace is used.
If the zDC is run as a started task, the ID must be capable of representing the zDC started task. Give the ID the same name as the zDC started task and add it to the access list of the RACF class STARTED with at least READ authority.
Change the dataset high-level qualifier represented by <hlq> in member
SZDTSAMP to the value you used in the SMP/E install of the OneAgent for z/OS procedure.
Run the COPYAGNT job to create the required z/OS Unix subdirectories under the home directory of the zDC started task.
Copy the z/OS zLocal binary file from
SZDTSAMP and set it to be executable.
If you're using the default home directory of
/u/dt/, the resulting file is
If the home directory has been changed, the paths in this job must be changed accordingly. File and directory names for z/OS zLocal are case-sensitive. The path names below the home directory cannot be changed.
COPYAGNT job should run under the user ID that was created for the zDC process, so that user ID owns the
dtzagent file. If this is inconvenient, you can also use the
chgrp commands to reset the owning user and group for the
dtzagent file after running
<hlq> is the high-level qualifier value you used in the SMP/E install of the OneAgent for z/OS procedure. If you are also planning on using the OneAgent IMS, you must also authorize the
For example, create a member named
APF FORMAT(DYNAMIC) APF ADD DSNAME(<hlq>.SZDTAUTH) VOLUME(XXXXXX)
Then issue the console command:
Copy the ZDCMEPC sample zDC started task procedure from SZDTSAMP to a system proclib used for started tasks.
The default name for the procedure is MEPC. Customize it as necessary for local standards. If a different subsystem name is chosen, change the SUBSYSTEM_ID() in the zDC SYSIN parameters to match. The proc and the two sysin members ZDCSYSIN and ZDCMEPCA that it uses contain dataset names that must be edited to replace
<hlq>. with the appropriate high-level qualifiers.
Update the zDC startup parameters that are supplied through the SYSIN DD of the zDC proc, which defaults to the
ZDCSYSIN member of
DTAGTCMD(/u/dt/agent/lib64/dtzagent): Defines the path to the OneAgent, if the default was not used.
nobootstrap=false: Defines the usage of the bootstrap OneAgent. Set
falseto use it or
trueto not use the bootstrap OneAgent. If not set, the default is to use the bootstrap OneAgent.
zremoteagent=<host>[:port]: Defines the IP address of the zRemote. The zRemote is a required parameter. The port is optional and defaults to
See the Fail over processing for zDC/zRemote section for additional information on using the fail over feature.
See the Non-bootstrap OneAgent section for additional information on using the non-bootstrap OneAgent.
See zRemote Agent for additional information on how to set up the zRemote.
Optional but recommended: Add a command to your system startup to automatically start the zDC subsystem at IPL.
zDC reference information
The following sections describe the basic operation and configuration parameters for the zDC.
zDC code module
The zDC manages the lifecycle of the zLocal and bootstrap code modules that run in the z/OS Unix environment:
- It starts the z/OS code module when the zDC initializes at system startup or is manually started.
- It terminates the code module when the zDC is stopped.
The filename of the code module is
COPYAGNT job creates the z/OS Unix file containing this program.
Non-bootstrap code module
By default the z/OS code module of the OneAgent uses the bootstrap mode. It means the code modules receives automatic updates. On every start of the zDC the code module queries Dynatrace for available update. If an update is available the code module automatically downloads it and updates itself.
If you don't want to receive automatic updates, you can run the z/OS code module in non-bootstrap mode. In that case it must be downloaded from Dynatrace. Complete the instructions to start the zDC and connect it to Dynatrace using the standard
dtzagent binary. The first time you connect to Dynatrace, do not use the
nobootstrap=true parameter. If you do use this parameter the first time, the zDC does not start successfully. You can comment it out for this first execution.
The filename for the non-bootstrap code module is
libdtzagent.so. When the zDC successfully connects to Dynatrace it is copied to the
<installation_dir>/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64 directory, where:
<installation_dir>: installation directory, by default
v.r.m: the release number.
bbbb: the build number.
SZDTSAMP dataset, the member
OCOPYAGT is a job that copies the
libdtzagent.so to the
/u/dt/agent/lib64 folder. When you uncomment or add the
nobootstrap=true parameter in the
ZDCSYSIN, the non-bootstrap code module that was copied into the
lib64 folder is used.
The following are examples of log messages in the zDC job output. The first example is with a bootstrap code module. The second example is from a zDC using a non-bootstrap code module.
2014-11-13 19:04:31 [009cb288] info [native] Server/Collector requests us to use Agent v.r.m.bbbb with a hash of 67c137cc48e340da68e1062f58996e95 2014-11-13 19:04:32 [009cb288] info [native] Loading Agent /u/dt/agent/downloads/v.r.m.bbbb/native/zos-s390-64/libdtzagent.so
2014-11-14 23:40:18 [009cb288] info [native] Configured to not download module - loading local module 2014-11-14 23:40:18 [009cb288] info [native] Loading Agent /u/dt/agent/lib64/libdtzagent.so
Failover processing for zDC/zRemote
This is not load balancing.
If you wish to have more than one zRemote running and zDC to switch to a defined active zRemote, if there are problems with the primary, then this can be accomplished with some limitations.
In your ZDCSYSIN you can update your
zremoteagent parameter to have a list of zRemote addresses.
nobootstrap=true in ZDCSYSIN to turn off bootstrapping
Limitations on this are as follows:
- Once the failover happens, it won't return to the primary until zDC is restarted or all but the primary zRemotes are stopped.
- Bootstrapping isn't supported when using failover processing. The zLocal writes the following error message when attempting to bootstrap during failover processing.
This limitation can however be circumvented as follows when there are updates available in the zRemote.
severe [native] Connection error (connect()/apr_sockaddr_info_get(), EDC9501I The name does not resolve for the supplied parameters.) while trying to bootstrap the zLocal.
- Update the
zremoteagent=parameter to point to the primary host. For example:
- Start the zDC. When the zDC starts with the above setting, it downloads the latest
- Copy the latest
Now the zDC environment is ready to be used in failover processing mode using the latest
- In ZDCSYSIN, set
- Specify the list of zRemote addresses in
- Restart the zDC.
The code module creates a log directory and log files in its working directory. It defaults to the home directory specified in the user ID's RACF z/OS Unix segment, so write permissions are required for this location
/u/dt/log/dt.log. You can control permissions assigned to files created by code module (
libdtzagent.so) by setting
DT_UMASK. The default is
zDC doesn't limit which user IDs can enter operator commands.
The zDC uses TCP/IP sockets communication, so it may be necessary to update your security system's access rules if they're configured to deny TCP/IP access by default.
General SYSIN dataset format
Positions 1 through 71 of each statement can contain parameter data. Positions 72 through 80 are ignored and can contain a sequence number, but are not required to do so.
Each parameter takes the form
KEYWORD(value). Each keyword and the requirements for the values associated with it are documented in the ZDCSYSIN member of the sample dataset SZDTSAMP.
zDC SYSIN parameters
All values that tailor the execution of the zDC reside in the dataset pointed to by DDNAME SYSIN.
The sample started task procedure (ZDCMEPC) refers to the ZDCSYSIN member of SZDTSAMP as shown below:
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
The dataset must contain 80 character records, blocked to a multiple of 80 bytes.
When you make a change to the SYSIN dataset, you must stop and restart the zDC for the change to take effect.
Commonly used parameters
||The name of the zDC subsystem. Specify the desired subsystem name here if
||The logging level produced by the zDC. Set this parameter to
||A number that defines the maximum amount of storage (in Mbytes) for messages that can be queued in the zDC shared memory object pending their writing to the OneAgent. The default is 10 MB. This default value may need to be increased if very high volumes of transactions are traced.|
||Used to change the z/OS Unix current directory where the code module (
||Requests the zLocal code module to offload OneAgent message data processing and transmission to a zIIP CPU if available.
||Overrides the default number and size of transaction buffers. Transaction buffers offer better performance for CICS and IMS agents by placing event messages related to each PurePath into dedicated buffers. Rather than send individual event messages for each PurePath as they occur, they are blocked into one or more buffers and sent together. Additionally, the buffer of related messages is processed more efficiently by the zRemote . The
If a parameter value must span multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.
A comment is specified by placing an asterisk in position 1 of a statement. The entire statement is considered part of the comment.
A comment can be enclosed between a slash-asterisk and an asterisk-slash:
/* This is a comment. */
Multiple lines can be contained within comments of this type.
Running multiple zDC systems on a single z/OS image
A zDC connects one or more CICS or IMS code modules running on the same z/OS system to a single Dynatrace cluster, so it may be necessary to configure multiple zDC-started tasks on a single z/OS system when different application groups are being traced by different Dynatrace clusters. At times, it may be useful to configure an alternate zDC for testing purposes as well. See Deploy OneAgent on multiple instances of zDC/zRemote for step-by-step instructions on adding an additional zDC/zRemote pair.
Only one zDC on a particular z/OS system can specify DEFAULT(YES) in its SYSIN parameters. Additional zDCs fail to initialize unless they specify DEFAULT(NO).
See Adding SIT override INITPARM for Non-Default ZDC sub-systems to configure a CICS Agent to connect to a non-default zDC.
Check maintenance level
Browse install library for
<hlq> is the High Level Qualifier of the SMPE target zone.
Browse the beginning of the ZDCDTAGT load module. You should see the header:
ZDCDTAGT 00BASE YYYYMMDD HH.MM VER nn.nn.00 COPYRIGHT (C)...
In this example, 00BASE indicates this is the GA version of the module without any maintenance. When service is applied, the field contains a problem number and release numbers; and dates vary over time.
ZDCDTAGT is the manager for the zLocal code module. It can produce diagnostic messages in the following format:
ZDC99<n><i> messages where n=loglevel and i=severity
zDC commands supported by the z/OS modify command
The fomat of the z/OS modify command is the following:
Modify <JobName or StcProcName>,DT1 <cmd>
Possbile commands are listed in the table below.
|DISPLAY||Display OneAgent subtask processor stats.|
|DTM%V||Force display (ZDC952) of utilization of the zLocal code module message queue.|
|LOG=n||Set loglevel to
|START||Start zLocal code module.|
|STDO||Append previously unreported lines from zLocal code module
|STOP||Stop zLocal code module.|
|ZLALOGLEVEL=NN||Set zLocal code module loglevel to NN (FINEST,FINER,FINE,CONFIG,INFO,WARNING,SEVERE,DEBUG,NONE). No default.|
zDC sample output from MODIFY MEPx,DT1 DISPLAY
The display skips most rows with zero counters if the log level is at or higher than the default level of 3. The log level can be changed for diagnostic purposes to greatly increase logging, including displaying the full set of counters
using MODIFY MEPx,DT1 LOG=1):
14:13:33.854126 ZDC027I MODIFY COMMAND RECEIVED 14:13:33.857174 ZDC028I DT1 DISPLAY 14:13:33-977432 ZDC994I ShoS Counters 35:Msgs sent from MAgent to zDC using unnamed pipe 1:QUEUE_MAX_EVENTS(EQH)High watermark elements used 128:EQH queue elements defined 128:EQH elements currently free 0:Internal Performance:XmPosts avoided 0:Internal Performance:XmPost Skip DupSrbs 1,758:zDC Elapsed time (approx sec) 0: Bytes read from DTM Msgs 49:SMO Msgs written 34:Internal:Redundant Post bypassed 15:Internal:Attempt Rd=Wr cursor for vStorage Perf. 15:z/OS Unix has no ready Msgs, waiting for work 48:z/OS Unix Msgs read 3,808: Bytes read from SMO Msgs 2,267:TBC Total Transactional Buffers 2,267:TBC number in Free queue 25:TBC Msgs written 25:TBC Msgs read 10:TBC TBCs written 10:TBC TBCs read 10:TBC Post bypass 41:TBC No Ready Q 1,905:TBC Bytes written 1,905:TBC Bytes read 2:TBC Age<=PingDelay*.5 2:TBC Age<=PingDelay*.75 2:TBC Age<=PingDelay*1 2:TBC Age<=PingDelay*1.25 2:TBC Age>=PingDelay*1.25 10,485,760:DTMSG_SMOSIZE Bytes allocated 9,248:DTMSG_SMOSIZE Bytes used 14:13:36-712702 ZDC995W DispTcb: 009D1438 0.900sec 14:13:36-712792 ZDC995W DispTcb: 009BBAD0 0.577sec 14:13:36-712832 ZDC995W DispTcb: 009BBCF0 0.013sec 14:13:36-712882 ZDC995W DispTcb: 009BBE88 0.012sec 14:13:36-714022 ZDC995W DispTcb: 009BE170 0.924sec 14:13:36-714062 ZDC995W DispTcb: 009BE390 0.590sec 14:13:36-714112 ZDC995W DispTcb: 009BE528 0.251sec
Rows that end with an exclamation mark (
!) should have zero counts.
Emergency zDC termination
zDC makes every effort possible to release system resources (most especially common storage) when it terminates. This is true if the termination is normal due to a shutdown command or abnormal due to a cancel command or a program abend.
Recovery (ESTAEX) routines are always in effect to trap abnormal terminations and to free all system resources.
However, there may be a situation where zDC cannot be cancelled due to a z/OS anomaly or failure, which precludes the functioning of the cancel command.
If you are unable to communicate with zDC and want to terminate it, try the commands below in the specified order until the zDC terminates. Go on to step 2 only if step 1 is unsuccessful, go on to step 3 only if step 2 is unsuccessful, and so on. To avoid the need to manually clean up system resources using the ZDCDELET process below, wait a short while between commands to permit dumps to be taken and system resources to be freed.
- Issue the
- Issue the
MODIFY jobname,SHUTDOWN IMMEDcommand.
- Issue the
- Issue the
Attempt to restart the zDC. If the restart is successful you can continue to run this job. If however the restart is not successful and zDC indicates that it cannot continue, you can execute the emergency zDC cleanup program named ZDCDELET.
ZDCDELET ZDCDELET is a stand-alone job executed by JCL, which attempts to terminate a specified ZDC job and release all system resources held by an ZDC instance. The JCL to execute ZDCDELET is shown below: //* //* PLACE YOUR JOB STATEMENT HERE //* //DELETE EXEC PGM=ZDCDELET,PARM=xxxx //STEPLIB DD DISP=SHR,DSN=<hlq>.LZDT630.SZDTAUTH The PARM= on the EXEC statement must specify the 4-character subsystem ID of the ZDC instance you wish to terminate. User ABEND 100 and 101 may occur and messages describing the abends are written to the joblog. User ABEND 100 indicates that the PARM= is not exactly 4 bytes long. User ABEND 101 indicates that the subsystem ID specified on the PARM= operand cannot be found in the system.
Messages ZDC400 through ZDC403 may be written to the jobl og as well. These can be found in the ZDC Messages section below.
Refer to the z/OS Agent Messages page for a list of all zDC messages.
Multiple TCP/IP stacks
A z/OS system with multiple TCP/IP stacks presents an additional complication. In this situation, an additional step is needed in the server job. See SZDTSAMP member ZDCMEPC.
//MEPC PROC ... //STEP0 EXEC PGM=BPXTCAFF,PARM='TCPIP_stack_name' //* //ZDCMAIN EXEC PGM=ZDCMAIN,REGION=0M,PARM='LANGUAGE=EN' //STEPLIB DD DISP=SHR,DSN=hlq.SZDTAUTH //* Notes: //* SYSPRINT is required in all cases. //* SYSPRINT must be allocated to spool, not disk, on JES2 systems. //* SYSPRIN3 is required on JES3 systems only. //SYSPRINT DD SYSOUT=* //SYSPRIN3 DD SYSOUT=* //SYSIN DD DISP=SHR,DSN=hlq.SZDTSAMP(ZDCSYSIN)
The BPXTCAFF program is used to associate a specific TCP/IP stack with the current address space. The PARM value is the job name used to start the desired TCP/IP stack.