Customize the zDC subsystem
Learn how to customize the zDC subsystem depending on your needs.
By default the zDC starts in bootstrap mode. This means the zLocal receives automatic updates. On every start of the zDC, the bootstrap component
dtzagent queries Dynatrace for available updates. If an update is available, the zLocal will be automatically downloaded and updated.
If you don't want to receive automatic updates, you can run the zDC in non-bootstrap mode. In such a case, zLocal must be downloaded from Dynatrace. Complete the instructions to start the zDC and connect it to Dynatrace using the standard
dtzagent binary. The first time you connect to Dynatrace, don't use the
nobootstrap=true parameter. If you do use this parameter the first time, the zDC won't start successfully. The zRemote must be up when starting the zDC for the first time to avoid
The filename for the zLocal is
libdtzagent.so. When the zDC successfully connects to Dynatrace,
libdtzagent.so is copied to the
<installation_dir>/agent/downloads/appmon/native/7.2.m.bbbb/zos-s390-64 directory, where:
<installation_dir>: installation directory, by default
m: the release number.
bbbb: the build number.
SZDTSAMP dataset, the member
OCOPYAGT is a job that copies the
libdtzagent.so to the
/u/dt/agent/lib64 folder. When you uncomment or add the
nobootstrap=true parameter in the
ZDCSYSIN, the zLocal module
libdtzagent.so that was copied into the
lib64 folder is used.
The following are examples of log messages in the zDC job output. The first example is when the
nobootstrap=false parameter is used where a zDC is started in bootstrap mode. The second example is when
nobootstrap=true is used where a zDC is started in non-bootstrap mode .
info (native) The bootstrap channel connected successfully, requesting version: v.r.m.bbbb info (native) Interprocess lock acquired for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so info (native) Fetching agent binary succeeded for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so
info (native) Configured to not download module - loading local module info (native) Start loading local agent binary info (native) Successfully loaded agent binary /u/dt/agent/lib64/libdtzagent.so
Failover processing for zDC/zRemote
This is not load balancing.
While there are some limitations, you can have multiple zRemotes running and have the zDC switch to a defined active zRemote if a problem is encountered with the primary zRemote.
To do this, update your
zremoteagent parameter in your
ZDCSYSIN so that it includes a list of zRemote addresses.
ZDCSYSIN to turn off bootstrapping
The following limitations apply:
- Once the failover happens, it won't return to the primary until the zDC is restarted (or until all but the primary zRemote are stopped).
- Bootstrapping isn't supported when using failover processing. The zLocal writes the following error message when attempting to bootstrap during failover processing.
This limitation can, however, be circumvented when updates are available in the zRemote.dns
severe [native] Connection error (connect()/apr_sockaddr_info_get(), EDC9501I The name does not resolve for the supplied parameters.) while trying to bootstrap the zLocal.
- Update the
zremoteagent=parameter to point to the primary host. For example:
- Start the zDC. When the zDC starts with the above setting, it downloads the latest
- Copy the
Now the zDC environment is ready to be used in failover processing mode using the latest
- In ZDCSYSIN, set
- Specify the list of zRemote addresses in
- Restart the zDC.
The bootstrap component and/or zLocal creates a log directory and log files in its working directory. It defaults to the home directory specified in the user ID's RACF z/OS USS segment, so write permissions are required for location
<installation_dir>/log/dt.log. You can control permissions assigned to files created by the bootstrap component or zLocal (
libdtzagent.so) by setting
DT_UMASK. The default is
The zDC doesn't limit which user IDs can enter operator commands.
The zDC uses TCP/IP socket communication, so it may be necessary to update your security system's access rules if they are configured to deny TCP/IP access by default.
General SYSIN dataset format
Positions 1 through 71 of each statement can contain parameter data. Positions 72 through 80 are ignored and can optionally contain a sequence number.
Each parameter uses
KEYWORD(value) format. Each keyword and the requirements for the values associated with it are documented in the
ZDCSYSIN member of the sample dataset
zDC SYSIN parameters
All values that tailor the execution of the zDC reside in the dataset pointed to by DDNAME SYSIN.
The sample started task procedure
ZDCMEPC refers to the
ZDCSYSIN member of
SZDTSAMP as shown below:
//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
The dataset must contain 80 character records, blocked to a multiple of 80 bytes.
When you make a change to the SYSIN dataset, you must stop and restart the zDC for the change to take effect.
Commonly used parameters
The name of the zDC subsystem. Specify the desired subsystem name here if
The logging level produced by the zDC. Set this parameter to
The maximum amount of storage (MB) for messages that can be queued in the zDC shared memory object while awaiting writing to the OneAgent. The default of 1 MB should be adequate in most cases when Transaction Buffering is enabled. However, when Transaction Buffering is disabled (not recommended), set the SMO size to 10 MB if very high volumes of transactions are traced.
Changes the z/OS USS current directory where the bootstrap component (
Overrides the default number and size of transaction buffers. Transaction buffers offer better performance for CICS and IMS code modules by placing event messages related to each distributed trace into dedicated buffers. Rather than send individual event messages for each distributed trace as they occur, they are blocked into one or more buffers and sent together. Additionally, the buffer of related messages is processed more efficiently by the zRemote.
If a parameter value must span multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.
A comment is specified by placing an asterisk in position 1 of a statement. The entire statement is considered part of the comment.
A comment can be enclosed between a slash-asterisk and an asterisk-slash:
/* This is a comment. */
Multiple lines can be contained within comments of this type.
Running multiple zDC systems on a single z/OS image
A zDC connects one or more CICS or IMS code modules running on the same z/OS system to a single Dynatrace cluster, so it may be necessary to configure multiple zDC-started tasks on a single z/OS system when different application groups are being traced by different Dynatrace clusters. At times, it may be useful to configure an alternate zDC for testing purposes as well. See Install multiple instances of zDC/zRemote for step-by-step instructions on adding an additional zDC/zRemote pair.
Only one zDC on a particular z/OS system can specify
DEFAULT(YES) in its
SYSIN parameters. Additional zDCs fail to initialize unless they specify
See Adding SIT override INITPARM for non-default zDC sub-systems to configure the CICS code module to connect to a non-default zDC.
Check maintenance level
Browse install library
<hlq> is the High Level Qualifier of the product dataset.
Browse the beginning of the
ZDCDTAGT load module. You should see the header:
ZDCDTAGT 00000000 YYYYMMDD HH.MM VER 1.nnn.00 COPYRIGHT (C)...
In this example,
1.nnn.00 indicates this is the GA version of the module without any maintenance. When service is applied, the field contains a sub version number like
1.nnn.01. The dates vary over time.
ZDCDTAGT is the manager for the zLocal. It can produce diagnostic messages in the following format:
n is log level and
i is severity.
zDC commands supported by the z/OS modify command
The format of the z/OS modify command is as follows:
Modify <JobName or StcProcName>,DT1 <cmd>
Possible commands are listed in the table below.
Display Dynatrace subtask processor stats.
Force display (ZDC952) of utilization of the zLocal message queue.
Query log level.
Set log level to
Start the zLocal.
Append previously unreported lines from zLocal
Stop the zLocal.
Set the log level for zLocal to NN (FINEST,FINER,FINE,CONFIG,INFO,WARNING,SEVERE,DEBUG,NONE). No default.
zDC sample output from MODIFY MEPx,DT1 DISPLAY
The display skips most rows with zero counters if the log level is at or higher than the default level of 3. The log level can be changed for diagnostic purposes to greatly increase logging, including displaying the full set of counters using
MODIFY MEPx,DT1 LOG=1.
14:13:33.854126 ZDC027I MODIFY COMMAND RECEIVED 14:13:33.857174 ZDC028I DT1 DISPLAY 14:13:33-977432 ZDC994I ShoS Counters 35:Msgs sent from MAgent to zDC using unnamed pipe 1:QUEUE_MAX_EVENTS(EQH)High watermark elements used 128:EQH queue elements defined 128:EQH elements currently free 0:Internal Performance:XmPosts avoided 0:Internal Performance:XmPost Skip DupSrbs 1,758:zDC Elapsed time (approx sec) 0: Bytes read from DTM Msgs 49:SMO Msgs written 34:Internal:Redundant Post bypassed 15:Internal:Attempt Rd=Wr cursor for vStorage Perf. 15:z/OS Unix has no ready Msgs, waiting for work 48:z/OS Unix Msgs read 3,808: Bytes read from SMO Msgs 2,267:TBC Total Transactional Buffers 2,267:TBC number in Free queue 25:TBC Msgs written 25:TBC Msgs read 10:TBC TBCs written 10:TBC TBCs read 10:TBC Post bypass 41:TBC No Ready Q 1,905:TBC Bytes written 1,905:TBC Bytes read 2:TBC Age<=PingDelay*.5 2:TBC Age<=PingDelay*.75 2:TBC Age<=PingDelay*1 2:TBC Age<=PingDelay*1.25 2:TBC Age>=PingDelay*1.25 10,485,760:DTMSG_SMOSIZE Bytes allocated 9,248:DTMSG_SMOSIZE Bytes used 14:13:36-712702 ZDC995W DispTcb: 009D1438 0.900sec 14:13:36-712792 ZDC995W DispTcb: 009BBAD0 0.577sec 14:13:36-712832 ZDC995W DispTcb: 009BBCF0 0.013sec 14:13:36-712882 ZDC995W DispTcb: 009BBE88 0.012sec 14:13:36-714022 ZDC995W DispTcb: 009BE170 0.924sec 14:13:36-714062 ZDC995W DispTcb: 009BE390 0.590sec 14:13:36-714112 ZDC995W DispTcb: 009BE528 0.251sec
Rows that end with an exclamation mark (
!) should have zero counts.
Emergency zDC termination
The zDC makes every effort possible to release system resources (most especially common storage) when it terminates. This is true if the termination is normal due to a shutdown command or abnormal due to a cancel command or a program abend.
Recovery (ESTAEX) routines are always in effect to trap abnormal terminations and to free all system resources.
However, there may be a situation where zDC cannot be cancelled due to a z/OS anomaly or failure, which precludes the functioning of the cancel command.
If you are unable to communicate with zDC and want to terminate it, try the commands below in the specified order until the zDC terminates. Go on to step 2 only if step 1 is unsuccessful, go on to step 3 only if step 2 is unsuccessful, and so on. To avoid the need to manually clean up system resources using the
ZDCDELET process below, wait a short while between commands to permit dumps to be taken and system resources to be freed.
- Issue the
- Issue the
MODIFY jobname,SHUTDOWN IMMEDcommand.
- Issue the
- Issue the
Attempt to restart the zDC. If the restart is successful you can continue to run this job. If however the restart is not successful and zDC indicates that it cannot continue, you can execute the emergency zDC cleanup program named
ZDCDELET is a stand-alone job, which attempts to terminate a specified zDC job and release all system resources held by a zDC instance. The JCL to execute
ZDCDELET is shown below:
//* //* PLACE YOUR JOB STATEMENT HERE //* //DELETE EXEC PGM=ZDCDELET,PARM=xxxx //STEPLIB DD DISP=SHR,DSN=<hlq>.R1nnnxx.SZDTAUTH
PARM= on the EXEC statement must specify the 4-character subsystem ID of the zDC instance you wish to terminate.
User ABEND 100 and 101 may occur and messages describing the abends are written to the job log.
User ABEND 100 indicates that the
PARM= is not exactly 4 bytes long.
User ABEND 101 indicates that the subsystem ID specified on the
PARM= operand cannot be found in the system.
ZDC403 may be written to the job log as well. These can be found in the zDC Messages section below.
Refer to the z/OS code module messages page for a list of all zDC messages.
Multiple TCP/IP stacks
A z/OS system with multiple TCP/IP stacks presents an additional complication. In this situation, an additional step is needed in the zDC job. See
//MEPC PROC ... //STEP0 EXEC PGM=BPXTCAFF,PARM='TCPIP_stack_name' //* //ZDCMAIN EXEC PGM=ZDCMAIN,REGION=0M,PARM='LANGUAGE=EN' //STEPLIB DD DISP=SHR,DSN=hlq.SZDTAUTH //* Notes: //* SYSPRINT is required in all cases. //* SYSPRINT must be allocated to spool, not disk, on JES2 systems. //* SYSPRIN3 is required on JES3 systems only. //SYSPRINT DD SYSOUT=* //SYSPRIN3 DD SYSOUT=* //SYSIN DD DISP=SHR,DSN=hlq.SZDTSAMP(ZDCSYSIN)
BPXTCAFF program is used to associate a specific TCP/IP stack with the current address space. The PARM value is the job name used to start the desired TCP/IP stack.
zDC user abends
A zDC may terminate with a user abend due to internal errors and initialization failures. All user abends, except for
U106, create a dump file. See the list of possible user abends below.
An internal error occurred due to zDC initialization failure.
zDC initialization failed and is unable to display the failure message.
ECSA storage is not available for the zDC.
A problem occurred during a RETRY operation from a recovery exit routine.
A queue task ended abnormally. This abend is accompanied by a
A queue task ended abnormally while shutdown was NOT in progress.
A fatal error occurred in TCPSP routine.
This abend might occur for various internal error reasons. It is accompanied by one of the following zDC error messages in the SYSLOG:
An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following error messages in the SYSLOG:
The zDC is unable to open SYSPRINT or SYSPRIN3 DD for displaying log messages.
An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following messages in the SYSLOG: