zDC reference

This page provides the basic operation and configuration parameters for the zDC.

zDC basic operation

The zDC manages the lifecycle of the zLocal and the bootstrap components that run in the z/OS Unix environment.

  • When the zDC initializes (system startup or manual started), it starts the bootstrap component dtzagent and the zLocal.
  • When the zDC terminates, the bootstrap component dtzagent and the zLocal are stopped.

The COPYAGNT job in SZDTSAMP creates the z/OS Unix file dtzagent.

Non-bootstrap mode

By default the zDC starts in bootstrap mode. This means the zLocal receives automatic updates. On every start of the zDC, the bootstrap component dtzagent queries Dynatrace for available updates. If an update is available, the zLocal will be automatically downloaded and updated.

If you don't want to receive automatic updates, you can run the zDC in non-bootstrap mode. In such a case, zLocal must be downloaded from Dynatrace. Complete the instructions to start the zDC and connect it to Dynatrace using the standard dtzagent binary. The first time you connect to Dynatrace, don't use the nobootstrap=true parameter. If you do use this parameter the first time, the zDC won't start successfully. The zRemote must be up when starting the zDC for the first time to avoid U103 abend.

The filename for the zLocal is libdtzagent.so. When the zDC successfully connects to Dynatrace, libdtzagent.so is copied to the <installation_dir>/agent/downloads/appmon/native/7.2.m.bbbb/zos-s390-64 directory, where:

  • <installation_dir>: installation directory, by default /u/dt.
  • m: the release number.
  • bbbb: the build number.

In the SZDTSAMP dataset, the member OCOPYAGT is a job that copies the libdtzagent.so to the /u/dt/agent/lib64 folder. When you uncomment or add the nobootstrap=true parameter in the ZDCSYSIN, the zLocal module libdtzagent.so that was copied into the lib64 folder is used.

The following are examples of log messages in the zDC job output. The first example is when the nobootstrap=false parameter is used where a zDC is started in bootstrap mode. The second example is when nobootstrap=true is used where a zDC is started in non-bootstrap mode .

info    (native) The bootstrap channel connected successfully, requesting version: v.r.m.bbbb
info    (native) Interprocess lock acquired for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so
info    (native) Fetching agent binary succeeded for /u/dt/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64/libdtzagent.so

Failover processing for zDC/zRemote

Disclaimer

This is not load balancing.

While there are some limitations, you can have multiple zRemotes running and have the zDC switch to a defined active zRemote if a problem is encountered with the primary zRemote.

To do this, update your zremoteagent parameter in your ZDCSYSIN so that it includes a list of zRemote addresses.

zremoteagent=<host>[:port];<host>[:port];<host>[:port];<host>[:port]

Set nobootstrap=true in ZDCSYSIN to turn off bootstrapping

nobootstrap=true

Limitations on this are as follows:

  • Once the failover happens, it won't return to the primary until zDC is restarted (or until all but the primary zRemote are stopped).
  • Bootstrapping isn't supported when using failover processing. The zLocal writes the following error message when attempting to bootstrap during failover processing.
    severe  [native] Connection error (connect()/apr_sockaddr_info_get(), EDC9501I The name does not resolve for the supplied parameters.) while trying to bootstrap the zLocal.
    
    This limitation can however be circumvented as follows when there are updates available in the zRemote.
    1. In ZDCSYSIN, set nobootstrap=false for bootstrapping.
    2. Update the zremoteagent= parameter to point to the primary host. For example: zremoteagent=<host>[:port].
    3. Start the zDC. When the zDC starts with the above setting, it downloads the latest libdtzagent.so to <installation_dir>/agent/downloads/appmon/native/v.r.m.bbbb/zos-s390-64.
    4. Copy the libdtzagent.so to location <installation_dir>/agent/lib64.
      Now the zDC environment is ready to be used in failover processing mode using the latest libdtzagent.so module.
    5. In ZDCSYSIN, set nobootstrap=true.
    6. Specify the list of zRemote addresses in zremoteagent=.
    7. Restart the zDC.

System security

The bootstrap component and/or zLocal creates a log directory and log files in its working directory. It defaults to the home directory specified in the user ID's RACF z/OS Unix segment, so write permissions are required for this location <installation_dir>/log/dt.log. You can control permissions assigned to files created by the bootstrap component or zLocal (dtzagent or libdtzagent.so) by setting DT_UMASK. The default is umask(022).

The zDC doesn't limit which user IDs can enter operator commands.

The zDC uses TCP/IP socket communication, so it may be necessary to update your security system's access rules if they are configured to deny TCP/IP access by default.

General SYSIN dataset format

Positions 1 through 71 of each statement can contain parameter data. Positions 72 through 80 are ignored and can contain a sequence number, but are not required to do so.

Each parameter takes the form KEYWORD(value). Each keyword and the requirements for the values associated with it are documented in the ZDCSYSIN member of the sample dataset SZDTSAMP.

zDC SYSIN parameters

All values that tailor the execution of the zDC reside in the dataset pointed to by DDNAME SYSIN.

The sample started task procedure ZDCMEPC refers to the ZDCSYSIN member of SZDTSAMP as shown below:

//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)

The dataset must contain 80 character records, blocked to a multiple of 80 bytes.

When you make a change to the SYSIN dataset, you must stop and restart the zDC for the change to take effect.

Commonly used parameters

Parameter(default value) Description
DTAGTCMD(<installation_dir>/agent/lib64/dtzagent) Additional values:
  • nobootstrap=true: Set it to start the zDC in non-bootstrap mode using zLocal module libdtzagent.so. If the nobootstrap=true value is not used, the bootstrap component is loaded and attached by the zDC.
  • name=<CM name>: The name of the zLocal to appear in log files. This name should be reflective of the fact that this is a zLocal module, and it could include the SMF id of the LPAR that it services. For example, name=zlocal_<lpar>should be different than the corresponding zRemote, so that the log files can be easily distinguished.
  • [zremoteagent=ipaddress:port]): The port parameter is optional and defaults to 8898. See zRemote for information on how to register the Windows service or start the Linux/Unix daemon with the right zdclistenerport, name, and config parameters.
  • loglevel=info: The loglevel used by the zLocal.
SUBSYSTEM_ID(MEPC) The name of the zDC subsystem. Specify the desired subsystem name here if MEPC is not appropriate.
DTLOGLEVEL(3) The logging level produced by the zDC. Set this parameter to 4 to suppress informational messages. It should not be set to less than 3 unless requested by Customer Support. Values less than 3 are used for diagnostic debugging.
DTMSG_SMOSIZE(1) A number that defines the maximum amount of storage (in Mbytes) for messages that can be queued in the zDC shared memory object pending their writing to the OneAgent. The default is 1 MB. This default should be adequate in most cases when Transaction Buffering is enabled. However when Transaction Buffering is disabled (not recommended), set the SMO size to 10 MB if very high volumes of transactions are traced.
DTCHDIR(/u/dt) Used to change the z/OS Unix current directory where the bootstrap component (dtzagent) creates temporary files for stdin, stdout, and stderr. It defaults to the home directory of the associated user ID.
ZIIP_ENABLE(YES) Requests the zLocal to offload OneAgent message data processing and transmission to a zIIP CPU if available. ZIIP_ENABLE(YES) makes all CPU time consumed by the zDC to process OneAgent data eligible to run on a zIIP assist processor. The relatively small amount of background activity that occurs when the zDC is idle is not affected by this option and continues to run on a general CPU.
DTMSG_TRANBUFSIZE(n,m) Overrides the default number and size of transaction buffers. Transaction buffers offer better performance for CICS and IMS OneAgents by placing event messages related to each PurePath into dedicated buffers. Rather than send individual event messages for each PurePath as they occur, they are blocked into one or more buffers and sent together. Additionally, the buffer of related messages is processed more efficiently by the zRemote . The n parameter is the number of buffers in thousands. For example, 2 = 2000 buffers. A zero value disables transaction buffers. The m parameter must be either 2 or 4 to specify 2K or 4K buffer size. We recommend the 4K buffer size, unless storage consumption is a significant concern. The minimum non-disabled value is 1,2. The maximum is 126,4 or 248,2 corresponding to total size of 512 Megabytes. An upper bound for the number of buffers that may be required is one for each IMS message region and MAXTASK times the number of CICS regions, but actual requirements are likely to be considerably smaller.

Continuation lines

If a parameter value must span multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.

Comments

A comment is specified by placing an asterisk in position 1 of a statement. The entire statement is considered part of the comment.

A comment can be enclosed between a slash-asterisk and an asterisk-slash:

/* This is a comment. */

Multiple lines can be contained within comments of this type.

Running multiple zDC systems on a single z/OS image

A zDC connects one or more CICS or IMS code modules running on the same z/OS system to a single Dynatrace cluster, so it may be necessary to configure multiple zDC-started tasks on a single z/OS system when different application groups are being traced by different Dynatrace clusters. At times, it may be useful to configure an alternate zDC for testing purposes as well. See Install multiple instances of zDC/zRemote for step-by-step instructions on adding an additional zDC/zRemote pair.

Only one zDC on a particular z/OS system can specify DEFAULT(YES) in its SYSIN parameters. Additional zDCs fail to initialize unless they specify DEFAULT(NO).

See adding SIT override INITPARM for non-default zDC sub-systems to configure the OneAgent for CICS to connect to a non-default zDC.

Check maintenance level

Browse install library <hlq>.R1nnnxx.SZDTAUTH, where <hlq> is the High Level Qualifier of the product dataset.

Browse the beginning of the ZDCDTAGT load module. You should see the header:

ZDCDTAGT 00000000 YYYYMMDD HH.MM VER 1.nnn.00 COPYRIGHT (C)...

In this example, 1.nnn.00 indicates this is the GA version of the module without any maintenance. When service is applied, the field contains a sub version number like 1.nnn.01; dates vary over time.

ZDCDTAGT

ZDCDTAGT is the manager for the zLocal. It can produce diagnostic messages in the following format:

ZDC99<n><i> messages where n=loglevel and i=severity

zDC commands supported by the z/OS modify command

The fomat of the z/OS modify command is the following:

Modify <JobName or StcProcName>,DT1 <cmd>

Possbile commands are listed in the table below.

Command Description
DISPLAY Display OneAgent subtask processor stats.
DTM%V Force display (ZDC952) of utilization of the zLocal message queue.
LOG=? Query loglevel.
LOG=n Set loglevel to n (0-8) where lower numbers increase the level of detail. Default is 3.
START Start the zLocal.
STDO Append previously unreported lines from zLocal stdo.log to SYSPRINT.
STOP Stop the zLocal.
ZLALOGLEVEL=NN Set the loglevel for zLocal to NN (FINEST,FINER,FINE,CONFIG,INFO,WARNING,SEVERE,DEBUG,NONE). No default.

zDC sample output from MODIFY MEPx,DT1 DISPLAY

The display skips most rows with zero counters if the log level is at or higher than the default level of 3. The log level can be changed for diagnostic purposes to greatly increase logging, including displaying the full set of counters using MODIFY MEPx,DT1 LOG=1.

14:13:33.854126 ZDC027I MODIFY COMMAND RECEIVED                         
14:13:33.857174 ZDC028I DT1 DISPLAY                                     
14:13:33-977432 ZDC994I ShoS Counters                                   
                  35:Msgs sent from MAgent to zDC using unnamed pipe      
                   1:QUEUE_MAX_EVENTS(EQH)High watermark elements used  
                 128:EQH queue elements defined                         
                 128:EQH elements currently free                        
                   0:Internal Performance:XmPosts avoided               
                   0:Internal Performance:XmPost Skip DupSrbs           
               1,758:zDC Elapsed time (approx sec)                      
                   0: Bytes read from DTM Msgs                          
                  49:SMO Msgs written                                   
                  34:Internal:Redundant Post bypassed                   
                  15:Internal:Attempt Rd=Wr cursor for vStorage Perf.   
                  15:z/OS Unix has no ready Msgs, waiting for work      
                  48:z/OS Unix Msgs read                                
               3,808: Bytes read from SMO Msgs      
               2,267:TBC Total Transactional Buffers   
               2,267:TBC number in Free queue          
                  25:TBC Msgs written                  
                  25:TBC Msgs read                     
                  10:TBC TBCs written                  
                  10:TBC TBCs read                     
                  10:TBC Post bypass                   
                  41:TBC No Ready Q                    
               1,905:TBC Bytes written                 
               1,905:TBC Bytes read                    
                   2:TBC Age<=PingDelay*.5             
                   2:TBC Age<=PingDelay*.75            
                   2:TBC Age<=PingDelay*1              
                   2:TBC Age<=PingDelay*1.25           
                   2:TBC Age>=PingDelay*1.25           
          10,485,760:DTMSG_SMOSIZE Bytes allocated                      
               9,248:DTMSG_SMOSIZE Bytes used                           
14:13:36-712702 ZDC995W  DispTcb:     009D1438                0.900sec
14:13:36-712792 ZDC995W  DispTcb:     009BBAD0                0.577sec
14:13:36-712832 ZDC995W  DispTcb:     009BBCF0                0.013sec
14:13:36-712882 ZDC995W  DispTcb:     009BBE88                0.012sec
14:13:36-714022 ZDC995W  DispTcb:     009BE170                0.924sec
14:13:36-714062 ZDC995W  DispTcb:     009BE390                0.590sec
14:13:36-714112 ZDC995W  DispTcb:     009BE528                0.251sec

Rows that end with an exclamation mark (!) should have zero counts.

Emergency zDC termination

The zDC makes every effort possible to release system resources (most especially common storage) when it terminates. This is true if the termination is normal due to a shutdown command or abnormal due to a cancel command or a program abend.

Recovery (ESTAEX) routines are always in effect to trap abnormal terminations and to free all system resources.

However, there may be a situation where zDC cannot be cancelled due to a z/OS anomaly or failure, which precludes the functioning of the cancel command.

If you are unable to communicate with zDC and want to terminate it, try the commands below in the specified order until the zDC terminates. Go on to step 2 only if step 1 is unsuccessful, go on to step 3 only if step 2 is unsuccessful, and so on. To avoid the need to manually clean up system resources using the ZDCDELET process below, wait a short while between commands to permit dumps to be taken and system resources to be freed.

  1. Issue the STOP jobname command.
  2. Issue the MODIFY jobname,SHUTDOWN IMMED command.
  3. Issue the CANCEL jobname command.
  4. Issue the FORCE jobname command.

Attempt to restart the zDC. If the restart is successful you can continue to run this job. If however the restart is not successful and zDC indicates that it cannot continue, you can execute the emergency zDC cleanup program named ZDCDELET.

ZDCDELET is a stand-alone job, which attempts to terminate a specified zDC job and release all system resources held by a zDC instance. The JCL to execute ZDCDELET is shown below:

//*
//* PLACE YOUR JOB STATEMENT HERE
//*
//DELETE EXEC PGM=ZDCDELET,PARM=xxxx
//STEPLIB DD DISP=SHR,DSN=<hlq>.R1nnnxx.SZDTAUTH

The PARM= on the EXEC statement must specify the 4-character subsystem ID of the zDC instance you wish to terminate. User ABEND 100 and 101 may occur and messages describing the abends are written to the joblog. User ABEND 100 indicates that the PARM= is not exactly 4 bytes long. User ABEND 101 indicates that the subsystem ID specified on the PARM= operand cannot be found in the system.

Messages ZDC400 through ZDC403 may be written to the joblog as well. These can be found in the zDC Messages section below.

zDC messages

Refer to the z/OS code module messages page for a list of all zDC messages.

Multiple TCP/IP stacks

A z/OS system with multiple TCP/IP stacks presents an additional complication. In this situation, an additional step is needed in the zDC job. See SZDTSAMP member ZDCMEPC.

//MEPC     PROC
...
//STEP0 EXEC PGM=BPXTCAFF,PARM='TCPIP_stack_name'
//*
//ZDCMAIN EXEC PGM=ZDCMAIN,REGION=0M,PARM='LANGUAGE=EN'
//STEPLIB   DD DISP=SHR,DSN=hlq.SZDTAUTH                  
//* Notes:                                                        
//* SYSPRINT is required in all cases.                            
//* SYSPRINT must be allocated to spool, not disk, on JES2 systems.
//* SYSPRIN3 is required on JES3 systems only.                    
//SYSPRINT  DD SYSOUT=*                                            
//SYSPRIN3  DD SYSOUT=*                                           
//SYSIN     DD DISP=SHR,DSN=hlq.SZDTSAMP(ZDCSYSIN)       

The BPXTCAFF program is used to associate a specific TCP/IP stack with the current address space. The PARM value is the job name used to start the desired TCP/IP stack.