Apache Agent troubleshooting

Apache starts but no Agent connects to the AppMon Collector

  • Windows: Start the master Web Server Agent service automatically registered by the installer.
  • Linux and Unix: Start the master Web Server Agent process.

If the problem persists, check the Agent log files (usually in <DT_HOME>/log) for output similar to the following:

2011-09-05 16:36:46 severe  [native] Failed to launch Webserver Agent
binary /home/labuser/GG/wsagent/agent/lib/dtwsagent: Permission denied

This indicates a permission problem with the Web Server Agent (dtwsagent). See Deployment and File Access Permissions for more information.

ModSecurity blocks the signal to the AppMon monitor (403 error)

A restrictive configuration of the web server ModSecurity plugin can block signals to the AppMon Monitor. This can result in Access forbidden (403) error codes for some web content. To prevent this issue, paste the following snippet into the Apache configuration file after the inclusion of the ModSecurity-config files.

<IfModule security2_module>

    # !!! This rule is to ensure that dynaTrace gets all signals !!!
    <LocationMatch 'dynaTraceMonitor$'>
        SecRuleEngine Off       
    </LocationMatch>

    # !!! All the following rules exist to not break the website !!!

    # default is 1500
    SecPcreMatchLimit 2000     

    # SQL injection, Detect SQL Comment Sequences
    SecRuleRemoveById 981231

    # SQL injection, Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded
    SecRuleRemoveById 981172

    # SQL injection, String Termination/Statement Ending Injection Testing
    SecRuleRemoveById 981318

    # SQL injection, Detects MySQL comments, conditions and ch(a)r injections
    SecRuleRemoveById 981240

    # HTTP policy, Restrict which content-types we accept (POST with custom Content-Type)
    SecRuleRemoveById 960010

    # if there are a lot of page problems due to blocked requests, use this
    #SecRuleRemoveByTag "WEB_ATTACK/SQL_INJECTION"
</IfModule>

Loading Apache Agent on SELinux

If you are using Linux with SELinux enabled, agents might not start because SELinux is blocking essential operations. This section shows possible ways to solve it for the Apache HTTPd Agent.

Check if SELinux is involved

If your Apache Agent is not starting, check whether SELinux is enabled by executing the sestatus command:

> sestatus

SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

If SELinux status is enabled and Current mode is enforcing, SELinux might block essential operations required to start the Apache Agent.

To check whether SELinux actually denies something important, take a look at the audit log. It is usually located in the /var/log/audit.log file. Denied operations are logged as type=AVC.

Right after an attempt to start the HTTPd + Agent, execute the ausearch -m avc --start recent command to look for recent audit log entries. If there are denied operations, ausearch command returns something like this:

time->Fri Feb 24 10:09:13 2017
type=SYSCALL msg=audit(1487927353.999:24): arch=c000003e syscall=2 success=no exit=-13 a0=7f750f17b7e0 a1=2 a2=7ffd800ab500 a3=20 items=0 ppid=2589 pid=2590 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1487927353.999:24): avc:  denied  { write } for  pid=2590 comm="httpd" name="dynaTraceWebServerSharedMemory" dev=dm-0 ino=404499 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=file
....
time->Fri Feb 24 10:09:18 2017
type=SYSCALL msg=audit(1487927358.556:34): arch=c000003e syscall=42 success=no exit=-13 a0=6 a1=7f750f17fb78 a2=10 a3=7ffd800ab380 items=0 ppid=2589 pid=2590 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1487927358.556:34): avc:  denied  { name_connect } for  pid=2590 comm="httpd" dest=9998 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

The first entry at 10:09:13 shows that the process with the id pid=2590, and the executable HTTPd exe="/usr/sbin/httpd", which is running in the unconfined_u:system_r:httpd_t:s0 domain, is denied write access to dynaTraceWebServerSharedMemory.

The second entry at 10:09:18 shows that the same HTTPd exe="/usr/sbin/httpd is denied connection to the destination TCP port 9998 tclass=tcp_socket, dest=9998.

There are several ways to resolve this:

  • Disable SELinux.
  • Create a policy module with audit2allow.
  • Create a policy module manually.

Disabling SELinux

To disable SELinux, edit the /etc/selinux/config file and set the SELINUX property to disabled.

SELINUX=disabled

Reboot your system for the changes to take effect.

Creating a policy module with audit2allow

If you don't want to completely disable SELinux, you can use audit2allow to create a policy module to grant required permissions. The tool can build a policy module based on specific contents of the audit.log file. The procedure is the following:

Switch to permissive mode, by executing the following command:

setenforce 0

Executing of the getenforce or sestatus command should now indicate that SELinux is running in the permissive mode:

> getenforce
Permissive

In permissive mode, SELinux doesn't deny any access, but logs actions are denied in enforcing mode.

Disable dontaudit rules.

In certain situations, AVC denials may not be logged. A policy can silence AVC denials by using dontaudit rules. To ensure we see all denials in the audit.log file, disable the dontaudit rules with the following command:

> semodule -DB

These are enabled later.

Start httpd + Agent. Start dtwsagent and httpd.

> ./dtwsagent
> service httpd start

Since SELinux is in permissive mode, everything should start now, but new SELinux denials are written to the audit.log file.

Create a policy module with audit2allow.

The new entries to the audit.log file can be used as input to audit2allow to create the module with the desired permissions:

> grep /usr/sbin/httpd | audit2allow -M apacheagent

Install the module:

> semodule -i apacheagent

Switch back to enforcing mode and restart HTTPd and Dynatrace Web Server Agent.

> setenforce 1

Restart HTTPd and Dynatrace Web Server Agent and everything should come up. Recheck audit log and verify that there are not any related denials.

The dontaudit rules can then be enabled again:

> semodule -B

Creating a policy module manually

You can also create a policy module manually, but is not recommended. The recommended approach is one of the two previously described: either disable SELinux completely or create a policy module with audit2allow.

Anyway, if you need to create a policy manually, the following example might help you. But it's just a demo with the purpose to show the general procedure of manual module creation.

Note

The following instructions below are verified on CentOS/RHEL 6 and with Apache HTTPd Agent only and might not be applicable to your system.

The strategy of this demo is the following:

  1. Recursively tag all files in the installation folder with a custom type dynatrace_agent_t.
  2. Permit full access to all files tagged dynatrace_agent_t for processes running in the httpd_t domain. This ensures that the agent can access all files in the installation folder and below, including the dynaTraceWebServerSharedMemory file.
  3. Permit processes, running in the httpd_t domain, to connect to TCP ports.

To create a policy module, create the two following files in an empty directory:

  • dynatrace_agent.fc: for the security contexts that should be applied to certain files/directories.
  • dynatrace_agent.te: for the type enforcement rules.

Content of the dynatrace_agent.te file:

policy_module(dynatrace_agent_Policy, 0.1.0)

require {
        type httpd_t;
        type port_t;
        class file { open read write create append unlink getattr execute setattr rename lock };
        class dir { open read write create search remove_name add_name getattr setattr rmdir };
        class tcp_socket { name_connect };
}

# declare the new file type
type dynatrace_agent_t;
files_type(dynatrace_agent_t)

# file/directory permissions
attribute dynatrace_agent_t_allowed_app;
allow dynatrace_agent_t_allowed_app dynatrace_agent_t:dir { open read search remove_name write add_name create getattr setattr rmdir };
allow dynatrace_agent_t_allowed_app dynatrace_agent_t:file { open read write create append unlink getattr execute setattr rename lock };

# permissions to connect to tcp socket
allow dynatrace_agent_t_allowed_app port_t:tcp_socket { name_connect };

# grant processes of domain httpd_t permissions specified for dynatrace_agent_t_allowed_app
typeattribute httpd_t dynatrace_agent_t_allowed_app;

Content of the dynatrace_agent.fc file:

# recursively label files in /opt/dynatrace-6.5 with our custom type dynatrace_agent_t
/opt/dynatrace-6.5(/.*)?        gen_context(unconfined_u:object_r:dynatrace_agent_t,s0)

Since the file labels change, existing rules may not fulfill their purpose because they were written for the original labels.

To build the SELinux module, execute:

> make -f /usr/share/selinux/devel/Makefile

This compiles the module to a policy package file dynatrace_agent.pp. A policy package can be loaded/unloaded similar to kernel modules:

> semodule -i dynatrace_agent.pp

To verify that the module was loaded correctly, list the installed modules with the selinux -l command:

> semodule -l | grep dynatrace
dynatrace_agent_Policy	0.1.0

To unistall the dynatrace_agent module, execute the following command:

> selinux -r  dynatrace_agent     ... uninstall module

Installing a module with the selinux -i command is persistent, so the module reloads after a system reboot.

The previous procedure of manual module creation might not be applicable for your system.