Lucy Monahan is a Principal Performance QA Engineer at Novell, and helps to manage their distributed Agile process.
One of the most important features of an application is to provide adequate security and protect secrets held within. Business Transactions used with Continuous Integration, Unit-, Feature- and Negative Testing specialized for security can detect known security vulnerabilities before your product goes to market.
Catch Secrets Written to Logging
Plaintext secrets written to a log file are a well-known vulnerability. Once an intruder gains access to a hard disk they can easily comb through log files to further exploit the system. Here a Business Transaction is used to search logging output to look for secrets.
Application data slips into logging in a variety of ways: lack of communication regarding which data is a secret, lingering early code, perhaps before a comprehensive logging strategy has been implemented, old debug statements or perhaps inadvertent inclusion via localization processing.
It’s a good idea to grep log files after your test run but that will not cover output to the server console, which may contain different content. For example, users may start the application using nohup which may write terminal logging to nohup.out. And starting an application with redirection of stdout and stderr will persist console output to a file, such as:
startserver.sh > /opt/logs/mylogfile.log 2>&1
Use Business Transactions to search logging content during your testing and trap secrets before they are revealed! And be sure to enable a high logging level since trace level may contain secrets that info level does not.
This Business Transaction uses two Measures for two of the org.apache.log4j.spi.Log4J LoggingEvent class constructors. Other methods that can be trapped include those from proprietary logging classes or classes associated with the auditing channel or retrieval of localized messages.
These Measures simply search the constructor’s message argument for the word “testuser” for purposes of demonstration:
The Business Transaction Filter defines both Measures with an OR operator to activate the Business Transaction:
We want to know about any instance of a secret being logged thus the Threshold Upper Severe is set to 1.0 occurrence. Each of the Business Transactions outlined here use this Threshold.
When the secret is written to logging the Business Transaction is activated:
The displayed columns in the Business Transaction can be customized to show counts of each Measure that matched the filter.
TIP: Running this Business Transaction against Install and Uninstall programs is highly recommended since a lot of secret data is requested and used during these processes.
From the Business Transaction Dashlet we can Drill-Down to the actual Transactional Trace (PurePath) and see where this secret was actually logged. The PurePath contains additional contextual information such as HTTP Parameters, method arguments, exceptions, log messages, database statements, remoting calls, …
Catch Exception Messages Containing Secrets
If your application prints Exceptions and their messages then you need to catch any secrets embedded within them. This Business Transaction uses a Measure to obtain the return value for java.lang.Throwable.getLocalizedMessage(). Depending on your application’s architecture a Measure for java.lang.Throwable.getMessage() may also be required.
With the Business Transaction defined, perform a negative testing test run that intentionally throws exceptions.
The Measure in this example simply searches for the word “authentication” in the getLocalizedMessage() return value for demonstration purposes:
Catch non-SSL Protocols
Any non-SSL traffic over the network has the potential for exposing secrets. Most applications requiring high security will disallow all non-SSL traffic. Your application needs to be tested to ensure that all requests are being made over SSL. Non-SSL connections can inadvertently be opened and used when different components within an application are handling their own connections or oversight in installation or configuration allowed the connections.
Unless you include an explicit check for non-SSL connections then they may be opened and used stealthily. One way to ensure that only SSL connections are being used is to trap any non-SSL connections being opened.
This Business Transaction uses a Measure to obtain the String value of the first argument to the constructor for javax.naming.ldap.InitialLdapContext. The first argument is a Hashtable and when SSL is enabled one of the entries contains the text “protocol=ssl”.
It’s worth noting that the presence of the “protocol=ssl” value is particular to the JVM implementation being used. Indeed, when SSL is not enabled the protocol value is omitted, hence the use of the “notcontains” operator for this Measure. Use a Method Sensor to capture the value being used in your JVM implementation to confirm the text token for your application environment.
What other protocols are being used in your application? You can write a similar Business Transaction to ensure that only SSL is being used for that protocol.
Catch Secrets Written to Print Statements
The above examples catch secrets written to Exceptions and logging. But a common debug technique is to use a print statement, such as System.out.println() in Java, rather than a logger function. A Business Transaction that catches print statements is recommended for the test suite.
Having said that, in Java java.lang.System.out.println() is not accessible in the usual way because “out” is a field in the System class. An alternative approach may be to use a Business Transaction with a Measure based on “Classname value” to trap all calls to java.lang.System. All related PurePaths for this Business Transaction would then merit review to assess security risks. This approach may or may not be feasible depending how often your application calls java.lang.System. Feasibility testing of this approach would thus be necessary.
The test application used here does not call java.lang.System so an example is not included. In any case, a companion test that performs a full text search of the source code tree for calls to java.lang.System.out.println() is highly recommended.
For each Business Transaction the process is:
- Define which entities within your application represent secrets (e.g. passwords, account numbers)
- Define which classes and methods will be used for the Measure definition
- Create a new Business Transaction using the Measure definition
- Ensure that the input data used in the test contains many secrets
Include these types of Business Transactions in continuous integration, unit, feature and negative testing that contain a diversity of secrets. These Business Transactions are not intended for load testing, however, since load testing may simply contain many instances of the same secret rather than a diversity of secrets and the result will flood your result set.
Security-related issues released into the field are costly. The customer’s sensitive data is at risk and the expense of security field patches and the damage done to the corporation’s reputation is high. Understanding your application’s architecture will help identify possible vulnerabilities and enable you to grow a suite of Business Transactions designed specifically to trap secrets and guarantee your application’s security.
Follow Up Links for Dynatrace Users
When you are a Dynatrace User you can check out the following links on our Community Portal that explain more on how Dynatrace does Business Transaction Management, how to improve your Continuous Integration Process and how to integrate Dynatrace in your Testing Environment.