Troubleshooting SSL monitoring issues

The NAM Probe provides a wide range of diagnostic information and tools that can help you resolve issues with SSL monitoring.

Before trying to find an answer to a specific question regarding SSL-related issues, you can use the built-in system diagnostics of Network Application Monitoring. Inspect the NAM Probe log files, especially rtm_perf_curr.log and check the system health reports. For more information, see Diagnostics.

Why is SSL not being decrypted, even though the NAM Probe has an SSL accelerator card and the SSL card has been initialized?

The SSL card needs to operate in the Logged on mode. For security reasons, after each machine reboot, the card reverts back to the Initialized mode. To re-activate the card, log in to the card using the user login and password.

How can I check whether SSL decryption is functioning properly?

  • To see full status information about the current SSL operation, execute the ssldecr status (SHOW SSLDECR STATUS for Classic AMD) rcon command. For more information, see SSL-related rcon commands.
  • To see historical information about SSL decryption, open the /var/log/adlex/rtm_perf.log file. Output from the ssldecr status command is written there every monitoring interval (default: 5 minutes).
  • When viewing NAM Server reports, note the number of SSL errors reported. In particular, if the error breakdown information shows a large number of “Other SSL errors”, this indicates that SSL decryption errors are a problem.

What should I do if the "ssldecr status" command does not return engine status as "OK" or if the incorrect engine is used?

To operate correctly, the engine and accelerator card should match. For example, when using a NITROX accelerator card, use the nitroxfips engine. If the engine status is not OK or an incorrect engine is listed as being in use, check the following:

  • Installation: perhaps the wrong upgrade file has been installed. For more information, see Installing the NAM Probe software.
  • Engine configuration. For more information, see Selecting and configuring SSL engine.
  • Authentication: some cards require that you perform a login action before they can operate. Refer to the configuration instructions for the card.

My SSL engine status is OK, but SSL decryption fails entirely, with no keys recognized. what is the likely cause?

The NAM Probe requires that the SSL card be in an authenticated mode. This allows the NAM Probe to gain access to RSA private keys stored in the card. One common problem is that when the NAM Probe is restarted, the user forgets to log in to the NAM Probe and launch the SSL card configuration utility to authenticate user access (unlock access to RSA keys). The engine status will be given as OK, meaning that the card itself is functioning correctly and the correct system driver is loaded, but the number of keys recognized will be 0 because the NAM Probe is not able to retrieve key information from the card.

>$ ssldecr status
SSL DECRYPTION STATUS:
        CONFIGURATION: Engine:openssl(native) status:OK
                Keys recognized=65 not recognized=0
                Engine states: blocked=0, initializations=1
        SESSIONS:
       ...

To avoid this problem, remember to log in to the NAM Probe and launch the SSL card configuration utility to authenticate user access (unlock access to RSA keys) after you restart the NAM Probe.

What should I do if the ssldecr status (SHOW SSLDECR STATUS for Classic AMD) command reports that some keys were not recognized?

This can happen if RSA private keys stored in .pem files or on the accelerator card do not match the keys used by the SSL servers being monitored. Private keys used by servers can change.

Classic AMD

In Classic AMD, investigate the problem further by executing the SHOW SSLDECR KEYS command in rcon and check which keys have an error status. For example:

>$ show ssldecr keys
Configuration of SSL private keys:
<key: s1.key, status: error (reading failed)>
<key: strange.key, type: file, size: 1024, status: OK (matched)>
Keys total: 2, ok: 1, failed: 1, matched: 1

If there are errors, check the following:

  • Is the keylist file in the correct format? If not, correct the entries.
    For more information, see Management of RSA private keys on NAM Probe.
  • If .pem files are to be used, are there the correct .pem files in /usr/adlex/config/keys ?
    If not, supply the missing files.
  • If .pem files are to be used, are there any typos in the file names in the keylist file?
    Correct the file names or paths as needed.
  • Are the .pem files encrypted?
    Open a key file and see whether the word ENCRYPTED appears near the top of the file.
    The keys stored on the disk may be in encrypted form. In this case, to make the keys available the administrator has to arrange for the keys to be decrypted before they can be read by the NAM Probe process. This requires a password (one per key file) and is accomplished using the kpadmin utility and the KPA daemon. For more information, see Using KPA to make keys available to the NAM Probe process.
  • If keys from the accelerator card are used, are the key IDs and names given in the proper format in keylist ? For more information, see Management of RSA private keys on NAM Probe.
  • If only keys from the accelerator are to be used, consider not using the keylist file at all by setting the ssl.import.all.keys.from.token configuration property to true. This ensures that all the keys on the card will be seen correctly regardless of any entries you might make in the keylist file. For more information, see Management of RSA private keys on NAM Probe.

What should I do if the "ssldecr status" command reports no sessions?

If the number of sessions is reported as 0, check the following:

  • Does your NAM Probe installation have a license for SSL decryption?
    If not, you need to obtain one.

  • Are there any SSL services defined?
    Remember that you need to define a service before you can monitor it.

    Classic AMD

    You can execute the SHOW SSLDECR STATUS command in rcon to list all the servers for which SSL decryption is active. The analyzer for the software service must specify “SSL with decryption”.

  • Is there any actual traffic for the servers for which SSL decryption is active?
    To find out, use the tcpdump command on the NAM Probe. For example:

    tcpdump 1000 "/ssl.tcp" "host 10.102.10.133 and port 443"
    

    or

    tcpdump 1000 "/ssl.tcp" "vlan and host 10.102.10.133 and port 443"
    

    and check whether there is any traffic captured in the /ssl.tcp file.

If "ssldecr status" ("SHOW SSLDECR STATUS" for Classic AMD) reports decryption errors, what do they mean and what can I do to fix the problem?

The following decryption errors can be reported:

  • “packet lost during payload data exchange”
    Your network may be losing packets; check mirrored ports.
  • “corrupted payload data packet”
    Some of the traffic is corrupted and may be incorrectly received by the NAM Probe, potentially because of network problems.
  • “decryption failed during payload data exchange”
    The symmetric decryption failed.
  • “no private key found”
    You do not have a private key for this session or you have not listed it correctly in the keylist file.
  • “packet lost during handshake”
    It may mean that your network is losing packets; check mirrored ports.
  • “corrupted handshake packet or incorrect handshake sequence”
    Some of the traffic is corrupted and may be incorrectly received by the NAM Probe.
  • “decryption broken during handshake”
    The symmetric decryption failed. If you are using monitoring software earlier than 12.4.10, this error means that the SSL traffic contains valid SSL sessions handshakes with no missing packets but the but no operations are reported. This is a Chrome client specific issue with TLS Session Hash extension. The workaround is to update your monitoring software to minimum 12.4.10 release.
  • “unsupported SSL version”
    Traffic encrypted with SSL 2.0 has been encountered. These protocol versions are not supported by the NAM Probe.
  • “unsupported SSL feature”
    An unsupported SSL feature has been encountered. The area the feature relates to and the count of occurrences is in brackets: unsupported cipher, compression, server key exchange.
  • “re-used sessions with no matching master session seen before”
    A so-called “short handshake” (a session with re-used ID) was observed, but the NAM Probe has no record of the original session (“long handshake”) that established the security credentials. Note that some such errors are normal if you restart the NAM Probe, which may cause some traffic not to be observed by the NAM Probe.
  • “incomplete SSL handshake”
    A TCP session was observed to terminate before a complete SSL handshake was seen. The server can refuse a connection and close the TCP session for various reasons. For example, this can occur if the client requested a particular version of SSL but the server requires a different version.
  • “terminated by alert”
    A fatal SSL alert arrived. Technically, this is alert detection and not an error.
  • “session not seen from the beginning”
    May be related to monitoring of sessions with missing start of session. Change your settings if required. For more information, see Advanced - persistent TCP sessions.

I suspect that I do not have all the private keys necessary for decryption (for example, I observe sessions “with no private key found”). how can I ensure that all the servers have their matching keys?

Classic AMD

Execute the SHOW SSLDECR SERVERS command in rcon to list the decryption information for each server. For example:

>$ SHOW SSLDECR SERVERS
Configuration for SSL servers:
<server: 10.102.10.133:443, certs seen: 1, keys used: 1, status: key(s) found>
<cert: [/C=PL/ST=woj pomorskie/L=TRICIT,//OU=LAB/CN=sdfds/emailAddress=sdklj@sdkjw.com], sent: 4,
key: strange.key>
Servers total: 1, keys required: 1, keys found: 1, keys missing: 0

For all servers, ensure that their key status is found. Also note the last summary line of the output, which states how many keys were required and how many keys were found or were missing.

There appear to be missing keys, but I know that I have provided all the necessary keys. How can I verify that the keys I have are correct?

Classic AMD

A monitored server may change its private key, making the key used by the NAM Probe obsolete. To prove that a key is correct, perform a test encryption/decryption using that key:

Use the SSLDECR CERTS rcon command to extract the public keys from the traffic being seen by the NAM Probe. For more information, see SSL-related rcon commands.

Perform a test encryption of a short text string, such as today's date, using extracted certificates. Use OpenSSL to encrypt the string. For example:

# date > txt
# cat txt
Wed Feb 3 16:13:01 CET 2010
# openssl rsautl -inkey /cert_192.168.207.162\:443_1.der -keyform der -certin -in txt -encrypt -out txt.enc

where /cert_192.168.207.162\:443_1.der is the file saved by the SSLDECR CERTS command used earlier.

Decrypt the encrypted file using the private key you want to test.

For example, using OpenSSL:

openssl rsautl -inkey /usr/adlex/config/keys/www2.prod.ramq.gov_decr1.pem -decrypt -out txt.decr -in txt.enc

If the key is correct, there should be no difference between the files txt and txt.decr .

You can also use the key stored on the card to decrypt the test file. To do that, use the rsautil utility residing in /usr/adlex/rtm/bin/ . (For full usage syntax of the utility, type rsautil -?)

In the following example, the first decryption succeeds and the second one fails. Note the last line with “decrypt simple failed”:

[root@hsekilx030 bin]# cd /usr/adlex/rtm/bin/
[root@hsekilx030 bin]# ./rsautil -e nitroxfips -t token -k 7 -f /root/DT_00000_42494/cert_153.88.134.201\:443_1.enc
L3 2010-06-02 09:33:13.270  0@ssldecr/rsaeng.cpp:320 RSA engine mode auto set to native
L2 2010-06-02 09:33:13.270  0@ssldecr/rsaeng.cpp:80 Openssl version: OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008,
L2 2010-06-02 09:33:13.270  0@ssldecr/rsaeng.cpp:84 Initializing OpenSSL in thread safe mode with 41 locks
L3 2010-06-02 09:33:13.271  0@./ssldecr/sslnitroxfips.h:29 NitroxFips: blocking mode: 0
L1 2010-06-02 09:33:13.271  0@ssldecr/rsautil.cpp:322 OK
L1 2010-06-02 09:33:13.271  0@ssldecr/rsautil.cpp:347 SSL RSA handler nitroxfips(native) created
L3 2010-06-02 09:33:13.282  0@ssldecr/rsautil.cpp:394 key ok: 7
L1 2010-06-02 09:33:13.291  0@ssldecr/rsautil.cpp:67 30 (0x1e) bytes at 0xbfa71824
0000 4d 6f 6e 20 4d 61 79 20 33 31 20 31 33 3a 33 32 Mon May 31 13:32
0010 3a 30 39 20 43 45 53 54 20 32 30 31 30 0a       :09 CEST 2010.
[root@hsekilx030 bin]# ./rsautil -e nitroxfips -t token -k 8 -f /root/DT_00000_42494/cert_153.88.134.201\:443_1.enc
L3 2010-06-02 09:33:20.125  0@ssldecr/rsaeng.cpp:320 RSA engine mode auto set to native
L2 2010-06-02 09:33:20.125  0@ssldecr/rsaeng.cpp:80 Openssl version: OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008,
L2 2010-06-02 09:33:20.125  0@ssldecr/rsaeng.cpp:84 Initializing OpenSSL in thread safe mode with 41 locks
L3 2010-06-02 09:33:20.125  0@./ssldecr/sslnitroxfips.h:29 NitroxFips: blocking mode: 0
L1 2010-06-02 09:33:20.125  0@ssldecr/rsautil.cpp:322 OK
L1 2010-06-02 09:33:20.125  0@ssldecr/rsautil.cpp:347 SSL RSA handler nitroxfips(native) created
L3 2010-06-02 09:33:20.137  0@ssldecr/rsautil.cpp:394 key ok: 8
L2 2010-06-02 09:33:20.152  0@ssldecr/rsautil.cpp:147 decrypt simple failed

For more information on loaded keys, execute the SHOW SSLDECR KEYS command in rcon.