Troubleshooting NAM Probe

When troubleshooting the NAM Probe, review the following.

NAM Probe fails to activate Broadcom card during interface identification

Reported issue: The NAM Probe fails to activate the Broadcom network card during the interface identification procedure

When a Broadcom 10Gb card is connected to a mismatched link speed, a warning message occurs:

WARNING: Autonegotiation is not supported for following inactive network interface(s): eth5 eth6 eth7
         Rtminst will attempt to match link speed.
         Please make sure there is a cable connected to every interface and press ENTER.

The cause of this speed mismatch is the Broadcom card's inability to automatically negotiate the link speed. Configuring the link speed requires that you perform an additional step.

Before performing this step, make sure that there is a cable connected to every interface then press [ENTER]. The NAM Probe then automatically adjust the card's speed configuration to match the link speed and attempts to identify the network interface again.

The process of matching the link speed to the card's speed will cycle through all interfaces unable to autonegotiate.

---------------------------------------------
The autonegotiation is not supported for eth5
---------------------------------------------
 Validating the link speed for eth5
 The link is not up for eth5
 Set alternate link speed for eth5
 Current speed: 10000 Mb/s
 New speed: 1000 Mb/s
 New link speed set successfully for eth5

Once the link speed has been successfully matched, the NAM Probe attempts to activate it.

eth5 has been activated
Note

If the link still cannot be activated, an error message displays.

*** Cannot activate eth6
*** Restoring default link speed
*** Are you sure there is a cable connected to every interface?
*** Connecting cables and repeating the link speed matching attempt may solve the problem.
*** Repeat matching link speed for eth6? (y/n)

In case where the speed cannot be successfully matched and the link activated, it is advised that you repeat the identification procedure while the Broadcom card is connected to its designated 10Gb link. The default link speed on the card will be automatically matched to the link speed of the connected cable.

Since the newly negotiated link speed is not optimal for the activated Broadcom 10Gb card, it is not saved as part of the network configuration. The NAM Probe attempts to autonegotiate the link speed once again, if it is restarted. If your targeted link speed is less than 10Gb and you require to force a specific link speed configuration, use the option for configuring link parameters located in the network configuration menu. For more information, see RTM configuration tool (rtminst).

No traffic seen on sniffing ports on PCIe cards

When you install a PCIe card on NAM Probe 11.5 and 11.6 on Red Hat Enterprise Linux 5.3, 5.4, or 5.5, you may see the following symptoms:

  • No traffic on sniffing ports when you run the traffic command.
  • In the /var/messages file, for each eth on the PCIe card, a message such as
     msi interrupt test failed, using legacy interrupt
     ...ethX...
    

If this occurs, set the kernel option pci=nomsi in the /boot/grub/grub.conf file and then reboot the machine.

Example:

kernel /boot/vmlinuz-2.6.18-128.el5PAE ro root=LABEL=/ vga=0x317 rhash_entries=8192 crashkernel=64M@16M pci=nomsi

Restarting monitoring process on Sun Fire X4450

Reported issue:

The rtm process keeps restarting approximately every 20 minutes and generating the following (or similar) information in the rtm.log file:

L3 2008-05-28 18:24:16.167  0@commsrv/CommServer.cpp:285 CommServer cl:3586 UNREGISTER_CLIENT
L0 2008-05-28 18:24:16.168  0@commsrv/CommServer.cpp:138 Client id=3586 unregistered

No free packet buffers size=1536

anlzr thread locked
probe version: ndw.10.3.200
os version: RHEL5 i386
compiled with: CFLAGS=-O3 -march=i686 -pipe  -fno-strict-aliasing -DLINUX26  -g3 -D_DEBUG -I. -I./include -I/usr/local/openssl-0.9.7/include -I/usr/kerberos/include -I./lib/libpcap-0.9.4 -DU_STATIC_IMPLEMENTATION -D_REENTRANT -D_LINUX_THREADS -DRTM_VPN -DNO_LICENSES  -Wall -Wno-format -W -Wpointer-arith -Wcast-qual -Wcast-align -Wuninitialized -Wparentheses
created: Tue May 20 11:25:30 CEST 2008
build: mkwap@cwpl-ap011-dev5:/home/mkwap/common/ndw/rtm
Begin Stack Frame Dump
/usr/adlex/rtm/bin/rtm[0x83e0470]
[0x97b420]
[0x97b402]
/lib/libc.so.6(nanosleep+0x46)[0x8c9846]
/lib/libc.so.6(usleep+0x3c)[0x9026ac]
/usr/adlex/rtm/bin/rtm[0x80a54f6]
/usr/adlex/rtm/bin/rtm[0x83e0d24]
/usr/adlex/rtm/bin/rtm[0x83e0f21]
/lib/libpthread.so.0[0x3fa43b]
/lib/libc.so.6(clone+0x5e)[0x908fde]
End Stack Frame Dump
All stack addresses list: 0x083e0470 0x0097b420 0x0097b402 0x008c9846 0x009026ac 0x080a54f6 0x083e0d24 0x083e0f21 0x003fa43b 0x00908fde
L3 2008-05-28 18:24:17.377  0@commsrv/CommServer.cpp:270 CommServer cl:3594 REGISTER_CLIENT_NO_DIAG

This issue is specific to Sun Fire X4450 hardware configuration. The CPU frequency scaling causes tsc (time stamp counter) clocksource to be unreliable. There are several other clocksource choices that can be used instead of the tsc. Use the following procedure to examine your system for the available clocksource and to modify the grub.conf file to specify a different clocksource.

Log in to your NAM Probe as root user.

Determine your current clocksource name. Execute the following command at the prompt:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

The response from the system should be: tsc

Examine the availability of clocksource options on your machine. Display the content of the available_clocksource object located in /sys/devices/system/clocksource/clocksource0/

The system response should list the available clocksource options:

[root@AMD ~]# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
acpi_pm jiffies tsc pit

If the acpi_pm clocksource is available, edit the /boot/grub/grub.conf file and add acpi_pm to the line describing kernel boot parameters.

The following line should be appended to each kernel line in the /boot/grub/grub.conf file:

clocksource=acpi_pm

Example of the grub.conf file with clocksource configured for acpi_pm using the mcedit command:

[root@AMD ~]# mcedit /boot/grub/grub.conf

The above example lists two kernels installed: (2.6.18-92.el5PAE) and (2.6.18-92.el15). You should append the same clocksource parameter for each kernel installed.

Save the modified grub.conf file and reboot the NAM Probe.

Once the NAM Probe reboots, log in as root user and confirm the current clocksource:

[root@AMD ~]# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
acpi_pm

snmp daemon stops responding while snmpd.log continues to consume disk space

Reported issue: When I use snmpwalk, the snmp daemon stops responding while snmpd.log continues to consume disk space (approximately 5 MB/sec).

For more information, visit the Red Hat support center at Red Hat Global Support Services. For a workaround:

Log in to your NAM Probe as a root user.

Edit the snmpd.config file by executing the following command: mcedit /etc/snmp/snmpd.conf

Append the following lines at the end of the file:

view all excluded mib-2.ip
view all excluded mib-2.host

These lines exclude the object identifiers (OIDs) that cause the daemon to stop responding.

Save the file and restart the snmpd daemon by executing the following command: service snmpd restart

snmpd dead but subsys locked

Reported issue: When I attempt to view snmpd status, I receive the following message: snmpd dead but subsys locked.

The reason why snmpd does not start is that the net-snmp, net-snmp-libs, or net-snmp-utils packages have different version numbers. Ensure that the versions numbers are uniform and that SELinux is disabled.

  1. For more information about identifying your adapter, see the Adapter & Driver ID Guide at: http://support.intel.com/support/network/adapter/pro100/21397.htm. For the latest Intel network drivers for Linux, refer to the following website. In the search field, enter your adapter name or type, or use the following link to search for your adapter: http://downloadcenter.intel.com/default.aspx.
  2. Before continuing, ensure that development software is installed. Check whether the compiler was removed from your system. For more information, see Managing Development Packages.