IPMI Sensor Monitoring Plugin

From Thomas-Krenn-Wiki
Jump to: navigation, search

Due to an interruption in the power feed, the first power supply is no longer supplying power. Icinga will report this problem immediately, thanks to the IPMI Sensor Monitoring Plugin.
This article describes how to configure the IPMI Sensor Monitoring Plugin in Nagios and Icinga. Using this plugin, the hardware status of a server can be monitored by Nagios or Icinga. As specific examples, fan rotation speeds, temperatures, voltages, power consumption, power supply performance and more will be monitored.

Download

The three-page article "Lebenszeichen" (German) from ADMIN Magazin issue 01/2011 will show you how to monitor your hardware reliably using the IPMI plugin.
The current plugin version is available at GitHub:

Plugin version 2 can be found there too:


The Ubuntu Package nagios-plugins-thomas-krenn from the Thomas Krenn Ubuntu Repository is a comfortable way to install check_ipmi_sensor. An overview about all Thomas-Krenn plugins is given at Monitoring Plugins for Thomas Krenn Server.

GitHub Support

Regarding the IPMI plugin, there are two GitHub repositories available for reporting issues:

Introductory Articles from ADMIN Magazin

You will find a three-page German article regarding this plugin at ADMIN Magazin issue 01/2011. As a reader of the Thomas Krenn Wiki, you can download the article as a PDF file from here exclusively, for free:

Configuring the Version 3.x Plugin

The following steps describe the configuration necessary for using the plugin.

Software Modifications

The IPMI Sensor Monitoring Plugin requires the following software components:

  • Nagios or Icinga
  • FreeIPMI
  • libipc-run-perl

Installing the Plugin

So that the plugin can be used, it must be installed in Nagios. If Nagios is used under Debian, the plugin must be copied to the following location:

  • /usr/lib/nagios/plugins/check_ipmi_sensor

sudo Konfiguration

Attention: The following configuration hast only to be done if the plugin is run locally/NRPE and not via network! The user nagios needs root privileges for calling ipmi-sensors, ipmi-sel and (optional) ipmi-fru:

$ sudo vi /etc/sudoers.d/check_ipmi_sensor
nagios ALL=(root) NOPASSWD: /usr/sbin/ipmi-sensors, /usr/sbin/ipmi-sel

Please check with '-vvv' which commands are run by the plugin!

Command Definition

Define the check_ipmi_sensor command in your commands.cfg (under Debian 5 in /etc/nagios3/commands.cfg, for example):

define command{
  command_name  check_ipmi_sensor
  command_line  $USER1$/check_ipmi_sensor -H $_HOSTIPMI_IP$ -f $ARG1$
  }

Host Definition

The IP address for the IPMI interface must be configured as a custom object variable in the respective host definition files[1] _ipmi_ip:

define host{
  use           linux-server
  host_name     centos4
  alias         centos4
  address       192.168.1.151
  _ipmi_ip      192.168.1.211
  }

Service Definition

Finally, the service definition requires the following for defining the path to the FreeIPMI configuration file as a parameter:

define service{
  use                  generic-service
  host_name            centos4
  service_description  IPMI
  check_command        check_ipmi_sensor!/etc/ipmi-config/ipmi.cfg
  }

FreeIPMI Configuration

The FreeIPMI configuration file contains the IPMI user name, password and channel privilege level:

username monitoring
password ao5$snNc!
privilege-level user

The configuration file may also contain additional FreeIPMI configuration parameters, like workaround-flags (in this example as a workaround for this problem: FreeIPMI query to Supermicro Systems fails via LAN channel):

username monitoring
password ao5$snNc!
privilege-level user
ipmi-sensors-workaround-flags nochecksumcheck,noauthcodecheck

Example output

The following example shows the output of the plugin when querying a 1U Intel Single-CPU CSE512 Servers (Version 3.*) with X9SCM-F Motherboard (IPMI Firmware 2.24):

adminuser@ubuntu-12-04:~$ cat ipmi.cfg
username monitoring
password ao5$snNc!
privilege-level user
adminuser@ubuntu-12-04:~$ ./check_ipmi_sensor -H 192.168.255.5 -f ipmi.cfg -v
IPMI Status: OK | 'System Temp'=27.00 'Peripheral Temp'=35.00 'FAN 1'=1725.00 'Vcore'=0.71 '3.3VCC'=3.36 '12V'=11.93 'VDIMM'=1.53 '5VCC'=5.09 '-12V'=-12.09 'VBAT'=3.14 'VSB'=3.34 'AVCC'=3.38
adminuser@ubuntu-12-04:~$ ./check_ipmi_sensor -H 192.168.255.5 -f ipmi.cfg -vv
IPMI Status: OK | 'System Temp'=27.00 'Peripheral Temp'=35.00 'FAN 1'=1725.00 'Vcore'=0.71 '3.3VCC'=3.36 '12V'=11.93 'VDIMM'=1.53 '5VCC'=5.09 '-12V'=-12.09 'VBAT'=3.14 'VSB'=3.34 'AVCC'=3.38
System Temp = 27.00 (Status: Nominal)
Peripheral Temp = 35.00 (Status: Nominal)
CPU Temp = 'Low' (Status: Nominal)
FAN 1 = 1725.00 (Status: Nominal)
Vcore = 0.71 (Status: Nominal)
3.3VCC = 3.36 (Status: Nominal)
12V = 11.93 (Status: Nominal)
VDIMM = 1.53 (Status: Nominal)
5VCC = 5.09 (Status: Nominal)
-12V = -12.09 (Status: Nominal)
VBAT = 3.14 (Status: Nominal)
VSB = 3.34 (Status: Nominal)
AVCC = 3.38 (Status: Nominal)
Chassis Intru = 'OK' (Status: Nominal)
adminuser@ubuntu-12-04:~$ ./check_ipmi_sensor -H 192.168.255.5 -f ipmi.cfg -vvv
------------- begin of debug output (-vvv is set): ------------
  script was executed with the following parameters:
    ./check_ipmi_sensor -H 192.168.255.5 -f ipmi.cfg -vvv
  check_ipmi_sensor version:
    3.1 2012-05-24
  FreeIPMI version:
    ipmi-sensors - 1.2.3
  FreeIPMI was executed with the following parameters:
    /usr/local/sbin/ipmi-sensors -h 192.168.255.5 --config-file ipmi.cfg --quiet-cache --sdr-cache-recreate --interpret-oem-data --output-sensor-state --ignore-not-available-sensors
  FreeIPMI return code: 0
  output of FreeIPMI:
ID   | Name            | Type              | State    | Reading    | Units | Event
4    | System Temp     | Temperature       | Nominal  | 27.00      | C     | 'OK'
71   | Peripheral Temp | Temperature       | Nominal  | 35.00      | C     | 'OK'
138  | CPU Temp        | OEM Reserved      | Nominal  | N/A        | N/A   | 'Low'
205  | FAN 1           | Fan               | Nominal  | 1725.00    | RPM   | 'OK'
540  | Vcore           | Voltage           | Nominal  | 0.75       | V     | 'OK'
607  | 3.3VCC          | Voltage           | Nominal  | 3.36       | V     | 'OK'
674  | 12V             | Voltage           | Nominal  | 11.93      | V     | 'OK'
741  | VDIMM           | Voltage           | Nominal  | 1.53       | V     | 'OK'
808  | 5VCC            | Voltage           | Nominal  | 5.09       | V     | 'OK'
875  | -12V            | Voltage           | Nominal  | -12.09     | V     | 'OK'
942  | VBAT            | Voltage           | Nominal  | 3.14       | V     | 'OK'
1009 | VSB             | Voltage           | Nominal  | 3.34       | V     | 'OK'
1076 | AVCC            | Voltage           | Nominal  | 3.38       | V     | 'OK'
1143 | Chassis Intru   | Physical Security | Nominal  | N/A        | N/A   | 'OK'

--------------------- end of debug output ---------------------
IPMI Status: OK | 'System Temp'=27.00 'Peripheral Temp'=35.00 'FAN 1'=1725.00 'Vcore'=0.75 '3.3VCC'=3.36 '12V'=11.93 'VDIMM'=1.53 '5VCC'=5.09 '-12V'=-12.09 'VBAT'=3.14 'VSB'=3.34 'AVCC'=3.38
System Temp = 27.00 (Status: Nominal)
Peripheral Temp = 35.00 (Status: Nominal)
CPU Temp = 'Low' (Status: Nominal)
FAN 1 = 1725.00 (Status: Nominal)
Vcore = 0.75 (Status: Nominal)
3.3VCC = 3.36 (Status: Nominal)
12V = 11.93 (Status: Nominal)
VDIMM = 1.53 (Status: Nominal)
5VCC = 5.09 (Status: Nominal)
-12V = -12.09 (Status: Nominal)
VBAT = 3.14 (Status: Nominal)
VSB = 3.34 (Status: Nominal)
AVCC = 3.38 (Status: Nominal)
Chassis Intru = 'OK' (Status: Nominal)
adminuser@ubuntu-12-04:~$ 

Configuring the Version 2.x Plugin

The performance data returned can be evaluated using graphical tools like PNP4Nagios. The increased performance of the second power supply can be seen in this example, according to which the power feed to the first power supply failed at 17:00. Thereafter, Power Supply 2 is the only source of power for the server.
The following steps will describe the configuration necessary for using the plugin.

Software Modifications

The IPMI Sensor Monitoring Plugin requires the following software components:

  • Nagios or Icinga
  • FreeIPMI from Version 0.5.1, use the legacy-output option with Version 0.8 or higher
  • gawk

Installing the Plugin

So that the plugin can be used, it must be installed in Nagios. If Nagios is used under Debian, the plugin must be copied to the following location:

  • /usr/lib/nagios/plugins/check_ipmi_sensor

Command Definition

Define the check_ipmi_sensor command in your commands.cfg (under Debian 5 in /etc/nagios3/commands.cfg, for example):

define command{
  command_name  check_ipmi_sensor
  command_line  $USER1$/check_ipmi_sensor -H $_HOSTIPMI_IP$ -f $ARG1$
  }

Host Definition

The IP address for the IPMI interface must be configured as a custom object variable in the respective host definition files[1] _ipmi_ip:

define host{
  use           linux-server
  host_name     centos4
  alias         centos4
  address       192.168.1.151
  _ipmi_ip      192.168.1.211
  }

Service Definition

Finally, the service definition requires the following for defining the path to the FreeIPMI configuration file as a parameter:

define service{
  use                  generic-service
  host_name            centos4
  service_description  IPMI
  check_command        check_ipmi_sensor!/etc/ipmi-config/ipmi.cfg
  }

FreeIPMI Configuration (up to version 0.7) *

The FreeIPMI configuration file contains the IPMI user name, password and channel privilege level, and may also contain additional FreeIPMI configuration parameters:

username monitor
password ao5$snNc!
privilege-level user

FreeIPMI Configuration (from version 0.8 and later) *

The legacy-output flag is necessary with FreeIPMI 0.8.* for ipmimonitoring (since the default command output changed as of this version). You can easily add this flag to the FreeIPMI configuration file:

username monitor
password ao5$snNc!
privilege-level user
ipmimonitoring-legacy-output on

In addition, you can add the ipmi-sensors-interpret-oem-data on flag. Thereby, OEM sensors will be interpreted (if the OEM sensors for your system are already known to FreeIPMI – such as with Querying the Temperature Sensor for CPU1 of the Supermicro X8DT3-F motherboard using FreeIPMI Version 1.0.1):

username monitor
password ao5$snNc!
privilege-level user
ipmimonitoring-legacy-output on
ipmi-sensors-interpret-oem-data on

If Unrecognized Events are reported by FreeIPMI, you can ignore them by adding the following option to the configuration file, if necessary:

ipmi-sensors-ignore-unrecognized-events on

If you are using the plugin without a configuration file, you can use the –O flag:

./check_ipmi_sensor -H 10.1.102.10 -U monitor -P ***** -L user -O '--legacy-output --interpret-oem-data'

Optional Plugin Extensions

Planned extensions for future version (not yet available):

  • for version 2.x and 1.x:
    • option to change performance data output to omit quotes and to use underscores instead, e.g. to return Fan_1=5432 instead of 'Fan 1'=5432 - this should allow Zenoss users to use this plugin (a Zenoss user reported that Zenoss has issues with creating graphs for data sources with quotes)
    • option to query only defined sensors (like a single sensor or a list of sensors)
    • option to exclude OEM sensors (see this posting the plugin's mailing list)
  • for version 1.x:
    • remove searching for "error" string as this causes problem when a sensor has the term "error" in its name, see this posting the plugin's mailing list
    • error handling for creation of SDR cache file
    • search/query for other locations of ipmitool in case it is not in /usr/bin/ (e.g. in /bin, /usr/bin, /usr/local/bin) - e.g. using IPMITOOL=$(which ipmitool)
    • if $ipmioutput contains "error" maybe add query if server is switched on -> error message could be more precise then
    • validate if an option to execute different ipmitool queries instead of 'ipmitool sdr', e.g. 'ipmitool sunoem sbled get SERVICE' (with additional info on how the output should look like) would be useful

References

  1. 1.0 1.1 http://nagios.sourceforge.net/docs/3_0/customobjectvars.html

Additional Information


Foto Werner Fischer.jpg

Author: Werner Fischer

Werner Fischer, working in the Web Operations & Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.


Related articles

GPU Sensor Monitoring Plugin
Setting IPMI Monitoring User
SMART Attributes Monitoring Plugin