Installing hardware monitoring system for HP Proliant servers

Jephe Wu - http://linuxtechres.blogspot.com

Environment: HP Proliant server with CentOS 5.3 x86_64 OS
Objective: Enable hardware monitoring for the server


Steps summary only for sending out SNMP trap
1. configure sendmail to be able to send out email through mailrelay host
2. configure ILO as below:

SNMP/Insight Manager Settings
Configure and Test SNMP Alerts

SNMP Alert Destination(s): 
iLO 2 SNMP Alerts:  Enabled Disabled
Forward Insight Manager Agent SNMP Alerts:  Enabled Disabled
SNMP Pass-thru:  Enabled Disabled

give snmp trap server for alert destination. and enable all of above options. Then you don't have to configure snmp trap setting in /etc/snmp/snmp.conf

Part I:
Steps: after using NFS method to install OS through ILO virtual media, then vi /etc/hosts to put the server real IP address as hostname as follows:

127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.0.0.1 logserver.domain.com logserver

1. download the Proliant Support Package file from HP website for your server model such as Proliant DL360G6 x86_64
2. untar the file
3. change /etc/redhat-release file content from 'CentOS release 5.3 (Final)' to 'Red Hat Enterprise Linux Server release 5.3'
4. run ./install.sh to install it
5. point your browser to http://serverip:2301/, use your root user password to login
6. vi /opt/hp/hp-snmp-agents/cma.conf trapemail line
trapemail /bin/mail -s 'logserver HP Insight Management Agents Trap Alarm' root


note: this is for sending out email notification for any snmp trap, but if I reboot server, it won't send out any email according to my test. Also, please remember to configure /etc/aliases to enable root user email aliases
7. add the following to /etc/snmp/snmpd.conf to enable insight management agents to talk to snmp trap server. If configured this, the point 6 is basically not needed actually.

trapcommunity public
trapsink snmptrapserveripaddressorhostname



Part II: setup another server as SNMP trap server
1. setup snmp trap server, make /etc/snmp/snmptrapd.conf as follows:

[root@jephe snmp]# more snmptrapd.conf
traphandle default /usr/bin/perl /usr/bin/traptoemail -s localhost -f hpsnmptrap@domain.com jephe.wu@domain1.com
traphandle default /usr/bin/perl /usr/bin/traptoemail -s localhost -f hpsnmptrap@domain.com anotheruser@domain1.com

2. start up snmptrapd services and make it automatic for next restart, that's it.

note: You can configure to send out email as well from snmp trap server after generating snmp trap, just use above Part I step 6 to configure it.

note:
1. to start the HP Array Configuration Utility, go to /opt/compaq/cpqacuxe/bld/, run cpqacuxe , then you can see the utility option in system management homepape at https://serverip:2381/ , but you can only use the https://127.0.0.1:2381 to use the utility for security reason. So you might need to ssh into the server with -X option, then run firefox to open browser locally.
 2. for configuring 'Accept SNMP packets from these hosts', the host list should be separated with semi-column ;

FAQ
1. when using PSP 8.0.0, the cma.log filling up, you will see a lot of messages like this:

netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:151 netsnmp_access_interface_container_load()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:199 netsnmp_access_interface_index_find()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:199 netsnmp_access_interface_index_find()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:279 netsnmp_access_interface_entry_create()


Solution=> 
1)/opt/compaq/nic/etc/cmanicd stop
2)Add 'cmanicd' to the exclude line in the /opt/compaq/cma.conf so that it won't run after next reboot/shutdown.