Linux SysAdmin and Oracle/MySQL DBA: September 2009

Jephe Wu - http://linuxtechres.blogspot.com

Environment: HP Proliant server with CentOS 5.3 x86_64 OS
Objective: Enable hardware monitoring for the server

Steps summary only for sending out SNMP trap
1. configure sendmail to be able to send out email through mailrelay host
2. configure ILO as below:

SNMP/Insight Manager Settings

Configure and Test SNMP Alerts

SNMP Alert Destination(s):
iLO 2 SNMP Alerts:	Enabled Disabled
Forward Insight Manager Agent SNMP Alerts:	Enabled Disabled
SNMP Pass-thru:	Enabled Disabled

give snmp trap server for alert destination. and enable all of above options. Then you don't have to configure snmp trap setting in /etc/snmp/snmp.conf

Part I:
Steps: after using NFS method to install OS through ILO virtual media, then vi /etc/hosts to put the server real IP address as hostname as follows:

127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
10.0.0.1 logserver.domain.com logserver

1. download the Proliant Support Package file from HP website for your server model such as Proliant DL360G6 x86_64
2. untar the file
3. change /etc/redhat-release file content from 'CentOS release 5.3 (Final)' to 'Red Hat Enterprise Linux Server release 5.3'
4. run ./install.sh to install it
5. point your browser to http://serverip:2301/, use your root user password to login
6. vi /opt/hp/hp-snmp-agents/cma.conf trapemail line
trapemail /bin/mail -s 'logserver HP Insight Management Agents Trap Alarm' root

note: this is for sending out email notification for any snmp trap, but if I reboot server, it won't send out any email according to my test. Also, please remember to configure /etc/aliases to enable root user email aliases
7. add the following to /etc/snmp/snmpd.conf to enable insight management agents to talk to snmp trap server. If configured this, the point 6 is basically not needed actually.

trapcommunity public
trapsink snmptrapserveripaddressorhostname

Part II: setup another server as SNMP trap server
1. setup snmp trap server, make /etc/snmp/snmptrapd.conf as follows:

[root@jephe snmp]# more snmptrapd.conf
traphandle default /usr/bin/perl /usr/bin/traptoemail -s localhost -f hpsnmptrap@domain.com jephe.wu@domain1.com
traphandle default /usr/bin/perl /usr/bin/traptoemail -s localhost -f hpsnmptrap@domain.com anotheruser@domain1.com

2. start up snmptrapd services and make it automatic for next restart, that's it.

note: You can configure to send out email as well from snmp trap server after generating snmp trap, just use above Part I step 6 to configure it.

note:
1. to start the HP Array Configuration Utility, go to /opt/compaq/cpqacuxe/bld/, run cpqacuxe , then you can see the utility option in system management homepape at https://serverip:2381/ , but you can only use the https://127.0.0.1:2381 to use the utility for security reason. So you might need to ssh into the server with -X option, then run firefox to open browser locally.
2. for configuring 'Accept SNMP packets from these hosts', the host list should be separated with semi-column ;

FAQ
1. when using PSP 8.0.0, the cma.log filling up, you will see a lot of messages like this:

netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:151 netsnmp_access_interface_container_load()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:199 netsnmp_access_interface_index_find()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:199 netsnmp_access_interface_index_find()
netsnmp_assert 1 == _access_interface_init failed if-mib/data_access/interface.c:279 netsnmp_access_interface_entry_create()

Solution=>
1)/opt/compaq/nic/etc/cmanicd stop
2)Add 'cmanicd' to the exclude line in the /opt/compaq/cma.conf so that it won't run after next reboot/shutdown.

Jephe Wu - http://linuxtechres.blogspot.com

Environment: HP Proliant server A with one com port which is com1, CentOS 5
Objective: enable login through com1 using null modem cable from another machine B which can be Windows or Linux. Also, we will try to login from ILO vsp to display kernel bootup message, BIOS and GRUB

Steps:

Part I - physically connect to com1 interface at the back side of the HP Proliant server and login to Linux server

1. connect null modem cable between A and B at com1 port (ttyS0), if it's com2, it's ttyS1
2. on Linux server A, vi /etc/inittab and append the followin line:

s0:2345:respawn:/sbin/agetty 115200 ttyS0 vt100

3. If machine B is Linux, use minicom to configure the serial port speed as 115200, then connect.

If your laptop doesn't have serial port, you can use 'usb to RS232 converter' cable (one side is usb interface, anther end of the cable is the male DB9 interface.

use 'lsusb' and dmesg to confirm Linux is using /dev/ttyUSB0 after inserting cable to Linux pc.

Configure minicom to use /dev/ttyUSB0 as com port and connect. If udev is running (such as ubuntu), then you don't have to create /dev/ttyUSB0, it will be automatic.

[root@jephe dev]# ./MAKEDEV ttyUSB0
[oracle@jephe dev]$ ls -l ttyUSB0
crw-rw---- 1 root uucp 188, 0 Nov 6 05:08 ttyUSB0

[root@jephe dev]# ./MAKEDEV ttyUSB1
[root@jephe dev]# ls -l ttyUSB1
crw-rw---- 1 root uucp 188, 1 Nov 6 05:08 ttyUSB1

4. If machine B is Windows, use putty to directly connect to serial port, set speed as 115200, then login

5. you can also use kermit to connect to serial port as follows:
kermit
set line /dev/ttyS0
set carrier-watch off
connect

FAQ:
a. how to change speed? - best practise.
vi /etc/inittab , for example, change 115200 to 9600, then comment out the line first, run 'init q' to re-read file, after that, uncomment it, issue 'init q' again.

b. how to enable root login through serail ports?
Just add both ttyS0 and ttyS1 or both into /etc/securetty.

Part II: How to use HP virtual serial port(VSP) to login Linux server

For HP Proliant DL serial servers, you can use VSP (Virtual Serial Port) with ILO2, you can basically ssh into ILO IP address with Administrator login. Add the following lines to /etc/inittab

s1:2345:respawn:/sbin/agetty 115200 ttyS1 vt100

then run 'init q' to enable it, after that, you can use vsp command to connect to this serial port login.
use ssh connect to ILO ip address:(if there's firewall in between, enable port 22)
for ILO version 1, just run 'REMCONS'.
for ILO2 , type 'vsp' to connect to ttyS0 or ttyS1 to get console screen.

According to http://blog.nachotech.com/?p=92 ,
A new feature crept in unannounced in the recent iLO 2 v1.77 firmware update from HP: You can now kill / terminate Virtual Serial Port (VSP) sessions that are open or hung. This is a very useful when you need to connect to the VSP but you (or someone else) is already connected from another location.

The new command to terminate VSP sessions is:

stop /system1/oemhp_vsp1

For reference, here are the two methods of starting VSP sessions in the iLO CLI:

vsp
or
start /system1/oemhp_vsp1

Before now, the only way to clear a hung VSP session was to reboot the iLO (a very intrusive action that takes 30-60 seconds), so this is a major improvement in usability of Virtual Serial Port sessions.

When ILO2 hangs, what you can do it :

a. make sure you are using the latest ILO2 firmware, if not , download one for your OS,then do online upgrade.

b. ssh into ILO, then run stop /system1/oemhp_vsp1

c. vi /etc/inittab, to comment out your virtual serial port line, then run init q

d. vi /etc/inittab again to uncomment that line , then run init q

e. try again 'vsp' command to access virtual serial port from ILO2 prompt

FAQ:
a. if you used ttyS0 and init q already, after that, you realize you should use ttyS1 for HP VSP serial port 2 which is 0x2F8.

You have to comment that line first, then int q, after that, change ttyS0 to ttyS1, otherwise, the ttyS0 is still listening , use ps ax to find out, please.

Part III - kernel console and GRUB serial console output
1. Enabling kernel console output so that you can view Linux server booting up messages on virtual console ttyS1 through vsp, then you should do this: (comment out splashimage and add two console options at kernel line for /etc/grub.conf)

default=0
timeout=5
#splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-164.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/VolGroup01/LogVol00 rhgb quiet console=tty0 console=ttyS1,115200
        initrd /initrd-2.6.18-164.el5.img

Note:This will send the console output to the serial port ttyS1 at a baud rate of 115200 as well as send the output to the regular console or "screen" tty0.
Remember: you have to put console=tty0 first then console=ttyS1 so that HP VSP can display those OK messages. Otherwise, those [OK] messages won't display on VSP.

2. Configure and use GRUB over Serial Console (redhat knowledge base)

The system's BIOS, bootloader (GRUB), Xen, Linux, and login access must each be individually configured for serial console access. It is not strictly necessary to have each component fully functional, but it can be quite useful.

For general information on serial console configuration under Linux, refer to the Remote Serial Console HOWTO at The Linux Documentation Project.

Enabling GRUB serial console output neither enables nor disables Xen or Linux serial capabilities, but may make remote management of the system more convenient by displaying GRUB prompts, menus, and actions over serial port and allowing remote GRUB management.

Add the following two lines to the GRUB configuration file /boot/grub/grub.conf will enable GRUB serial output.

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1

terminal --timeout=10 serial console

Note that when both the serial port and the local monitor and keyboard are enabled, the text "Press any key to continue" will appear at both. Pressing a key on one device will cause GRUB to display to that device. The other device will see no output. If no key is pressed before the timeout period expires, the system will boot to the default GRUB boot entry.

FAQ.
a. How to capture Linux installation screen
You can add the following to linux prompt when you boot from Redhat/CentOS installation CD:

linux kernel console=ttyS0, 115200 console=tty0

then you minicom or putty from another machine to connect to serial port at the same baud rate, you should be able to see and capture the screens.

Part IV: BIOS over serial redirection (vsp, hyper terminal or minicom)

under 'system option', set embedded serial port as com1 and virtual serial port as com2
under 'BIOS serial console & EMS', set BIOS serial console port as Auto(auto choose vsp or physical com1), and BIOS serial console band rate as 115200

You can user a null modem calbe to physically connect a PC/server to this HP Proliant server, and use Windows hyper terminal or minicom from Windows or Linux to connect to com1 to get BIOS over serial redirection or use gtkterm, cutecom etc GUI terminal program
or
use HP Virtual Serial Port (VSP) after ssh into ILO2 from Xshell or putty, then type in 'vsp'.

It will display the following message at the end of the bootup screen:

For access via BIOS Serial Consoleller
Press "ESC+1" to continue
Press "ESC+9" for ROM-Based Setup Utility
Press "ESC+0" for System Maintenance Menu
Press "ESC+@" for Network Boot

for ESC+9, you can use 'show config script' or 'show config options' under rbsu> (Rom-Based Setup Utility) prompt.

You can also type Esc 8 for configuring raid controller (instead of F8, use Esc +8 in vt100 terminal)

You'd better to use standard 80 (column) X 24 (rows) window for putty to get the better display over serial console when using VSP. This includes better grub menu display with highlighted selection bar, full bios startup messages, kernel console and service OK messages etc. It will display nicely in 80x24 standard size putty window through vsp.

For Esc + 9 (bios setup through serial console), you can use the following commands to show/set bios serial console baud rate:

show config script

set config bios serial console baud rate 115200

show config bios serial console baud rate

Part V: Summary of a typical HP Proliant server serial configuration

1. BIOS serial console redirection etc
Refer to part IV

2. /etc/grub.conf (for showing kernel message and GRUB on serial console)

default=0
timeout=8
#splashimage=(hd0,0)/grub/splash.xpm.gz
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1

terminal --timeout=10 serial console

title Linux Boot (2.4 custom)
        root (hd0,0)
        kernel /vmlinuz-2.4.cust ro root=LABEL=/ rhgb quiet console=tty0 console=ttyS0,115200

initrd /initrd-2.4.cust.img

note:

a. --unit=0 or --unit=1  (0 is for ttyS0 and 1 is for ttyS1 )

b. remove out 'rhgb quiet' will display all kernel hardware detecting messages.


3. serial console login (/etc/inittab)

s0:2345:respawn:/sbin/agetty 115200 ttyS0 vt100

s1:2345:respawn:/sbin/agetty 115200 ttyS1 vt100

4. enable direct serial port login

add ttyS0 and ttyS1 to /etc/securetty

Part VI: Windows EMS and VSP

The following is from

http://www.netexpertise.eu/en/windows/hp-ilo-console.html

Emergency Management Services (EMS), you can get a command line prompt and run some basic

basic such as configuring IP address and reboot Windows server.

You can modify boot.ini file as follows

right click on 'my computer', advanced, startup and recovery, click on settings, 'edit'

to add the following to boot loader part:
redirect=COM2
redirectbaudrate=115200
and append /redirect to the partition part

The example configuration is something like this:
[boot loader]
timeout=2
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
redirect=COM2
redirectbaudrate=115200
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows xxxxxx Edition" \ /noexecute=optout /fastdetect /redirect

then you can use ssh into ILO, then vsp to get EMS > prompt

note: Please check Windows operating system version for correct syntax for enabling EMS. referring to
http://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/EMS.doc
it might be EMSPort and EMSBandRate instead of redirect and redirectbandrate

Part VII: How to login serial console between 2 Linux server each other

You can connect a null modem cable between 2 Linux server, both at com1. Enable console login in /etc/inittab, then you can login from serial console each other.

For example, server A and B, when you try to login server B serial console from A, you need to comment out the line in /etc/inittab on server A, then run 'init q', then use minicom to connect to server B serial console. The same way for server A.

When you enable serial console login for Linux server, after init q, you should be seeing the following from ps command before you try to connect to it using minicom.
5617 ttyS0 Ss+ 0:00 /sbin/agetty 115200 ttyS1 vt100

If you are seeing something like this:
5617 ttyS0 Ss+ 0:00 /bin/login --

You should kill 5617 process first, make sure it's showing 5617 ttyS0 Ss+ 0:00 /sbin/agetty 115200 ttyS1 vt100, which means it's waiting for minicom connection, otherwise, it will not see login prompt from minicom

This way, you can use minicom to login server A and server B each other, as long as the client (login from) is not using ttySx in /etc/inittab, and the server(the one minicom tries to login) is using ttySx in /etc/inittab and it's waiting for connection (ps | grep tty - shows it's waiting )

References:

1. HP Proliant server VSP documentation (bios serial console redirection, etc
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf

2. Redhat knowledgebase - How do I set up a serial terminal and/or console in Red Hat Enterprise Linux?
http://kbase.redhat.com/faq/docs/DOC-7213

3. HP support website document - HP Integrated Lights-Out 2 (iLO 2) - Unable to Use Virtual Serial Port (VSP) http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=1146658&prodTypeId=18964&objectID=c01702793

4. HP BIOS Serial Console user guide - http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00440332/c00440332.pdf
note: Check above guide for vt100 special keys (such as using ESC + 8 for F8 )

5. HP ROM-Based Setup Utility User Guide (RBSU) - http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00191707/c00191707.pdf

6. Setup a serial console - http://www.howtoforge.com/setting_up_a_serial_console

7. http://www.linux.org/docs/ldp/howto/Remote-Serial-Console-HOWTO/index.html

8. iLo2 / VSP on RHEL5 missing console messages- http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1284427257960+28353475&threadId=1436569
note: put console=tty0 first then console=ttyS1, 115200 to display OK messages in VSP

9. How do I setup serial terminal and serial console for my xen kernel? (Reaht knowledge base DOC-9979)

For Xen kernels, the kernel and module line need to be modified. Remember that com1 = ttyS0, com2 = ttyS1, etc.

kernel /boot/xen.gz-2.6.18-6.el5fvfix com1=9600,8n1
module /boot/vmlinuz-2.6.18-6.el5fvfixxen ro root=LABEL=RHEL5_x86_64 console=tty0 console=ttyS0,9600n8
module /boot/initrd-2.6.18-6.el5fvfixxen.img

For example, here is the /etc/grub.conf:

#boot=/dev/sda
default=0
timeout=15
#splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

title Red Hat Enterprise Linux Server (2.6.17-1.2519.4.21.el5xen)
 root (hd0,0)
 kernel /xen.gz-2.6.17-1.2519.4.21.el5 com1=115200,8n1
 module /vmlinuz-2.6.17-1.2519.4.21.el5xen ro root=/dev/VolGroup00/LogVol00 
console=tty0 console=ttyS0,115200n8
 module /initrd-2.6.17-1.2519.4.21.el5xen.img

Modify the above to match the enviroment in which the serial terminal and serial console are being set up for.

10. Useful commands:
a. setserial -a /dev/ttyS0
b. setserial /dev/ttyS0 uart 16550A port 0x0408 irq 4

Linux SysAdmin and Oracle/MySQL DBA

Make sendmail working in a non-DNS environment

Installing hardware monitoring system for HP Proliant servers

How to login from serial port under Linux

Total Pageviews

Bookmark

Search This Blog

Followers

keywords

All Posts

About me