How to build Oracle VM Manager 3.1.1

Jephe Wu -

Environment: Dell R610 server, Oracle Linux 6.3, Oracle VM Manager 3.1.1 with patch 399, Oracle database enterprise edition with the latest patchset, Ksplice
Objective: install a fully patched Oracle VM Manager 3.1.1 production host

1. BIOS config
Enable logical processor(hyper trending), virtualization, Turbo mode, disable C1E, C states, node interleaving(enable NUMA)

In Oracle VM server 3.1.1, to check if C-state is enabled or not, use the following command to check if the total C-state is not more than 2.

# xenpm get-cpuidle-states | grep total | uniq
total C-states : 2

or in Linux, check it :
C-state: cat /proc/acpi/processor/CPU*/power
P-state: cat /sys/devices/system/cpu/cpu*/cpufreq/stats/total_tran
         cat /sys/devices/system/cpu/cpu*/cpufreq/stats/time_in_state

# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
node 0 size: 49139 MB
node 0 free: 33832 MB
node distances:
node 0
0: 10

How to disable NUMA in Linux OS and Oracle:
pass numa=off to kernel boot parameter in grub.conf, you can check the current kernel parameter in /proc/cmdline.
According to MOS: Enable Oracle NUMA support with Oracle Server Version [ID 864633.1]:  Starting with, Oracle NUMA support is disabled by default.
example output: NUMA is disabled.

[root@util01 ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
CPU socket(s):         2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Stepping:              2
CPU MHz:               2660.346
BogoMIPS:              5319.79
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-23

[root@util01 ~]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
node 0 size: 49139 MB
node 0 free: 40698 MB
node distances:
node   0 
  0:  10 

power mode - use maximum performance mode
References:  check general bios settings and power management settings part.

2.  Configuring iDRAC ip address, netmask and gateway

After you have done this part, you are able to change it through DRAC and dell openmanage for Linux later on. The default DRAC username and password is root/calvin.

How to reset system by sshing into DRAC?
# racadm serveraction hardreset

3.  OS installation through kickstart by foreman

Preparing tftp/dhcp/[dns] server first by PXELinux, DNSmasq or cobber for foreman, we use traditional PXELinux here.
service tftp
    socket_type     = dgram
    protocol        = udp
    wait            = yes
    user            = root
    server          = /usr/sbin/in.tftpd
    server_args     = -vvv -s /tftpboot
    disable         = no
    per_source      = 11
    cps             = 100 2
    flags           = IPv4

under /tftpboot/pxelinux.cfg, put the following kernel files:

-rw-r--r-- 1 root root 30481341 Aug 20 15:27 OL-6.3-x86_64-initrd.img
-rw-r--r-- 1 root root  3986992 Aug 20 15:27 OL-6.3-x86_64-vmlinuz

# more /tftpboot/pxelinux.cfg/default
default linux
label linux
localboot 0  (important, boot into local hard disk if tftp failed, although DHCP is okay)

Preparing foreman
# more /etc/foreman/settings.yaml
:modulepath: /etc/puppet/modules/
:tftppath: /tftpboot/
:unattended: true

Preparing foreman-proxy

# cat /etc/foreman-proxy/settings.yml| grep -v ^# | grep -v ^$
:daemon: true
:daemon_pid: /var/run/foreman-proxy/
:port: 8443
:tftp: true
:tftproot: /tftpboot
:dns: false
:dhcp: true
:dhcp_vendor: isc
:dhcp_config: /etc/dhcp/dhcpd.conf
:dhcp_leases: /var/lib/dhcpd/dhcpd.leases
:puppetca: true
:puppet: true
:log_file: /tmp/proxy.log
:log_level: DEBUG

Preparing DHCP server

#  more /etc/dhcp/dhcpd.conf
allow booting;
allow bootp;
omapi-port 7911;
deny unknown-clients;

subnet netmask {
  option broadcast-address;
  option routers;
  option domain-name "jephe";
    option domain-name-servers;
  filename "pxelinux.0";


After adding util01.jephe by foreman, it appears in /var/lib/dhcpd/dhcpd.leases as follows:

server-duid "xxxxxx";

host util01.jephe {
  hardware ethernet d4:be:d9:ef:aa:bb;
        supersede server.filename = "pxelinux.0";
        supersede = aa:bb:cc:dd; #actually, it's hex code of tftp server ip addr in /etc/foreman-proxy/settings.yml
        supersede host-name = "util01.jephe";

access DeLL iDRAC at https://dracipaddress , system, properties, EMBEDDED NIC MAC ADDRESSES to get onboard nic1 MAC address
so that you can configure foreman.

4. foreman configuration in GUI interface
kickstart config for Oracle Linux 6.3:
lang en_US
keyboard us
url --url http://repo.jephe/OL/6.3/os/x86_64/
rootpw --iscrypted $6$ruyiAJ82a/9Bazd7$zspMbTz4e7omVlb12uTa.bjuvc8SuxcGoVHgJv3Y3r1m/4.7tONDUzPVpJpYWA85uXaaaaaaaaaaaaa
firewall --disabled
selinux --disabled
authconfig --enableshadow --enablemd5
timezone --utc Australia/Sydney
bootloader --location=mbr
<%= @host.diskLayout %>
# Oracle recommended tigervnc instead of its own one

%post --log=/root/kickstart-post.log
echo "Syncing system date and update bios date"
ntpdate puppet.jephe
clock --systohc

# Disable most things. Puppet will activate these if required.
echo "Disabling various system services..."
<% %w{autofs gpm atd abrt-ccpp abrt-oops abrtd acpid cpuspeed rpcbind rpcgssd rpcidmapd sendmail cups iptables ip6tables auditd arptables_jf xfs pcmcia isdn rawdevices hpoj bluetooth

openibd avahi-daemon avahi-dnsconfd hidd hplip pcscd restorecond mcstrans rhnsd yum-updatesd}. each do |service| -%>
  /sbin/chkconfig --level 345 <%= service %> off 2>/dev/null
<% end -%>
echo "Running puppet..."
#/usr/sbin/puppetd --debug --no-daemonize --verbose --onetime

# Force the right hosts file
echo "generating correct /etc/hosts"
IP=`ifconfig em1 | grep "inet " | cut -d : -f 2 | cut -d " " -f 1`
cat > /etc/hosts <<EOF
# Do not remove the following line, or various programs
# that require network functionality will fail.     localhost.localdomain localhost
::1           localhost6.localdomain6 localhost6
`echo -n $IP`         $(hostname)  $(hostname -s)

# generate dell openmanage script
cat > /root/ <<END
export http_proxy=http://puppet.jephe:8080
wget -q -O - | bash
yum -y -e 0 install srvadmin-all
yum -y -e 0 install dell_ft_install
yum -y -e 0 install \$(bootstrap_firmware)
echo "now you can run  inventory_firmware then update_firmware"

chmod +x /root/

echo "Informing Foreman that we are built.."
wget -q -O /dev/null --no-check-certificate http://puppet.jephe:3000/unattended/built

After install dell openmanager, you can access it through https://serveripaddress:1311, login with root password

a. refresh cdrom installation anaconda config for VM server 3.1.1 (use eth4 as ovsagent interface and add to vlan 1001)

eula Accepted
lang en_US.UTF-8
keyboard us
network --device eth4 --bootproto static --ip --netmask --gateway --nameserver, --hostname util01.jephe
ovsagent --iscrypted X12345TkAjLZk+TLABCDEFGHJIg=
ovsmgmntif eth4.1001
rootpw --iscrypted $1$abcde$O5ngOGU.0eXsXIfk6QB2B.
firewall --enabled --port=21:tcp --port=22:tcp --port=53:udp --port=53:tcp --port=80:tcp --port=2049:tcp --port=5900-7999:tcp --port=8002:tcp --port=8003:tcp --port=8899:tcp --port=7777:tcp
authconfig --enableshadow --enablemd5
selinux --disabled
timezone --utc Australia/Sydney
bootloader --location=mbr --dom0_mem=5800 --driveorder=sda
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
#clearpart --all --drives=sda
#part /boot --fstype ext3 --size=100 --ondisk=sda
#part / --fstype ext3 --size=3072 --ondisk=sda
#part swap --size=1024 --ondisk=sda


[root@util01 init.d]# openssl passwd -1 -salt abcde "password"

According to Oracle support, you can use the following in %post to display on screen
%post --log=/root/kickstart-post.log
#actually, if you specify tee /root/ks-post_install.log, you can ignore this --log as it won't generate anything inside

exec < /dev/tty3 > /dev/tty3
chvt 3

) 2>&1 | /usr/bin/tee /root/ks-post_install.log
chvt 1

b. without static ip address, use generate kickstart config file for everything
eula Accepted
lang en_US.UTF-8
keyboard us
timezone Australia/Sydney
ovsagent --iscrypted XM3Ve5TkAjLZAAAAAAAAAAAAAAA
ovsmgmntif eth0
url --url http://repo.jephe/xen/3.1.1/os/x86_64/
rootpw --iscrypted $1$/NL6hNln$mVk8AAAAAAAAAAAAAA.
firewall --disabled
selinux --disabled
authconfig --enableshadow --enablemd5
timezone  Australia/Sydney
bootloader --location=mbr
<%= @host.diskLayout %>

%post --log=/root/kickstart-post.log
echo "Informing Foreman that we are built.."
wget -q -O /dev/null --no-check-certificate http://puppet.jephe:3000/unattended/built

And diskLayout is as follows:
clearpart --all --drives=sda
part /boot --fstype ext3 --size=100 --ondisk=sda
part / --fstype ext3 --size=3072 --ondisk=sda
part swap --size=1024 --ondisk=sda

c. /tftpboot/pxelinux.cfg/01-MACaddress

[root@puppet pxelinux.cfg]# more 01-d4-ae-52-aa-bb-cc
default linux
label linux
kernel boot/OVS-OVS3.1-vmlinuz
append initrd=boot/OVS-3.1-OVS3.1-initrd.img ks=http://puppet.jephe:3000/unattended/provision ksdevice=bootif network kssendmac
ipappend 2

or use the following for specific ip address
append initrd=boot/OVS-3.1-OVS3.1-initrd.img ks=http://puppet.jephe:3000/unattended/provision ksdevice=bootif
network --device eth5 --bootproto static --ip --netmask --gateway --nameserver, --hostname xens01.jephe kssendmac

or you can use general pxelinux.cfg, but in foreman provision script, use if-then to dynamic generate ipaddress for different hostname as folows:
echo "configuring correct IP address for bond1.2091 for `hostname`"
if [ `hostname` == 'xens01.jephe' ]
  cat >> ifcfg-bond1.91 <<EOF
elif [ `hostname` == 'xens02.jephe' ]
   cat >> ifcfg-bond1.91 <<EOF

5. ksplice and ULN
quickstart guide:
firstly, get access key for ksplice,
then run the following command

wget -N
sh install-uptrack 123445555kklkkddkdkakdk
uptrack-upgrade -y

configuring https_proxy in /etc/uptrack/uptrack.conf

Uninstall of Uptrack is very easy. You should consider removing the updates with
uptrack-remove --all followed by #yum remove uptrack.

[root@util01 uptrack]# yum remove uptrack
Loaded plugins: dellsysid, rhnplugin, security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package uptrack.noarch 0:1.2.2-0.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

 Package                                        Arch                                          Version                                               Repository                                               Size
 uptrack                                        noarch                                        1.2.2-0.el6                                           @ksplice-uptrack                                        1.1 M

Transaction Summary
Remove        1 Package(s)

Installed size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : uptrack-1.2.2-0.el6.noarch                                                                                                                                                                     1/1
warning: /etc/uptrack/uptrack.conf saved as /etc/uptrack/uptrack.conf.rpmsave
  Verifying  : uptrack-1.2.2-0.el6.noarch                                                                                                                                                                     1/1

  uptrack.noarch 0:1.2.2-0.el6                                                                                                                                                                                  


Note: this will clear /var/cache/uptrack directory .

go to to register which binds your SSO to ULN, then run uln_register --proxy= to register Oracle Linux to ULN, proxy is optional, during text gui interface, it will ask you SSO, CSI etc information for register.

run yum repolist to check if the following are listed.

ol6_x86_64_UEK_latest                                                                                          Latest Unbreakable Enterprise Kernel for Oracle Linux 6 (x86_64)                                                                                             71
ol6_x86_64_latest                                                                                              Oracle Linux 6 Latest (x86_64)  

then run yum update to update system.

6.  install Dell openmanage after os install
refer to

Install OpenManage Server Administrator

Step 1: wget -q -O - | bash
Step 2: yum -y install srvadmin-all
Step 3: Start  Systems Management Data Engine - /etc/init.d/dataeng start
Step 4: Start webinterface - /etc/init.d/dsm_om_connsvc start

Your system homepage is now availible via https://hostname:1311, where hostname is the hostname of your server.

Install Firmware/Bios Management Tools

Step 1: yum install dell_ft_install
Step 2: yum install $(bootstrap_firmware)

Now run either inventory_firmware or inventory_firmware_gui to view your firmware versions, and perform firmware upgrades. Also, you can access server ip dell openmanage at https://serveripaddress:1311/

7. Dell Nagios plugin installation for hardware monitoring

Go to to download epel

rpm -Uvh
then download perl-Config-Tiny rpm from public yum repository:

then install Dell openmanage nagios plugin by running

yum install nagios-plugins-openmanage

[root@util02 yum.repos.d]# /usr/lib64/nagios/plugins/check_openmanage
OK - System: 'PowerEdge R610 II', SN: '5HZXXX', 48 GB ram (6 dimms), 1 logical drives, 2 physical drives

8. Silent install database

unzip downloaded Oracle database 64bit, unzip it, find *.rsp file

[root@util01 database]# cat inst.rsp | grep -v ^$ | grep -v ^# | grep -v =$

[root@outil01 database]# mkdir -p /u01/app/oracle/
[root@util01 database]# chown oracle:oinstall /u01/ -R
[root@util01 database]# su - oracle
[oracle@util01 database]$ pwd
[oracle@util01 database]$ ./runInstaller -ignoreSysPrereqs -force -silent -responseFile /home/oracle/database/inst.rsp

9. configure database environment

create /usr/local/bin/oraset
[oracle@util01 bin]$ ls -l oraset
-rw-r--r-- 1 oracle oinstall 149 Aug 27 16:54 oraset
[oracle@util01 bin]$ more oraset
export PATH=$PATH:/home/oracle/bin
export ORACLE_SID=orcl

. oraenv

[oracle@util01 bin]$ ls -l ~/.bashrc
-rw-r--r-- 1 oracle oinstall 149 Aug 27 16:54 /home/oracle/.bashrc

[oracle@util01 bin]$ more ~/.bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc

# User specific aliases and functions

. /usr/local/bin/oraset >/dev/null 2>&1

Note: if you don't put >/dev/null 2>&1, the scp file to this host from other server may fail.

[root@util01 init.d]# more oracle
# chkconfig: 35 98 10
# description: Starts and stops Oracle processes 

# startup/shutdown sequence,  the lower the number is , the high priority it gets, for shutdown, 5 shutdown first, 10 shutdown later


case "$1" in 
    su - oracle -c "dbstart;lsnrctl start;emctl start dbconsole"
  touch $LOCKFILE
    su - oracle -c "emctl stop dbconsole;lsnrctl stop;dbshut"
    rm -f $LOCKFILE

Note: those red color part is very important, otherwise, auto shutdown won't work although auto start will be working as Redhat-alike system doesn't think oracle is running when rebooting if it cannot see oracle in /var/lock/subsys/, so ignore it.

# chkconfig --add oracle
# chkconfig oracle --level 35 on

You also need to modify dbstart and dbshut file as follows:

[oracle@util01 bin]$ grep -B 1 ORACLE_HOME_LISTNER dbstart

[oracle@util01 bin]$ grep -B 1 ORACLE_HOME_LISTNER dbshut

[root@util01 init.d]# ls -l oracle
-rwxr-xr-x 1 root root 354 Aug 30 14:31 oracle

a more complete auto stop/start file is as follows: Please refer to

# Run-level Startup script for the Oracle Instance and Listener
# chkconfig: 345 91 19
# description: Startup/Shutdown Oracle listener and instance


# if the executables do not exist -- display error

if [ ! -f $ORA_HOME/bin/dbstart -o ! -d $ORA_HOME ]
        echo "Oracle startup: cannot start"
        exit 1

# depending on parameter -- startup, shutdown, restart 
# of the instance and listener or usage display 

case "$1" in
        # Oracle listener and instance startup
        echo -n "Starting Oracle: "
        su - $ORA_OWNR -c "$ORA_HOME/bin/lsnrctl start"
        su - $ORA_OWNR -c $ORA_HOME/bin/dbstart
        touch /var/lock/subsys/oracle
        echo "OK"
  # Oracle listener and instance shutdown
        echo -n "Shutdown Oracle: "
        su - $ORA_OWNR -c "$ORA_HOME/bin/lsnrctl stop"
        su - $ORA_OWNR -c $ORA_HOME/bin/dbshut
        rm -f /var/lock/subsys/oracle
        echo "OK"
        $0 stop
        $0 start
        echo "Usage: $0 start|stop|restart|reload"
        exit 1
exit 0

10. patch to the latest patchset
download OPATH latest version first:
How To Download And Install The Latest OPatch Version [ID 274526.1]

install opatch

oracle@util01 ~]$ cd $ORACLE_HOME
[oracle@util01 db_1]$ mv OPatch/ OPatch.orig
[oracle@util01 db_1]$ unzip ~/

11. Patch Oracle database to the latest pathset

sqlplus / as sysdba
sql> shutdown immediate;
sql> exit
lsnrctl stop
emctl stop dbconsole

[oracle@util01 13923374]$ export PATH=$PATH:/u01/app/oracle/product/11.2.0/db_1/OPatch
[oracle@util01 13923374]$ opatch apply

Check the readme.html file from the patch, run below for database upgrade.
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
sql> connect / as sysdba
sql> startup
sql> @catbundle_PSU_ORCL_ROLLBACK.sql
sql> quit

check logfile for any errors: $ORACLE_HOME/cfgtoollogs/catbundle/catbundle_PSU_ORCL_ROLLBACK_timestamp.log

12. install Oracle VM Manager
Download OVMM original and patch 399 ISO

Preparing config.yml for silent OVMM installation

[root@util01 oracle]# more config.yml
install : No
host : localhost
sys_password : oracle
port : 1521
sid : orcl
mgmt_port : 8080
user : ovs
password : oracle

install : Yes
user : weblogic
password : oracle

install : Yes

install : Yes

install : Yes
user : admin
password : oracle

install : Yes

install : Yes

cleanup : Yes

install : Yes

13  Preparing Oracle environment

[root@util01 ovmm]# mount Oracle_VM_Manager_3.1.1_64bit.iso ovmm -o loop
[root@util01 ovmm]# ./
Adding group 'oinstall' with gid '54323' ...
groupadd: group 'oinstall' already exists
Adding group 'dba'
groupadd: group 'dba' already exists
Adding user 'oracle' with user id '54322', initial login group 'dba', supplementary group 'oinstall' and  home directory '/home/oracle' ...
User 'oracle' already exists ...
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba)
Creating user 'oracle' succeeded ...
Verifying user 'oracle' OS prerequisites for Oracle VM Manager ...
oracle    soft        nofile                    8192
oracle    hard       nofile                    8192
oracle    soft        nproc                    4096
oracle    hard       nproc                   4096
oracle    soft        core                       unlimited
oracle    hard       core                       unlimited
Setting  user 'oracle' OS limits for Oracle VM Manager ...
Altered file /etc/security/limits.conf
Original file backed up at /etc/security/limits.conf.orabackup
Verifying & setting of user limits succeeded ...
Modifying iptables for OVM
Adding rules to enable access to:
     7001  : Oracle VM Manager http
     7002  : Oracle VM Manager https
     15901 : Oracle VM Manager VM console proxy
     54321 : Oracle VM Manager core
     54322 : Oracle VM Manager core via SSL
       123 : NTP
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
Rules added.

when OVMS and OVMM are not on the same network, port 7001 is needed ,
OVMM events shows BindingMismatchEvent critical python bindings mismatch .

14. actual installation of OVMM 3.1.1
/ --config=/home/oracle/config.yml --installtype=production –assumeyes

15.  Upgrade Oracle VM manager to 399
mount iso file as -o loop, then
#./ as root

access OVM at  http://ip:7001/ovm/console

16. database archive log mode and rman
oracle@util01 u01]$ sqlplus / as sysdba
SQL*Plus: Release Production on Wed Aug 22 15:41:19 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> alter system set log_archive_dest_1='location=/u01/oraarch' scope=both;
SQL> alter system set log_archive_format='%t_%s_%r.arc' scope=spfile;
SQL> shutdown immediate;
SQL> startup mount;
SQL> alter database archivelog;
SQL> alter database open;

$ rman target /
Rman> show all;
Rman> configure controlfile autobackup on;
Rman> backup database plus archivelog format ‘/u01/orarman/%U’

17. after renaming database host, re-generate em repository
modify the following files first.


[oracle@util01 bin]$ ./emca -deconfig dbcontrol db -repos drop
[oracle@util01 bin]$ ./emca -config dbcontrol db -repos create

18. known issue
If you configured vlan for em1 or bond0(em1 and em2), you might not able to start ovmm service with https port 7002.
The solutions is to disable ipv6 as follows:

a. put the following in /etc/modprobe/ipv6.conf
options ipv6 disable=1

b. append the following into /etc/sysconfig/network

19. references:

kickstart -  4 steps to update dell firmware from Linux  Nagios plugin to check dell openmanage

No comments:

Post a Comment