How to Install Oracle VM Server 3.2 and 3.3 from USB Key

Jephe Wu - http://linuxtechres.blogspot.com


Objective:  to install Oracle VM server 3.2.8 or 3.3.2 from USB key for those server without DVD/CDROM drive

Steps on Linux Mint 17 

Preparing USB key

Make partition by fdisk /dev/sdb1, make it as W95 FAT32 (b) type

fdisk /dev/sdb

L , b , wq

format it by running mkfs -t vfat /dev/sdb1

use unetbootin 

install unetbootin software from Mint 17 software manager. 

copy ISO file into /dev/sdb1

mount /dev/sdb1 under Linux mint , copy OVS3.2.8 iso file to it directly, without creating any folder.

boot from USB to install OVMS 3.2.8

choose installation method 'Hard drive', selection partiton /dev/sdb1 which contain installation image 

at partition type screen, deselect sdb

at 'boot loader configuration' screen, choose 'change drive order' to use sda, then use sda to install MBR

Steps on Windows

Install unetbootin

format USB key as FAT format

use unetbootin to write iso file to it as above

copy iso file into USB key directly

FAQ

format it as ext2 or ext3 and use unetbootin to write iso file doesn't boot if you didn't install extlinux package from Mint software manager.

Notes

The size of OVM 3.3 iso file size is now doubled compared to OVMS 3.2.9. 1GB USB key is now not able to fit.

How to decide ethX layout after Oracle VM Server 3.2 Installation

Jephe Wu - http://linuxtechres.blogspot.com

Objective: to decide ethX layout remotely after Oracle VM Server 3.2 installation.
Environment: Oracle VM server 3.2.9

If you are in front of the physical server, you can decide ethX layout easily by running 'ethtool -p ethX' etc to blink each NIC light. e.g. ethtool -p eth0

How Oracle VM server to decide which NIC for which ethX
based on pci bus address from low to high to assign eth0, eth1 etc

Theory behind

• Os will assign ethX sequence based on NIC PCI bus address sequence
• 4 NICs onboard are actually 2 dualports cards
• For additional PCI card, the ports near the bottom pins are getting high pci bus number, from ethX is assigned from top to bottom.
• Sometimes, pci bus address has primary-secondary relationships, 

Check steps

Get server model, pci NIC slot direction, PCI numbering sequence physically etc first

/sys/class/net 

[root@cpnrglobovms13 device]# more /sys/class/net/eth0/device/label
Intel 82576 Ethernet Controller.

to decide pci bus address and ethX alighment

ll /sys/class/net

List ethX bus information - ethtool -i ethX

[root@ovms01 ~]# for i in `ifconfig -a | grep ^eth | awk '{print $1}' | xargs`; do echo -n "$i " && ethtool -i $i| grep -i bus ; done
eth0 bus-info: 0000:01:00.0
eth1 bus-info: 0000:01:00.1
eth2 bus-info: 0000:02:00.0
eth3 bus-info: 0000:02:00.1
eth4 bus-info: 0000:04:00.0
eth5 bus-info: 0000:04:00.1
eth6 bus-info: 0000:07:00.0
eth7 bus-info: 0000:07:00.1
eth8 bus-info: 0000:08:00.0
eth9 bus-info: 0000:08:00.1

lspci to get ethernet NICs - lspci | grep -i eth

[root@ovms01 ~]# lspci | grep -i eth
01:00.0 Ethernet controller: Broadcom Corporation Device 165f
01:00.1 Ethernet controller: Broadcom Corporation Device 165f
02:00.0 Ethernet controller: Broadcom Corporation Device 165f
02:00.1 Ethernet controller: Broadcom Corporation Device 165f
04:00.0 Ethernet controller: Intel Corporation Device 154d (rev 01)
04:00.1 Ethernet controller: Intel Corporation Device 154d (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

lspci bridge to secondary  - lspci -vvv | grep primary

05:00 -> 06:00 -> 07:00 and 08:00

[root@hmspglobovms01 device]# lspci -vvv | grep primary
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
Bus: primary=00, secondary=09, subordinate=09, sec-latency=0
Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
Bus: primary=00, secondary=0b, subordinate=0f, sec-latency=0
Bus: primary=00, secondary=10, subordinate=10, sec-latency=32
Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
Bus: primary=0b, secondary=0c, subordinate=0f, sec-latency=0
Bus: primary=0c, secondary=0d, subordinate=0e, sec-latency=0
Bus: primary=0c, secondary=0f, subordinate=0f, sec-latency=0
Bus: primary=0d, secondary=0e, subordinate=0e, sec-latency=0

dmidecode to list PCI slot information 

[root@ovms01 ~]# dmidecode -t 9
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x0900, DMI type 9, 17 bytes
System Slot Information
Designation: PCI1
Type: x8 <OUT OF SPEC>
Current Usage: In Use
Length: Long
Characteristics:
3.3 V is provided
PME signal is supported
Bus Address: 0000:04:00.0

Handle 0x0901, DMI type 9, 17 bytes
System Slot Information
Designation: PCI2
Type: x16 <OUT OF SPEC>
Current Usage: In Use
Length: Long
Characteristics:
3.3 V is provided
PME signal is supported
Bus Address: 0000:05:00.0

dmidecode to get onboard device information

[root@ovms01 ~]# dmidecode -t 41  or dmidecode -t slot
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x2900, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:01:00.0

Handle 0x2901, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:01:00.1

Handle 0x2902, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 3
Type: Ethernet
Status: Enabled
Type Instance: 3
Bus Address: 0000:02:00.0

Handle 0x2903, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 4
Type: Ethernet
Status: Enabled
Type Instance: 4
Bus Address: 0000:02:00.1

lspci to get ethernet NIC information
[root@ovms01 ~]# lspci | grep -i ethernet

dmidecode to get PCI card information
[root@ovms01 ~]# dmidecode -t slot | grep -e Designation -e Bus

use ethtool to decide bus info
[root@ovms10 ~]# for i in `ifconfig -a | grep ^eth | awk '{print $1}' | xargs`; do echo -n "$i " && ethtool -i $i| grep -i bus ; done 
eth0 bus-info: 0000:01:00.0
eth1 bus-info: 0000:01:00.1
eth2 bus-info: 0000:04:00.0
eth3 bus-info: 0000:04:00.1
eth4 bus-info: 0000:42:00.0
eth5 bus-info: 0000:42:00.1
eth6_rename bus-info: 0000:07:00.0
eth7_rename bus-info: 0000:07:00.1

dmesg to get PCI slot cards MAC address

[root@ovms01 ~]# dmesg | grep 'PCI Express'

pciehp: PCI Express Hot Plug Controller Driver version: 0.4
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
ixgbe 0000:08:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a6:80
ixgbe 0000:08:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a6:81
ixgbe 0000:0e:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a5:ac
ixgbe 0000:0e:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a5:ad
ixgbe 0000:0f:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:58:20
ixgbe 0000:0f:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:58:21
bnx2 0000:01:00.0: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 36, node addr d4:ae:52:e6:83:a0
bnx2 0000:01:00.1: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 48, node addr d4:ae:52:e6:83:a2
bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 37, node addr d4:ae:52:e6:83:a4
bnx2 0000:02:00.1: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 49, node addr d4:ae:52:e6:83:a6

to get each PCI bus address MAC address:

[root@ovms1 ~]# dmesg | grep 02:00.0 | grep 'PCI Express'

Kickstart to swap eth2/eth3 and eth6/eth7 for better reading if necessary

echo "fixing ethX device sequence"
ETH2=`dmesg | grep Express | grep "30:00.0" | awk '{print $NF}'`
ETH3=`dmesg | grep Express | grep "30:00.1" | awk '{print $NF}'`
ETH6=`dmesg | grep Express | grep "88:00.0" | awk '{print $NF}'`
ETH7=`dmesg | grep Express | grep "88:00.1" | awk '{print $NF}'`


cd /etc/sysconfig/network-scripts
sed -i -e "s/^HWADDR.*/HWADDR=$ETH6/g" ifcfg-eth2
sed -i -e "s/^HWADDR.*/HWADDR=$ETH7/g" ifcfg-eth3
sed -i -e "s/^HWADDR.*/HWADDR=$ETH2/g" ifcfg-eth6
sed -i -e "s/^HWADDR.*/HWADDR=$ETH3/g" ifcfg-eth7

Draw ethX layout at last step

example below:

PCI1

RAID Controller



PCI2

00:42.00.1(eth5)(pin) 00:42.00.0(eth4)



PCI3

00:04:00.0(eth2) 00:04:00.1(eth3)(pin)






eth0

00:01:00.0

eth1

00:01:00.1

eth6 (X)

00:07:00.0

eth7 (X)

00:07:00.1

Ultimate way to associate ethX with PCI bus address

[root@ovms10 rules.d]# more 99-ethernet.rules

KERNEL=="eth*", ID=="0000:01:00.0", name="eth0"

KERNEL=="eth*", ID=="0000:01:00.1", name="eth1"

KERNEL=="eth*", ID=="0000:04:00.0", name="eth2"

KERNEL=="eth*", ID=="0000:04:00.1", name="eth3"

KERNEL=="eth*", ID=="0000:42:00.0", name="eth4"

KERNEL=="eth*", ID=="0000:42:00.1", name="eth5"

KERNEL=="eth*", ID=="0000:07:00.0", name="eth6"

KERNEL=="eth*", ID=="0000:07:00.1", name="eth7"

[root@ovms10 rules.d]# ethtool -i eth0

driver: ixgbe

version: 3.4.8-k

firmware-version: 4.2-8

bus-info: 0000:01:00.0

Keep Oracle VM server ethX name consistent across reboot

Jephe Wu - http://linuxtechres.blogspot.com

Problem: random assigned ethX name with MAC address and renamed ethX to something else, different /etc/issue output after each reboot

Environment: Oracle VM server 3.1.1, PCI card with multiple NIC ports


Steps:

1. use DRAC and lspci to check pci bus info

Open DRAC, network device, check each MAC address for NIC


2. draw network diagram for each NIC MAC address


3. assign ethX to pci slot 

create 99-ethernet.rules under /etc/udev/rules.d as follows:


KERNEL=="eth*", ID=="0000:05:05.0", NAME="eth0"
KERNEL=="eth*", ID=="0000:0b:00.0", NAME="eth1"

Note: The ID can be determined by running lspci | grep Eth and prefixing the appropriate number with 0000: or else by using ethtool -i <DEVICE> | grep bus-info.

If using udev rules like this in RHEL5, kudzu should be disabled (chkconfig kudzu off) so that it does not interfere by modifying the /etc/sysconfig/network-scripts/ifcfg-* files.

4. remove HWADDR from ifcfg-ethX

cd /etc/sysconfig/network-scripts/
for i in ifcfg-eth*;do sed -i 's#HWADDR#\#HWADDR#g' $i;done

5. use real driver in /etc/modprobe.conf

cd /etc/
vi modprobe.conf to use correct kernel driver for ethX

ethtool -i eth0
or lspci -vvv 0000:04:00.0

Reference

RHEL5: How to make NIC names persistent across reboots

https://access.redhat.com/site/solutions/16411

How to mount Oracle VM guest image and fsck it

Jephe Wu - http://linuxtechres.blogspot.com

Objective: modify and fix Oracle VM guest OS image file
Environment: Oracle VM server 3.1.1, Oracle Linux 5.8 64bit OS image


Steps:

1. mount system image as follows

[root@ovms03 db02]# file /OVS/Repositories/0004fb00000300009471703af52bc769/VirtualDisks/0004fb00001200006ff9f87b9279c6dd.img
/OVS/Repositories/0004fb00000300009471703af52bc760/VirtualDisks/0004fb00001200006ff9f87b9279c6dr.img: x86 boot sector; partition 1: ID=0x83, active, starthead 0, startsector 2048, 208896 sectors; partition 2: ID=0x83, starthead 0, startsector 210944, 24954880 sectors, code offset 0x48
Note: there are two partitions inside this image file.

[root@ovms03 db02]#  mount -o offset=$((512*210944)) /OVS/Repositories/0004fb00000300009471703af52bc760/VirtualDisks/0004fb00001200006ff9f87b9279c6dr.img /mnt/a
[root@ovms03 db02]#  mount -o offset=$((512*2048))
/OVS/Repositories/0004fb00000300009471703af52bc760/VirtualDisks/0004fb00001200006ff9f87b9279c6dr.img /mnt/a/boot/

2. chroot to it and modify accordingly
cd /mnt/a
chroot .
chkconfig --list
vi /etc/hosts
umount /mnt/a/boot
umount /mnt/a

3. fsck image file if required.

[root@ovms03 db02]# losetup /dev/loop0 /OVS/Repositories/0004fb00000300009471703af52bc769/VirtualDisks/0004fb00001200006ff9f87b9279c6dr.img -o $((512*210944))
[root@ovms03 goepsdb02]# fsck /dev/loop0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/: clean, 147679/2985984 files, 955713/3119360 blocks

[root@ovms03 goepsdb02]# losetup -d /dev/loop0

4. Other useful commands:

xm create vm.cfg

xm list

xm shutdown id

xm destroy id

How to startup OL6 as PVM under Oracle VM 2.X

Jephe Wu - http://linuxtechres.blogspot.com

Problem:  Oracle VM 2.2 cannot startup default OL 6.3 as PVM (Oracle Linux 6.3 VM is created from downloaded assembly .ova file), although it can startup as HVM.


Error message:
After xm create vm1.cfg, it immediately return error like this:

 Error: Boot loader didn't return any data

Reason:  Oracle VM 2.2 cannot read ext4 /boot partition to find kernel to boot up as PVM
Soluton:  bootup this OL 6.3 on Oracle VM 3 hypervisor,  then convert /boot to ext3, then copy system.img to Oracle VM 2 to use.

Steps:
1. check image file 
[root@dr01 xen]# file vm1.img
vm1.img: x86 boot sector, GRand Unified Bootloader (0.94); partition 1: ID=0x83, active, starthead 0, startsector 2048, 208896 sectors; partition 2: ID=0x83, starthead 0, startsector 210944, 20760576 sectors; partition 3: ID=0x82, starthead 63, startsector 20971520, 4194304 sectors, code offset 0x48

[root@dr01 xen]# mount dr01.img -o offset=$((512*2048)) /tmp/a
mount: unknown filesystem type 'ext4'

note: /boot partition is ext4 which is not recongized by Oracle VM server 2.2

2. boot up image on OVM3.1.1 then convert /boot from ext4 to ext3

#boot up OL 6.3 64bit VM as PVM on Oracle VM server 3.1.1
cp /boot /boot2 -va
umount /boot
mkfs -t ext3 /dev/xvda1
vi /etc/fstab to change /boot partition type from ext4 to ext3
mount /boot
cp /boot2/* /boot/ -va
sync
poweroff

3. boot up new image with 'xm create vm1.cfg' as follows

[root@dr01 xen]# more vm1.cfg
vif = ['type=netfront,bridge=fe', 'type=netfront,bridge=stor']
OVM_simple_name = 'vm1'
disk = ['file:/xen/vm1.img,xvda,w']
uuid = '0004fb00-0006-0010-07aa-15a5c3b8c419'
on_reboot = 'restart'
cpu_weight = 27500
memory = 4096
cpu_cap = 0
maxvcpus = 4
OVM_high_availability = False
maxmem = 4096
OVM_description = ''
on_poweroff = 'destroy'
on_crash = 'restart'
bootloader = '/usr/bin/pygrub'
name = 'vm1'
guest_os_type = 'default'
vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,keymap=en-us']
vcpus = 4
OVM_os_type = 'None'
OVM_cpu_compat_group = ''
OVM_domain_type = 'xen_pvm'

4. file system corrupted for / sometimes during reboot and give error message like
xvda: barrier or flush: disabled

Please refer to  http://lzone.de/blkfront+barrier+empty+write+xvda+op+failed　
Ubuntu 12.04 on Xen: blkfront: barrier: empty write xvda op failed
Solution: to add barrier=0 to mount options for all your local file system.