How to decide ethX layout after Oracle VM Server 3.2 Installation

Jephe Wu - http://linuxtechres.blogspot.com

Objective: to decide ethX layout remotely after Oracle VM Server 3.2 installation.
Environment: Oracle VM server 3.2.9

If you are in front of the physical server, you can decide ethX layout easily by running 'ethtool -p ethX' etc to blink each NIC light. e.g. ethtool -p eth0

How Oracle VM server to decide which NIC for which ethX
based on pci bus address from low to high to assign eth0, eth1 etc

Theory behind

• Os will assign ethX sequence based on NIC PCI bus address sequence
• 4 NICs onboard are actually 2 dualports cards
• For additional PCI card, the ports near the bottom pins are getting high pci bus number, from ethX is assigned from top to bottom.
• Sometimes, pci bus address has primary-secondary relationships, 

Check steps

Get server model, pci NIC slot direction, PCI numbering sequence physically etc first

/sys/class/net 

[root@cpnrglobovms13 device]# more /sys/class/net/eth0/device/label
Intel 82576 Ethernet Controller.

to decide pci bus address and ethX alighment

ll /sys/class/net

List ethX bus information - ethtool -i ethX

[root@ovms01 ~]# for i in `ifconfig -a | grep ^eth | awk '{print $1}' | xargs`; do echo -n "$i " && ethtool -i $i| grep -i bus ; done
eth0 bus-info: 0000:01:00.0
eth1 bus-info: 0000:01:00.1
eth2 bus-info: 0000:02:00.0
eth3 bus-info: 0000:02:00.1
eth4 bus-info: 0000:04:00.0
eth5 bus-info: 0000:04:00.1
eth6 bus-info: 0000:07:00.0
eth7 bus-info: 0000:07:00.1
eth8 bus-info: 0000:08:00.0
eth9 bus-info: 0000:08:00.1

lspci to get ethernet NICs - lspci | grep -i eth

[root@ovms01 ~]# lspci | grep -i eth
01:00.0 Ethernet controller: Broadcom Corporation Device 165f
01:00.1 Ethernet controller: Broadcom Corporation Device 165f
02:00.0 Ethernet controller: Broadcom Corporation Device 165f
02:00.1 Ethernet controller: Broadcom Corporation Device 165f
04:00.0 Ethernet controller: Intel Corporation Device 154d (rev 01)
04:00.1 Ethernet controller: Intel Corporation Device 154d (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

lspci bridge to secondary  - lspci -vvv | grep primary

05:00 -> 06:00 -> 07:00 and 08:00

[root@hmspglobovms01 device]# lspci -vvv | grep primary
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
Bus: primary=00, secondary=09, subordinate=09, sec-latency=0
Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
Bus: primary=00, secondary=0b, subordinate=0f, sec-latency=0
Bus: primary=00, secondary=10, subordinate=10, sec-latency=32
Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
Bus: primary=0b, secondary=0c, subordinate=0f, sec-latency=0
Bus: primary=0c, secondary=0d, subordinate=0e, sec-latency=0
Bus: primary=0c, secondary=0f, subordinate=0f, sec-latency=0
Bus: primary=0d, secondary=0e, subordinate=0e, sec-latency=0

dmidecode to list PCI slot information 

[root@ovms01 ~]# dmidecode -t 9
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x0900, DMI type 9, 17 bytes
System Slot Information
Designation: PCI1
Type: x8 <OUT OF SPEC>
Current Usage: In Use
Length: Long
Characteristics:
3.3 V is provided
PME signal is supported
Bus Address: 0000:04:00.0

Handle 0x0901, DMI type 9, 17 bytes
System Slot Information
Designation: PCI2
Type: x16 <OUT OF SPEC>
Current Usage: In Use
Length: Long
Characteristics:
3.3 V is provided
PME signal is supported
Bus Address: 0000:05:00.0

dmidecode to get onboard device information

[root@ovms01 ~]# dmidecode -t 41  or dmidecode -t slot
# dmidecode 2.10
SMBIOS 2.7 present.
# SMBIOS implementations newer than version 2.6 are not
# fully supported by this version of dmidecode.

Handle 0x2900, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:01:00.0

Handle 0x2901, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:01:00.1

Handle 0x2902, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 3
Type: Ethernet
Status: Enabled
Type Instance: 3
Bus Address: 0000:02:00.0

Handle 0x2903, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Integrated NIC 4
Type: Ethernet
Status: Enabled
Type Instance: 4
Bus Address: 0000:02:00.1

lspci to get ethernet NIC information
[root@ovms01 ~]# lspci | grep -i ethernet

dmidecode to get PCI card information
[root@ovms01 ~]# dmidecode -t slot | grep -e Designation -e Bus

use ethtool to decide bus info
[root@ovms10 ~]# for i in `ifconfig -a | grep ^eth | awk '{print $1}' | xargs`; do echo -n "$i " && ethtool -i $i| grep -i bus ; done 
eth0 bus-info: 0000:01:00.0
eth1 bus-info: 0000:01:00.1
eth2 bus-info: 0000:04:00.0
eth3 bus-info: 0000:04:00.1
eth4 bus-info: 0000:42:00.0
eth5 bus-info: 0000:42:00.1
eth6_rename bus-info: 0000:07:00.0
eth7_rename bus-info: 0000:07:00.1

dmesg to get PCI slot cards MAC address

[root@ovms01 ~]# dmesg | grep 'PCI Express'

pciehp: PCI Express Hot Plug Controller Driver version: 0.4
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
ixgbe 0000:08:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a6:80
ixgbe 0000:08:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a6:81
ixgbe 0000:0e:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a5:ac
ixgbe 0000:0e:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:a5:ad
ixgbe 0000:0f:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:58:20
ixgbe 0000:0f:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:19:58:21
bnx2 0000:01:00.0: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 36, node addr d4:ae:52:e6:83:a0
bnx2 0000:01:00.1: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 48, node addr d4:ae:52:e6:83:a2
bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 37, node addr d4:ae:52:e6:83:a4
bnx2 0000:02:00.1: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 49, node addr d4:ae:52:e6:83:a6

to get each PCI bus address MAC address:

[root@ovms1 ~]# dmesg | grep 02:00.0 | grep 'PCI Express'

Kickstart to swap eth2/eth3 and eth6/eth7 for better reading if necessary

echo "fixing ethX device sequence"
ETH2=`dmesg | grep Express | grep "30:00.0" | awk '{print $NF}'`
ETH3=`dmesg | grep Express | grep "30:00.1" | awk '{print $NF}'`
ETH6=`dmesg | grep Express | grep "88:00.0" | awk '{print $NF}'`
ETH7=`dmesg | grep Express | grep "88:00.1" | awk '{print $NF}'`


cd /etc/sysconfig/network-scripts
sed -i -e "s/^HWADDR.*/HWADDR=$ETH6/g" ifcfg-eth2
sed -i -e "s/^HWADDR.*/HWADDR=$ETH7/g" ifcfg-eth3
sed -i -e "s/^HWADDR.*/HWADDR=$ETH2/g" ifcfg-eth6
sed -i -e "s/^HWADDR.*/HWADDR=$ETH3/g" ifcfg-eth7

Draw ethX layout at last step

example below:

PCI1

RAID Controller



PCI2

00:42.00.1(eth5)(pin) 00:42.00.0(eth4)



PCI3

00:04:00.0(eth2) 00:04:00.1(eth3)(pin)






eth0

00:01:00.0

eth1

00:01:00.1

eth6 (X)

00:07:00.0

eth7 (X)

00:07:00.1

Ultimate way to associate ethX with PCI bus address

[root@ovms10 rules.d]# more 99-ethernet.rules

KERNEL=="eth*", ID=="0000:01:00.0", name="eth0"

KERNEL=="eth*", ID=="0000:01:00.1", name="eth1"

KERNEL=="eth*", ID=="0000:04:00.0", name="eth2"

KERNEL=="eth*", ID=="0000:04:00.1", name="eth3"

KERNEL=="eth*", ID=="0000:42:00.0", name="eth4"

KERNEL=="eth*", ID=="0000:42:00.1", name="eth5"

KERNEL=="eth*", ID=="0000:07:00.0", name="eth6"

KERNEL=="eth*", ID=="0000:07:00.1", name="eth7"

[root@ovms10 rules.d]# ethtool -i eth0

driver: ixgbe

version: 3.4.8-k

firmware-version: 4.2-8

bus-info: 0000:01:00.0