Preparing Oracle database ASM diskgroup from EMC storage luns

Jephe Wu - http://linuxtechres.blogspot.com

Environment:  RHEL6.4 64bit, HP DL380, Oracle database 11.2.0.3 RAC, ASM, iscsi software initiator and multipath, EMC storage luns
Objective: create EMC Storage luns to present to 2 RAC nodes through iscsi and multipath to be used by Oracle ASM, use udev instead of Oracle ASMLIB


Steps
1. decide how many luns and groups of luns to be created on EMC storage
let's say we will create the following RAC database
database name: racdb
instance names: racdb1 and racdb2
cluster name/scan name:  racdb-scan.domain.com

data lun: data1 and data2, asm disk group: +DATA, external redundancy, for data files and temp files only
redo lun: redo1 and redo2, asm disk group:  +REDO, external redundancy, for redo logs and control files
reco lun: reco1 and reco2, asm disk group: +RECO, external redundancy, for rman backups
arch lun: arch1 and arch2, asm disk group: +ARCH, external redundancy, for archive logs only
ocr/votedisk lun: grid1 and grid2, 5G each, asm disk group: +GRID, external redundancy

2. setting up iscsi software initiator on RHEL6.4
We have installed one additional 4-port NIC on HP DL380 so that we can have a pair of NIC for bonding.

PCI NIC: eth4 eth5 eth6 eth7
on board:  eth3 eth2 eth1 eth0

bond0:  eth0 and eth4
bond1: eth1 and eth5
eth2 and eth6 are in different subnet for iscsi initiator use

eth2: 192.168.0.1/29
eth6: 192.168.0.2/29
EMC: 192.168.0.5/29

change initiator name on each node to be same as server name
[root@db1 iscsi]# more initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:db1

on each node: discovery and setup auto iscsi login:

iscsiadm -m discovery -t st -p 192.168.0.5

will output for target information xxxx, then

iscsiadm -m node -T xxxx -p 192.168.0.5

iscsiadm -m node -T xxxx -p 192.168.0.5 -l

other useful iscsi commands:

iscsiadm -m session [-P1 | -P3]

login all Luns exported on the target: iscsiadm -m discovery -t st -l

node and discovery database directory:  /var/lib/iscsi

disconnect: iscsiadm -m node -T target_iqn_name -p ipaddress -u

remove target from database so it won't connect upon reboot: iscsiadm -m node -T target_iqn_name -p ipaddress --op delete

delete all bindings for iface0: iscsiadm -m node -I iface0 --op=delete

logout all target: iscsiadm -m node -U all

login all node: iscsiadm -m node -L all

rescan lun:  iscsiadm -m node -p 192.168.40.10 --rescan

Change default iscsi replacement timeout from 120s to 15s based on redhat recommendation:

node.conn[0].timeo.noop_out_interval = 5


node.conn[0].timeo.noop_out_timeout = 10

The replacement_timeout setting will control how long to wait for session re-establishment before failing pending SCSI commands up to multipath:

node.session.timeo.replacement_timeout = 15



And add this statement to the device section in /etc/multipath.conf:

features                "1 queue_if_no_path"





Setting up DM-Multipath




References


http://www.reactivated.net/writing_udev_rules.html 

Installation 

# yum install device-mapper-multipath

# cp /usr/share/doc/device-mapper-multipath*/multipath.conf /etc

whitelist local disk sda

# /lib/udev/scsi_id --whitelisted --replace-whitespace –-device=/dev/sda
3600508b1001030353434363646301289

Uncomment and modify the blacklist section within the /etc/multipath.conf file to include
the scsi id of the local disk on the system. Once complete, save the changes made to 
the multipath.conf file.

blacklist { 
wwid 3600508b1001030353434363646301289

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" 

devnode "^hd[a-z]"

devnode "^sd[a]$"  #add this for making sure sda is blacklisted

#The reason for blacklisting a second time with a regular expression is in case the scsi_id program fails to read the WWID 
#from sector zero of a local device. See the blacklist section in the configuration example. The 
#WWID line and devnode “^sd[a]$” line both serve as a blacklist for device sda.

}

# Start the multipath daemon.

# service multipathd start

Starting multipathd daemon: [ OK ]

6. Enable the multipath daemon to ensure it is started upon boot time.
# chkconfig multipathd on

7. Identify the dm- device, size, and WWID of each device mapper volume for Oracle 
data disks and recovery disks. In this example, volume mpathb is identified via the 
following command:
# multipath -ll

8. Uncomment the defaults section found within the /etc/multipath.conf file.
defaults { 
udev_dir /dev 
polling_interval 10 
path_selector "round-robin 0"

features  "1 queue_if_no_path"   ( add this line , see https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/iscsi-settings-dmmultipath.html) 

path_grouping_policy multibus 

getuid_callout "/lib/udev/scsi_id --whitelisted 
--device=/dev/%n" 

prio alua 
path_checker readsector0 

rr_min_io 100 

max_fds 8192 

rr_weight priorities 

failback immediate 

# no_path_retry fail   # comment out this as we already put 'feature "1 queue if _no_path"' . see https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/queueifnopath_issues.html 
user_friendly_names yes 
}





9. find out the mapping for lun name and wwid


refer to http://grangerx.wordpress.com/2013/03/19/linux-rhel-6-centos-6-determining-the-san-lun-id-for-an-emc-vnx5700-presented-lun/

# more /root/lun.sh

for x in /dev/dm-* ; do export lunid=$( /usr/bin/sg_inq --page=0x83 ${x} | grep "^ 00" | cut -d" " -f17 | awk '{ print strtonum( "0x" $1 ) };' ) ; export wwid=$( /sbin/scsi_id --whitelisted --replace-whitespace --device=${x} ) ; echo ${lunid} == ${x} == ${wwid}; done

Eventually, run /root/lun.sh on both nodes to confirm the consistency on both nodes

10. Uncomment the multipath section found within the /etc/multipath.conf file and create 
an alias for each device mapper volume in order to enable persistent naming of those 
volumes. Once complete, save the changes made to the multipath.conf file.

multipaths

 {

  multipath {

wwid 3600c0ff000d7e7a899d8515101000000

alias data1

}

multipath {

wwid 3600c0ff000dabfe5a7d8515101000000

alias data2

}

 multipath {

wwid 3600c0ff000d7e7a8dbd8515101000000

alias reco1

}

multipath {

 wwid 3600c0ff000dabfe5f4d8515101000000

alias reco2

 } 

bla bla

}

10.Restart the device mapper multipath daemon.
# service multipathd restart
ok 
Stopping multipathd daemon: [ OK ] 
Starting multipathd daemon: [ OK ]

11. verify it by multipath -ll

12. reload udev rules

Reload udev command:
udevcontrol reload_rules



Partitioning Device Mapper Shared Disks

Partitioning of the device mapper shared disks is only required when using Oracle ASMLib.

For the simplification of ensuring to meet all requirements, this reference architecture creates 
a partition for each device mapper volume. For each device mapper volume, create a partition
using parted. An example is shown below.

# parted /dev/mapper/db1 mklabel gpt mkpart primary "1 -1"


Information: You may need to update /etc/fstab.

Once the partition is created, a newly created device mapper device is created as db1p1.

# ls -l /dev/mapper/data1p1

 lrwxrwxrwx. 1 root root 8 Apr 16 15:15 /dev/mapper/data1p1 -> ../dm-11

NOTE: A newly created partition requires the alias name followed by p1 such as data1p1 seen 
above. If p1 is missing, please run the following command to add the partition mappings to 
the device mapper disks. And you should run command below on another node to generate all *.p1 files

# kpartx -a /dev/mapper/data1

Setting up udev

1. As the root user, identify the Device Mapper Universally Unique IDentifier (DM_UUID) 
for each device mapper volume. The example below shows the DM_UID for the 
partitions of the volumes labeled db1,db2,fra, and redo. 



# for i in data1p1 data2p1 ... reco1p1 reco2p1; do printf "%s %s\n" "$i" "$(udevadm 
info --query=all --name=/dev/mapper/$i | grep -i dm_uuid)"; done 



data1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a899d8515101000000 

data2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5a7d8515101000000 


reco1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a8dbd8515101000000 

reco2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5f4d8515101000000

2. Create a file labeled 99-oracle-asmdevices.rules within /etc/udev/rules.d/

3. Within 99-oracle-asmdevices.rules file, create rules for each device similar to the 
example below:
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-3600c0ff000dabfe5f4d8515101000000",OWNER="oracle",GROUP="oinstall",MODE="0660"

KERNEL=="dm-*",ENV{DM_UUID}=="<enter-value-according-to-your-environment>",OWNER="oracle",GROUP="oinstall",MODE="0660"

4. Save the file labeled 99-oracle-asmdevices.rules

5. reboot to check if the owner and group are changed.