Preparing Oracle database ASM diskgroup from EMC storage luns

Jephe Wu - http://linuxtechres.blogspot.com

Environment:  RHEL6.4 64bit, HP DL380, Oracle database 11.2.0.3 RAC, ASM, iscsi software initiator and multipath, EMC storage luns
Objective: create EMC Storage luns to present to 2 RAC nodes through iscsi and multipath to be used by Oracle ASM, use udev instead of Oracle ASMLIB


Steps
1. decide how many luns and groups of luns to be created on EMC storage
let's say we will create the following RAC database
database name: racdb
instance names: racdb1 and racdb2
cluster name/scan name:  racdb-scan.domain.com

data lun: data1 and data2, asm disk group: +DATA, external redundancy, for data files and temp files only
redo lun: redo1 and redo2, asm disk group:  +REDO, external redundancy, for redo logs and control files
reco lun: reco1 and reco2, asm disk group: +RECO, external redundancy, for rman backups
arch lun: arch1 and arch2, asm disk group: +ARCH, external redundancy, for archive logs only
ocr/votedisk lun: grid1 and grid2, 5G each, asm disk group: +GRID, external redundancy

2. setting up iscsi software initiator on RHEL6.4
We have installed one additional 4-port NIC on HP DL380 so that we can have a pair of NIC for bonding.

PCI NIC: eth4 eth5 eth6 eth7
on board:  eth3 eth2 eth1 eth0

bond0:  eth0 and eth4
bond1: eth1 and eth5
eth2 and eth6 are in different subnet for iscsi initiator use

eth2: 192.168.0.1/29
eth6: 192.168.0.2/29
EMC: 192.168.0.5/29

change initiator name on each node to be same as server name
[root@db1 iscsi]# more initiatorname.iscsi 
InitiatorName=iqn.1994-05.com.redhat:db1

on each node: discovery and setup auto iscsi login:

iscsiadm -m discovery -t st -p 192.168.0.5
will output for target information xxxx, then
iscsiadm -m node -T xxxx -p 192.168.0.5
iscsiadm -m node -T xxxx -p 192.168.0.5 -l

other useful iscsi commands:
iscsiadm -m session [-P1 | -P3]
login all Luns exported on the target: iscsiadm -m discovery -t st -l
node and discovery database directory:  /var/lib/iscsi
disconnect: iscsiadm -m node -T target_iqn_name -p ipaddress -u
remove target from database so it won't connect upon reboot: iscsiadm -m node -T target_iqn_name -p ipaddress --op delete
delete all bindings for iface0: iscsiadm -m node -I iface0 --op=delete
logout all target: iscsiadm -m node -U all
login all node: iscsiadm -m node -L all
rescan lun:  iscsiadm -m node -p 192.168.40.10 --rescan
Change default iscsi replacement timeout from 120s to 15s based on redhat recommendation:
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 10
The replacement_timeout setting will control how long to wait for session re-establishment before failing pending SCSI commands up to multipath:
node.session.timeo.replacement_timeout = 15
And add this statement to the device section in /etc/multipath.conf:
features                "1 queue_if_no_path"

Setting up DM-Multipath

References

Installation 

# yum install device-mapper-multipath
# cp /usr/share/doc/device-mapper-multipath*/multipath.conf /etc

whitelist local disk sda

# /lib/udev/scsi_id --whitelisted --replace-whitespace –-device=/dev/sda 3600508b1001030353434363646301289
Uncomment and modify the blacklist section within the /etc/multipath.conf file to include the scsi id of the local disk on the system. Once complete, save the changes made to  the multipath.conf file.
blacklist {  wwid 3600508b1001030353434363646301289
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" 
devnode "^hd[a-z]"
devnode "^sd[a]$"  #add this for making sure sda is blacklisted
#The reason for blacklisting a second time with a regular expression is in case the scsi_id program fails to read the WWID  #from sector zero of a local device. See the blacklist section in the configuration example. The  #WWID line and devnode “^sd[a]$” line both serve as a blacklist for device sda.
}
# Start the multipath daemon.
# service multipathd start
Starting multipathd daemon: [ OK ]
6. Enable the multipath daemon to ensure it is started upon boot time. # chkconfig multipathd on
7. Identify the dm- device, size, and WWID of each device mapper volume for Oracle  data disks and recovery disks. In this example, volume mpathb is identified via the  following command: # multipath -ll
8. Uncomment the defaults section found within the /etc/multipath.conf file. defaults {  udev_dir /dev  polling_interval 10  path_selector "round-robin 0"
path_grouping_policy multibus 
getuid_callout "/lib/udev/scsi_id --whitelisted  --device=/dev/%n" 
prio alua  path_checker readsector0 
rr_min_io 100 
max_fds 8192 
rr_weight priorities 
failback immediate 
# no_path_retry fail   # comment out this as we already put 'feature "1 queue if _no_path"' . see https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/queueifnopath_issues.html  user_friendly_names yes  }
9. find out the mapping for lun name and wwid
# more /root/lun.sh
for x in /dev/dm-* ; do export lunid=$( /usr/bin/sg_inq --page=0x83 ${x} | grep "^ 00" | cut -d" " -f17 | awk '{ print strtonum( "0x" $1 ) };' ) ; export wwid=$( /sbin/scsi_id --whitelisted --replace-whitespace --device=${x} ) ; echo ${lunid} == ${x} == ${wwid}; done
Eventually, run /root/lun.sh on both nodes to confirm the consistency on both nodes
10. Uncomment the multipath section found within the /etc/multipath.conf file and create  an alias for each device mapper volume in order to enable persistent naming of those  volumes. Once complete, save the changes made to the multipath.conf file.
multipaths
 {
  multipath {
wwid 3600c0ff000d7e7a899d8515101000000
alias data1
}
multipath {
wwid 3600c0ff000dabfe5a7d8515101000000
alias data2
}
 multipath {
wwid 3600c0ff000d7e7a8dbd8515101000000
alias reco1
}
multipath {
 wwid 3600c0ff000dabfe5f4d8515101000000
alias reco2
 } 
bla bla
}
10.Restart the device mapper multipath daemon. # service multipathd restart ok  Stopping multipathd daemon: [ OK ]  Starting multipathd daemon: [ OK ]
11. verify it by multipath -ll
12. reload udev rules
Reload udev command: udevcontrol reload_rules

Partitioning Device Mapper Shared Disks

Partitioning of the device mapper shared disks is only required when using Oracle ASMLib.
For the simplification of ensuring to meet all requirements, this reference architecture creates  a partition for each device mapper volume. For each device mapper volume, create a partition using parted. An example is shown below.
# parted /dev/mapper/db1 mklabel gpt mkpart primary "1 -1"
Information: You may need to update /etc/fstab.
Once the partition is created, a newly created device mapper device is created as db1p1.
# ls -l /dev/mapper/data1p1
 lrwxrwxrwx. 1 root root 8 Apr 16 15:15 /dev/mapper/data1p1 -> ../dm-11
NOTE: A newly created partition requires the alias name followed by p1 such as data1p1 seen  above. If p1 is missing, please run the following command to add the partition mappings to  the device mapper disks. And you should run command below on another node to generate all *.p1 files
# kpartx -a /dev/mapper/data1
Setting up udev
1. As the root user, identify the Device Mapper Universally Unique IDentifier (DM_UUID)  for each device mapper volume. The example below shows the DM_UID for the  partitions of the volumes labeled db1,db2,fra, and redo. 
# for i in data1p1 data2p1 ... reco1p1 reco2p1; do printf "%s %s\n" "$i" "$(udevadm  info --query=all --name=/dev/mapper/$i | grep -i dm_uuid)"; done 
data1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a899d8515101000000 
data2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5a7d8515101000000 
reco1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a8dbd8515101000000
reco2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5f4d8515101000000
2. Create a file labeled 99-oracle-asmdevices.rules within /etc/udev/rules.d/
3. Within 99-oracle-asmdevices.rules file, create rules for each device similar to the  example below: KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-3600c0ff000dabfe5f4d8515101000000",OWNER="oracle",GROUP="oinstall",MODE="0660"
KERNEL=="dm-*",ENV{DM_UUID}=="<enter-value-according-to-your-environment>",OWNER="oracle",GROUP="oinstall",MODE="0660"
4. Save the file labeled 99-oracle-asmdevices.rules
5. reboot to check if the owner and group are changed.