Jephe Wu - http://linuxtechres.blogspot.com
Objective: increase existing ASM disk group size
Environment: RHEL 6.4 64bit, Oracle 11.2.0.4, iscsi, multipath, udev and ASM, EMC storage
Concept: increase disk group size by increasing ASM disk size itself or adding some more ASM disk
Part I: Add additional Lun as ASM disk
Steps:
1. create a new Lun in EMC storage
2. use the following methods to find out the newly created disk name ,e.g. /dev/sdae etc
dmesg
more /proc/partitions (the new lines at the bottom)
ls -l /sys/block
ls -l /dev/disk/by-uuid/
3. find out the WWID
scsi_id --whitelisted --replace-whitespace –-device=/dev/sdae
multipath -ll
4. vi /etc/multipath.conf to use alias for newly added ASM lun, e.g. db1
then run multipathd reload to reload the configuration
4. find out the lunid and wwid for each Lun so we know which volume is which
It's best to get lunid from storage administrator so that we can confirm which volume is which, otherwise, try to run the following command to find out. If sg_inq command is not available, yum install sg3_utils first
for x in /dev/dm-* ; do export lunid=$( /usr/bin/sg_inq --page=0x83 ${x} | grep "^ 00" | cut -d" " -f17 | awk '{ print strtonum( "0x" $1 ) };' ) ; export wwid=$( /sbin/scsi_id --whitelisted --replace-whitespace --device=${x} ) ; echo ${lunid} == ${x} == ${wwid}; done
for each device mapper volume. The example below shows the DM_UID for the
partitions of the volumes labeled db1,db2,fra, and redo.
# for i in db1p1 ; do printf "%s %s\n" "$i" "$(udevadm
info --query=all --name=/dev/mapper/$i | grep -i dm_uuid)"; done
db1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a899d8515101000000
7. Within 99-oracle-asmdevices.rules file, create rules for each device similar to the
example below:
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-3600c0ff000dabfe5f4d8515101000000",OWNER="oracle",GROUP="oinstall",MODE="0660"
- If one has made a change and
don't want to reboot the system then can utilize udevadm trigger instead.
Specify --type and--action or it will effectively
work like start_udev.
- One can even trigger only
specific devices like below;
8. make partition for new Lun, only run on one of RAC node if any
Information: You may need to update /etc/fstab.
Once the partition is created, a newly created device mapper device is created as db1p1.
# ls -l /dev/mapper/db1p1
lrwxrwxrwx. 1 root root 8 Apr 16 15:15 /dev/mapper/db1p1 -> ../dm-11
NOTE: A newly created partition requires the alias name followed by p1 such as db1p1 seen
above. If p1 is missing, please run the following command to add the partition mappings to
the device mapper disks.
# kpartx -a /dev/mapper/db1
9. if there's multiple paths on one system, you might need to run partprobe to update all available path for created partitions, and also run kpartx -a /dev/mapper/db1 on another RAC node
ALTER DISKGROUP data_finance ADD DISK
'/dev/mapper/db1p1';
2. make OS recognize the new size
To re-scan logical units on a system that uses multipathing, execute the aforementioned command for each sd device (i.e. sda, sdb, and so on) that represents a path for the multipathed logical unit. To determine which devices are paths for a multipath logical unit, use multipath -ll; then, find the entry that matches the logical unit being changed. It is advisable that you refer to the WWID of each entry to make it easier to find which one matches the logical unit being changed.
echo 1 > /sys/block/sdax/device/rescan
echo 1 > /sys/block/sday/device/rescan
echo 1 > /sys/block/sdaz/device/rescan
echo 1 > /sys/block/sdba/device/rescan
Instead you can use /usr/bin/rescan-scsi-bus.sh script for rescanning all SAN LUNs. The sg3_utils package provides the rescan-scsi-bus.sh script, which can automatically update the logical unit configuration of the host as needed (after a device has been added to the system). The rescan-scsi-bus.sh script can also perform an issue_lip on supported devices. For more information about how to use this script, refer to rescan-scsi-bus.sh --help.
Now, run multipath -F followed by multipath -v2 mpathX to reload the mutipath configuration.
If the LUNs are configured over iSCSI protocol, the following command can be used to get the new LUN size:
# iscsiadm -m session -R
3. If it's multipath device, run kpartx -a /dev/mapper/mpathX if necessary
4. multipathd -k"resize map <multipath_device>"
5. reload multipath configuration : multipathd reload or multipathd restart, then run miltipath -ll to see new size.
Note: if you enabled 'queue if no path' option, then do not use multipathd -k'resize map mpatha'
6. resize /dev/mapper/mpathX by running resize2fs if necessary
7. check and resize ASM disk group
select name, total_mb, usable_file_mb from v$asm_diskgroup;
alter diskgroup DATADG resize all;
References:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/DM_Multipath/index.html#online_device_resize
http://www.novell.com/support/kb/doc.php?id=7009660
https://access.redhat.com/solutions/127103
http://www.jk-47.com/2012/08/extending-an-existing-lun-w-oracle-asm/
https://orainternals.wordpress.com/2012/08/29/do-you-need-asmlib/
Posts
