Showing posts with label snapshot. Show all posts
Showing posts with label snapshot. Show all posts

How to make netapp Oracle database snapshot copy crash-consistent


Jephe Wu - http://linuxtechres.blogspot.com

Objective: understanding Netapp point-in-time snapshot Oracle backup without putting in the hot backup mode


Crash-consistent snapshot copies should only be considered under special circumstances where requirements restrict the use of standard backup methods (such as rman or hot backup mode)

Oracle backup overview:
----------------------
physical backup and logical backup

physical backup can be classified as consistent backup or inconsistent backup

consistent backup means controlfile and data file are checkpointed with same SCN, only possible when database is cleanly shut down, no matter it's in nonarchivelog or archivelog mode

Besides the standard 3 methods for backup: cold/offline backup, rman backup and online/hot backup(user-managed backup), Oracle recently certify the third party snapshot copy technology as one of options of backup/recovery as long as it's crash consistent

In the past, Oracle did not support or recommend the use of a snapshot copy created of an online active
database without the database or tablespaces being put in backup mode. The risk was thought to be
the danger of mixing old archive logs with current archive logs, which can lead to data corruption or
potentially destroy the production database.

According to MOS note 604683.1, the snapshot of an online database not in backup mode can be
deemed valid and supported if and only if all of the following requirements are strictly satisfied:
•  Oracle’s recommended restore and recovery operations are followed. 
•  Database is crash consistent at the point of the snapshot. 
•  Write ordering is preserved for each file within a snapshot. 


Oracle recovery overview:
------------------------
instance recovery and media recovery
instance recovery is automatic done by Oracle itself, it requires redo log file only.
Media recovery requires archived redo log.
Media recovery has complete recovery and incomplete recovery


An incomplete recovery of the whole database is usually required in the following situations:
•  Data loss caused by user errors
•  Missing archived redo log, which prevents complete recovery
•  Physical loss or corruption of online redo logs
•  No access to current control file

When performing incomplete recovery, the types of media recovery are available.


  • Time-based recovery  Recovers the data up to a specified point in time. 
  • Cancel-based recovery  Recovers until you issue the CANCEL statement (not available when using Recovery Manager). 
  • Change-based recovery  Recovers until the specified SCN. 
  • Log sequence recovery  Recovers until the specified log sequence number (only available when using Recovery Manager). 

What's the crash consistent?
------------------------------
It's point-in-time(PIT) image of Oracle database, looks like it crashed due to power outage, instance crash or shutdown abort etc, it requires instance recovery after restart database, not media recovery. Netapp snapshot generate Point-In-time image for database.


How to make snapshot crash-consistent?
---------------------------------------
1. all databqase files(controlfile, datafile, online redo log) are in single volume, then snapshot will generate crash-consistent image.
Note: Not require archived logs to be in the same volume.

If a database has all of its files (control files, data files, online redo logs, and archived logs) contained
within a single NetApp volume, then the task is straightforward. A Snapshot copy of that single volume
will provide a crash-consistent copy.

2. use crash consistent group by snapmanager/snapdrive etc if database cross different volumes
e.g. data volume and log volume, data captured by snapshot for data volume must exist in log volume first because Oracle always makes sure it writes to redo log first before writing associated data buffer cache to data file.


Starting from SnapDrive for unix 2.2, SnapDrive supports the feature of consistency groups provided
by Data ONTAP (beginning with version 7.2 and higher). This feature is necessary for creating a
consistent Snapshot copy across multiple controller/volumes.

In an environment where all participating controllers support consistency groups, SnapDrive will use a
Data ONTAP consistency group as the preferred (default) method to capture multicontroller/volume
Snapshot copies.

SnapDrive can simplify the creation of a consistency group Snapshot copy when there are
multiple file systems.

snapdrive snap create -fs /u01/oradata/prod /u02/oradata/prod -snapname snap_prod_cg 


a. POINT-IN-TIME COPY OF THE DATABASE 

After the database is opened, no future redo logs beyond this snapshot
can be applied.

Open resetlogs operation is recommended to avoid potential mixing of existing
archive logs and new archive logs. and start a new incarnation and log ID:

1. SHUTDOWN IMMEDIATE 
2.  STARTUP MOUNT 
3.  RECOVER DATABASE UNTIL CANCEL 
4.  ALTER DATABASE OPEN RESETLOGS; 

b. FULL DATABASE RECOVERY WITH ZERO DATA LOSS 

Restore the snapshot of only the data files. Do not overwrite the current control files, current redo
logs, and current archived logs.

run commands below to fully recover database by applying archived and online redo logs
1. recover automatic database;
2. alter database open;

c. point-in-time(PIT) database recovery
PIT requires the presence of current controlfile, current online redo logs and archived logs.

only restore data files and run the following commands:
1. startup mount

Identify the minimum SCN we have to recover to by script @scandatafile.sql

SQL> @scandatafile  
File 1 absolute fuzzy scn = 861391  
File 2 absolute fuzzy scn = 0  
File 3 absolute fuzzy scn = 0  
File 4 absolute fuzzy scn = 0  
Minimum PITR SCN = 861391  

PL/SQL procedure successfully completed. 


scandatafiles.sql 
# scans all files and update file headers with meta information  
# depending on number and sizes of files, the scandatafile procedure can be 
a  
# time consuming operation.  
# create a script, “scandatafile”, with the following content 

spool scandatafile.sql  
set serveroutput on  
declare  
 scn number(12) := 0;  
 scnmax number(12) := 0;  
begin  
 for f in (select * from v$datafile) loop  
 scn := dbms_backup_restore.scandatafile(f.file#);  
 dbms_output.put_line('File ' || f.file# ||' absolute fuzzy scn = ' || 
scn);  
 if scn > scnmax then scnmax := scn; end if;  
 end loop;  

 dbms_output.put_line('Minimum PITR SCN = ' || scnmax);  
end; 


If the minimum PITR SCN is zero, then database is not required for further recovery, it can to opened now.
if it's no zero, database must be recovered to at least that SCN and onwards.

2. RECOVER AUTOMATIC DATABASE UNTIL CHANGE [Minimum PITR SCN or higher]  
or 
ALTER DATABASE RECOVER DATABASE UNTIL CHANGE [Minimum PITR SCN or higher] 

3. ALTER DATABASE OPEN RESETLOGS 

References:

--------------
1. Using Crash-Consistent Snapshot Copies as Valid Oracle Backups - http://media.netapp.com/documents/tr-3858.pdf
2. MOS Supported Backup, Restore and Recovery Operations using Third Party Snapshot Technologies [ID 604683.1]

How to use LVM snapshot to clone CentOS 5 server

Jephe Wu - http://linuxtechres.blogspot.com

Environment: 36G IDE hard disk with CentOS 5.5 default installation which has /boot as /dev/hda1 and the rest are on LVM, this is server A.
Objective: to clone to another same kind of machine by using LVM snapshot. This is server B.
Tools used: RIP(Recovery Is Possible) CD V8.9, create LVM snapshot for / partition before using tar to copy to another machine.

Concept:
1. use RIP to boot up the destination machine
2. create partitions same as the original one and make file system for all partitions
3. use external hard disk or thumb drive on original machine to extend the volume group
4. create LVM snapshot logical volume on original machine before using tar to copy the whole file system
5. use tar on destinaion machine to copy over the whole file system from original machine
6. install grub on destiniation machine. reboot


Steps:
1.  use RIP to boot up the server B, choose skip the keyboard map.
2.  setting up the environment of RIP for network use
login as root without password
passwd root
ifconfig eth0 10.0.0.2 netmask 255.255.255.0 up
/usr/sbin/sshd


3.  copy over the partition configurations from server A (10.0.0.1/24)

ssh 10.0.0.1 'sfdisk -d /dev/hda' | sfdisk [--force] /dev/hda
or
use fdisk -ul /dev/hda on server A to get the sector layout, then configure it on server B

mkfs -t ext3 /dev/hda1
mkfs -t ext3 /dev/VolGroup00/LogVol00
mkswap /dev/VolGroup00/LogVol01
e2label /boot /dev/hda1


4. create snapshot on server A for intact tar backup over ssh.
4.1 extend the current volume group first
use external hard disk or thumb drive as the extra space for snapshot, as long as the thumb drive can hold the extra changes between the time you created snapshot and the time you finish then delete snapshot, thumb drive doesn't have to be same size as hard disk.

How LVM snapshot works?
As soon as you create a snapshot, LVM creates a pool of blocks. I believe that this pool also contains a full copy of the metadata of the volume. When writes happen to the main volume, the block being overwritten is copied to this new pool and the new block is written to the main volume. This is the 'copy-on-write'. Because of this, the more data that gets changed between when a snapshot was taken and the current state of the main volume, the more space will get consumed by that snapshot pool.

4.2 create snapshot (I used a 4G thumb drive as /dev/sda1)

vgextend VolGroup00 /dev/sda1
vgdisplay -v
lvcreate -n backup -l 126 VolGroup00
mkfs -t ext3 /dev/VolGroup00/backup
mkdir /backup
mount /dev/VolGroup00/backup /backup/
vcreate -l 126 -s -n rootsnapshot /dev/VolGroup00/LogVol00


4.3 to remove snapshot later, do:
umount /backup/
lvremove /dev/VolGroup00/backup
vgreduce VolGroup00 /dev/hda1

or
pvremove /dev/sda1
vgreduce --removemissing
vgreduce --removemissing VolGroup00


Note: If the physical volume is still used you will have to migrate the data to another physical volume using pvmove.

5.  copying the whole file system to server B
on server B:
mount /dev/VolGroup00/LogVol00 /mnt/hda2
mount /dev/hda1 /mnt/hda2/boot
scp 10.0.0.1:/etc/passwd /etc/
scp 10.0.0.1:/etc/group /etc/
ssh 10.0.0.1 'cd /backup; tar cvpf - .' | tar xvpf -


6. make grub
cd /mnt/hda2
chroot .
[MAKEDEV hda]
[vgchange -a y] to activate all LVM - optional
grub-install hd0


7. References:

http://www.howtoforge.com/linux_lvm_snapshots