How to use looping in Linux bash script

Jephe Wu - http://linuxtechres.blogspot.com


Objective: list all kinds of Linux bash looping script which is useful for bash programming.
let computer do work for you.

Concept: 
Very often, we just need to run some script by loop to finish work


Examples


1. infinite loop

while true;do ...done
while [ 1 ]; do...done
while : ; do ....done

watch ls -l /tmp/file

2.  List a range of number 

list each line of /etc/hosts file
while read line;do echo $line;done < /etc/hosts

list number 1 to 10
for i in {1..10};do echo $i;done
for i in $(seq 1 10);do echo $i;done
for i in `seq 1 10`;do echo $i;done

list number 1 to 10 with same width
for i in {01..10};do echo $i;done
for i in $(seq -w 1 10);do echo $i;done
for i in `seq -w 1 10`;do echo $i;done

list number 1 to 10 with only odd number
for i in {1..10..2};do echo $i;done

list number 1 to 10 with only even number
for i in {2..10..2};do echo $i;done

Use just echo to list 1 to 10
echo {1..10}
echo {01..10}

Use just echo to list a-z and A-Z
echo {a..z}
echo {A..Z}

tranditional method to list 1 to 10
i=1;while [ $i -le 10 ];do echo $i;i=$[$i+1];done

i=1;while (( $i < 11 )); do echo $i; let i++; done
for (( i=0 ; i < 11 ; i++ )) ; do  echo $i ; done


3. Examples
Use loop to touch file a.txt, b.txt until z.txt?
for i in {a..z};do touch $i.txt;done

Set up bonding in RHEL6

Jephe Wu - http://linuxtechres.blogspot.com

Environment: CentOS 6 x86_64 on HP Blade Proliant BL460c G6, 2 builtin broadcom NIC and a NC364m Quad Port 1Gb NIC for c-class BladeSystem
Objective: after using builtin NIC to install OS , then configure bonding for 2 of quad port NIC.

Steps:

1. OS installation
You can download http://mirror.optus.net/centos/6/isos/x86_64/CentOS-6.0-x86_64-netinstall.iso and install OS through http URL from Internet.

2. bonding configuration

Firstly, disable network manager if NM is running.
service NetworkManager stop
chkconfig NetworkManager off

a. /etc/udev/rules.d/70-persistent-net.ruels
make sure it contains all interfaces you have in the system and don't configure bond0 inside.

b. /etc/sysconfig/network-scripts/

configuring ifcfg-bond0 as follows: (assuming ip address is 192.168.0.1)
       
[root@db03 network-scripts]# more ifcfg-bond0
DEVICE="bond0"
BROADCAST="192.168.0.255"
GATEWAY="192.168.0.254"
IPADDR="192.168.0.1"
NETMASK="255.255.255.0"
ONBOOT="yes"
USERCTL=no
BONDING_OPTS="mode=0 miimon=100 downdelay=300 updelay=300 max_bonds=4"

note: you must use capital letter for left side variables, especially BONDING_OPTS, otherwise, you might not be able to see slave devices from command output of 'ifconfig'.
Do not configure BONDING_OPTS in /etc/modprobe.d/bonding.conf as suggested by Redhat.

c. /etc/sysconfig/network-scripts/ifcfg-ethX

[root@db03 network-scripts]# more ifcfg-eth*
::::::::::::::
ifcfg-eth0
::::::::::::::
DEVICE=eth0
BOOTPROTO=none
MASTER=bond0
HWADDR=00:01:02:03:04:00
ONBOOT=yes
SLAVE=yes
USERCTL=no
::::::::::::::
ifcfg-eth1
::::::::::::::
DEVICE=eth1
BOOTPROTO=none
HWADDR=00:01:02:03:04:01
ONBOOT=no
USERCTL=no
::::::::::::::
ifcfg-eth2
::::::::::::::
DEVICE=eth2
BOOTPROTO=none
MASTER=bond0
HWADDR=00:01:02:03:04:02
ONBOOT=yes
SLAVE=yes
USERCTL=no
::::::::::::::
ifcfg-eth3
::::::::::::::
DEVICE=eth3
BOOTPROTO=none
HWADDR=00:01:02:03:04:03
ONBOOT=no
USERCTL=no
::::::::::::::
ifcfg-eth4
::::::::::::::
DEVICE="eth4"
HWADDR="00:00:00:00:00:04"
NM_CONTROLLED="no"
ONBOOT="no"
::::::::::::::
ifcfg-eth5
::::::::::::::
DEVICE="eth5"
HWADDR="00:00:00:00:00:05"
NM_CONTROLLED="no"
ONBOOT="no"
note: we connected network cables for eth0 and eth2 only, so only use these 2 ports for bonding.

d. /etc/modprobe/bonding.conf
[root@db03 network-scripts]# more /etc/modprobe.d/bonding.conf
alias eth0 e1000e
alias eth1 e1000e
alias eth2 e1000e
alias eth3 e1000e
alias eth4 bnx2x
alias eth5 bnx2x
alias scsi_hostadapter cciss
alias scsi_hostadapter1 lpfc
alias bond0 bonding

3. Testing and debug
a. more /proc/net/bonding/bond0
b. ethtool eth0


4. References
a. RHEL6 deployment guide
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-networkscripts-interfaces-chan.html

b. Redhat knowledge base DOC-48159 for 'How do I configure the bonding device on Red Hat Enteprise Linux 6" or search bonding in access.redhat.com knowledge base.

How to convert CentOS root file system to XFS

Jephe Wu - http://linuxtechres.blogspot.com


As you know, the CentOS is a clone of Redhat Enterprise Linux. CentOS doesn't support XFS file system by default which is the same as RHEL, but CentOS provides additional kernel which has built-in XFS support. You can download it to replace the existing kernel to easily get XFS support in kernel space. But the problem is how to make your /, /usr,/var etc partition as XFS file system since you already used ext3 during installation.

This article is to guide you to do that, the following is my testing environments:

CentOS 4.1
partitions:
/dev/hda1 -/boot (ext3)
/dev/hda2 - swap
/dev/hda3 - / (ext3)
/dev/hda5 - /usr (ext3)
/dev/hda6 -/var (ext3)
/dev/hda7 - /tmp (ext3)
/dev/hda8 - /serverdata (ext3)(for storing data later, it's empty after installation)


Our objective:
convert all partitions except for /boot from default file system ext3 to XFS. Grub doesn't support XFS so we leave /boot as ext3.

Clone and XFS convert concept:
a. download XFS-enabled kernel from CentOS and install it
b. use /serverdata partition to save all server OS partitions such as /boot, /,/usr,/var,/tmp
c. boot up server using RIP(Recovery Is Possible) CD
d. format all partitions as XFS
e. copy back all data from /serverdata
f. change /etc/fstab and /boot/initrd image file
g. done. reboot

Steps:

a. download the latest RIP CD at http://www.tux.org/pub/people/kent-robotti/looplinux/rip/
download Non-X version is enough

b. download the XFS enabled supported kernel from CentOS website and install it to the server
# rpm -ivh xfs-enabled-kernel
# vi /etc/grub.conf to make sure it will boot with the new kernel next time
# reboot (make sure it can boot normally with new kernel)

c. copy the whole server OS to /serverdata, make xfs file system then copy back
c.1 reboot server with RIP CD after upgrading to xfs enabled kernel, you might want to choose the second option to skip keyboard map
c.2 login as root without password
c.3 make /serverdata partition as XFS first and backup the whole OS to /serverdata
mkfs -t xfs /dev/hda8
cd /mnt
mount /dev/hda3 hd
cd hd
mount /dev/hda1 boot
mount /dev/hda5 usr
mount /dev/hda6 var
mount /dev/hda7 tmp
mount /dev/hda8 serverdata
chroot .
tar --exclude ./boot --exclude ./proc --exclude ./serverdata -cpf - .| (cd serverdata; tar xvpf -)

c.4 umount all partitions then make xfs file systems
exit (exit from chroot environement)
cd /mnt/hd
umount usr
umount var
umount tmp
umount serverdata
umount boot
cd ..
umount hd

mkfs -t xfs -L / /dev/hda3
mkfs -t xfs -L /usr /dev/hda5
mkfs -t xfs -L /var /dev/hda6
mkfs -t xfs -L /tmp /dev/hda7
note: -L to make label for partition, be sure to use the same label name in /etc/fstab.

c.5 mount them again like c.3
cd /mnt
mount /dev/hda3 hd
cd hd
mount /dev/hda1 boot
mount /dev/hda5 usr
mount /dev/hda6 var
mount /dev/hda7 tmp
mount /dev/hda8 serverdata

c.6 copy back the whole OS from backup
(cd serverdata; tar cpf - . ) | tar xvpf -

d. make necessary changes
d.1 change /etc/fstab
cd /mnt/hd
chroot .
vi /etc/fstab (to change file system for those changed from ext3 to xfs, if you didn't use -L option in above mkfs -t xfs for /, /usr,/var,/tmp partitions, you also need to change the LABEL= to the real device name like from LABEL=/usr to /dev/hda5)

d.2 change /boot/initrd image file
cd /mnt/hd
chroot .
cp /boot/initrd.img /tmp/a (use the correct name for your initrd image, I use /boot/initrd.img here)
cd /tmp
zcat a > a1
mkdir a1.dir
cd a1.dir
cpio -iv < ../a1
modify something

vi init ( change ext3 to xfs, add xfs.ko module in init as well as lib/ folder) as follows:

[root@linuxtechres ]# more init

#!/bin/nash
mount -t proc /proc /proc
setquiet
echo Mounted /proc filesystem
echo Mounting sysfs
mount -t sysfs none /sys
echo Creating /dev
mount -o mode=0755 -t tmpfs none /dev
mknod /dev/console c 5 1
mknod /dev/null c 1 3
mknod /dev/zero c 1 5
mknod /dev/hda3 b 3 3 ---> added, hda3 is the root file system device
mkdir /dev/pts
mkdir /dev/shm
echo Starting udev
/sbin/udevstart
echo -n "/sbin/hotplug" > /proc/sys/kernel/hotplug
echo "Loading jbd.ko module"
insmod /lib/jbd.ko
echo "Loading ext3.ko module"
insmod /lib/ext3.ko
echo "Loading xfs.ko module"

insmod /lib/xfs.ko ---> added - please remember to copy xfs.ko file from /lib/modules/kernel_version/kernel/fs/xfs/xfs.ko to /lib

/sbin/udevstart
echo Creating root device
mkrootdev /dev/root
umount /sys
echo Mounting root filesystem
mount -o defaults --ro -t xfs /dev/root /sysroot ---> change ext3 to xfs
mount -t tmpfs --bind /dev /sysroot/dev
echo Switching to new root
switchroot /sysroot
umount /initrd/dev

re-generate initrd.img with xfs support

find . | cpio -co > ../c
cd ..
gzip c
cp c.gz /boot/initrd.xfs.img

change /etc/grub.conf to use this new initrd.xfs.img file.

d.3 install grub
cd /mnt/hd
chroot .
grub-install /dev/hda

After that, umount all partitions, reboot the server.

Clone Linux server with LVM2 partition

Jephe Wu - http://linuxtechres.blogspot.com


You might want to clone your production Linux server to another, but the production server is having LVM2 partition which is enabled by default installation.


1. My test environment is as follows:

Source:
Dell Latitude C510
CentOS 5
IDE 30G HDD
/dev/hda1 mounted as /boot
/dev/VolGroup00/LogVol00 mounted as /
/dev/VolGroup00/LogVol01 is swap partition

Destination:
Acer TravelMate 603TER
IDE 20G HDD

2. Cloning concept
a. use RIP(Recovery Is Possible) CD to boot up Acer laptop
b. make /boot normal partition and LVM2 partition
c. create physical volume, volume group and logical volumes
d. mount logical volumes then clone all files over from the source
e. modify /boot/initrd image file if the destination VG and LV names are different from the source ones, it's recommended to use the same name so that this step can be ignored
f. modify /etc/fstab and /etc/sysconfig/network-scripts/ifcfg-ethX etc
g. done

3. Cloning steps
a. download RIP CD from http://www.tux.org/pub/people/kent-robotti/looplinux/rip/
download smaller non-X version which is about 37M since we don't need X window for cloning

b. boot up the Acer laptop using RIP cd, choose boot up option 2 to skip keyboard map

c. login as root without password

d. Set up the ip address for Acer
# ifconfig eth0 192.168.0.3 up (assuming the source server IP is 192.168.0.2 and they are on the same network)

e. Make partitions for Acer
If the both the source and destination hard disks are the same size, you can just use the following command to clone the partition table over.
# ssh 192.168.0.2 'sfdisk -d /dev/hda' | sfdisk /dev/hda

In my case, I have to do it manually since my Acer HDD is smaller. So I created /dev/hda1(type:83) and /dev/hda2 as LVM(type: 8e)

f. Create logical volumes and make file systems on Acer
# pvcreate /dev/hda2 (initialize it)
# vgcreate VolGroup00 /dev/hda2 (use the same volume group name as the source so that you don't have to modify /boot/initrd image file later)
# vgdisplay -v (to find out the total PE numbers and available space to create logical volumes)
# lvcreate -l 4617 -n LogVol00 VolGroup00 (to create / partition )
# lvcreate -l 128 -n LogVol01 VolGroup00 (for swap partition inside LVM)

note : you don't have to create the same number of extend for each logical volume, you can specify any value, this setting is not fixed somewhere in source server, so you can still be able to startup destination server later.

# mkfs -t ext3 /dev/VolGroup00/LogVol00
# mkswap /dev/VolGroup00/LogVol01
# mkfs -t ext3 /dev/hda1 (for /boot)
# e2label /dev/hda1 /boot (for labelling /boot partition, to be the same as the source which is indicated in /etc/fstab)


g. Cloning over everything from the source
cd /mnt
mkdir hd
mount /dev/VolGroup00/LogVol00 hd
cd hd
mkdir boot proc sys ( boot is a separated partition, and proc and sys are empty folder for kernel memory information)
mount /dev/hda1 boot
ssh 192.168.0.2 'cd /; tar --exclude ./proc --exclude ./sys -cpf - .'| tar xvpf -

h. Make necessary changes for Acer
cd /mnt/hd;chroot .

vi /etc/fstab(optional) (since you are using the same volume group and logical volume names, so you can ignore this step, otherwise, change the names to the new ones)

vi /etc/grub.conf (optional) (since your are using the same volume group and logical volume names, so you can ignore this step, otherwise, change the all root= line to the new logical volume path)

vi /etc/sysconfig/network-scripts/ifcfg-ethx (to change IP address and comment out hardware address line)

i. Install grub for Acer
cd /mnt/hd
chroot .
grub-install hd0
exit

change necessary configuration such as /etc/sysconfig/networking-scripts/ifcfg-eth0 mac address

j. Finishing up
cd /mnt/hd
umount boot
cd /
umount /mnt/hd
reboot

Note:

1. if you forgot to do something, you can always reboot the destination server with RIP CD again, the following are the steps:
login as root after RIP bootup
sh /etc/rc.d/rc.lvm2 start (to activate all volume groups)
(if not, use 'vgchange -a y' to activate all volume groups
cd /mnt
mount /dev/VolGroup00/LogVol00 hd
cd hd
mount /dev/hda1 boot
chroot .
now you can do whatever forgotten changes

2. If you used the different volume group and logical volume names, you need to do some additional changes:

2.1 change /boot/initrd-2.6.18-8.el5.img before rebooting destination
cd /mnt
mount /dev/VolGroup00/LogVol00 hd
cd hd
mount /dev/hda1 boot
chroot .
cp /boot/initrd-2.6.18.el5.img /boot/intrd-2.6.18-el5.img.orig -va (backup it first)
cp /boot/initrd-2.6.18-8.el5.img /tmp/a.img
cd /tmp
zcat a.img > a
mkdir a.dir
cd a.dir
cpio -iv < ../a vi init (to change VolGroup00 to vg0 and LogVol00 to lv0 and LogVol01 to lv1 provided you used the names vg0, lv0 and lv1) find . | cpio -co > ../b
cd ..
gzip b (zip file b as b.gz)
cp b.gz /boot/initrd-2.6.18-8.el5.img
sync
exit
cd /
umount /mnt/hd/boot
umount /mnt/hd
reboot

2.2 change /etc/fstab to reflect the new volume group and logical volume names
2.3 change /etc/grub.conf root= line to use the new names

How to use ntfsresize from the command line

Jephe Wu http://linuxtechres.blogspot.com/


In order to resize Windows ntfs partition for whatever reason(e.g. shrink it for installing Linux in free space for multi boot), you should get ntfsresize tool. I recommend you download RIP(Recovery Is Possible) rescue CD(non-X version is enough).

The following is the step by step guide. Actually, the original article is from http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html but I couldn't access it anymore, so I also copy it on my blog for easier access for other people.

I'm not responsible for any data loss it could cause.

1. get RIP CD, boot up your Windows server with it, you might want to choose the second GRUB options to skip keyboard map

2. check your Windows ntfs partition first

# fdisk -l /dev/hda Disk /dev/hda: 255 heads, 63 sectors, 2480 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 2479 19912536 7 HPFS/NTFS

Only one partition, it's /dev/hda1 and NTFS, also marked as bootable

3. Find out where you could resize

# ./ntfsresize --info /dev/hda1 ntfsresize v1.9.4 NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 20390432768 bytes (20391 MB) Current device size: 20390436864 bytes (20391 MB) Checking filesystem consistency ... 100.00 percent completed Accounting clusters ... Space in use : 7851 MB (38.5%) Collecting shrinkage constrains ... Estimating smallest shrunken size supported ... You might resize at 7850958848 bytes or 7851 MB (freeing 12540 MB).

So we could free over 12 GB disk space using NTFS.

4. Make an ntfsresize test run, using the --no-action option

# ./ntfsresize --no-action --size 11000M /dev/hda1 ntfsresize v1.9.4 NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 20390432768 bytes (20391 MB) Current device size: 20390436864 bytes (20391 MB) New volume size : 10999996416 bytes (11000 MB) Checking filesystem consistency ... 100.00 percent completed Accounting clusters ... Space in use : 7851 MB (38.5%) Collecting shrinkage constrains ... Needed relocations : 265947 (1090 MB) Schedule chkdsk NTFS consistency check at Windows boot time ... Resetting $LogFile ... (this might take a while) Relocating needed data ... 100.00 percent completed Updating $BadClust file ... Updating $Bitmap file ... Updating Boot record ... The read-only test run ended successfully.

Everything looks good, let's go on

5. Resize NTFS

# # ./ntfsresize --size 11000M /dev/hda1 ntfsresize v1.9.4 NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 20390432768 bytes (20391 MB) Current device size: 20390436864 bytes (20391 MB) New volume size : 10999996416 bytes (11000 MB) Checking filesystem consistency ... 100.00 percent completed Accounting clusters ... Space in use : 7851 MB (38.5%) Collecting shrinkage constrains ... Needed relocations : 265947 (1090 MB) WARNING: Every sanity check passed and only the DANGEROUS operations left. Please make sure all your important data had been backed up in case of an unexpected failure! Are you sure you want to proceed (y/[n])? y Schedule chkdsk NTFS consistency check at Windows boot time ... Resetting $LogFile ... (this might take a while) Relocating needed data ... 100.00 percent completed Updating $BadClust file ... Updating $Bitmap file ... Updating Boot record ... Syncing device ... NTFS had been successfully resized on device '/dev/hda1'. You can go on to resize the device e.g. with 'fdisk'. IMPORTANT: When recreating the partition, make sure you 1) create it with the same starting disk cylinder 2) create it with the same partition type (usually 7, HPFS/NTFS) 3) do not make it smaller than the new NTFS filesystem size 4) set the bootable flag for the partition if it existed before Otherwise you may lose your data or can't boot your computer from the disk!

6. Optionally you could check your NTFS integrity

# ./ntfsresize --info --force /dev/hda1 ntfsresize v1.9.4 NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 10999992320 bytes (11000 MB) Current device size: 20390436864 bytes (20391 MB) Checking filesystem consistency ... 100.00 percent completed Accounting clusters ... Space in use : 7851 MB (71.3%) Collecting shrinkage constrains ... Estimating smallest shrunken size supported ... You might resize at 7850958848 bytes or 7851 MB (freeing 3149 MB).

Notice that the volume (NTFS) size indeed became 11000 MB but
the device (partition) size is still 20399 MB
7. Repartition the disk

• do backup first

# sfdisk -d /dev/hda > hda.pt # saves the partition table # dd if=/dev/hda of=hda.mbr bs=512 count=1 # saves the MBR

• Follow the steps below in order.
  
  • List the partition table. If you should see the correct partition types (NTFS),
  • Temporarily delete the entry for the 1st partition. This is an in-memory operation only.
  • Recreate the 1st partition entry at the same starting cylinder (sector) and using a larger size than above. This is very important because partitioning tools, fdisk included, may round down slightly the provided value to cylinder boundary. This would cause unbootable Windows. The theoretical maximum cylinder size currently is less than 140 MB hence this extra size should be always enough. We also use 11140 MB, it must be safe.
  • Set the partition type to NTFS (type 7).
  • Mark it bootable if it was also marked before.
  • Print the partition table again to check everything is right.
  • Commit the whole process by writing it to disk.

  # fdisk /dev/hda Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 2480 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 2479 19912536 7 HPFS/NTFS Command (m for help): d Partition number (1-4): 1 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-2480, default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-2480, default 2480): +11140M Command (m for help): t Partition number (1-4): 1 Hex code (type L to list codes): 7 Changed system type of partition 1 to 7 (HPFS/NTFS) Command (m for help): a Partition number (1-4): 1 Command (m for help): p Disk /dev/hda: 255 heads, 63 sectors, 2480 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 1355 10884006 7 HPFS/NTFS Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

• Next step is also optional but strongly recommended. We check again, using ntfsresize, if we can still access our NTFS. For this purpose one must use both the --info and --force options again.

  # ./ntfsresize --info --force /dev/hda1 ntfsresize v1.9.4 NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 10999992320 bytes (11000 MB) Current device size: 11145222144 bytes (11146 MB) Checking filesystem consistency ... 100.00 percent completed Accounting clusters ... Space in use : 7851 MB (71.3%) Collecting shrinkage constrains ... Estimating smallest shrunken size supported ... You might resize at 7850958848 bytes or 7851 MB (freeing 3149 MB).

  
  Excellent, still no error.

If you got error, you might need to give a bit more partition size for above fdisk operation.

• Reboot to Windows to check everything is right (e.g. pressing [Ctrl]-[Alt]-[Del]). Don't get scared when during the boot process Windows will check the filesystem. It was scheduled by ntfsresize for extra safety. If the partition is a system partition then Windows will restart automatically after the filesystem check.

Now you should have unallocated disk space that all Linux distribution installations must be able to handle.

Use Linux to reset Windows server administrator password

Jephe Wu http://linuxtechres.blogspot.com


It's possible you might forget your Windows server administrator password or it's locked after you tried too many times. But don't worry, there's a nice tool 'chntpw' which can help you to reset and unlock it. The following example was tested on Windows 2003 server.

1. download the Linux rescue CD - RIP(Recovery Is Possible)
note: to reduce download time. Non-X version is enough.
2. reboot your Windows server with this CD, you might want to choose option 2 to skip keyboard map
3. login as root without password
4. run the following commands to mount Windows 2003 server partition as read-write
cd /mnt

mount -t ntfs-3g /dev/sda1 win -o force (assuming your Windows partition is /dev/sda1, you can use fdisk command to find out)
note: to make sure you mounted Windows partition as read/write, you can try to run command
touch /mnt/win/testfile to confirm it won't give you any error.

cd win/WIN2K03/system32/config

chntpw SAM

then follow the screen instruction
first, answer y to reset/unlock the counters
then enter * to blank password
finally, enter y all the way to finish

5. finishing up
cd /mnt
sync
umount win
reboot

Note:
1. you can use chntpw SAM -l to list all Windows users
2. use chntpw SAM -u user1 to unlock/reset user1 password
3. http://www.tuxera.com/community/ntfs-3g-download/  (NTFS 3G website)

Cloning a CD under Linux command line

Jephe Wu(linuxtechres.blogspot.com)

It's easy to clone a standard CD under Linux command line, the following are the steps:

1. find out the volume size (assuming your cdrom is /dev/hdc)
# isoinfo -d -i /dev/hdc
record down volume size (e.g. 53488)

2. generate md5sum from CD
# dd if=/dev/hdc bs=2048 count=53488 | md5sum

3. generating iso file from the existing CD
# dd if=/dev/hdc of=/linuxtechres.iso bs=2048 count=53488 conv=notrunc

4. generate md5sum from iso file
# md5sum /linuxtechres.iso
make sure the output is the same as the step 2

5. burn the iso file

# cdrecord dev=/dev/hdc speed=10 padsize=63s -pad -dao -v -eject /linuxtechres.iso

note:
a. always use padsize=63s -pad -dao to generate a CDR which can be used anywhere.
b. do not use the maximum speed your cd writer can support, use a bit lower speed to write.

6. verify the CD
a. put in CDR, run command 'isoinfo -d -i /dev/hdc' to find out volume size value, let's say it's 53488
b. dd if=/dev/hdc bs=2048 count=53488 | md5sum
make sure the output is the same as step 2 and 4.

That's it.

Cloning Windows server - using RIP and ntfsclone

Jephe Wu http://linuxtechres.blogspot.com


RIP(Recovery Is Possible) CD can also be used for cloning Windows NTFS partition. If the source and destination servers have the same kind of hardwares, you can use the RIP and ntfsclone which comes with RIP CD to clone Windows NTFS partition.

Use ntfsclone to clone Windows server through network

on the source server(HP DL360)
1) properly shutdown your Windows. Boot from RIP CD, choice the second bootup option which loads everything in memory and skip keyboard map
2) ifconfig eth0 192.168.0.1 up (you can set any ip address, I use 192.168.0.1 here)
4) passwd root (to change root password for remote ssh login later)
5) /usr/sbin/sshd (startup ssh daemon)
6) check ntfs partition information with commands below: (optional)
ntfsinfo -m -v /dev/cciss/c0d0p1 | less
ntfsinfo -i -v /dev/cciss/c0d0p1 | less
7) you might need to use command 'ntfsfix' to fix some basic problems such as Windows didn't shutdown properly before cloning.
ntfsfix /dev/cciss/c0d0p1

on the destination server:
1) boot from RIP CD too, same as what you did on the source
2) use 'fdisk' to make new partition(the size must be equal or greater than the server) (use id 7 (hpfs/ntfs).
you can also use 'sfdisk' to clone the partition table over like below if you'd like to have the exact same partition size.
ifconfig eth0 192.168.0.2 up
ssh 192.168.0.1 'sfdisk -d /dev/cciss/c0d0' | sfdisk /dev/cciss/c0d0
3) clone over the MBR code first(excluding partition table info)
ssh 192.168.0.1 'dd if=/dev/cciss/c0d0 count=1 bs=446' | dd of=/dev/cciss/c0d0
It might doesn't bootup properly if the destination server has the different kind of hard disk, you can try step 5 then to install MBR.
4) clone the ntfs partition content over
ssh 192.168.0.1 'ntfsclone -s -o - /dev/cciss/c0d0p1' | ntfsclone -r - -O /dev/cciss/c0d0p1
reboot
5) in case it doesn't boot up. try to boot from windows CD to recovery console mode to run 'fixmbr' to install the proper MBR

Note: if it doesn't boot up, in order to find out it's due to MBR or ntfsclone issue, you can boot with the RIP CD, then choose 'Boot MBR on first hard disk' option to check if Windows can boot up properly.

note: you can also use RIP and MBRFix program to easily fix the MBR, please refer my another article at http://linuxtechres.blogspot.com/2008/09/use-rip-cd-and-mbrfix-to-easily-fix.html

Use ntfsclone to backup Windows ntfs partition
If you want to do backup for the source Windows ntfs partition, you can do the following on the source after booting up with RIP CD

a. backup partition table
sfdisk -d /dev/cciss/c0d0 | ssh remoteserver 'cat > /path/to/c0d0_sfdisk-d'
for restore, use
ssh remoteserver 'cat /path/to/c0d0_sfdisk-d' | sfdisk /dev/cciss/c0d0

b. backup MBR
dd if=/dev/cciss/c0d0 count=1 bs=512 | ssh remoteserver 'dd of=/path/to/mbr_dd'
for restore, use
ssh remoteserver 'dd if=/path/to/mbr_dd' | dd of=/dev/cciss/c0d0

c. backup ntfs partition /dev/cciss/c0d0p1
ntfsclone -s -o - /dev/cciss/c0d0p1 |gzip -c | ssh remoteserver 'cat > /path/to/c0d0p1_ntfsclone.gz'
for restore, use
ssh remoteserver 'gzip -dc /path/to/c0d0p1_ntfsclone.gz' | ntfsclone -r - -O /dev/cciss/c0d0p1



note: If the original hard disk is bad, you might want to use the parameters below during backup.

--force --ignore-fs-check

References:
a. http://edoceo.com/exemplar/ntfsclone-transfer-windows

Cloning a Linux server - general method

Jephe Wu http://linuxtechres.blogspot.com


Last time, we talked about using dd to do Linux server cloning for the same kind of hardwares - Cloning a Linux server - scenery 1. This time, we will introduce another general method for doing Linux server cloning.

I assume the source is using the normal partition which means it's not using software raid, LVM etc. For cloning Linux server with LVM partition, please check my another article at http://linuxtechres.blogspot.com/2007/08/clone-linux-server-with-lvm2-partition.html . My test environment is as follows:

the source:
a. HP DL380 (Intel Xeon CPU)
b. Redhat Enterprise Linux 4

the destination:
a. HP DL385G2 AMD64

The following are the cloning steps:

1. boot up the destination server from RIP(Recovery Is Possible) CD
You can choose the second options in GRUB bootup menu to load everything into memory and skip keyboard map option, then you can eject CD out before doing the following if you'd like to.

2. configure the IP address for the destination server( we use 192.168.0.101 here)
ifconfig eth0 192.168.0.101 up

2. copy over the 3 files from the source(assuming server name is linuxtechres, the IP is 192.168.0.100)
scp 192.168.0.100:/etc/passwd /etc/
scp 192.168.0.100:/etc/shadow /etc/
scp 192.168.0.100:/etc/group /etc/

3. clone the partition table over using sfdisk
Assuming /dev/cciss/c0d0p1 is / partition, /dev/cciss/c0d0p2 is the /boot partition.

ssh 192.168.0.100 'sfdisk -d /dev/cciss/c0d0' | sfdisk /dev/cciss/c0d0
mkfs -t ext3 /dev/cciss/c0d0p1
e2label /dev/cciss/c0d0p1 /
mkfs -t ext3 /dev/cciss/c0d0p2
e2label /dev/cciss/c0d0p2 /boot
Note: check your source server /etc/fstab so that you know why we use e2label to label the partition / and /boot. That's the default label name by Redhat Enterprise Linux for / and /boot partitions.
cd /mnt
mount /dev/cciss/c0d0p1 hd
cd hd
mkdir boot proc sys mnt ...
mount /dev/cciss/c0d0p2 boot
note: if you have some more partitions like /usr and /home etc, make sure you make file system for all partitions and give the correct label as what the source has, similar as the above steps.

4. clone everything over now
cd /mnt/hd
ssh 192.168.0.100 'cd /; tar --exclude ./proc --exclude ./sys --exclude ./mnt -cpf - .' | tar xvpf -
note: you might want to exclude some data folder or partitions also so that you can finish the cloning process earlier, after that, you can copy the data folder/partition separately. see the step 8
cd etc
rm -f blkid.tab mtab (because they will be generated by OS each time it boots up)
vi sysconfig/network-scripts/ifcfg-eth* ( to comment out hardware address lines and change ip address line to the destination IP)

5. install GRUB (if both the source and the destination are x86 32bit)
cd /mnt/hd
chroot .; grub_install hd0
In my case, I skiped this step and use step 7 to install GRUB since the destination is AMD64 CPU

6. umount those partition that you mounted before
exit (exit from chroot environment)
cd /mnt/hd
umount boot mnt
cd /mnt
umount hd
sync;sync;sync
reboot

7. install GRUB on AMD64 servers
If you are cloning from x86_32 server to x86_64 server like AMD64, you have to
boot from the first x86_64 RedHat Enterprise Linux 4 CD to install GRUB
chroot /mnt/sysimage;grub_install hd0

8. rsync over the data partitions
after the OS boots up, you can use rsync to copy the data folder or partition over.
rsync -e ssh -P -avz source_ip_addr:/data/ /data/

That's it. Now you can bring up one more server which is exactly the same as the source.

Cloning a Linux server - use dd

Jephe Wu http://linuxtechres.blogspot.com

One of the benefits of using Linux is that you don't have to install it every time, you can ALWAYS clone it despite the destination server hardware is the same as the source ones or not. There're two situations as follows:

1) exactly same hardwares.
2) different hardwares

For each situation, there're different ways to clone the servers. Today, I'll introduce how to clone server by using command dd.

In order to use dd. Generally, the destination server will be having the same kind of server model as the source, or at least the same kind of network card and hard disk.

dd will clone everything from source to destination, sector by sector. dd doesn't care about what kind of OS, it can apply to Windows too. The advantage is that you only need to use one command, the disadvantage is that it might take a bit longer since it will copy every sector of your hard disk, even there's no data on those sectors. If the time is not important to you, and the hardware is the almost same, it might be the best choice for you to clone servers.

The following is the steps I use.

a. keep the minimum application running on the source if possible.
Shutdown those services as long as you can. The idea here is to prevent anything from writing things to the hard disk during cloning process. However, according to my experience, it works well even if you don't shutdown anything, just keep the source server running as normal while cloning.

b. use RIP(Recovery is possible) CD to boot up the destination server.
You can download RIP CD from here. The smaller non-X version is enough, around 37M. Burn it to CD, boot up the destination server.

Normally, I choose the second GRUB bootup option for RIP so that I can eject out the CD after boot up, everything will be in memory only.

c. after booting up RIP, login as root(no password is needed), running the following commands to set up IP address:
ifconfig eth0 ip_address_here up (IP address here should be the same network segment as the source)

d. run the following commands to clone:
ssh source_server_ip 'dd if=/dev/sda' | dd of=/dev/sda

note: if the hard disk is HP hardware raid, it might be /dev/cciss/c0d0. Just replace it with yours.

Now you have to wait until it finish.

Note: the dd won't give you any process report as long as the things go well. However, you can use the command

pkill -USR1 ^dd$

to get the progress report, for more info, you can refer to this excellent article at
http://www.redhatmagazine.com/2007/08/16/tips-from-an-rhce-how-can-i-make-dd-give-me-a-progress-report/
(How to make dd to give me progress report)

If you want to check remotely if it finish or not. Then you can run the following commands after setting up ip on destination server and before doing actual cloning:

passwd root (to change root password)
/usr/sbin/sshd (to start sshd daemon for remote login)

You can also boot up the source using RIP CD, then use the above commands to change root password for the source and start sshd daemon for you login from the destination server.

That's it. I'll introduce other methods for cloning servers for same kind of hardwares later.