Showing posts with label rsync. Show all posts
Showing posts with label rsync. Show all posts

How to setup an ideal rsync task between two Linux servers

                                            by Jephe Wu - http://linuxtechres.blogspot.com

 

Summary

There's some issue for syncing server a(source) to server b (dest.) which caused server b partition full, in order to solve the problem, we need to adjust some rsync parameters for cronjob task.


Original rsync task under root cronjob

*/30 * * * *  /usr/bin/rsync --numeric-ids -avr --exclude=.snapshot 192.168.1.10::share/* /share

Note: this has caused server b /share partition full due to it won't delete files on destination site.


In order to make a full sync working again, we will need to manually run task below which will delete first to free up space, then sync files again.

/usr/bin/rsync --numeric-ids -avr --delete --delete-before --ignore-errors --exclude=.snapshot 192.168.1.10::share/* /share


The rsync daemon config on server a is below:

[root@servera share]# more /etc/rsyncd.conf

reverse lookup = false

[share]
  path = /share
  numeric ids = yes
  max connections = 10
  list = true



After full sync completed, we will change cronjob to below:


*/30 * * * *  /usr/bin/rsync --numeric-ids -avr --delete --delete-before --ignore-errors --exclude=.snapshot 192.168.1.10::share/* /share >> /var/log/rsync-cron.log

Note: sometimes, you will encounter error message 'IO error encountered -- skipping file deletion' for rsync command, you can use '--ignore-errors' option as above to avoid this issue.



Common Linux tools usage

Jephe Wu - http://linuxtechres.blogspot.com/


  • bash  - if
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html
  • vi
30G - go to 30th  line
30| - go to 30th column
change a file to remove the ending new line
vi file , :%s#$\n# #g
  • sed (delete one line from the file itself)
for i in B*;do sed -i /A443/d $i;done 
JEPHE=A1234; for i in B*;do sed -i "/$JEPHE/d" $i;done
  • Tar
Backup files and exclude a list of files from a file:
tar --exclude-from=/path/to/excludedfilelist -cvpzf  file.tar.gz *

note: inside excludedfilelist, give file or directory name line by line, don't put / for directories behind.

delete a file from a tar archive:
tar --delete --file=file.tar tobedeletedfile

transfer sparse file on the network:


tar cvzSpf - *|ssh jephe@remoteserver '(cd /path/to; tar xzSpf -)'


  • rsync
use rsync to transfer specified files under some directories.
rsync -av -r  --include-from=include.txt  /cygdrive/e/ root@10.0.0.1:/data/backup/

$ cat include.txt
+ a
+ a/b/
+ b
+ b/c/
+ a/b/*.dat
+ b/c/*.exe
- *

/usr/bin/rsync --timeout=600 -v --progress --include=*.gpg --exclude=* -a -e ssh --delete /data/db 10.0.0.1:/data/db/ > /tmp/dbbackup

Note: refer to http://samba.anu.edu.au/ftp/rsync/rsync.html


transfer-root directory referes to the source directory on the source server.

Here are some examples of exclude/include matching:

o --exclude "*.o" would exclude all filenames matching *.o
o --exclude "/foo" would exclude a file called foo in the transfer-root directory
o --exclude "foo/" would exclude any directory called foo
o --exclude "/foo/*/bar" would exclude any file called bar two levels below a directory called foo in the transfer-root directory
o --exclude "/foo/**/bar" would exclude any file called bar two or more levels below a directory called foo in the transfer-root directory
o --include "*/" --include "*.c" --exclude "*" would include all directories and C source files
o --include "foo/" --include "foo/bar.c" --exclude "*" would include only foo/bar.c (the foo/ directory must be
explicitly included or it would be excluded by the "*")

o --exclude "*/foo/" would exclude any directory called foo which is one level below the transfer root directory
o --exclude "**/foo/" would exclude any directory called foo which is one or more levels below the transfer-root directory
  •  curl & wget
download a iso file  -  curl -CO complete_download_url and wget -c complete_download_url
check web response header  - curl -I www.domain.com

  • nc
nc -v -z -s 10.0.0.1 10.0.10.1 25
Connection to 10.0.10.1 25 port [tcp/smtp] succeeded!
for udp, use:
nc -vuz destination_ip_addr 53  (udp might always report it's successful when using -z option)

  • socat  (linux or cygwin)
$ socat -d -d tcp4-listen:25,bind=192.168.100.20,fork,reuseaddr tcp4:smtp.gmail.com:25,bind=192.168.100.20 &

  • awk
ls -l *.3.gz.gpg  | awk '{total += $5} END { print total}'
2298937968

db2 list application | awk '($3==495)'  => print the third column which equals 495

  • netstat
netstat -tunelp  --list listening tcp and udp port numbers
netstat -natup
  • ssh port forwarding 
Local port forwarding
$ ssh jephe@server1 -L 1234:10.0.0.1:22 [-g]  (-g means allows remote hosts to connect to local forwarded ports)
$ ssh jephe@localhost -p 1234

Remote port forwarding
$ ssh jephe@office_server -R 1234:10.0.0.2:22  (on office server, the user can connect to localhost at port 1234 to access the local ssh server at port 22)

or


$ ssh jephe@office_server -R 2222:server_on_internet:22 (when user ssh to port 2222 to office server, it actually goes to internet server ssh port)

If you want to anyone from your office network to access office server at port 2222 which will be forwarded to server_on_internet ssh server, uncomment GatewayPorts line as
GatewayPorts yes


or
$ ssh jephe@office_server -R 80:your_home_web_server:80 (your home web server is not necessarily same as your home ssh client pc, can be another server)


Reference: http://souptonuts.sourceforge.net/sshtips.htm

You can use Windows cygwin ssh server plus ssh remote port forwarding to achieve something. Assume you have lease line connected to remote office, you can only ssh into remote office, now you need to install Linux in one of machine, you can use this method to install from local office http server.

Use above local and remote port forwarding + cygwin + openssh + putty + proxytunnel at http://proxytunnel.sourceforge.net/ , you can do a lot of things you might think it's impossible before.

If your company only allows to use proxy to access Internet and you control one of ssh server on Internet, then you got the power to access office network from home.
  • rpm
rpm -Uvh --root=/tmp/ --nodeps /mnt/iso/CentOS/glibc-2.5-34.x86_64.rpm 
or
  1. Make a temporary directory to extract the rpm in and copy the rpm into the directory: 
    
     mkdir tempdir
     cp bash.rpm tempdir
     

  2. Execute rpm2cpio in the temporary directory: 
    
     cd tempdir
     rpm2cpio bash.rpm | cpio -idmv
     
  3. show architecture
 rpm -qf /bin/ls --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n'

  • stty and setterm/xset
stty erase ^H  (ctrl V H at the same time) 

setterm -blank nn will tell the console driver to blank the screen after nn minutes of inactivity. (With nn = 0, screensaving is turned off. In some old kernels this first took effect after the next keyboard interrupt.)

The s option of xset(1) will set the X screensaving parameters: xset s off turns off the screensaver, xset s 10 blanks the screen after 10 minutes.
  • ssh 
ssh user@host -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
ssh-keygen -F 10.0.3.1  - find this host from .ssh/known_hosts file
ssh-keygen -R 10.0.3.1  - remove this host from .ssh/known_hosts file
 
$ sshfs jephe@server:/path/to /mnt/sshfs/ 

Working in a ssh shell that used forwarding:
Supported escape sequences:

~. – terminate connection
~B – send a BREAK to the remote system
~C – open a command line
~R – Request rekey (SSH protocol 2 only)
~^Z – suspend ssh
~# – list forwarded connections
~& – background ssh (when waiting for connections to terminate)
~? – this message
~~ – send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
~. and ~# are particularly useful.

  • ftp (windows)
literal pasv (to change to passive mode under Windows ftp CLI) 

  • sftp
$ sftp -b /tmp/1 -o port=2222 jephe@1.2.3.4

$ more 1
put /etc/hosts

$ more 1
rm hosts
  • grep/cat
grep -v "^#" /etc/httpd/conf/httpd.conf | cat -s | less
  • tcpdump 
    Collecting a TCP dump from the server using a command like the following:
    tcpdump -s0 -w /tmp/tcpdump.pcap -i any host <client ip> and port 80

    and generating traffic (HTTP or LDAP) to the server captures evidence of the server not responding to the TCP SYN packets.  The output file can be analyzed with a command like
    tcpdump -r /<path>/<to>/tcpdump.pcap
For monitoring openbsd firewall PF rules, you can use 'tcpdump -n -e -ttt -i pflog0' to see which pf rules is matching the traffic, pass or block. use 'pfctl -sr' to get output of in-memory rules, the first block or pass rule will be rule 0, followed by rule 1, rule 2 and so on.
  • Tshark
     tshark -i eth0 -f 'host 10.0.2.1' -w /tmp/upstream.cap -S
  • screen
 screen -S main (manually)

/home/jephe/.bash_profile contact 'screen -D -R main'. this way, if I login server, it will disconnect then reconnect my screen session 'main' automatically every time. So we keep one session only.

Cloning a Linux server - general method

Jephe Wu http://linuxtechres.blogspot.com


Last time, we talked about using dd to do Linux server cloning for the same kind of hardwares - Cloning a Linux server - scenery 1. This time, we will introduce another general method for doing Linux server cloning.

I assume the source is using the normal partition which means it's not using software raid, LVM etc. For cloning Linux server with LVM partition, please check my another article at http://linuxtechres.blogspot.com/2007/08/clone-linux-server-with-lvm2-partition.html . My test environment is as follows:

the source:
a. HP DL380 (Intel Xeon CPU)
b. Redhat Enterprise Linux 4

the destination:
a. HP DL385G2 AMD64

The following are the cloning steps:

1. boot up the destination server from RIP(Recovery Is Possible) CD
You can choose the second options in GRUB bootup menu to load everything into memory and skip keyboard map option, then you can eject CD out before doing the following if you'd like to.

2. configure the IP address for the destination server( we use 192.168.0.101 here)
ifconfig eth0 192.168.0.101 up

2. copy over the 3 files from the source(assuming server name is linuxtechres, the IP is 192.168.0.100)
scp 192.168.0.100:/etc/passwd /etc/
scp 192.168.0.100:/etc/shadow /etc/
scp 192.168.0.100:/etc/group /etc/

3. clone the partition table over using sfdisk
Assuming /dev/cciss/c0d0p1 is / partition, /dev/cciss/c0d0p2 is the /boot partition.

ssh 192.168.0.100 'sfdisk -d /dev/cciss/c0d0' | sfdisk /dev/cciss/c0d0
mkfs -t ext3 /dev/cciss/c0d0p1
e2label /dev/cciss/c0d0p1 /
mkfs -t ext3 /dev/cciss/c0d0p2
e2label /dev/cciss/c0d0p2 /boot
Note: check your source server /etc/fstab so that you know why we use e2label to label the partition / and /boot. That's the default label name by Redhat Enterprise Linux for / and /boot partitions.
cd /mnt
mount /dev/cciss/c0d0p1 hd
cd hd
mkdir boot proc sys mnt ...
mount /dev/cciss/c0d0p2 boot
note: if you have some more partitions like /usr and /home etc, make sure you make file system for all partitions and give the correct label as what the source has, similar as the above steps.

4. clone everything over now
cd /mnt/hd
ssh 192.168.0.100 'cd /; tar --exclude ./proc --exclude ./sys --exclude ./mnt -cpf - .' | tar xvpf -
note: you might want to exclude some data folder or partitions also so that you can finish the cloning process earlier, after that, you can copy the data folder/partition separately. see the step 8
cd etc
rm -f blkid.tab mtab (because they will be generated by OS each time it boots up)
vi sysconfig/network-scripts/ifcfg-eth* ( to comment out hardware address lines and change ip address line to the destination IP)

5. install GRUB (if both the source and the destination are x86 32bit)
cd /mnt/hd
chroot .; grub_install hd0
In my case, I skiped this step and use step 7 to install GRUB since the destination is AMD64 CPU

6. umount those partition that you mounted before
exit (exit from chroot environment)
cd /mnt/hd
umount boot mnt
cd /mnt
umount hd
sync;sync;sync
reboot

7. install GRUB on AMD64 servers
If you are cloning from x86_32 server to x86_64 server like AMD64, you have to
boot from the first x86_64 RedHat Enterprise Linux 4 CD to install GRUB
chroot /mnt/sysimage;grub_install hd0

8. rsync over the data partitions
after the OS boots up, you can use rsync to copy the data folder or partition over.
rsync -e ssh -P -avz source_ip_addr:/data/ /data/

That's it. Now you can bring up one more server which is exactly the same as the source.