Keep both Zabbix 1.8 and 2.2 running for real world testing

Jephe Wu - http://linuxtechres.blogspot.com

Objective: keep both Zabbix 1.8 and 2.2 system running for same monitored hosts for real world testing.
Challenge: make zabbix 2.2 proxies to be able to monitor same set of agent hosts without adding new proxies into /etc/zabbix/zabbix_agentd.conf
Environment: CentOS 6.4 64bit, Zabbix 1.8(old system) and Zabbix 2.2(new system), Percona server 5.5(both old and new), Zabbix proxy 1.8(old system) and Zabbix proxy 2.2(new system)

Zabbix monitoring network is at 192.168.0.0/24, monitored hosts are at different network, we use about 10 zabbix proxies for monitoring. All agent hosts are configured to use these 10 zabbix proxies for monitoring.


Network diagram

Steps:
1.  Preparing Zabbix 2.2 system server/VM first 

We can prepare Zabbix 2.2 web server VM, zabbix 2.2 server VM and all zabbix 2.2 Proxies first.

2. Online clone Percona server 5.5 db01 to db02
refer to http://linuxtechres.blogspot.com.au/2014/02/preparing-mysql-slave-database-by.html

3.  Enable ip_forward and iptables NAT on existing proxies
For proxy servers prox01,prox02,....prox10, run:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -p tcp -o eth0 -j SNAT --to 192.168.0.100

Note: this will enable ip forward for any traffic received from eth0, then the destination is not for prox01 itself

4.  on new proxies, enable default gateway to old proxies
on each proxy server 2.2, change default gateway to coresponding old 1.8 proxy server IP address 

5.  on each new proxy server, configure hostname as old proxy name
e.g. still use hostname=prox01 on prox01new

This way, new proxy will get data through old proxy server iptables IP Masquerading.

6. go to new Zabbix 2.2 web GUI to modify SMTP server setting to avoid sending double alert together with old 1.8 Zabbix system.

How to do migration to 2.2 after testing


1.  shutdown each proxy, bring up new coresponding proxy with same IP address
2.  remove default gateway, use normal default gateway instead.

Preparing mysql slave database by online cloning Percona master database server

Jephe Wu - http://linuxtechres.blogspot.com
Objective: use Xtrabackup from Percona server 5.5 to online create  slave database from master.
Environment: CentOS 6.4 64bit, Percona server 5.5 for both master(192.168.0.1) and slave(192.168.0.2), almost same hardware specs for master and slave


Steps:
1. Backup from master
# innobackupex --user=root --password=password /data/cloneforslave/ --no-timestamp  
(innobackupex will create /data/cloneforslave folder automatically)
# xtrabackup --prepare --target-dir=/data/cloneforslave/
# cd /data/cloneforslave
# more xtrabackup_binlog_info
mysql-bin.129290 14072915

2. transfer to slave 
#rsync --progress -avz /data/cloneforslave/ slaveserver:/data/cloneforslave --delete  
(create /data/cloneforslave directory first on slaveserver)
3. Prepareing slave 
cd /var/lib/mysql/
mv data data.old
cp /data/cloneforhmspzdb05 /var/lib/mysql/data -va
chown mysql:mysql -R data
vi /etc/my.cnf # to change server_id from 1 to 2, optional you can add read-only parameter 
/etc/init.d/mysql start

4. setting up slave
mysql -uroot -ppassword zabbix
mysql> stop slave;
mysql> reset slave;
mysql> change master to master_host='192.168.0.1', master_user='replication', master_password='replication_password', master_log_file='mysql-bin.129290', master_log_pos=14072915;
mysql> start slave
mysql> show slave status\G

5.References
http://www.percona.com/doc/percona-xtrabackup/2.1/howtos/setting_up_replication.html
mysql tuning tool: tuning-primer.sh