How to recover deleted file under Linux

Jephe Wu - http://linuxtechres.blogspot.com

Objective: undelete httpd access log file while httpd process is still running
Environment:  CentOS 6.5 64bit

Cases Study:
Sometime, you deleted a big httpd log file while httpd is still running. As long as httpd process doesn't exit, you still can recover removed deleted httpd log file.

Apply to: Recover deleted file, data recovery under Linux,  restore deleted file,  undo deletion

Steps:


1. For testing purpose, list access_log file content first for confirmation

[root@www httpd]# pwd
/var/log/httpd
[root@www httpd]# more access_log
192.168.1.15 - - [24/Jul/2014:09:02:25 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.15 - - [25/Jul/2014:02:29:59 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:58 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:55:26 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:56:11 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"

2. purposely delete /var/log/httpd/access_log, then check lsof

[root@www httpd]# cp access_log /tmp
[root@www httpd]# rm -f access_log
[root@www httpd]# lsof | grep -i deleted
httpd     1146    root   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1177  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1178  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1179  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1180  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1181  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1182  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1183  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1184  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)

Note: above httpd process 1146 is running under root which is parent process of httpd.

3. find out the FD of parent process of httpd then retrieve deleted access log file

[root@www fd]# cd /proc/1146/fd
[root@www fd]# ll
total 0
lr-x------ 1 root root 64 Dec  4 06:40 0 -> /dev/null
l-wx------ 1 root root 64 Dec  4 06:40 1 -> /dev/null
l-wx------ 1 root root 64 Dec  4 06:40 10 -> /var/log/httpd/ssl_error_log
l-wx------ 1 root root 64 Dec  4 06:40 11 -> /var/log/httpd/access_log (deleted)
l-wx------ 1 root root 64 Dec  4 06:40 12 -> /var/log/httpd/ssl_access_log
l-wx------ 1 root root 64 Dec  4 06:40 13 -> /var/log/httpd/ssl_request_log
l-wx------ 1 root root 64 Dec  4 06:40 2 -> /var/log/httpd/error_log
lr-x------ 1 root root 64 Dec  4 06:40 3 -> /dev/urandom
lrwx------ 1 root root 64 Dec  4 06:40 4 -> socket:[9492]
lrwx------ 1 root root 64 Dec  4 06:40 5 -> socket:[9493]
lrwx------ 1 root root 64 Dec  4 06:40 6 -> socket:[9496]
lrwx------ 1 root root 64 Dec  4 06:40 7 -> socket:[9497]
lr-x------ 1 root root 64 Dec  4 06:40 8 -> pipe:[9509]
l-wx------ 1 root root 64 Dec  4 06:40 9 -> pipe:[9509]

[root@www fd]# cp 11 /var/log/httpd/access_log.deleted

[root@www fd]# cat /var/log/httpd/access_log.deleted
192.168.1.15 - - [24/Jul/2014:09:02:25 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.15 - - [25/Jul/2014:02:29:59 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:58 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:55:26 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:56:11 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"


4. compare both files by diff and md5sum

[root@www fd]# diff /tmp/access_log /var/log/httpd/access_log.deleted

[root@www fd]# md5sum /tmp/access_log /var/log/httpd/access_log.deleted
9aa2e8c58ecfb689d3ada71bb6ec5815  /tmp/access_log
9aa2e8c58ecfb689d3ada71bb6ec5815  /var/log/httpd/access_log.deleted

Note: in my test environment, there's no active web access traffic so both log files are the same.

5. free up disk space
When httpd is running and access log file is deleted, the disk space won't be freed up, df -h and du reported disk free space do not match, you can use lsof -i deleted to find out if there's a big file was deleted to decide which process you need to restart to retrieve back the free space, in this case, I need to restart httpd process to free up space after restore back access log file.

How to install a new server ssl certificate with chain certificate

Jephe Wu - http://linuxtechres.blogspot.com

Environment: CentOS 5.5 32bit web server with Apache httpd installed. We need to renew one of ssl certificate for our client.
Objective: apply for new ssl cert without shutting down server


Steps:
1) preparing csr file for submit to CA
2) configure new cert file and sslcertificatechainfile directive in Apache
3) run command 'kill -USR1 ppid_of_httpd'
4) use 'stat' command to check if new cert files being read (compare file access time before and after running stat)
5) use browser to check server certificate information to confirm new cert has been placed.

How to use looping in Linux bash script

Jephe Wu - http://linuxtechres.blogspot.com


Objective: list all kinds of Linux bash looping script which is useful for bash programming.
let computer do work for you.

Concept: 
Very often, we just need to run some script by loop to finish work


Examples


1. infinite loop

while true;do ...done
while [ 1 ]; do...done
while : ; do ....done

watch ls -l /tmp/file

2.  List a range of number 

list each line of /etc/hosts file
while read line;do echo $line;done < /etc/hosts

list number 1 to 10
for i in {1..10};do echo $i;done
for i in $(seq 1 10);do echo $i;done
for i in `seq 1 10`;do echo $i;done

list number 1 to 10 with same width
for i in {01..10};do echo $i;done
for i in $(seq -w 1 10);do echo $i;done
for i in `seq -w 1 10`;do echo $i;done

list number 1 to 10 with only odd number
for i in {1..10..2};do echo $i;done

list number 1 to 10 with only even number
for i in {2..10..2};do echo $i;done

Use just echo to list 1 to 10
echo {1..10}
echo {01..10}

Use just echo to list a-z and A-Z
echo {a..z}
echo {A..Z}

tranditional method to list 1 to 10
i=1;while [ $i -le 10 ];do echo $i;i=$[$i+1];done

i=1;while (( $i < 11 )); do echo $i; let i++; done
for (( i=0 ; i < 11 ; i++ )) ; do  echo $i ; done


3. Examples
Use loop to touch file a.txt, b.txt until z.txt?
for i in {a..z};do touch $i.txt;done

How to use innobackupex to online backup innodb database for Percona 5.5 or MariaDB 10.0

Jephe Wu - http://linuxtechres.blogspot.com

Environment: Percona server 5.5 running on both server db01 and db02,  full backup folder on db02 is /data/fullbackup, incremental backup folder on db02 is /data/incremental, mysql datadir on both server is /srv/mysql/data

This post is applicable to any MySQL, MariaDB database with innodb engine, not only Percona server .

Objective: online backup innodb database from db1 to db2 by Percona innobackupex

Concept:
Daily backup is incremental backup. On every Wednesday, we will apply incremental backup into mysql database on db02.



Steps

1. initial full backup on db01 and apply logs
We will do a full backup on db01 then apply transaction logs to make it consistent:

# innobackupex --user=root --password=password /data/fullbackup --slave-info --no-timestamp  --parallel=4
# innobackupex --apply-log --redo-only --use-memory=4G /data/fullbackup

2. Copy whole directory /data/fullbackup to db2 

# cd /data/fullbackup
# ssh db02 'rm -fr /data/fullbacukp/*'
# tar cpf - . | ssh db02 'cd /data/fullbackup; tar xvpf - '

3.  copy back to datadir on db02
ssh into db2 as root, assume mysql data directory on db02 is /srv/mysql/data, ssh into db02 to run commands below which performs the restoration of a backup to database server's datadir

# rm -fr /srv/mysql/data/*
# innobackupex --copy-back /data/fullbackup
# chown mysql:mysql -R /srv/mysql/data'

Now you can start up mysql from db02.

4. incremental backup done by cronjob script daily

[root@db01 cron.d]# more /root/bin/innobackupex.sh
#!/bin/sh
# do incremental backup with stream on db01 daily as follows
LSN=`ssh db02 'grep to_lsn /data/fullbackup/xtrabackup_checkpoints' | cut -d ' ' -f 3`
LOG=/var/log/innobackupex.log

(innobackupex --user=root --password=password  --use-memory=4G --incremental --incremental-lsn=$LSN --stream=xbstream ./ | ssh root@db02 "cat - | xbstream -x -v -C  /data/incremental" ) > $LOG 2>&1


# apply incremental log with --redo-only on db02
ssh db02 'innobackupex --apply-log --redo-only --use-memory=4G /data/fullbackup --incremental-dir=/data/incremental' >> $LOG 2>&1

# clean up
ssh db02 'rm -fr /data/incremental/*'

# actual restore on db02, only run on every Wednesday
DAY=`date +%w`
if [ $DAY -eq 3 ];then

ssh db02 '/etc/init.d/mysql stop; sync; sleep 10'
ssh db02 'rm -fr /srv/mysql/data/*;innobackupex --copy-back /data/fullbackup;chown mysql:mysql -R /srv/mysql/data'
# Note: you can use --move-back instead of above --copy-back if you are confident

ssh db02 '/etc/init.d/mysql restart'
mutt -s "db01 incremental innobackpuex - `tail -1 $LOG`" jephewu@gmail.com < $LOG
fi

5. setup cronjob on db01 to run from Mon-Fri on 8am daily

[root@db01 cron.d]# more /etc/cron.d/innobackupex
0 8 * * 1-5 root /root/bin/innobackupex.sh >/dev/null 2>&1

6. References
http://www.percona.com/doc/percona-xtrabackup/2.1/innobackupex/innobackupex_script.html

How to upgrade Mysql 5.1 to MariaDB 10.0.X under CentOS 6.5 without using mysqldump

Jephe Wu - http://linuxtechres.blogspot.com

Environment:  CentOs 6.5 64bit with default mysql 5.1 server and innodb engine for database.
Objective:  Upgrade database to MariaDB 10 without dumping table contents.

Concept:  Tried to upgrade directly from Mysql 5.1 to MariaDB 10 but it doesn't work, we got errors in mysql log file. However, upgrade to Percona server 5.5 first then upgrade to MariaDB 10 works.


Steps

upgrade to Percona 5.5 server first
run the following commands to upgrade mysql 5.1 to Percona 5.5,

refer to http://www.percona.com/doc/percona-server/5.5/installation/yum_repo.html 

yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm

/etc/init.d/mysql stop
rpm -e mysql mysql-server
yum install Percona-Server-client-55 Percona-Server-server-55

/etc/init.d/mysql start

mysql_upgrade -uroot -ppassword

upgrade Percona 5.5 to MariaDB 10.0.X

Refer to https://downloads.mariadb.org/mariadb/repositories/#mirror=aarnet_pty_ltd&distro=CentOS&distro_release=centos6-amd64&version=10.0 

vi /etc/yum.repos.d/MariaDb.repo and put into following

# MariaDB 10.0 CentOS repository list - created 2014-12-10 08:39 UTC # http://mariadb.org/mariadb/repositories/ [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/10.0/centos6-amd64 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1

 /etc/init.d/mysql stop
sudo yum install MariaDB-server MariaDB-client
/etc/init.d/mysql start

mysql_upgrade -uroot -ppassword

Truncate vcenter SQL Server database before the size reach 10G limit

Jephe Wu - http://linuxtechres.blogspot.com

Environment: SQL server 2008 Express edition on vcenter server, no SSMS installed by default
Objective:  learn how to truncate sql server database size in SQL Server Management Studio so that it won't exceed 10G limit, otherwise, database engine service will stop after 10G.


Steps:

1.  check the current database size
run the following query in new query window in SSMSE

exec sp_spaceused

2. stop Vmware VirtualCenter Service

Just click on Vmware VirtualCenter Server, then stop it, then it will also stop some other services due to dependency.

3. Stop SQL Server (VIM_SQLEXP) service from sql server configuration manager 

4. Backup data and log files for database VIM_SQLEXP somewhere else
Must make sure database service is stopped first then take backup.

5. Install SSMS 
Install locally on the same server or copy 2 database files over to another server which has SSMS installed.

refer to http://blogs.msdn.com/b/bethmassi/archive/2011/02/18/step-by-step-installing-sql-server-management-studio-2008-express-after-visual-studio-2010.aspx for installation

During installation of SSMS, choose 'Perform a new installation of SQL Server 2008' as follows:

Use SSMSE to connect to local Vcenter SQL express database

6. type in 'localhost' to connect

if error is showing it's unable to connect to database, go to SQL server configuration manager, SQL server network configuration, Protocols for VIM_SQLEXP,

enable IP1 and IP3 (changed 'Enabled' from no to yes), also, give tcp Port from null to 1433

If you copied data .mdf and log .ldf files to another server which has SSMS installed, then attach database then choose those 2 copied files

When you attach database to another full edition of SQL server,  SQL server must be the same version as original vcenter database 

7.  Check tables disk usage
Right click on database VIM_VCDB, reports, standard reports, disk usage by table to check table disk usage, normally, VPX_EVENT table should be very big.

8. Change event.maxage and task.maxAge to 7 days if it's not set yet. You may also set to 30 days as per suggested by Vmware KB

go to dbo.VPX.PARAMETER table to edit event.maxAge to 7 days, and also for task.maxAge.

9. To truncate the data in the VPX_EVENT table:

• 1. Connect to SQL Database and log in with the appropriate credentials.
  2. Expand databases ,VIM_VCDB ,Tables.
  3. Right-click the dbo.VPX_PARAMETER table and click Open.
  4. Modify event.maxAge to 7, and modify the event.maxAgeEnabled value to true.
  5. Modify task.maxAge to 7, and modify the task.maxAgeEnabled value to true.
  6. Run the built-in stored procedure:
• Navigate to VIM_VCDB - Programmability - Stored Procedures.
• Right-click dbo.cleanup_events_tasks_proc and click Execute Stored Procedure.
• This purges the data from the vpx_event, vpx_event_arg, and vpx_task tables based on the date specified for maxAge.

Note: this step might take around one hour time, be patient, do not touch anything once click

When this has successfully completed, close SQL Management Studio and start the VMware Virtual Center Server service.

10. truncate all performance tables - optional

right click on VIM_VCDB database, new query, then paste the following code to truncate all performance tables.
refer to http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1007453

Declare @current_table varchar(100) declare @sqlstatement nvarchar(4000) --move declare cursor into sql to be executed set @sqlstatement = 'Declare table_cursor CURSOR FOR SELECT name FROM  sys.tables where name like ''VPX_HI%'' or name like ''VPX_SAMPLE%''' exec sp_executesql @sqlstatement OPEN table_cursor FETCH NEXT FROM table_cursor INTO @current_table WHILE @@FETCH_STATUS = 0 BEGIN set @sqlstatement = 'truncate table ' + @current_table exec sp_executesql @sqlstatement FETCH NEXT FROM table_cursor INTO @current_table END CLOSE table_cursor DEALLOCATE table_cursor

11. Shrink database size
right click database, tasks, Shrink - database

12. References

Purging old data from the database used by VMware vCenter Server 4.x and 5.x(1025914) - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025914