How to recover deleted file under Linux

Jephe Wu - http://linuxtechres.blogspot.com

Objective: undelete httpd access log file while httpd process is still running
Environment:  CentOS 6.5 64bit

Cases Study:
Sometime, you deleted a big httpd log file while httpd is still running. As long as httpd process doesn't exit, you still can recover removed deleted httpd log file.

Apply to: Recover deleted file, data recovery under Linux,  restore deleted file,  undo deletion

Steps:


1. For testing purpose, list access_log file content first for confirmation

[root@www httpd]# pwd
/var/log/httpd
[root@www httpd]# more access_log
192.168.1.15 - - [24/Jul/2014:09:02:25 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.15 - - [25/Jul/2014:02:29:59 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:58 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:55:26 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:56:11 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"

2. purposely delete /var/log/httpd/access_log, then check lsof

[root@www httpd]# cp access_log /tmp
[root@www httpd]# rm -f access_log
[root@www httpd]# lsof | grep -i deleted
httpd     1146    root   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1177  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1178  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1179  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1180  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1181  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1182  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1183  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)
httpd     1184  apache   11w      REG              253,0     1282     261208 /var/log/httpd/access_log (deleted)

Note: above httpd process 1146 is running under root which is parent process of httpd.

3. find out the FD of parent process of httpd then retrieve deleted access log file

[root@www fd]# cd /proc/1146/fd
[root@www fd]# ll
total 0
lr-x------ 1 root root 64 Dec  4 06:40 0 -> /dev/null
l-wx------ 1 root root 64 Dec  4 06:40 1 -> /dev/null
l-wx------ 1 root root 64 Dec  4 06:40 10 -> /var/log/httpd/ssl_error_log
l-wx------ 1 root root 64 Dec  4 06:40 11 -> /var/log/httpd/access_log (deleted)
l-wx------ 1 root root 64 Dec  4 06:40 12 -> /var/log/httpd/ssl_access_log
l-wx------ 1 root root 64 Dec  4 06:40 13 -> /var/log/httpd/ssl_request_log
l-wx------ 1 root root 64 Dec  4 06:40 2 -> /var/log/httpd/error_log
lr-x------ 1 root root 64 Dec  4 06:40 3 -> /dev/urandom
lrwx------ 1 root root 64 Dec  4 06:40 4 -> socket:[9492]
lrwx------ 1 root root 64 Dec  4 06:40 5 -> socket:[9493]
lrwx------ 1 root root 64 Dec  4 06:40 6 -> socket:[9496]
lrwx------ 1 root root 64 Dec  4 06:40 7 -> socket:[9497]
lr-x------ 1 root root 64 Dec  4 06:40 8 -> pipe:[9509]
l-wx------ 1 root root 64 Dec  4 06:40 9 -> pipe:[9509]

[root@www fd]# cp 11 /var/log/httpd/access_log.deleted

[root@www fd]# cat /var/log/httpd/access_log.deleted
192.168.1.15 - - [24/Jul/2014:09:02:25 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.15 - - [25/Jul/2014:02:29:59 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:29 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:54:58 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:55:26 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
192.168.1.5 - - [04/Aug/2014:19:56:11 +1000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"


4. compare both files by diff and md5sum

[root@www fd]# diff /tmp/access_log /var/log/httpd/access_log.deleted

[root@www fd]# md5sum /tmp/access_log /var/log/httpd/access_log.deleted
9aa2e8c58ecfb689d3ada71bb6ec5815  /tmp/access_log
9aa2e8c58ecfb689d3ada71bb6ec5815  /var/log/httpd/access_log.deleted

Note: in my test environment, there's no active web access traffic so both log files are the same.

5. free up disk space
When httpd is running and access log file is deleted, the disk space won't be freed up, df -h and du reported disk free space do not match, you can use lsof -i deleted to find out if there's a big file was deleted to decide which process you need to restart to retrieve back the free space, in this case, I need to restart httpd process to free up space after restore back access log file.