How to use looping in Linux bash script

Jephe Wu - http://linuxtechres.blogspot.com


Objective: list all kinds of Linux bash looping script which is useful for bash programming.
let computer do work for you.

Concept: 
Very often, we just need to run some script by loop to finish work


Examples


1. infinite loop

while true;do ...done
while [ 1 ]; do...done
while : ; do ....done

watch ls -l /tmp/file

2.  List a range of number 

list each line of /etc/hosts file
while read line;do echo $line;done < /etc/hosts

list number 1 to 10
for i in {1..10};do echo $i;done
for i in $(seq 1 10);do echo $i;done
for i in `seq 1 10`;do echo $i;done

list number 1 to 10 with same width
for i in {01..10};do echo $i;done
for i in $(seq -w 1 10);do echo $i;done
for i in `seq -w 1 10`;do echo $i;done

list number 1 to 10 with only odd number
for i in {1..10..2};do echo $i;done

list number 1 to 10 with only even number
for i in {2..10..2};do echo $i;done

Use just echo to list 1 to 10
echo {1..10}
echo {01..10}

Use just echo to list a-z and A-Z
echo {a..z}
echo {A..Z}

tranditional method to list 1 to 10
i=1;while [ $i -le 10 ];do echo $i;i=$[$i+1];done

i=1;while (( $i < 11 )); do echo $i; let i++; done
for (( i=0 ; i < 11 ; i++ )) ; do  echo $i ; done


3. Examples
Use loop to touch file a.txt, b.txt until z.txt?
for i in {a..z};do touch $i.txt;done

Basic Oracle DBA Task List

Jephe Wu - http://linuxtechres.blogspot.com

Objective: some basic Oracle DBA daily task scripts and performance checking


Performance Issue: Most of (more 90%) performance issues are caused by Application. Few (less than 10%) of issues are caused by resource limitation, OS/RDBMS configuration and other reasons.

• CPU bottleneck 

According to page http://www.dba-oracle.com/t_high_cpu.htm

It is not a cause for concern when the user + system CPU values approach 100 percent. This just means that the CPUs are working to their full potential. The only metric that identifies a CPU bottleneck is when the run queue (r value) exceeds the number of CPUs on the server.

• IO bottleneck

The Wa value of vmstat command indicates how many percent of the CPU time is being used waiting for database I/O. But it does not always indicate I/O bottleneck.

• Memory bottleneck 

According to page http://www.dba-oracle.com/t_tuning_cpu_usage_vmstat.htm

Oracle parameters such as sort_area_size can greatly increase the amount of RAM allocated to each connected user's PGA.

a UNIX process may page-in when the UNIX program is starting or is accessing parts of its code that it had not used before.

With UNIX virtual memory we are always anticipating running out of RAM memory, and a page-out is a method for being ready for a subsequent page-in

In sum, page-out (po) operations are a normal part of virtual memory operation, but page-in (pi) operations indicate that the server has excessive RAM demands. Because Linux will put least used part of memory back to swap

• Cronjob scripts 

1. use vmstat 10 to get data and the following is the script:

-bash-3.2# cat /home/oracle/bin/vmstat.sh 
#!/bin/sh
DATE=`date +%Y%m%d`
function vm {
  while read line
  do
    printf "$line"
    date '+ %m-%d-%Y %H:%M:%S'
  done
}
find /var/adm/logs/vmstat.log* -type f -mtime +30 -exec rm -f {} \;
vmstat 10 | vm >> /var/log/vmstat/vmstat.log.$DATE

then put cronjob as follows to run vmstat 
0 0 * * * sleep 1;kill `ps -ef | grep -v  grep  |grep vmstat.sh | awk '{print $2}'`;sleep 3;nohup /home/oracle/bin/vmstat.sh &

 2.  most used cpu and wait time


#!/bin/sh
export ORACLE_SID=PROD
export ORACLE_BASE=/db01/oracle
export ORACLE_HOME=/db01/oracle/product/11.2.0
PATH=/usr/local/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/home/oracle/bin:/usr/bin/X11:/sbin:/db01/oracle/product/11.2.0/bin
LD_LIBRARY_PATH=/db01/oracle/product/11.2.0/lib
export PATH LD_LIBRARY_PATH

DATE=`date +%Y%m%d`
LOGFILE=/var/adm/logs/oracle_perf.log.$DATE

find /var/adm/logs/oracle_perf.log.* -type f -mtime +30 -exec rm -f {} \;
sqlplus -S / as sysdba @/home/oracle/bin/oracle_perf.sql >> $LOGFILE


oracle@prod1 (PROD)> cat /home/oracle/bin/oracle_perf.sql 
set line 32000
select to_char(sysdate, 'YYYYMMDD hh24:mi:ss') tm from dual;

# logon within 4 hours, last half hour, cpu and db time .
SELECT s.sid, s.serial#, p.spid as "OS PID", s.username, s.module, st.value/100 as "DB Time (sec)"
, stcpu.value/100 as "CPU Time (sec)", round(stcpu.value / st.value * 100,2) as "% CPU"
FROM v$sesstat st, v$statname sn, v$session s, v$sesstat stcpu, v$statname sncpu, v$process p
WHERE sn.name = 'DB time' -- CPU
AND st.statistic# = sn.statistic#
AND st.sid = s.sid
AND  sncpu.name = 'CPU used by this session' -- CPU
AND stcpu.statistic# = sncpu.statistic#
AND stcpu.sid = st.sid
AND s.paddr = p.addr
AND s.last_call_et < 1800 -- active within last 1/2 hour
AND s.logon_time > (SYSDATE - 240/1440) -- sessions logged on within 4 hours
AND st.value/100 > 30 order by st.value;

# time waited sort for logon within 4 hours, last half an hour, for db file sequential read time waited , sorting
SELECT s.sid, s.serial#, p.spid as "OS PID", s.username, s.module, se.time_waited
FROM v$session_event se, v$session s, v$process p
WHERE se.event = 'db file sequential read' 
AND s.last_call_et < 1800 -- active within last 1/2 hour
AND s.logon_time > (SYSDATE - 240/1440) -- sessions logged on within 4 hours
AND se.sid = s.sid
AND s.paddr = p.addr
ORDER BY se.time_waited;
exit;

3. long queries

# long query , run more than 60s, active, not background type.

select s.username,s.type,s.sid,s.serial#,s.last_call_et seconds_running,q.sql_text from v$session s 
join v$sql q
on s.sql_id = q.sql_id
 where status='ACTIVE'
 and type <> 'BACKGROUND'
 and last_call_et> 60
order by sid,serial#

• 
  

• 4. checking blocking session.

SQL> select process,sid, blocking_session from v$session where blocking_session is not null;

or according to http://stackoverflow.com/questions/622289/how-to-check-oracle-database-for-long-running-queries?answertab=votes#tab-top

select
  object_name, 
  object_type, 
  session_id, 
  type,                 -- Type or system/user lock
  lmode,        -- lock mode in which session holds lock
  request, 
  block, 
  ctime                 -- Time since current mode was granted
from
  v$locked_object, all_objects, v$lockwhere
  v$locked_object.object_id = all_objects.object_id AND
  v$lock.id1 = all_objects.object_id AND
  v$lock.sid = v$locked_object.session_idorder by
  session_id, ctime desc, object_name/

SQL> select SERIAL# from v$session where SID=123;

SERIAL#

———-

380

SQL> alter system kill session ’380,123′;

• 5. long transaction more than 6 seconds

sqlplus / as sysdba

sql> select *from v$session_longops;


select * from (
  select opname, target, sofar, totalwork,
         units, elapsed_seconds, message
  from v$session_longops order by start_time desc)
where rownum <=1;
http://www.gplivna.eu/papers/v$session_longops.htm

# long transaction and still running ones, based on http://stackoverflow.com/questions/622289/how-to-check-oracle-database-for-long-running-queries?answertab=votes#tab-top

COLUMN percent FORMAT 999.99 
SELECT sid, to_char(start_time,'hh24:mi:ss') stime, 
message,( sofar/totalwork)* 100 percent FROM v$session_longopsWHERE sofar/totalwork < 1
/

6.  oracle performance tuning - To find sessions generating lots of redo

methods:  refer to: How to Find Sessions Generating Lots of Redo or Archive logs [ID 167492.1]

1. Query V$SESS_IO

This view contains the column BLOCK_CHANGES which indicates

   how much blocks have been changed by the session. High values indicate a 

   session generating lots of redo.

SELECT s.sid, s.serial#, s.username, s.program,

 i.block_changes

  FROM v$session s, v$sess_io i

  WHERE s.sid = i.sid [and block_changes > 1000 ]

  ORDER BY 5 desc, 1,2,3,4;

# Run the query multiple times and examine the delta between each occurrence

   of BLOCK_CHANGES. Large deltas indicate high redo generation by the session.

2) Query V$TRANSACTION. This view contains information about the amount of

   undo blocks and undo records accessed by the transaction (as found in the 

   USED_UBLK and USED_UREC columns).

  The query you can use is:

SELECT s.sid, s.serial#, s.username, s.program, 

 t.used_ublk, t.used_urec

FROM v$session s, v$transaction t

 WHERE s.taddr = t.addr

ORDER BY 5 desc, 6 desc, 1, 2, 3, 4;

   Run the query multiple times and examine the delta between each occurrence

   of USED_UBLK and USED_UREC. Large deltas indicate high redo generation by 

   the session.

Basic Oracle names:

Linux bash scripting comman usage guide

Jephe Wu - http://linuxtechres.blogspot.com

Objective: When you need to write bash script, you need to follow certain steps, I've summarized some tips which I've been using for the past.



Parameter and print out script usage:
The first thing after #!/bin/sh line should be usage for your script, especially when your script needs the parameters:

if  [ $# -eq 1 ];then echo "$0 username";exit 1;fi
note: your script requires username as command parameter, so above line will force you to give one parameter after the command.

Path definition:
You should export PATH variables first like this:
export PATH=/usr/bin:/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin

Common tips in the script:

• Let history command show date and time

# export HISTTIMEFORMAT="%h/%d - %H:%M:%S "

# history | less

• check command return code 

command
if [ $? -ne 0 ];then
commands
exit
fi

• recursive variable

 A="DATABASE/script.sh";B=${A/DATABASE/orcl};echo $B

it will return value: orcl/script.sh

• while loop to read each line from /path/to/file and process it
  

while read line
do
echo $line
done < /path/to/file

or while IFS=\| read left right
do
echo $left $right
done < /path/to/filename

# another method
i=1
LINES=`wc -l /path/to/file | awk '{print $1}'`
while [ $i -le $LINES ]
do
iLINE=`sed -n $i'p' /path/to/file`
echo $iLINE
i=$[$i+1]
done

• specify return value
  

if [ $? -eq 0 ];then
return 0
exit
else
return 2
exit
fi

•  seq command usage in for loop
  

 for i in `seq -w 1 10`;do echo $i;done
01 01 03 04 05 06 07 08 09 10

• Always use "bash -n scriptname" to verify if your script got any syntax error.

•  check if the file size is 0
  

if [ ! -s /path/to/filename ];then commands ;fi
note: if the /path/to/file filesize is not zero, then run commands

• use mktemp and mkdtemp 

You can use mktemp and mkdtemp to create temporary file or directorie variables
# TMPFILE1=`mktemp`
# TMPDIR1=`mktempdir`

• how to let ls function as cd

alias ls=cd

or

ls () { builtin cd ; }

note: how to let ccd function as cd;pwd;foo

alias foo=`echo testing builtin`
ccd () { builtin cd "$1";pwd;foo; }

so ccd /tmp command will cd to /tmp, then print current working directory and print string 'testing builtin'


common tips:

1. batch rename file

rename 1.txt and 2.txt to 1.bat and 2.bat

ls *.txt | sed 's#\(.*\).txt#mv \1.txt \1.bat#'  | sh 
     
    2.  use bc to calculate from CLI

# echo "scale=2;34359730176/1024/1024/1024" | bc
31.99
# echo "ibase=10;obase=16;10" | bc
A
# echo "ibase=16;obase=A;A" | bc
10

  3.  force root user to run script
if [ "$(id -u)" != "0" ]; then
echo "Only root can run this script"
exit 1
fi


4. vmstat with timestamp


#!/bin/sh
# more vmstat.sh
function eg {
  while read line
  do
    printf "$line"
    date '+ %m-%d-%Y %H:%M:%S'
  done
}

find /var/adm/logs/vmstat.log* -type f -mtime +15 -exec rm -f {} \;
vmstat 10 | eg >> /var/adm/logs/vmstat.log.`date +%Y%m%d`

in crontab:

0 0 * * * sleep 1;kill `ps -ef | grep -v  grep  |grep vmstat.sh | awk '{print $2}'`;sleep 3;nohup /home/oracle/bin/vmstat.sh &


5. variables in for loop


servers=("root@server1" "root@server2" )
for server in ${servers[*]}
do
ssh -n $server .....
    balabala
done

or


Files=(
"primary.xml.gz"
)
for file in ${Files[*]}

do
   balah
done

6. create sparse file and detect it


[root@oratest ~]# dd if=/dev/zero of=filea bs=1M count=0 seek=5
0+0 records in
0+0 records out
0 bytes (0 B) copied, 3.8969e-05 seconds, 0.0 kB/s
You have new mail in /var/spool/mail/root
[root@oratest ~]# ls -l filea
-rw-r--r-- 1 root root 5242880 Jul 19 12:25 filea
[root@oratest ~]# du -sm filea
0 filea
[root@oratest ~]# ls -lh filea -s
0 -rw-r--r-- 1 root root 5.0M Jul 19 12:25 filea