use wget to keep session cookie and use form-based file upload

#!/bin/sh
wget --no-check-certificate -O file1 --save-cookies cookies.txt --keep-session-cookies https://URL/
COOKIE=`cat cookies.txt tail -1 awk'{print$NF}'`
USER=admin
PASS=password
CORP=corpname
HASH=`echo -n $CORP$USER$PASS sha1sum awk '{print $1}'`
wget --debug --no-check-certificate -O file2 --load-cookies cookies.txt --post-data "LoginId=$USER&Password=$PASS&Corporation=$CORP&sessionid=${COOKIE}" https://URL/

wget --debug --no-check-certificate -O file3 --load-cookies cookies.txt --header="Content-Type: multipart/form-data; boundary=FILEUPLOAD" --post-file postfile https://url/

sleep 10

wget --debug --no-check-certificate -O file4 --load-cookies cookies.txt https://url2/
wget --debug --no-check-certificate -O file5 --load-cookies cookies.txt --post-data "module=a&screen=b&action=c" https://url3/

note:
1. use httpliveheader firefox addon to get all the http header and request/response
2. sleep 10 to wait for the processing finish
3. --save-cookies cookies.txt --keep-session-cookies -> make multiple wget command to be like in the same browser session.
4. for upload text file, that's to say, use ' form-based file upload in html', the RFC is at
http://www.faqs.org/rfc/rfc1867.html
5. for postfile content:

-bash-3.1# more postfile
--FILEUPLOAD
Content-Disposition: form-data; name="module"
a
--FILEUPLOAD
Content-Disposition: form-data; name="screen"
b
--FILEUPLOAD
Content-Disposition: form-data; name="action"
c
--FILEUPLOAD
Content-Disposition: form-data; name="description"
test
--FILEUPLOAD
Content-Disposition: form-data; name="fileName"; filename="data.txt"Content-Type: text/plain
20070201 0758 a1000 I
20070201 2031 a1000 O
--FILEUPLOAD--
note: with --FILEUPLOAD, the separator is just 'FILEUPLOAD', and the last line is --FILEUPLOAD--

system rescue basic

1. add init=/bin/sh to grub or lilo to quickly bootup system , but ctrl-c or ctrl-z might not work
2. mount -t proc none /proc to get command like ps to work, or anything else use /proc file system
3. mount -o remount,rw /
4 .fuser -am /mnt/muic
fuser -km /mnt/music
ps axwwl | grep processidnumber
lsof /mnt/music

5. dd if=/dev/hda of=filename bs=512 count=1
restore partition table:
dd if=filename of=/dev/hda bs=1 count=64 skip=446 seek=446

to cloning bad disk
dd if=/dev/hda of=/dev/hdb conv=noerror,sync
ddrescue --max-retries=-1 /dev/hda /dev/hdb

6. recover lost partition
gpart /dev/hdb (guess partition table)
gpart -b filename -W /dev/hda /dev/hda
note: scan /dev/hda ,backup the existing MBR, then write to /dev/hda

7. in case the bad block on hard disk is in journal file system, so remove journal might help
tune2fs -f -O ^has_journal /dev/hda1
tune2fs: attempt to read block from file system resulted in short read while reading journal inode

# debugfs /dev/hda1
debugfs: features
defugfs : open /dev/hda1
debugfs: quit

OpenBSD basic

• ip forwarding is off by default, enable it by putting the following into /etc/rc.local

sysctl net.inet.ip.forward=1
note: see http://www.openbsd.org/faq/pf.nat.html

or put into /etc/sysctl.conf
net.inet.ip.forwarding=1

• set up ip alias
  

put into /etc/hostname.fxp0 or other file /etc/hostname.bge0

# cat /etc/hostname.fxp0
inet 192.168.0.2 255.255.255.0
inet alias 192.168.0.3 255.255.255.0

manually add:
ifconfig fxp0 inet alias 192.168.0.3 netmask 255.255.255.0

delete:
ifconfig fxp0 inet -alias 192.168.0.3 netmask 255.255.255.0

to view these aliases: use 'ipconfig -A'

• pf disable and enable

pfctl -e (enable)
pfctl -d (disable)

use pfctl -e to confirm it's running

• pf show rules up

pfctl -s rules
pfctl -s nat
pfctl -s all

• static routing , put it into /etc/rc.local
  

/sbin/route add -net 192.168.0.0/16 192.168.4.251
/sbin/route delete -net 192.168.0.0/16

put gateway ip address into /etc/mygate:

10.0.0.1

• to mount USB under openbsd

dmesg
mkdir /mnt/usb
mount -t msdos /dev/sd0i /mnt/usb

• mount hard disk under openbsd

in case you mess up, you can boot from openbsd cdrom, then 'boot -s' to single user mode, then choose 'shell', then mount hard disk:
cd /mnt
mkdir hd
mount /dev/wd0a hd

• openbsd 4.5 tips and tricks - www.openbsd101.com

• how to use serial port as bootup and login terminal

1. http://www.openbsd.org/faq/faq7.html
   

note: basically, you just put 'set tty com0' in /etc/boot.conf(create it), then edit the line that reads:

     tty00   "/usr/libexec/getty std.9600"   unknown off

to read something like:

     tty00   "/usr/libexec/getty std.9600"   vt220   on secure

note: for using baud 115200, change like this:
put additional line in /etc/boot.conf: stty com0 115200
then
change /etc/ttys -> tty00 line ,change std.9600 to std.115200

freebsd 6.3 knowledge

• static routes  (add into /etc/rc.conf)

static_routes="lan mumoffice foo"
route_lan="-net 192.168.1.0/24 192.168.1.254"

route_mumoffice="-net 10.0.0.0/8 10.30.110.5"
route_foo="-host 1.2.3.4 5.6.7.8"


note:
1.cannot write as follows:
static_routes="lan"
route_lan="-net 192.168.1.0/24 192.168.1.254"
static_routes="mumoffice"
route_mumoffice="-net 10.0.0.0/8 10.30.110.5"

otherwise, only mumoffice static routes will be added.

2. manually add static routing
route add -net 10.0.0.0/8 10.30.110.5
route add -host 1.2.3.4 4.5.6.7

note: for RHEL 5, static routing:

To set static routes, use the /etc/sysconfig/network-scripts/route-<interface> files which read at network service initialization time. For example to add static route for eth0, create a file /etc/sysconfig/network-scripts/route-eth0 and add the routes as explained below.

There are two possible formats for this file.  The first is with ip command arguments and the second is with  network/netmask directives.

Format 1:

For ip commands, the ifup-route script supplies ip route add and the contents of the file are all parameters necessary to set up the route. For example, to set up a default route, the file would contain the following:

default via X.X.X.X dev bond0
10.10.10.0/24 via X.X.X.X dev bond0

• mounting usb thumb drive
  

mkdir /mnt/usb
mount -t msdos /dev/da0s1 /mnt/usb

• ifconfig in /etc/rc.conf

ifconfig_fxp0="inet 1.2.3.4 netmask 255.255.255.0"
for alias on fxp0, put:

ifconfig_em0="inet 10.20.110.2 netmask 255.255.255.192 "
ifconfig_em0_alias0="inet 10.20.1.226 netmask 255.255.255.0"
ifconfig_em0_alias1="inet 10.20.1.227 netmask 255.255.255.0"
ifconfig_em0_alias2="inet 10.20.1.228 netmask 255.255.255.0"

manual way:
add alias: ifconfig em0 inet alias 10.20.1.226 netmask 255.255.255.0
remove alias: ifconfig em0 inet -alias 10.20.1.226 netmask 255.255.255.0

note: enabling ip alias is normally required for doing port forwarding (rdr)

• gateway enable

in order to automatic enable a freebsd box as gateway machine which normally has 2 NICs and it will do ip forwarding, put the following into /etc/rc.conf
gateway_enable="yes"
note:
for manually enabling gateway: run command : sysctl -w net.inet.ip.forwarding=1

• DNS and ssh slowness response issue

vi /etc/nssswitch.conf to take out 'dns' from hosts line

• add user 'jephe' to allow to use 'su -'

vi /etc/group to add user 'jephe' to 'wheel' group

• enable raid1 mirror for hp lp1000r server  (refer to http://www.freebsd.org/doc/handbook/geom-mirror.html )

1. we assume /dev/da0 is the first hard disk which has OS installed , we need to add /dev/da1 which is the second hard disk to the raid1 mirror system
2. run commands below
# sysctl kern.geom.debugflags=16
#gmirror label -v -b round-robin gm0 /dev/da0
#echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0, :%s#da0#mirror\/gm0#g)

bsd# more /etc/fstab
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
# reboot
after reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status
for adding the second hard disk /dev/da1 to raid array, run
# gmirror forget gm0 (optional, depends)
# gmirror insert gm0 /dev/da1

Note: testing raid1
after rebuilding finished (gmirror status to check), you can actually shutdown da0, just use da1 to boot up.
I have tested okay.

• compile kernel to enable ipsec and carp
  

cd /usr/src/sys/i386/conf
cp GENERIC /root/MYKERNEL
ln -sf /root/MYKERNEL
vi MYKERNEL to add the following:

# enable IPSEC
options IPSEC
options IPSEC_ESP
options IPSEC_DEBUG

# or enable CARP
device pf
device pflog
device pfsync
device carp

cd /usr/src
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
reboot

• put in /etc/rc.conf for enabling pf

pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflog_logfile="/var/log/pflog"
pflog_flags=""

• mounting cdrom and usb
  

mount /cdrom ( see /etc/fstab)
insert usb thumb drive
mkdir /mnt/usb
mount -t msdos /dev/da0s1 /mnt/usb

• error message '/dev/pf no such file or device'

solution: add pf_load="YES" to /boot/loader.conf, then reboot or run 'kldload pf' with reboot

• add alias

ifconfig bge0 inet alias 10.0.202.80 netmask 255.255.255.0

to remove alias
ifconfig bge0 inet -alias 10.0.202.80 netmask 255.255.255.0
to review alias
# ifconfig (without parameters)

how to do basic configuration and setup for freebsd 6.3 pf firewall

Jephe Wu - http://linuxtechres.blogspot.com

Environment: FreeBSD PF firewall.
Objective: understanding Packet Filter firewall


Steps:

1. OS installation part

use just first CD of FreeBSD 6.3.
1. use 'F - DD' mode to configure partition since we are using the server dedicated for FreeBSD only, no other OS.
2. use 'Automatic' mode to make partitions
3. select 'full binary, doc and kernel source only' category to install

2. user and dns configuration

During OS installation, you need to create a normal user (e.g. jephe), because you cannot login as root after OS finishs installation. Also, in order to 'su - ' as root, you have to add user 'jephe' to 'wheel' group

If your environment doesn't have DNS server, you have to wait for a long time before getting a login prompt when you try to ssh into the server. The solution is to vi /etc/nsswitch.conf to take out 'dns' from hosts line.

3. setup up PF firewall

After OS installation, it's no firewall enabled by default, you can configure the following to enable it:

a. put to /etc/rc.conf:
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflogd_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""


b. edit the default /etc/pf.conf
ext_if="fxp0" # replace with actual external interface name i.e., dc0
int_if="fxp1" # replace with actual internal interface name i.e., dc1
scrub in all  # normalize all packets
block in log all  # default deny policy


# enable ssh from the specific internal IP to internal NIC of firewall
pass in on $int_if proto tcp from 192.168.0.2 to $int_if port 22 keep state

# enable all outgoing traffic on internet NIC for tcp and udp
pass out on $int_if proto { tcp, udp } all keep state

#enable incoming ssh from trusted external IP
pass in on $ext_if from x.y.z.a keep state
pass out on $ext_if from $ext_if to x.y.z.a keep state

how to do port forwarding(rdr) in freebsd 6.1

purpose: setup a port forwarding firewall on freebsd 6.1

environment: fxp0 is for external NIC interface(192.168.1.1/24), fxp1 is internal one(192.168.0.1/24), servera is on the lan
steps:

• configuring ip alias on fxp0

# ifconfig fxp0 inet alias 192.168.1.2 netmask 255.255.255.0
# ifconfig fxp0 inet alias 192.168.1.3 netmask 255.255.255.0

• add configuration into /etc/rc.conf

ifconfig_fxp0="inet 192.168.1.1 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 192.168.1.2 netmask 255.255.255.0"
ifconfig_fxp0_alias1="inet 192.168.1.3 netmask 255.255.255.0"

• in /etc/pf.conf, after 'scrub in', before 'block all'

rdr on $fxp0 proto tcp from any to 192.168.1.2 port 3389 -> $servera
or
rdr on $fxp0 proto tcp from any to 192.168.1.2 port 3389 -> $servera port 3390

# servera
#pass external user to servera at port 3389 because after RDR, the destination become servera instead of 192.168.1.2, no need to specify RDR tcp ip pair rules because that will happen before traffic going through PF rules.
pass in quick on $fxp0 proto tcp from any to $servera port 3389 flags S/SA keep state (after rdr)
note: here uses $servera, not 192.168.1.2 which is the external ip alias, because you don't have to specify rules for RDR, here is after RDR rules)
pass out quick on $fxp1 proto tcp from any to $servera port 3389 flags S/SA keep state (direct pass after rdr tcpip payload to servera, servera needs to have routing going back to the source)
==========================
note: this method, the servera will see the real source ip, if you want to NAT also after doing RDR, NAT the source ip to the internal IP of 192.168.0.1, you can add the following to the just after above RDR or just before RDR:

nat on $fxp1 proto tcp from any to $servera - > 192.168.0.1


This way, the servera will see the traffic is coming from 192.168.0.1

but the pf rules will change also, after NAT, the sourceip become 192.168.0.1,so it becomes:
pass in quick on $fxp0 proto tcp from any to $servera port 3389 flags S/SA keep state (after rdr)
pass out quick on $fxp1 proto tcp from 192.168.0.1 to $servera port 3389 flags S/SA keep state (after nat, servera only needs to reply back to internal ip of firewall which is 192.168.0.1.

Important rules:
RDR happens before the actual PF filtering rules, once traffic coming in, it goes to RDR first before going through filtering rules.

NAT happens before the actual PF filtering rules also, before traffic going out the interface, it applies to NAT first, then applying the actual ip pf filtering rules.

RDR->incoming pf rules->|Server Interface 1_>NAT-> Server Interface 2|->outgoing pf rules

configuring sudosh and sendmail masquerading, as well as working with syslog-ng

Jephe Wu - http://linuxtechres.blogspot.com

Objective: Basically, we’d like to only allow IS staffs to remote ssh into production servers and use sudo sudosh or sudo –u db2inst1 sudosh to switch to root or db2inst1 so that every command including vi keystroke will be logged, at the same time, email will be triggered whenever anyone use sudo to switch to root or db2inst1 user.

Use sudosh2 for accountability, swatch for sending alert out once someone logs in db2inst1 or root. Use openssh allowusers syntax to only allow system administrators to remote ssh.

Environment: RHEL 5 or CentOS 5


Steps:
1. sudosh2 software installation and configuration

Search google for sudosh2 then download the software sudosh2-1.0.4. Please refer to website
http://www.shortcutsolutions.net/sudosh2-software/55-introduction-sudosh2-user-shell-auditing.html
for installation steps which is mentioned on http://sourceforge.net/projects/sudosh2/

# tar xvfz sudosh2-1.0.4.tar.gz
# cd sudosh2-1.0.4
# ./configure
# make
# make install
# sudosh

Important: run sudosh command once to create necessary directories first.

configuration file for sudo is at /etc/sudosh.conf

2. visudo
ssh into server and become root as normal
# useradd jephe -c "Jephe Wu"
# passwd jephe
# visudo

add the following to the /etc/sudoers

Cmnd_Alias SUDOSH=/usr/local/bin/sudosh
jephe ALL=(root)SUDOSH, (db2inst1)SUDOSH


Note: Cmnd_Alias line must be on the top of jephe ALL line, otherwise, sudo sudosh doesn't work


Please refer to http://www.shortcutsolutions.net/sudosh2-shell-auditing-software/89-configuration-of-sudosh2.html for sudosh2 configuration and installation

3. ssh configuration
vi /etc/ssh/sshd_config to add the followings:

PermitRootLogin no
Protocol 2
banner /etc/motd.ssh
allowusers jephe anotheruser

then put the security warning message into /etc/motd.ssh

then run 'sshd -t' to test ssh daemon configuration syntax
then run 'service sshd restart' to restart sshd
then run 'ssh localhost' to test it.

4. su configuration
vi /etc/pam.d/su to uncomment the following line to require only members in wheel group are able to su
auth required /lib/security/$ISA/pam_wheel.so use_uid

5. Sendmail

For enabling sending email from database server to centralized mail server mailrelay, we need to put the following to the default sendmail.mc as follows:

e.g. servername is server.domain.com, we need to masquerade all sender address including envelope address as from root@server.domain.com to root@newdomain.com, then deliver it to the centralized mail server mailrelay.

Dnl EXPOSED_USER(`root’)dnl
MASQUERADE_AS(newdomain.com’)dnl
FEATURE(masquerade_envelope)dnl
FEATURE(masquerade_entire_domain)dnl
MASQUERADE_DOMAIN(localhost)dnl
MASQUERADE_DOMAIN(loclahost.localdomain)dnl
MASQUERADE_DOMAIN(domain.com)dnl

Then put the following into /etc/mail/mailertable, after that, run
makemap hash /etc/mail/mailertable < /etc/mail/mailertable

newdomain.com relay:[mailrelay]

Then put the following into /etc/hosts
10.0.0.1 mailrelay

enable root alias as follows:
add the following line to /etc/alias then run newalias
root: jephe_wu@newdomain.com

   1. Note:
      a. no need to give SMART_HOST since we only need to relay newdomain.com to the centralized mailrelay server.
      b. No need to disable only listening on localhost since we only need to send out email, not for receiving.

so, now, any email sending to local user root will be delivered to jephe_wu@newdomain.com on the mailrelay server.

6. Swatch
register your redhat subscription using rhn_register (interactive, require to type in username and password for RHN)
or there's no need to register if you are using CentOS 5

install rpmforge rpm from DAG website at http://dag.wieers.com/rpm/FAQ.php#B ,
wget http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
rpm -Uvh http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

put the proxy setting in /etc/yum.conf like
proxy=http://10.0.0.1:3128

yum install swatch
then the rpm packages will be downloaded at /var/lib/yum and install it

Put the following to /etc/swatch.conf

watchfor /sudo:.*/
mail root,subject=--- DB1 sudo alert! ---

watchfor /su:.*/
mail root,subject=--- DB1 su alert!---

watchfor /login: ROOT LOGIN.*/
mail root,subject=---DB1 console login alert!---

note: must use small letter watchfor , not Watchfor, small letter mail root, not Mail root.
If swatch exits from ps command after a while, use 
/usr/bin/swatch -c /etc/swatch.conf -t /var/log/secure
to run and try to ssh then check if there's any error.



put the following lines to /etc/rc.local
/usr/bin/swatch –c /etc/swatch.conf –t /var/log/secure --daemon

change /etc/logrotate.conf for week 52 to rate a year and compress

configure /etc/logrotate.d/syslog

use the following
/var/log/messages /var/log/mailog /var/log/spooler /var/log/secure /var/log/boot.log /var/log/cron {
sharedscripts
potstrotate
/bin/kill –HUP `cat /var/log/syslogd.pid 2> /dev/null` 2> /dev/null true
##added by Jephe####
kill -9 `ps -ef | grep -e swatch -e '/usr/bin/tail -n 0' | grep -v grep | awk '{print $2}'`
sleep 5
/usr/bin/swatch -c /etc/swatch.conf -t /var/log/secure --daemon
###end#####
endscript
}

finally, run command in /etc/rc.local
/usr/bin/swatch –c /etc/swatch.conf –t /var/log/secure --daemon

===========
syslog-ng and swatch standard input
Objective: use syslog-ng default log output file to pipeline to swatch

    * syslog-ng configuration
destination swatch {
program("/usr/bin/swatch -c /etc/swatch.conf --read-pipe=\"cat /dev/fd/0\"");
};

#send all logs to swatch
log { source(s_sys); destination(swatch); };


    * /etc/swatch.conf
watchfor /sudo:.*/
mail root,subject=--- SERVER1 sudo alert! ---

watchfor /su:.*/
mail root,subject=--- SERVER1 su alert! ---

watchfor /login: ROOT LOGIN.*/
mail root,subject=--- SERVER1 console login alert! ---


note:
1. no need to run swatch as deamon
2. no need to modify /etc/logrotate.d/syslog
3. /dev/fd/0 is different from /dev/fd0 which is floppy disk



References:
Central Loghost Mini-HOWTO at http://www.campin.net/newlogcheck.html

note:
1. SEC(http://www.estpak.ee/~risto/sec/) is better than swatch