Jephe Wu - http://linuxtechres.blogspot.com
Secure Firewall is very important for company network, it protects all valuable resources on the internal LAN. To avoid the single point of failure, it's always good practice to implement firewall fail-over. FreeBSD is a good choice to act as a firewall because it's popular, secure and support packet filter(PF) which is ported from OpenBSD, as well as CARP, pfsync. Software raid tool - gmirror in FreeBSD is extremely easy to configure.
Environment
1. 2 HP netserver lp1000r (running Freebsd 6.2)
2. each lp1000r server has 2 built-in network ports, one connects to Internet, the another connects to internal network
3. each lp1000r
3. web servers behind the firewalls using internal IP addresses (running CentOS 4)
Objective
1. when firewall1 is down, firewall2 will be taking over. This should be transparent to the end web user.
2. since each hp netserver lp1000r has 2 idential hard disks(18G), we need to build it as raid-1 mirroring. In case any one of hard disk dies, the firewall is still able to work.
3. 2 web servers are also DNS server, SMTP server and NTP client which means we need to configure the relevant PF rules on each firewall.
Naming conversion
hostname for firewall1: bsd1 (master firewall)
domain name: maxposs.com
external network: a.b.c.192/28 (replace a.b.c. with your own IPs)
default gateway: a.b.c.193
fxp0: a.b.c.204
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.5
carp1: 10.0.0.1
hostname for firewall2: bsd2 (backup firewall)
domain name: maxposs.com
external network: a.b.c.192/28
default gateway: a.b.c.193
fxp0: a.b.c.205
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.6
carp1: 10.0.0.1
hostname for internal web server 1: web1
ext0: 10.0.0.7
gateway: 10.0.0.1 (carp1 on firewalls)
services: web, smtp, dns, ntp client, ssh client
hostname for internal web server 2: web2
ext0: 10.0.0.8
gateway: 10.0.0.1 (carp1 on firewall)
services: web, smtp, dns, ntp client, ssh client
Freebsd 6.2 OS Installation
I'm using the default installation for FreeBSD, slide a for /, slide b for swap, slide d for /var and slide e for /usr.
Freebsd 6.2 raid-1 mirroring
Since there're 2 firewalls that needs to install, It's better to install one firewall, then the clone everything to the another one.
We installed FreeBSD OS on the first hard disk on bsd1, then use the following steps to add the second hard disk to become a raid-1 mirrored system.
# sysctl kern.geom.debugflags=16
# gmirror label -v -b round-robin gm0 /dev/da0
# echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0)
bsd1# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
/dev/mirror/gm0s3d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
# reboot
After reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status.
For adding the second hard disk to raid array, run
#gmirror forget gm0
# gmirror insert gm0 /dev/da1
Compiling kernel to enable carp and pfsync interfaces
During OS installation, we installed kernel developer packages so that we can do kernel
compilation for carp and pfsync later.
First of all, backup your current original kernel first.
# cp -pr /boot/kernel /boot/kernel.orig
note:After the whole system is stablized, you might want to backup the working kernel again
# cp -pr /boot/kernel /boot/kernel.good
# cd /usr/src/sys/i386/conf
# cp GENERIC MYKERNEL
# vi MYKERNEL (to add the following lines)
device pf
device pfsync
device pflog
device carp
note: you cannot just add 'device pfsync' without adding 'device pf' first.
If you want to be able to use ALTQ then the following as well:
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ
options ALTQ_NOPCC
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL# reboot
In case the new kernel doesn't boot up, please refer to http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-trouble.html#KERNELCONFIG-NOBOOT
Clone the FreeBSD OS to another firewall/ Replacing failed hard disk with a new one
shutdown bsd1 firewall, put the 2 hard disks to the left slot on each firewall, then
insert empty 2 hard disks to the right slot on each firewall, then run the following
commands to add the second hard disk to raid1 array
# gmirror forget gm0
# gmirror insert gm0 /dev/da1
note: you must run the first command, otherwise you will get error message 'not all
hard disks connected'
Configuring CARP and Pfsync on firewall
/etc/rc.conf on bsd1:
gateway="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"
/etc/rc.conf on bsd2
gateway="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 advskew 100 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 advskew 100 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 advskew 100 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"
note:
1. maxposs is the password, must be same for the same vhid (virtual host ID)
2. advskew 100 on bsd2 makes the carp advertisement packet less frequent than bsd1 so
it will be backup firewall whenever the election happens.
/etc/sysctl.conf on both bsd1 and bsd2
add the following lines
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
#if one interface fails then all will fail over
net.inet.carp.preempt=1
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
/etc/pf.conf on both bsd1 and bsd2
int_if="fxp1"
ext_if="fxp0"
lo_if="lo0"
int_network="10.0.0.0/24"
web1 = "10.0.0.7/32"
web2 = "10.0.0.8/32"
int_www = "{ $web1, $web2 }"
int_ns = "{ $web1, $web2 }"
ext_www = "a.b.c.201"
ext_ns = "a.b.c.200"
ext_smtp="a.b.c.206"
ssh_client1 = "x.y.z.1/32"
ssh_client2 = "x.y.z.2/32"
# Normalization
scrub in all
# NAT for all
#nat on $ext_if from $int_network to any -> $ext_if
# above is commented, for using physical external interface for outgoing smtp nat
nat on $ext_if from $int_network to any -> $ext_smtp
# using virtual interface for outgoing smtp nat
# web service rdr
rdr on fxp0 proto tcp from any to $ext_www port 80 -> $int_www round-robin
rdr on fxp0 proto tcp from any to $ext_www port 443 -> $int_www round-robin
# dns rdr
rdr on fxp0 proto udp from any to $ext_ns port 53 -> $int_ns port 53
# default rule
block in log all
# Allow all Loopback
pass quick on $lo_if all
# Allow pfsync Updates In/Out
pass quick on $int_if proto pfsync keep state
# Allow CARP Advertisements In/Out
pass quick on {$ext_if, $int_if} proto carp keep state
# dns incoming traffic
pass in log quick on fxp0 proto udp from any to $int_www port = 53 keep state
pass out quick on fxp1 proto udp from any to $int_www port = 53 keep state
# dns outgoing traffic
pass out log quick on fxp0 proto udp from fxp0 to any port = 53 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 53 keep state
# smtp outgoing traffic from physical interface fxp0 and virtual interface $ext_smtp
pass out quick on fxp0 proto tcp from fxp0 to any port = 25 keep state
# for allowing smtp traffic from virtual external interface to any
pass out quick on fxp0 proto tcp from $ext_smtp to any port = 25 keep state
pass in quick on fxp1 proto tcp from $int_www to any port = 25 keep state
# ssh outgoing traffic
pass out quick on fxp0 proto tcp from fxp0 to $ssh_clients port = 22 keep state
pass in quick on fxp1 proto tcp from $int_www to $ssh_clients port = 22 keep state
# ntp outgoing traffic
pass out quick on fxp0 proto udp from fxp0 to any port = 123 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 123 keep state
# web incoming traffic
pass in quick on fxp0 proto tcp from any to $int_www port {80,443} keep state
pass out quick on fxp1 proto tcp from any to $int_www port {80,443} keep state
- for outgoing traffic NAT, also can do NAT on carp alias interface as follows
nat on $ext_if from $int_network to any -> 1.2.3.4
then enable carp alias interface as follows:
# ifconfig carp0 alias 1.2.3.4 netmask 255.255.255.0
note: for disable alias, run
# ifconfig carp0 -alias 1.2.3.4
note: you even can define NAT only for port 25 as follows in /etc/pf.conf
nat on $ext_if from $int_network to any port 25 -> 1.2.3.4
Posts
