Setup IPSec VPN between 2 FreeBSD 6.3 servers

Environment:
1 HP lp1000r with 2 18G hard disk is sitting in datacenter; another PC is sitting in office. Both are running FreeBSD 6.3.

lp1000r: vpndc has IP pair d.1.2.3 k/di.1.2.3 in datacenter (d for datacenter, the behind network is di.1.2.0/24)
another pc:vpnsg has IP pair o.1.2.3/oi.1.2.3 in office (o for office, the behind network is oi.1.2.0/24)
2 firewalls doing port forwarding and seperate DMZ and internal office/datacenter network

___Internetl corporate network____"FW"oi.1.2.4____oi.1.2.0/24---oi.1.2.3"vpnsg"o.1.2.3 ++++++++d.1.2.3"vpndc"di.1.2.3__di.1.2.0/24____di.1.2.4"FW"____internal datacenter network___

Objective:
Setup VPN over IPSec between 2 FreeBSD servers so that it can connect 2 private network through Internet.

Steps:

• install OS

Use the first CD of FreeBSD 6.3, use 'F - DD' mode to configure partition since we are using FreeBSD for the dedicated server, use 'Automatic' mode of making partitions. use 'full binary, doc and kernel source only' category to install.

• SSH and PF configuration

Just after installing OS, by default ,it's no firewall enabled, you can configure the following to enable ssh on internal NIC.
=================
ext_if="fxp0"
int_if="fxp1"
scrub in all
block in log all
pass in on $int_if proto tcp from any to $int_if port 22 keep state
pass out on $int_if proto { tcp, udp } all keep state
=================
for final pf.conf on vpndc and can be used for ipsec, I use:
ext_if="fxp0"
int_if="fxp1"
scrub in all
block in log all

# allow icmp type 3 code 4
pass in log quick on $ext_if inet proto icmp all icmp-type unreach code 4        
pass in log quick on $int_if inet proto icmp all icmp-type unreach code 4       
pass out log quick on $ext_if inet proto icmp all icmp-type unreach code 4      
pass out log quick on $int_if inet proto icmp all icmp-type unreach code 4     

block in log quick on $ext_if proto icmp from any to any
block in log quick on $int_if proto icmp from any to any
block out log quick on $ext_if proto icmp from any to any
block out log quick on $int_if proto icmp from any to any

# for $int_if
# allow ssh to vpndc itself from FW on inside DMZ
pass in on $int_if proto tcp from di.1.2.4 to $int_if port 22 keep state

# allow the whole DMZ can reply back tcpip packet to the office uses, actually, this rule might contain the above ssh one
pass in on $int_if from di.1.2.0/24 keep state

#allow office users' request going to DMZ in datacenter
pass out on $int_if from any to di.1.2.0/24 keep state
####for $ext_if ,almost same as above
pass in on $ext_if from o.1.2.3 to $ext_if keep state
pass out on $ext_if from $ext_if to o.1.2.3 keep state

• DNS and ssh slow response issue

If your environment doesn't have DNS server, the ssh startup and login will be very slow, to address this issue, you can disable DNS as folllows:
vi /etc/nsswitch.conf
to take out 'dns' from hosts line

• add user jephe for 'su -'

vi /etc/group
to add user 'jephe' to 'wheel' group

• enable raid1 mirror for hp lp1000r server

to enable raid1 mirror, do the following:
We assume /dev/da0 is the first hard disk we installed OS on, we need to add /dev/da1 which is the second hard disk to the raid1 mirror system.
# sysctl kern.geom.debugflags=16
# gmirror label -v -b round-robin gm0 /dev/da0
# echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0, :%s#da0#mirror\/gm0#g)

bsd1# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
/dev/mirror/gm0s3d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
# reboot
After reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status.
For adding the second hard disk /dev/da1 to raid array, run
#gmirror forget gm0 (optional, depends)
#gmirror insert gm0 /dev/da1

• compile kernel to enable ipsec

cd /usr/src/sys/i386/conf
cp GENERIC /root/MYKERNEL
ln -sf /root/MYKERNEL
vi MYKERNEL to add the following:

options IPSEC
options IPSEC_ESP
options IPSEC_DEBUG

cd /usr/src
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
reboot

• install racoon

download it from Internet, it's an open source software
put it under /home/jephe
tar xvfz ipsec-tools-0.7.tar.gz
cd ipsec-tools-0.7
./configure ;make
su - as root to make install
binary racoon will be installed to /usr/local/sbin/racoon
then you need to put racoon.conf.sample to /usr/local/etc/racoon.conf
and psk.txt.sample to /usr/local/etc/psk.txt

you can run 'find . -name "psk.txt*" and 'find . -name "racoon.conf*" from ipsec-tools-0.7 directory to find out the path of two files psk.txt.sample and racoon.conf.sample.

• configure racoon

You can use command 'strings /usr/local/sbin/racoon | grep racoon.conf' to find out the path of racoon.conf
then copy racoon.conf.sample there as racoon.conf.
after that, copy psk.txt.sample to /usr/local/etc also, then vi racoon.conf to change the following line:

path pre_shared_key "/usr/local/etc/psk.txt";

So, change it to use /usr/local/etc/psk.txt, before it was /usr/local/etc/v6/psk.txt .

chown root:wheel psk.txt
chmod 600 psk.txt

the above commands are very important, otherwise, after you run racoon, it won't establish vpn normally.
inside psk.txt, configure one line for peer end, make it looks like this:

remotesiteipaddress sharedkey

• configuring /etc/rc.conf on vpndc
  

#enable pf
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflogd_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

#enable ipsec
ipsec_enable="YES"
ipsec_file="/etc/ipsec.conf"

#for gif interface on datacenter freebsd 6.3
###
gif_interfaces="gif0"
gifconfig_gif0="d.1.2.3 o.1.2.3"
ifconfig_gif0="inet di.1.2.3 oi.1.2.3 netmask 0xffffffff"
static_routes="vpn"
route_vpn="-net oi.1.2.0/24 oi.1.2.3 "

• configure /etc/ipsec.conf

spdadd d.1.2.3/32 o.1.2.3/32 ipencap -P out ipsec esp/tunnel/d.1.2.3-o.1.2.3/require;
spdadd o.1.2.3/32 d.1.2.3 ipencap -P in ipsec esp/tunnel/o.1.2.3-d.1.2.3/require;

spdadd di.1.2.0/24 oi.1.2.0/24 any -P out ipsec esp/tunnel/d.1.2.3-o.1.2.3/require;
spdadd oi.1.2.0/24 di.1.2.0/24 any -P in ipsec esp/tunnel/o.1.2.3-d.1.2.3/require;

• configure /etc/rc.local

vpn# more rc.local
# start up racoon, the default configuration file is /usr/local/etc/racoon.conf, you can run 'strings /usr/local/sbin/racoon | grep racoon.conf' to know that.

/usr/local/sbin/racoon [ -l /var/log/racoon.log ]

• reboot server and 'racoon' should be started automatically , after that , try to ping each other, e.g. from office freebsd server vpnsg, ping di.1.2.3, then sniff the traffic on datacenter freebsd server using command 'tcpdump -n -i fxp0 host o.1.2.3'

• same examples of configuration
  

$ netstat -nr
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default x.x.x.x UGS 2 34 fxp0
127.0.0.1 127.0.0.1 UH 0 90 lo0
oi.1.2.0/24 oi.1.2.3 UGS 8874 8877 gif0
oi.1.2.3 di.1.2.3 UH 8 7 gif0
di.1.2.0/24 link#2 UC 0 0 fxp1
d.1.2.0/24 link#1 UC 0 0 fxp0

note: from above routing table, going to office internal network oi.1.2.0/24 will be sent to gif0 and office vpnsg inside ip oi.1.2.3; and going to oi.1.2.3 will be sent to di.1.2.3 and gif0

CARP pfsync and gmirror for firewall failover and raid1 mirror under FreeBSD 6.2

Jephe Wu - http://linuxtechres.blogspot.com


Secure Firewall is very important for company network, it protects all valuable resources on the internal LAN. To avoid the single point of failure, it's always good practice to implement firewall fail-over. FreeBSD is a good choice to act as a firewall because it's popular, secure and support packet filter(PF) which is ported from OpenBSD, as well as CARP, pfsync. Software raid tool - gmirror in FreeBSD is extremely easy to configure.

Environment
1. 2 HP netserver lp1000r (running Freebsd 6.2)
2. each lp1000r server has 2 built-in network ports, one connects to Internet, the another connects to internal network
3. each lp1000r
3. web servers behind the firewalls using internal IP addresses (running CentOS 4)


Objective
1. when firewall1 is down, firewall2 will be taking over. This should be transparent to the end web user.
2. since each hp netserver lp1000r has 2 idential hard disks(18G), we need to build it as raid-1 mirroring. In case any one of hard disk dies, the firewall is still able to work.
3. 2 web servers are also DNS server, SMTP server and NTP client which means we need to configure the relevant PF rules on each firewall.

Naming conversion
hostname for firewall1: bsd1 (master firewall)
domain name: maxposs.com
external network: a.b.c.192/28 (replace a.b.c. with your own IPs)
default gateway: a.b.c.193
fxp0: a.b.c.204
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.5
carp1: 10.0.0.1

hostname for firewall2: bsd2 (backup firewall)
domain name: maxposs.com
external network: a.b.c.192/28
default gateway: a.b.c.193
fxp0: a.b.c.205
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.6
carp1: 10.0.0.1

hostname for internal web server 1: web1
ext0: 10.0.0.7
gateway: 10.0.0.1 (carp1 on firewalls)
services: web, smtp, dns, ntp client, ssh client

hostname for internal web server 2: web2
ext0: 10.0.0.8
gateway: 10.0.0.1 (carp1 on firewall)
services: web, smtp, dns, ntp client, ssh client

Freebsd 6.2 OS Installation
I'm using the default installation for FreeBSD, slide a for /, slide b for swap, slide d for /var and slide e for /usr.

Freebsd 6.2 raid-1 mirroring
Since there're 2 firewalls that needs to install, It's better to install one firewall, then the clone everything to the another one.

We installed FreeBSD OS on the first hard disk on bsd1, then use the following steps to add the second hard disk to become a raid-1 mirrored system.

# sysctl kern.geom.debugflags=16
# gmirror label -v -b round-robin gm0 /dev/da0
# echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0)

bsd1# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
/dev/mirror/gm0s3d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
# reboot
After reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status.
For adding the second hard disk to raid array, run
#gmirror forget gm0
# gmirror insert gm0 /dev/da1

Compiling kernel to enable carp and pfsync interfaces
During OS installation, we installed kernel developer packages so that we can do kernel
compilation for carp and pfsync later.

First of all, backup your current original kernel first.
# cp -pr /boot/kernel /boot/kernel.orig

note:After the whole system is stablized, you might want to backup the working kernel again
# cp -pr /boot/kernel /boot/kernel.good


# cd /usr/src/sys/i386/conf
# cp GENERIC MYKERNEL
# vi MYKERNEL (to add the following lines)
device pf
device pfsync
device pflog
device carp
note: you cannot just add 'device pfsync' without adding 'device pf' first.

If you want to be able to use ALTQ then the following as well:

options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ
options ALTQ_NOPCC
# cd /usr/src

# make buildkernel KERNCONF=MYKERNEL 

# make installkernel KERNCONF=MYKERNEL
# reboot

In case the new kernel doesn't boot up, please refer to http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-trouble.html#KERNELCONFIG-NOBOOT

Clone the FreeBSD OS to another firewall/ Replacing failed hard disk with a new one
shutdown bsd1 firewall, put the 2 hard disks to the left slot on each firewall, then
insert empty 2 hard disks to the right slot on each firewall, then run the following
commands to add the second hard disk to raid1 array
# gmirror forget gm0
# gmirror insert gm0 /dev/da1
note: you must run the first command, otherwise you will get error message 'not all
hard disks connected'


Configuring CARP and Pfsync on firewall

/etc/rc.conf on bsd1:
gateway="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""


cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"

/etc/rc.conf on bsd2
gateway="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""


cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 advskew 100 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 advskew 100 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 advskew 100 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"

note:
1. maxposs is the password, must be same for the same vhid (virtual host ID)
2. advskew 100 on bsd2 makes the carp advertisement packet less frequent than bsd1 so
it will be backup firewall whenever the election happens.

/etc/sysctl.conf on both bsd1 and bsd2
add the following lines

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

#if one interface fails then all will fail over
net.inet.carp.preempt=1

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536


/etc/pf.conf on both bsd1 and bsd2

int_if="fxp1"
ext_if="fxp0"
lo_if="lo0"

int_network="10.0.0.0/24"

web1 = "10.0.0.7/32"
web2 = "10.0.0.8/32"

int_www = "{ $web1, $web2 }"
int_ns = "{ $web1, $web2 }"

ext_www = "a.b.c.201"
ext_ns = "a.b.c.200"
ext_smtp="a.b.c.206"

ssh_client1 = "x.y.z.1/32"
ssh_client2 = "x.y.z.2/32"


# Normalization
scrub in all


# NAT for all
#nat on $ext_if from $int_network to any -> $ext_if
# above is commented, for using physical external interface for outgoing smtp nat
nat on $ext_if from $int_network to any -> $ext_smtp
# using virtual interface for outgoing smtp nat

# web service rdr
rdr on fxp0 proto tcp from any to $ext_www port 80 -> $int_www round-robin
rdr on fxp0 proto tcp from any to $ext_www port 443 -> $int_www round-robin

# dns rdr
rdr on fxp0 proto udp from any to $ext_ns port 53 -> $int_ns port 53

# default rule
block in log all

# Allow all Loopback
pass quick on $lo_if all

# Allow pfsync Updates In/Out
pass quick on $int_if proto pfsync keep state

# Allow CARP Advertisements In/Out
pass quick on {$ext_if, $int_if} proto carp keep state

# dns incoming traffic
pass in log quick on fxp0 proto udp from any to $int_www port = 53 keep state
pass out quick on fxp1 proto udp from any to $int_www port = 53 keep state

# dns outgoing traffic
pass out log quick on fxp0 proto udp from fxp0 to any port = 53 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 53 keep state

# smtp outgoing traffic from physical interface fxp0 and virtual interface $ext_smtp
pass out quick on fxp0 proto tcp from fxp0 to any port = 25 keep state
# for allowing smtp traffic from virtual external interface to any
pass out quick on fxp0 proto tcp from $ext_smtp to any port = 25 keep state
pass in quick on fxp1 proto tcp from $int_www to any port = 25 keep state

# ssh outgoing traffic
pass out quick on fxp0 proto tcp from fxp0 to $ssh_clients port = 22 keep state
pass in quick on fxp1 proto tcp from $int_www to $ssh_clients port = 22 keep state

# ntp outgoing traffic
pass out quick on fxp0 proto udp from fxp0 to any port = 123 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 123 keep state

# web incoming traffic
pass in quick on fxp0 proto tcp from any to $int_www port {80,443} keep state
pass out quick on fxp1 proto tcp from any to $int_www port {80,443} keep state

• for outgoing traffic NAT, also can do NAT on carp alias interface as follows

put the following line to nat part in /etc/pf.conf
nat on $ext_if from $int_network to any -> 1.2.3.4
then enable carp alias interface as follows:
# ifconfig carp0 alias 1.2.3.4 netmask 255.255.255.0
note: for disable alias, run
# ifconfig carp0 -alias 1.2.3.4

note: you even can define NAT only for port 25 as follows in /etc/pf.conf
nat on $ext_if from $int_network to any port 25 -> 1.2.3.4