Jephe Wu - http://linuxtechres.blogspot.com
Objective: allow the users on the LAN who are using Squid proxy (10.0.0.1) to be able to access ftp sites.
Environment: OpenBSD 4.5(1.2.3.4/10.0.0.2), Squid client(10.0.0.1) behind this OpenBSD firewall
Internet[1.2.3.4]OpenBSD4.5[10.0.0.2]<->[10.0.0.1]Squid/Web/DNS server
<->[10.0.0.10]sysadmin pc
Steps:
1. modify /etc/rc.conf to enable ftpproxy
vi /etc/rc.conf to change ftpproxy_flag from NO to YES
2. enable pf.conf for ftp outgoing and incoming web/dns requests
ext_if="fxp1"
int_if="fxp0"
set block-policy return
set loginterface $ext_if
set skip on lo
# scrub incoming pcakets like you cannot set both SYN and FIN
scrub in all
# ftpproxy
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
# Redirect ftp traffic to proxy
rdr pass proto tcp from any to any port ftp -> 127.0.0.1 port 8021
# let squid proxy act as web server and dns server
rdr pass on $ext_if proto tcp from any to 1.2.3.4/32 port {80,443} -> 10.0.0.1
rdr pass on $ext_if proto udp from any to 1.2.3.4/32 port 53 -> 10.0.0.1
# squid proxy server can go to anywhere
nat pass on $ext_if from 10.0.0.1 to any -> 1.2.3.4
# setup a default deny policy
block in all
# activate spoofing protection for all interfaces
block in quick from urpf-failed
# anchor for ftpproxy
anchor "ftp-proxy/*"
# pass tcp, udp, and icmp out on the external (Internet) interface.
# tcp connections will be modulated, udp/icmp will be tracked statefully
pass out modulate state
antispoof quick for { lo $int_if }
pass in quick on $ext_if inet proto icmp all icmp-type { echorep, timex, unreach }
pass in quick on $ext_if proto udp to 1.2.3.4 port 53 keep state
pass in quick on $ext_if proto tcp to 1.2.3.4 port {80,443} synproxy state
# use synproxy for internal host 10.0.0.0/24
pass in quick on $int_if proto tcp from 10.0.0.0/24 to $int_if port ssh synproxy state
# allow admin pc for anything
pass in quick on $int_if from 10.0.0.10/24
3. startup ftp proxy
ftp-proxy
pfctl -f /etc/pf.conf
References:
a. http://www.cyberciti.biz/faq/freebsd-opebsd-pf-firewall-ftp-configuration/
How to enable ftp client to access ftp server behind FreeBSD firewall
Understand Packet Filter(PF) Firewall
Jephe Wu - http://linuxtechres.blogspot.com
Environment: OpenBSD 4.8, FreeBSD 7.1
Objective: understanding how PF firewall works and varous important rules and parameters
Concepts:
PF is enabled by default on OpenBSD 4.6 and newer releases. In OpenBSD 4.1 and later, the keep state option became the implicit default for all filter rules
1. block in log all # default deny policy
a. Above rule doesn't have 'quick' option, which means it will still continue to traverse to the end of the rules in /etc/pf.conf until it meets one rule with quick option. If the traffic doesn't meet any rest of rules, then this block rule will take it, so the result is 'block'.
# if above rule becomes 'block in log quick all' , then it will block everything for incoming traffic due to 'quick' option, it won't look down anymore
b. By default, the PF rules will pass traffic unless it's blocked by default policy or specific rules
c.'log' option indicates that it will record those matching information to pflog file, if you use 'tcpdump -n -e -ttt -i pflog0' to monitor, it will see those matching information for this block rule.
The following message is from PF FAQ page at http://www.openbsd.org/faq/pf/filter.html
Each packet is evaluated against the filter ruleset from top to bottom. By default, the packet is marked for passage, which can be changed by any rule, and could be changed back and forth several times before the end of the filter rules. The last matching rule "wins". There is an exception to this: The quick option on a filtering rule has the effect of canceling any further rule processing and causes the specified action to be taken.
2. Keep state and modulate state
a. In OpenBSD 4.1 and later, the keep state option became the implicit default for all filter rules.You can use 'pfctl -sr' to check running rules for this option.
According to PF FAQ page, by storing information about each connection in a state table, PF is able to quickly determine if a packet passing through the firewall belongs to an already established connection. If it does, it is passed through the firewall without going through ruleset evaluation.
b. When a rule creates state, the first packet matching the rule creates a "state" between the sender and receiver. Now, not only do packets going from the sender to receiver match the state entry and bypass ruleset evaluation, but so do the reply packets from receiver to sender.
pass out on fxp0 proto tcp from any to any
This rule allows any outbound TCP traffic on the fxp0 interface and also permits the reply traffic to pass back through the firewall. Keeping state significantly improves the performance of your firewall as state lookups are dramatically faster than running a packet through the filter rules.
c. The modulate state option works just like keep state except that it only applies to TCP packets. The modulate state option can be used in rules that specify protocols other than TCP; in those cases, it is treated as keep state.
Keep state on outgoing TCP, UDP, and ICMP packets and modulate TCP ISNs:
pass out on fxp0 proto { tcp, udp, icmp } from any to any modulate state
Another advantage of keeping state is that corresponding ICMP traffic will be passed through the firewall.
d. keep state for UDP
PF simply keeps track of how long it has been since a matching packet has gone through. If the timeout is reached, the state is cleared.
e. TCP SYN Proxy
According to PF FAQ page at http://www.openbsd.org/faq/pf/filter.html.
Normally when a client initiates a TCP connection to a server, PF will pass the handshake packets between the two endpoints as they arrive. PF has the ability, however, to proxy the handshake. With the handshake proxied, PF itself will complete the handshake with the client, initiate a handshake with the server, and then pass packets between the two. The benefit of this process is that no packets are sent to the server before the client completes the handshake. This eliminates the threat of spoofed TCP SYN floods affecting the server because a spoofed client connection will be unable to complete the handshake.
The TCP SYN proxy is enabled using the synproxy state keywords in filter rules. Example:
pass in on $ext_if proto tcp to $web_server port www synproxy state
Here, connections to the web server will be TCP proxied by PF.
Because of the way synproxy state works, it also includes the same functionality as keep state and modulate state.
The SYN proxy will not work if PF is running on a bridge(4).
3. Flag S/SA (default for tcp)
a. To have PF inspect the TCP flags during evaluation of a rule, the flags keyword is used with the following syntax:
flags check/mask
flags any
The mask part tells PF to only inspect the specified flags and the check part specifies which flag(s) must be "on" in the header for a match to occur. Using the any keyword allows any combination of flags to be set in the header.
pass in on fxp0 proto tcp from any to any port ssh flags S/SA
pass in on fxp0 proto tcp from any to any port ssh
As flags S/SA is set by default, the above rules are equivalent, Each of these rules passes TCP traffic with the SYN flag set while only looking at the SYN and ACK flags. A packet with the SYN and ECE flags would match the above rules, while a packet with SYN and ACK or just ACK would not.
4. block drop in or block return in
a. by default, block uses 'drop', you can specify 'return' in
* drop - packet is silently dropped.
* return - a TCP RST packet is returned for blocked TCP packets and an ICMP Unreachable packet is returned for all others.
For example:
$ tcptraceroute -n 10.0.5.226
traceroute to 10.0.5.226 on TCP port 80 (http), 30 hops max
1 10.0.5.226[closed] 0.344ms 0.307ms 0.287ms
for the following rule, if without 'return', it will print 30 rows of asterisk.
block return in log quick on $int_if proto tcp from 10.0.10.0/24 to any port 80
5. PF Firewall Redundancy with CARP and pfsync
http://www.openbsd.org/faq/pf/carp.html
6. PF options and best practise example (http://www.openbsd.org/faq/pf/example1.html)
a. set block-policy return
set loginterface fxp0 #turn statistics logging "on" for the external interface
set skip on lo or set skip on {lo enc0} # ipencap communication goes through enc0 interface.
scrub in all # Reassembles fragment IP packets
scrub out all
b. block in log # setup a default deny policy
block in quick from urpf-failed # activate spoofing protection for all interfaces
# pass tcp, udp, and icmp out on the external (Internet) interface.
# tcp connections will be modulated, udp/icmp will be tracked statefully
c. pass out quick modulate state # We'll opt to filter the inbound traffic only. Outbound packets can avoid being checked, for improving performance.
d. antispoof quick for { lo $int_if } # It is good to use the spoofed address protection:
e. Now open the ports used by those network services that will be available to the Internet. First, the traffic that is destined to the firewall itself:
pass in on egress inet proto tcp from any to (egress) \
port $tcp_services
Specifying the network ports in the macro $tcp_services makes it simple to open additional services to the Internet by simply editing the macro and reloading the ruleset. UDP services can also be opened up by creating a $udp_services macro and adding a filter rule, similar to the one above, that specifies proto udp.
f. The next rule catches any attempts by someone on the Internet to connect to TCP port 80 on the firewall. Legitimate attempts to access this port will be from users trying to access the network's web server. These connection attempts need to be redirected to COMP3:
pass in on egress inet proto tcp to (egress) port 80 \
rdr-to $comp3 synproxy state
For an added bit of safety, we'll make use of the TCP SYN Proxy to further protect the web server.
g. ICMP traffic needs to be passed:
pass in inet proto icmp all icmp-type $icmp_types
Similar to the $tcp_services macro, the $icmp_types macro can easily be edited to change the types of ICMP packets that will be allowed to reach the firewall. Note that this rule applies to all network interfaces.
for example:
pass in on $ext inet proto icmp all icmp-type { echorep, timex, unreach } pass in on $ext inet proto icmp all icmp-type unreach code 4 # at least allow this, it's a must
pass in on $ext inet proto tcp from any to any port { = smtp, = http, = https, = ssh } h. Now traffic must be passed to and from the internal network. We'll assume that the users on the internal network know what they are doing and aren't going to be causing trouble. This is not necessarily a valid assumption; a much more restrictive ruleset would be appropriate for many environments.
pass in on $int_if
TCP, UDP, and ICMP traffic is permitted to exit the firewall towards the Internet due to the earlier "pass out" line. State information is kept so that the returning packets will be passed back in through the firewall.
i. using synproxy
pass in quick on $ext_if proto tcp to {10.0.4.7,10.0.4.8} port {80,443} synproxy state
pass in quick on $int_if proto tcp from {10.0.0.1,10.0.0.200} to $int_if port ssh synproxy state
7. packet filter rules for OpenBSD VPN and ipsec protocol
The following indicates that how the OpenBSD VPN rules matches the vpn traffic, fxp1 is internal NIC, fxp0 is facing Internet, enc0 is VPN virtual NIC.
# tcpdump -n -e -ttt -i pflog0
tcpdump: listening on pflog0
rule 57/0(match): pass in on fxp1: 10.0.0.1.56752 > 10.204.0.8.www: S 1328159562:1328159562(0) win 0 [ttl 1]
rule 54/0(match): pass out on enc0: 10.0.0.1.56752 > 10.204.0.8.www: S 1328159562:1328159562(0) win 0 [ttl 1]
rule 47/0(match): pass out on fxp0: esp 1.2.3.4 > 5.6.7.8 spi 0x945D0A23 seq 1 len 76
IPSec utilizes protocol UDP port 500(isakmp) for key exchange -- and port 4500/UDP for NAT-Traversal (ipsec-nat-t) as well as protocols ESP on Internet facing NIC.
Also, IPENCAP communication which goes through enc0 interface.
pass log quick on enc0 proto ipencap all keep state
or
set skip on {lo enc0}
8. pfctl usage
a. enable and disable
pfctl -sa # show status, if it's disabled, then
pfctl -e # enable it
pfctl -d # disable it
# it doesn't actually load a ruleset. The ruleset must be loaded separately.
b. check syntax of /etc/pf.conf
# pfctl -nf /etc/pf.conf
c. list rules and states etc
# pfctl -sr # list current rules in memory, the first row is rule 0, the second rule is rule 1, and so on
note: when using pfctl -sr, it list the actual rules in memory, it may find that the 'keep state' is the default for tcp protocol. For example:
# grep 22 /etc/pf.conf
pass in log quick proto tcp from any to any port 22
# pfctl -sr
block drop in log all
pass in log quick proto tcp from any to any port = ssh flags S/SA keep state
# above pfctl -sr actually expands the port 22 passing rule with 'flags S/SA keep state'.
# pfctl -ss Show the current state table
# pfctl -si Show filter stats and counters
# pfctl -sa Show EVERYTHING it can show
# pfctl -sa | grep tcp.established
tcp.established 86400s (note: 24 hours)
d. tcpdump for troubleshooting
# tcpdump -n -e -ttt -i pflog0 # realtime monitor traffic is passed or block by which rules provided the log option is enabled for that rule.
9. example on /etc/pf.conf
ext_if="fxp1"
int_if="fxp0"
set block-policy return
set loginterface $ext_if
set skip on lo
# scrub incoming pcakets like you cannot set both SYN and FIN
scrub in all
#assume 1.2.3.4 is our external IP for web servers
rdr pass on $ext_if proto tcp from any to 1.2.3.4/32 port {80,443} -> 10.0.0.1
# nat pass rule
nat pass on $ext_if proto icmp from 10.0.0.1 to any -> 1.2.3.4
nat pass on $ext_if from any to any port {53,25} -> 1.2.3.4
# setup a default deny policy
block in all
# activate spoofing protection for all interfaces
block in quick from urpf-failed
# pass tcp, udp, and icmp out on the external (Internet) interface.
# tcp connections will be modulated, udp/icmp will be tracked statefully
pass out modulate state
antispoof quick for { lo $int_if }
# for path mtu discovery
pass in quick on $ext_if inet proto icmp all icmp-type { echorep, timex, unreach }
# for dns server sitting on DMZ to serve internet, as well as web server
pass in quick on $ext_if proto udp to 10.0.0.2 port 53 keep state
pass in quick on $ext_if proto tcp to {10.0.0.1} port {80,443} synproxy state
# for internal admin pc to ssh into firewall
pass in quick on $int_if proto tcp from {10.0.0.20,10.0.0.21} to $int_if port ssh synproxy state
how to do basic configuration and setup for freebsd 6.3 pf firewall
Jephe Wu - http://linuxtechres.blogspot.com
Environment: FreeBSD PF firewall.
Objective: understanding Packet Filter firewall
Steps:
1. OS installation part
1. use 'F - DD' mode to configure partition since we are using the server dedicated for FreeBSD only, no other OS.
2. use 'Automatic' mode to make partitions
3. select 'full binary, doc and kernel source only' category to install
2. user and dns configuration
If your environment doesn't have DNS server, you have to wait for a long time before getting a login prompt when you try to ssh into the server. The solution is to vi /etc/nsswitch.conf to take out 'dns' from hosts line.
3. setup up PF firewall
a. put to /etc/rc.conf:
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflogd_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
b. edit the default /etc/pf.conf
ext_if="fxp0" # replace with actual external interface name i.e., dc0
int_if="fxp1" # replace with actual internal interface name i.e., dc1
scrub in all # normalize all packets
block in log all # default deny policy
# enable ssh from the specific internal IP to internal NIC of firewall
pass in on $int_if proto tcp from 192.168.0.2 to $int_if port 22 keep state
# enable all outgoing traffic on internet NIC for tcp and udp
pass out on $int_if proto { tcp, udp } all keep state
#enable incoming ssh from trusted external IP
pass in on $ext_if from x.y.z.a keep state
pass out on $ext_if from $ext_if to x.y.z.a keep state
Setup IPSec VPN between 2 FreeBSD 6.3 servers
Environment:
1 HP lp1000r with 2 18G hard disk is sitting in datacenter; another PC is sitting in office. Both are running FreeBSD 6.3.
lp1000r: vpndc has IP pair d.1.2.3 k/di.1.2.3 in datacenter (d for datacenter, the behind network is di.1.2.0/24)
another pc:vpnsg has IP pair o.1.2.3/oi.1.2.3 in office (o for office, the behind network is oi.1.2.0/24)
2 firewalls doing port forwarding and seperate DMZ and internal office/datacenter network
___Internetl corporate network____"FW"oi.1.2.4____oi.1.2.0/24---oi.1.2.3"vpnsg"o.1.2.3 ++++++++d.1.2.3"vpndc"di.1.2.3__di.1.2.0/24____di.1.2.4"FW"____internal datacenter network___
Objective:
Setup VPN over IPSec between 2 FreeBSD servers so that it can connect 2 private network through Internet.
Steps:
- install OS
- SSH and PF configuration
=================
ext_if="fxp0"
int_if="fxp1"
scrub in all
block in log all
pass in on $int_if proto tcp from any to $int_if port 22 keep state
pass out on $int_if proto { tcp, udp } all keep state
=================
for final pf.conf on vpndc and can be used for ipsec, I use:
ext_if="fxp0"
int_if="fxp1"
scrub in all
block in log all
# allow icmp type 3 code 4
pass in log quick on $ext_if inet proto icmp all icmp-type unreach code 4
pass in log quick on $int_if inet proto icmp all icmp-type unreach code 4
pass out log quick on $ext_if inet proto icmp all icmp-type unreach code 4
pass out log quick on $int_if inet proto icmp all icmp-type unreach code 4
block in log quick on $ext_if proto icmp from any to any
block in log quick on $int_if proto icmp from any to any
block out log quick on $ext_if proto icmp from any to any
block out log quick on $int_if proto icmp from any to any
# for $int_if
# allow ssh to vpndc itself from FW on inside DMZ
pass in on $int_if proto tcp from di.1.2.4 to $int_if port 22 keep state
# allow the whole DMZ can reply back tcpip packet to the office uses, actually, this rule might contain the above ssh one
pass in on $int_if from di.1.2.0/24 keep state
#allow office users' request going to DMZ in datacenter
pass out on $int_if from any to di.1.2.0/24 keep state
####for $ext_if ,almost same as above
pass in on $ext_if from o.1.2.3 to $ext_if keep state
pass out on $ext_if from $ext_if to o.1.2.3 keep state
- DNS and ssh slow response issue
vi /etc/nsswitch.conf
to take out 'dns' from hosts line
- add user jephe for 'su -'
to add user 'jephe' to 'wheel' group
- enable raid1 mirror for hp lp1000r server
We assume /dev/da0 is the first hard disk we installed OS on, we need to add /dev/da1 which is the second hard disk to the raid1 mirror system.
# sysctl kern.geom.debugflags=16
# gmirror label -v -b round-robin gm0 /dev/da0
# echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0, :%s#da0#mirror\/gm0#g)
bsd1# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
/dev/mirror/gm0s3d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
# reboot
After reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status.
For adding the second hard disk /dev/da1 to raid array, run
#gmirror forget gm0 (optional, depends)
#gmirror insert gm0 /dev/da1
- compile kernel to enable ipsec
cp GENERIC /root/MYKERNEL
ln -sf /root/MYKERNEL
vi MYKERNEL to add the following:
options IPSEC
options IPSEC_ESP
options IPSEC_DEBUG
cd /usr/src
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
reboot
- install racoon
put it under /home/jephe
tar xvfz ipsec-tools-0.7.tar.gz
cd ipsec-tools-0.7
./configure ;make
su - as root to make install
binary racoon will be installed to /usr/local/sbin/racoon
then you need to put racoon.conf.sample to /usr/local/etc/racoon.conf
and psk.txt.sample to /usr/local/etc/psk.txt
you can run 'find . -name "psk.txt*" and 'find . -name "racoon.conf*" from ipsec-tools-0.7 directory to find out the path of two files psk.txt.sample and racoon.conf.sample.
- configure racoon
then copy racoon.conf.sample there as racoon.conf.
after that, copy psk.txt.sample to /usr/local/etc also, then vi racoon.conf to change the following line:
path pre_shared_key "/usr/local/etc/psk.txt";
So, change it to use /usr/local/etc/psk.txt, before it was /usr/local/etc/v6/psk.txt .
chown root:wheel psk.txt
chmod 600 psk.txt
the above commands are very important, otherwise, after you run racoon, it won't establish vpn normally.
inside psk.txt, configure one line for peer end, make it looks like this:
remotesiteipaddress sharedkey
- configuring /etc/rc.conf on vpndc
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflogd_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
#enable ipsec
ipsec_enable="YES"
ipsec_file="/etc/ipsec.conf"
#for gif interface on datacenter freebsd 6.3
###
gif_interfaces="gif0"
gifconfig_gif0="d.1.2.3 o.1.2.3"
ifconfig_gif0="inet di.1.2.3 oi.1.2.3 netmask 0xffffffff"
static_routes="vpn"
route_vpn="-net oi.1.2.0/24 oi.1.2.3 "
- configure /etc/ipsec.conf
spdadd o.1.2.3/32 d.1.2.3 ipencap -P in ipsec esp/tunnel/o.1.2.3-d.1.2.3/require;
spdadd di.1.2.0/24 oi.1.2.0/24 any -P out ipsec esp/tunnel/d.1.2.3-o.1.2.3/require;
spdadd oi.1.2.0/24 di.1.2.0/24 any -P in ipsec esp/tunnel/o.1.2.3-d.1.2.3/require;
- configure /etc/rc.local
# start up racoon, the default configuration file is /usr/local/etc/racoon.conf, you can run 'strings /usr/local/sbin/racoon | grep racoon.conf' to know that.
/usr/local/sbin/racoon [ -l /var/log/racoon.log ]
- reboot server and 'racoon' should be started automatically , after that , try to ping each other, e.g. from office freebsd server vpnsg, ping di.1.2.3, then sniff the traffic on datacenter freebsd server using command 'tcpdump -n -i fxp0 host o.1.2.3'
- same examples of configuration
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default x.x.x.x UGS 2 34 fxp0
127.0.0.1 127.0.0.1 UH 0 90 lo0
oi.1.2.0/24 oi.1.2.3 UGS 8874 8877 gif0
oi.1.2.3 di.1.2.3 UH 8 7 gif0
di.1.2.0/24 link#2 UC 0 0 fxp1
d.1.2.0/24 link#1 UC 0 0 fxp0
note: from above routing table, going to office internal network oi.1.2.0/24 will be sent to gif0 and office vpnsg inside ip oi.1.2.3; and going to oi.1.2.3 will be sent to di.1.2.3 and gif0
CARP pfsync and gmirror for firewall failover and raid1 mirror under FreeBSD 6.2
Jephe Wu - http://linuxtechres.blogspot.com
Secure Firewall is very important for company network, it protects all valuable resources on the internal LAN. To avoid the single point of failure, it's always good practice to implement firewall fail-over. FreeBSD is a good choice to act as a firewall because it's popular, secure and support packet filter(PF) which is ported from OpenBSD, as well as CARP, pfsync. Software raid tool - gmirror in FreeBSD is extremely easy to configure.
Environment
1. 2 HP netserver lp1000r (running Freebsd 6.2)
2. each lp1000r server has 2 built-in network ports, one connects to Internet, the another connects to internal network
3. each lp1000r
3. web servers behind the firewalls using internal IP addresses (running CentOS 4)
Objective
1. when firewall1 is down, firewall2 will be taking over. This should be transparent to the end web user.
2. since each hp netserver lp1000r has 2 idential hard disks(18G), we need to build it as raid-1 mirroring. In case any one of hard disk dies, the firewall is still able to work.
3. 2 web servers are also DNS server, SMTP server and NTP client which means we need to configure the relevant PF rules on each firewall.
Naming conversion
hostname for firewall1: bsd1 (master firewall)
domain name: maxposs.com
external network: a.b.c.192/28 (replace a.b.c. with your own IPs)
default gateway: a.b.c.193
fxp0: a.b.c.204
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.5
carp1: 10.0.0.1
hostname for firewall2: bsd2 (backup firewall)
domain name: maxposs.com
external network: a.b.c.192/28
default gateway: a.b.c.193
fxp0: a.b.c.205
carp0: a.b.c.201
carp0_alias0: a.b.c.200
fxp1: 10.0.0.6
carp1: 10.0.0.1
hostname for internal web server 1: web1
ext0: 10.0.0.7
gateway: 10.0.0.1 (carp1 on firewalls)
services: web, smtp, dns, ntp client, ssh client
hostname for internal web server 2: web2
ext0: 10.0.0.8
gateway: 10.0.0.1 (carp1 on firewall)
services: web, smtp, dns, ntp client, ssh client
Freebsd 6.2 OS Installation
I'm using the default installation for FreeBSD, slide a for /, slide b for swap, slide d for /var and slide e for /usr.
Freebsd 6.2 raid-1 mirroring
Since there're 2 firewalls that needs to install, It's better to install one firewall, then the clone everything to the another one.
We installed FreeBSD OS on the first hard disk on bsd1, then use the following steps to add the second hard disk to become a raid-1 mirrored system.
# sysctl kern.geom.debugflags=16
# gmirror label -v -b round-robin gm0 /dev/da0
# echo geom_mirror_load=YES >> /boot/loader.conf
# vi /etc/fstab (to change all /dev/da0 to /dev/mirror/gm0)
bsd1# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s2b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s4d /usr ufs rw 2 2
/dev/mirror/gm0s3d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
# reboot
After reboot, you can use command 'gmirror status' or 'gmirror list' to check the raid1 status.
For adding the second hard disk to raid array, run
#gmirror forget gm0
# gmirror insert gm0 /dev/da1
Compiling kernel to enable carp and pfsync interfaces
During OS installation, we installed kernel developer packages so that we can do kernel
compilation for carp and pfsync later.
First of all, backup your current original kernel first.
# cp -pr /boot/kernel /boot/kernel.orig
note:After the whole system is stablized, you might want to backup the working kernel again
# cp -pr /boot/kernel /boot/kernel.good
# cd /usr/src/sys/i386/conf
# cp GENERIC MYKERNEL
# vi MYKERNEL (to add the following lines)
device pf
device pfsync
device pflog
device carp
note: you cannot just add 'device pfsync' without adding 'device pf' first.
If you want to be able to use ALTQ then the following as well:
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ
options ALTQ_NOPCC
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL# reboot
In case the new kernel doesn't boot up, please refer to http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-trouble.html#KERNELCONFIG-NOBOOT
Clone the FreeBSD OS to another firewall/ Replacing failed hard disk with a new one
shutdown bsd1 firewall, put the 2 hard disks to the left slot on each firewall, then
insert empty 2 hard disks to the right slot on each firewall, then run the following
commands to add the second hard disk to raid1 array
# gmirror forget gm0
# gmirror insert gm0 /dev/da1
note: you must run the first command, otherwise you will get error message 'not all
hard disks connected'
Configuring CARP and Pfsync on firewall
/etc/rc.conf on bsd1:
gateway="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"
/etc/rc.conf on bsd2
gateway="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
cloned_interfaces="carp0 carp1"
ifconfig_carp0="vhid 1 advskew 100 pass maxposs a.b.c.201/28"
ifconfig_carp0_alias0="vhid 1 advskew 100 pass maxposs a.b.c.200/28"
ifconfig_carp1="vhid 2 advskew 100 pass maxposs 10.0.0.1/24"
ifconfig_pfsync0="up syncif fxp1"
note:
1. maxposs is the password, must be same for the same vhid (virtual host ID)
2. advskew 100 on bsd2 makes the carp advertisement packet less frequent than bsd1 so
it will be backup firewall whenever the election happens.
/etc/sysctl.conf on both bsd1 and bsd2
add the following lines
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
#if one interface fails then all will fail over
net.inet.carp.preempt=1
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
/etc/pf.conf on both bsd1 and bsd2
int_if="fxp1"
ext_if="fxp0"
lo_if="lo0"
int_network="10.0.0.0/24"
web1 = "10.0.0.7/32"
web2 = "10.0.0.8/32"
int_www = "{ $web1, $web2 }"
int_ns = "{ $web1, $web2 }"
ext_www = "a.b.c.201"
ext_ns = "a.b.c.200"
ext_smtp="a.b.c.206"
ssh_client1 = "x.y.z.1/32"
ssh_client2 = "x.y.z.2/32"
# Normalization
scrub in all
# NAT for all
#nat on $ext_if from $int_network to any -> $ext_if
# above is commented, for using physical external interface for outgoing smtp nat
nat on $ext_if from $int_network to any -> $ext_smtp
# using virtual interface for outgoing smtp nat
# web service rdr
rdr on fxp0 proto tcp from any to $ext_www port 80 -> $int_www round-robin
rdr on fxp0 proto tcp from any to $ext_www port 443 -> $int_www round-robin
# dns rdr
rdr on fxp0 proto udp from any to $ext_ns port 53 -> $int_ns port 53
# default rule
block in log all
# Allow all Loopback
pass quick on $lo_if all
# Allow pfsync Updates In/Out
pass quick on $int_if proto pfsync keep state
# Allow CARP Advertisements In/Out
pass quick on {$ext_if, $int_if} proto carp keep state
# dns incoming traffic
pass in log quick on fxp0 proto udp from any to $int_www port = 53 keep state
pass out quick on fxp1 proto udp from any to $int_www port = 53 keep state
# dns outgoing traffic
pass out log quick on fxp0 proto udp from fxp0 to any port = 53 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 53 keep state
# smtp outgoing traffic from physical interface fxp0 and virtual interface $ext_smtp
pass out quick on fxp0 proto tcp from fxp0 to any port = 25 keep state
# for allowing smtp traffic from virtual external interface to any
pass out quick on fxp0 proto tcp from $ext_smtp to any port = 25 keep state
pass in quick on fxp1 proto tcp from $int_www to any port = 25 keep state
# ssh outgoing traffic
pass out quick on fxp0 proto tcp from fxp0 to $ssh_clients port = 22 keep state
pass in quick on fxp1 proto tcp from $int_www to $ssh_clients port = 22 keep state
# ntp outgoing traffic
pass out quick on fxp0 proto udp from fxp0 to any port = 123 keep state
pass in quick on fxp1 proto udp from $int_www to any port = 123 keep state
# web incoming traffic
pass in quick on fxp0 proto tcp from any to $int_www port {80,443} keep state
pass out quick on fxp1 proto tcp from any to $int_www port {80,443} keep state
- for outgoing traffic NAT, also can do NAT on carp alias interface as follows
nat on $ext_if from $int_network to any -> 1.2.3.4
then enable carp alias interface as follows:
# ifconfig carp0 alias 1.2.3.4 netmask 255.255.255.0
note: for disable alias, run
# ifconfig carp0 -alias 1.2.3.4
note: you even can define NAT only for port 25 as follows in /etc/pf.conf
nat on $ext_if from $int_network to any port 25 -> 1.2.3.4
Posts
