Site-to-Site IPSec VPN in FreeBSD 7 and OpenBSD 4

Jephe Wu -

Objective: understanding ipsec vpn and how to configure it in FreeBSD and OpenBSD
Environment: FreeBSD 7.2 and OpenBSD 4.6, the sender ip is, the peer ip is,


The IPsec suite is an open standard. IPsec uses the following protocols to perform various functions:[2][3]

Authentication is performed by computing a cryptographic hash-based message authentication code over nearly all the fields of the IP packet (excluding those which might be modified in transit, such as TTL or the header checksum), and stores this in a newly-added AH header and sent to the other end.

AH use a special hashing algorithm and a specific key known only to the source and the destination. A security association between two devices is set up that specifies these particulars so that the source and destination know how to perform the computation but nobody else can. On the source device, AH performs the computation and puts the result (called the Integrity Check Value or ICV) into a special header with other fields for transmission. The destination device does the same calculation using the key the two devices share, which enables it to see immediately if any of the fields in the original datagram were modified (either due to error or malice).

data origin authentication, which enables the recipient to verify that messages have not been tampered with in transit (data integrity) and that they originate from the expected sender (authenticity). 
    We want to not only protect against intermediate devices changing our datagrams, we want to protect against them examining their contents as well. For this level of private communication, AH is not enough; we need to use the Encapsulating Security Payload (ESP) protocol.
      • Security associations (SA) provide the bundle of algorithms and data that provide the parameters necessary to operate the AH and/or ESP operations.
      A Security Association (SA) is the establishment of shared security attributes between two network entities to support secure communication. In a nutshell, an SA is a logical group of security parameters, that enable the sharing of information to another entity.

      IPSec provides the following security services:
      traffic is encrypted to ensure that only the legitimate receiver is able to access the data transmitted.
      Connectionless integrity
      ensures that no modifications were made to the data while in transit across the network.
      Data origin authentication
      the receiver is able to verify that data actually originates from the claimed source.
      Detection and rejection of replays (anti-replay)
      duplicate IP datagrams are detected and processed only once.

      These security services are provided at the IP layer  (layer  3 of the  OSI  model),  thus protecting  all
      protocols that may be carried over IP, including IP itself.

      IPSec can be run in either tunnel mode or transport mode.
      • Tunnel mode is most commonly used between gateways, or at an end-station to a gateway
      • Transport mode is used between end-stations or between an end-station and a gateway, if the gateway is being treated as a host, in which the gateway is the actual destination.
      Transport Mode provides a secure connection between two endpoints as it encapsulates IP's payload, while Tunnel Mode encapsulates the entire IP packet to provide a virtual "secure hop" between two gateways. The latter is used to form a traditional VPN, where the tunnel generally creates a secure tunnel across an untrusted Internet. 
        A secure VPN requires both authentication and encryption. We know that ESP is the only way to provide encryption, but ESP and AH both can provide authentication:

        ESP+Auth is used in Tunnel mode to fully encapsulate the traffic on its way across an untrusted network, protected by both encryption and authentication in the same thing.

        To accomplish security service, IPSec must perform (at least) the following tasks:
        1. They must agree on a set of security protocols to use, so that each one sends data in a format the other can understand.
        2. They must decide on a specific encryption algorithm to use in encoding data.
        3. They must exchange keys that are used to “unlock” data that has been cryptographically encoded.
        4. Once this background work is completed, each device must use the protocols, methods and keys previously agreed upon to encode data and send it across the network.

        1. why phrase 1 and phrase2?
        Phase 1 is used to set up a protected channel just
        between the two gateway machines. This channel is then used for
        the phase 2 negotiation traffic (i.e. encrypted & authenticated).

        'Phase 2' defines which connections the daemon should establish.
        These connections contain the actual "IPsec VPN" information.

        2. ipsecctl since OpenBSD 3.8
        A new command starts with version 3.8: ipsecctl.
        a simple VPN can be setup by editing /etc/ipsec.conf, then you can start up VPN as follows:
        isakmpd -K ipsecctl -f /etc/ipsec.conf
        note: -K      When this option is given, isakmpd does not read the policy con-
                     figuration file and no keynote(4) policy check is accomplished.
                     This option can be used when policies for flows and SA establish-
                     ment are arranged by other programs like ipsecctl(8) or bgpd(8).
        use 'ipsecctl -sa' to get the status of the ipsec flows and associations:

        or you can
        Set this up to start automatically at reboot in /etc/rc.conf
        PF=YES   # use pfctl -sr to get the running pf rules

        in /etc/rc.local,put
        ipsecctl -f /etc/ipsec.conf

        3. FreeBSD-specific requirements
        a. Recompiled FreeBSD kernel with IPSec support according to FreeBSD Handbook.
        b. IPsec-Tools port (/usr/ports/security/ipsec-tools) installed on FreeBSD host.
        c. manually create directory /usr/local/etc/

        OpenBSD has required applications and IPSec support compiled into the GENERIC kernel, so there's no need to do anything.

        4. encryption and authentication choices
        Default racoon configuration available from FreeBSD's port uses 3des cipher for encryption and hmac_sha1 algorithm for authentication in both phases.

        Default settings for isakmpd on OpenBSD are: aes and hmac-sha1 for phase 1 and aes and hmac-sha2-256 for phase 2.

        5. protocols used in ipsec vpn
        50 - Encapsulation Header (ESP)
        51 - Authentication Header (AH)
        4 - IP-ENCAP(ipencap protocol)

        500/udp - Internet Key Exchange (IKE)
        4500/udp - NAT-Traversal

        esp     50      IPSEC-ESP       # Encap Security Payload [RFC2406]
        ah      51      IPSEC-AH        # Authentication Header [RFC2402]
        ipencap 4       IP-ENCAP        # IP encapsulated in IP (IP in IP)

        OpenBSD: use udp 500 and esp protocol. Here is example:

        set skip on { lo enc0 } #skip on enc0, you can enable enc0 and filter traffic
        block on em0
        pass  in on em0 proto udp from to port 500
        pass  in on sk0 proto esp from to
        pass out on em0 proto udp from to port 500
        pass out on sk0 proto esp from to

        Freebsd: uses 500/UDP, ESP and -- additionally -- IPENCAP traffic.
        pass in quick on dc0 proto udp from port = isakmp to port = isakmp
        pass in quick on dc0 proto esp from to
        pass out quick on dc0 proto udp from port = isakmp  to port = isakmp
        pass out quick on dc0 proto esp from to

        pass in quick on dc0 proto ipencap from to
        pass out quick on dc0 proto ipencap from to
        block quick on dc0

        1. openbsd 4.6 - isakmpd method

        a. root@SHGFWHA9 # grep remote_vpngw /etc/pf.conf
        vpn_wan_ip   = "{}"
        remote_vpngw = "{}"         
        pass in  log quick on $vpn_wan_if proto {tcp,udp,icmp} from $remote_vpngw to $vpn_wan_ip keep state
        pass in  log quick on $vpn_wan_if proto esp from $remote_vpngw to $vpn_wan_ip   keep state
        pass in  log quick on $vpn_wan_if proto udp from $remote_vpngw port { 500, 4500 } to   $remote_vpngw port { 500, 4500 } keep state

        b. isakmpd and ipsec

        root@SHGFWHA9 # grep -v ^# /etc/isakmpd/isakmpd.conf | grep -v ^$
        Retransmits=            5
        Exchange-max-time=      120
        Default-phase-1-lifetime= 86400,28800:86400
        DPD-check-interval=     30  #DPD means Dead Peer Detection

        [Phase 1] =         ISAKMP-peer1

        [Phase 2]
        Connections=            IPsec-peer1,IPsec-peer2

        Phase=                  1
        Transport=              udp
        Configuration=          Default-main-mode
        Authentication=         abcdef

        Phase=                  2
        ISAKMP-peer=            ISAKMP-peer1
        Configuration=          Default-quick-mode
        Local-ID=               Net-local
        Remote-ID=              Net-peer1

        ID-type=                IPV4_ADDR_SUBNET

        ID-type=                IPV4_ADDR_SUBNET

        DOI=                    IPSEC
        EXCHANGE_TYPE=          ID_PROT
        Transforms=             DES-SHA

        DOI=                    IPSEC
        EXCHANGE_TYPE=          QUICK_MODE
        Suites=                 QM-ESP-DES-SHA-SUITE

        c. when you run ps with isakmpd method, you should see something below:

        root@SHGFWHA9 # ps ax | grep isakmpd | grep -v grep
        26976 ??  Is      0:42.69 isakmpd: monitor [priv] (isakmpd)
         8410 ??  S       0:00.40 isakmpd

        d. get vpn status in OpenBSD
        # ipsecctl -sa

        2. OpenBSD 4.6 - ipsec method

        a. /etc/ipsec.conf
        ike esp from to peer
          main auth hmac-sha1 enc blowfish group modp1024
          quick auth hmac-sha2-256 enc blowfish group modp1024

        b. refer to concepts second point to start up VPN in ipsec method

        3. FreeBSD 7.2 - ipsec/racoon method
        Refer to handbook link below to setup IPSec VPN on FreeBSD
        a. more /etc/ipsec.conf
        spdadd ipencap -P out ipsec esp/tunnel/;
        spdadd ipencap -P in ipsec esp/tunnel/;
        more /etc/ipsec.conf
        spdadd any -P out ipsec esp/tunnel/;
        spdadd any -P in ipsec esp/tunnel/;

 and are the IP addresses and netmasks that identify the network or hosts that this policy will apply to. In this case, we want it to apply to traffic between these two hosts. ipencap tells the kernel that this policy should only apply to packets that encapsulate other packets. -P out says that this policy applies to outgoing packets, and ipsec says that the packet will be secured.

        b. /usr/local/etc/racoon.conf
        vpn# grep -v ^# /usr/local/etc/racoon.conf | grep -v ^$
        path pre_shared_key "/usr/local/etc/psk.txt" ;
        log debug2;
        remote anonymous
            exchange_mode main,base;
            lifetime time 24 hour ;
            proposal {
                encryption_algorithm des;
                hash_algorithm sha1;
                authentication_method pre_shared_key ;
                dh_group 2 ;
            proposal_check strict;
        sainfo anonymous
            lifetime time 12 hour ;
            encryption_algorithm 3des, cast128, blowfish 448, des, rijndael ;
            authentication_algorithm hmac_sha1,hmac_md5 ;
            compression_algorithm deflate ;
        note: proposal part is phrase 1, sainfo part is phrase 2.

        c. vpn# more psk.txt    abcdefg

        d. vpn# more /etc/rc.local
        /usr/local/sbin/racoon -l /var/log/racoon.log
        # racoon -F (put racoon in foreground to debug)

        e. vpn# /etc/rc.conf

        vpnsg# more /etc/rc.conf




        ifconfig_xl0="inet netmask"
        ifconfig_fxp0="inet  netmask"


        # is the remote peer vpn gateway internal NIC ip address

        f. /etc/pf.conf

        vpnsg# cat /etc/pf.conf | grep -v ^# | grep -v ^$
        ext_if="fxp0"    # fxp0 ip is, gateway is
        int_if="xl0"    # xl0 ip is
        scrub in all
        set skip on lo0
        block in log all
        pass out log quick keep state
        pass in log quick on $ext_if inet proto icmp all icmp-type { echorep, timex, unreach }
        pass in log quick on $int_if inet proto icmp all icmp-type { echorep, timex, unreach }
        pass  in log quick on $int_if proto tcp from to $int_if port 22 keep state
        pass  in log quick on $int_if from  keep state
        pass  in log quick on $ext_if from to $ext_if  keep state

        g. get status of vpn in FreeBSD
        # setkey -D  # dump SAD entries
        # setkey -DP # dump PAD entries

        note: the following is from
        Security Policies, Security Associations and Associated Databases
        To manage all of this complexity, IPSec is equipped with a flexible, powerful way of specifying how different types of datagrams should be handled. To understand how this works, we must first define two important logical concepts:
        • Security Policies: A security policy is a rule that is programmed into the IPSec implementation that tells it how to process different datagrams received by the device. For example, security policies are used to decide if a particular packet needs to be processed by IPSec or not; those that do not bypass AH and ESP entirely. If security is required, the security policy provides general guidelines for how it should be provided, and if necessary, links to more specific detail.

          Security policies for a device are stored in the device's Security Policy Database (SPD).

        • Security Associations: A Security Association (SA) is a set of security information that describes a particular kind of secure connection between one device and another. You can consider it a "contract", if you will, that specifies the particular security mechanisms that are used for secure communications between the two.

          A device's security associations are contained in its Security Association Database (SAD).
        It's often hard to distinguish the SPD and the SAD, since they are similar in concept. The main difference between them is that security policies are general while security associations are more specific. To determine what to do with a particular datagram, a device first checks the SPD. The security policies in the SPD may reference a particular security association in the SAD. If so, the device will look up that security association and use it for processing the datagram.

        1.  (An Illustrated Guide to IPsec)
        2.  (wiki ipsec) 
        3. ( FreeBSD handbook VPN over IPSec)