BSD DevCenter
oreilly.comSafari Books Online.Conferences.


VPNs and IPSec Demystified
Pages: 1, 2


The IPSec standard was designed to add security to the IP protocol. It does this by using protocols which add additional headers, also called encapsulations, to an IP packet. Since IPSec is an Internet standard, there are RFCs (Request For Comments) for IPSec and the protocols it uses. If you are interested in the inner workings of IPSec, the following RFCs are useful references and can be found at

RFC 2401 	IPSec
RFC 2402 	AH
RFC 2406 	ESP
RFC 2409 	IKE

I'll give a brief overview of each, so you will have the necessary information you'll need to understand the next article when I demonstrate configuring an IPSec VPN using a FreeBSD system. Let's start with the acronyms and then see how they fit together to create an IPSec VPN.

AH is the Authentication Header protocol. It provides integrity by ensuring that none of the bits in the portion of the packet being protected were tampered with during transit. I won't get into the details of which portion of the packet is protected and where in the packet the AH header is added as that depends upon the encryption mode and is covered in detail, with diagrams, in the RFC. Additionally, using AH can be problematic, especially if the packet passes through a NAT device. NAT changes the IP address in an IP packet to allow a private source address to access the Internet. Since the packet changes, it won't match the AH checksum. Also, AH was designed only to provide integrity. It does not provide privacy by encrypting the contents of the packet.

ESP is the Encapsulating Security Protocol and it provides both integrity and privacy. In transport mode, the ESP header is placed between the original, clear text IP header and the TCP or UDP header. In tunnel mode, the ESP header is placed between the new IP header and the original, completely encrypted IP packet.

Related Reading

Virtual Private Networks
By Charlie Scott, Paul Wolfe, Mike Erwin

Since both AH and ESP add an additional header, they have an associated protocol ID so the IP header knows to look for an AH or an ESP header. You may remember from TCP Protocol Layers Explained that each type of header has a number associated with it. For example, TCP is 6 and UDP is 17. If you are using IPSec behind a firewall, you have to remember to create a rule allowing the AH and/or ESP protocol ID. AH has a protocol ID of 51, and ESP has a protocol ID of 50. When making your firewall rule, remember that a protocol ID is not the same as a port number.

The third protocol used by IPSec is IKE, or the Internet Key Exchange protocol. As its name suggests, this protocol is used to manage the exchange of keys between the two VPN peers. While it is possible to generate the key manually, it is better and certainly much more scalable to allow the IKE protocol to do this for you automatically. Remember, you want the keys to change often, and you don't want to rely on yourself to remember to find time to do this manually on a regular basis. However, do remember to create a rule in your firewall to allow UDP port 500 as this is the port used by IKE.

The SA, or Security Association, is the IPSec term for a connection. When a VPN is established, one SA pair is created for each protocol used (i.e. one for AH and one for ESP). SAs are created in pairs as each SA is one directional, so two are required (one for each direction). These SA pairs are stored at each peer. If your peer has SAs, it means that a VPN tunnel has successfully been established.

Since each peer is capable of establishing multiple tunnels with multiple peers, the SAs have a unique number so you can tell which SAs belong to which peers. That number is called an SPI or Security Parameter Index.

SAs are stored in a database which is called, surprisingly enough, the SAD or Security Association Database. We will see this database again when we set up an IPSec VPN.

Each IPSec peer also has a second database, the SPD or Security Policy Database. This database contains the policy you have pre-configured on the peer. Most VPN implementations allow you to create multiple policies so you can choose the algorithm combinations which are suited to each peer with which you want to establish.

What sort of configuration goes into a policy?

  1. The symmetric algorithm(s) you are willing to use to encrypt/decrypt the data
  2. The cryptographic checksum(s) you are willing to use to ensure the integrity of the data
  3. The type of authentication method you are willing to use to authenticate the peer. Pre-shared secrets or RSA digital certificates are the most common forms of authentication.
  4. Whether to use tunnel mode or transport mode
  5. Which Diffie Hellman group to use
  6. How often to re-authenticate the peer
  7. How often to exchange the key used to encrypt/decrypt the data
  8. Whether or not to use PFS
  9. Whether to use AH, ESP, or both

When creating the policy, it is usually possible to create an ordered list of algorithms and Diffie Hellman groups. The first to match on both peers will be used. Remember, it is very important to ensure that everything in the policy will allow for a match on the peer. Even if everything else in the policy matches but one portion does not match, the peers will be unable to negotiate the VPN. If you are configuring a VPN between two different types of systems, you should take the time to research which algorithms each system supports, so you can choose the most secure policy from what is available.

Phase One and Phase Two

Now, let's see how all of this works together. Establishing and maintaining an IPSec VPN tunnel occurs in two phases. In Phase One, the two peers negotiate an authentication method, encryption algorithm, hash algorithm, and Diffie Hellman group. They also authenticate each other. All of this can occur in either three clear text packets (known as aggressive mode) or in six clear text packets (known as main mode). Assuming this is successful, a Phase 1 SA (also called an IKE SA) is created and the peers move on to Phase Two.

Also in FreeBSD Basics:

Fun with Xorg

Sharing Internet Connections

Building a Desktop Firewall

Using DesktopBSD

Using PC-BSD

In Phase Two, the keying material is generated, and the two peers agree upon the policy that will be used. This mode, which is called quick mode, differs from Phase One in that it can't occur until after Phase One is successful and all of the Phase Two packets are encrypted. This makes troubleshooting a little more difficult as it is possible for Phase One to be successful but Phase Two to fail. If Phase Two is successful, it will result in a Phase 2 SA, also called an IPSec SA, and the tunnel will be fully established.

When does all of this occur? The very first time a packet arrives at a peer with a destination in another encryption domain, the peer will initiate Phase One with the peer associated with that other encryption domain. Assuming both peers successfully establish the tunnel, the tunnel will stay up, ready to receive packets. However, the peers will re-authenticate each other and re-compare their policies at a pre-configured time. This time is known as the Phase One or IKE SA lifetime.

The peers will also re-negotiate the key that is used to encrypt and decrypt the data at another pre-configured time known as the Phase Two or IPSec SA lifetime. The Phase Two lifetime is shorter than the Phase One lifetime as you do want to change the key often. A typical Phase Two lifetime is 60 minutes. A typical Phase One lifetime is 24 hours.

It is your job to ensure both peers are configured with the same lifetimes. If they are not, it is possible for the tunnel to be established initially, but then cease to work when one of the mis-matched lifetime periods arrives. You will also encounter strange problems if the Phase One lifetime is shorter than the Phase Two lifetime. If a previously working tunnel appears to hang, one of the first things to check is the two lifetime settings on each peer. Also, if you do change a policy on a peer, it won't take effect until the next time Phase One occurs. If you want your change to take effect immediately, you need to remove the SAs for that tunnel from the SAD. This will force a renegotiation using your new policy parameters.

We now have enough background information to create an IPSec VPN on your FreeBSD box. A demonstration of the required configurations is the subject of the next article.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Sponsored by: