Published on (
 See this if you're having trouble printing code examples

FreeBSD Basics VPNs and IPSec Demystified

by Dru Lavigne

So far in the cryptosystems series, we have taken a look at general cryptographic terminology and the SSH cryptosystem (including configuration). In today's article, I'll start off with how VPNs work and then concentrate on the IPSec standard.

A VPN, or Virtual Private Network, is a cryptosystem that allows you to secure your data as it travels over an insecure network such as the Internet. While this may sound similar to the SSH cryptosystem, VPNs have a different purpose. SSH was designed to allow a user to login securely to and remotely administer another computer. A VPN is designed to allow a user to access transparently the resources of a network. As far as the user is concerned, she will be able to do anything she normally would be able to do, even when she is away from the network. Because of this, VPNs are popular with telecommuters and with offices that need to share resources over physically separate locations.

VPN Tunnels

Before configuring a VPN, you should be aware of the terminology it uses and some of the configuration pitfalls. Let's start with some terminology. A VPN always consists of a point-to-point link known as a tunnel. The tunnel itself occurs over the insecure network which is usually an Internet connection. A point-to-point link means that it is always between two computers which are referred to as peers. Each peer is responsible for encrypting the data before it enters the tunnel and decrypting the data as it leaves the tunnel.

Even though the VPN tunnel is always between two peers, each peer can set up tunnels with other peers. For example, if three telecommuters were to establish a VPN to the same office, there would be three separate VPN tunnels to that office. However, each tunnel would share the same office peer. This can occur because it is possible for a peer to encrypt and decrypt data on behalf of an entire network as seen in Figure 1:

Figure 1 -- a VPN gateway to a network

When this occurs, the VPN peer is also known as a VPN gateway, and the network behind the gateway is referred to as the encryption domain. It makes sense to use a gateway for several reasons. First, all users are required to go through the same device, which makes it easier to administer a security policy and control which traffic is allowed in and out of the network. Second, it would quickly become unworkable to set up separate tunnels for every PC a user wanted to access. (Remember, a tunnel is a point-to-point link). With a gateway, the user initiates a tunnel to the gateway and is then allowed access to the network, or encryption domain, behind that gateway.

It is interesting to note that no encryption occurs within the encryption domain. This is because that portion of the network is considered to be secure and under your administrative control while the Internet is considered to be insecure and beyond your control. This also makes sense when using two VPN gateways to connect two offices. It ensures all data is encrypted as it traverses the insecure link connecting the two offices. Figure 2 shows a VPN connecting two offices:

Figure 2 -- a secure network over an insecure network

Network A is considered to be the encryption domain of VPN Gateway A, and Network B is the encryption domain of VPN Gateway B. When a user on Network A wishes to send data to Network B, VPN Gateway A will encrypt the data and send it over the VPN tunnel. VPN Gateway B will decrypt the data and send it to the destination in Network B.

Remember transport mode and tunnel mode from Cryptographic Terminology 101? Whenever two VPN gateways are used to connect two networks, they always use tunnel mode. This means that the entire IP packet is encrypted and a new IP header is added. That new IP header contains the IP addresses of the two VPN gateways. This adds an extra benefit in that a packet sniffer will only see the IP addresses of the gateways. There is no way to identify a source computer in the first encryption domain or a destination computer in the other encryption domain.

Compare this to Figure 1, which shows the other use of a VPN, allowing remote users with laptops and users who work from home to access the resources at the office network. In order for this to work, the user needs VPN client software installed on their PC to negotiate a VPN tunnel with the office VPN gateway. In this scenario, tunnel mode is still used as the remote user wants to access the resources within the encryption domain rather than the resources on the VPN gateway itself. The only time transport mode is used is when one computer wants to access another computer directly.

There are many options for VPN gateways and VPN clients. There are hardware VPN appliances and VPN software that can be installed on routers or PCs. Your FreeBSD system comes with software that allow you to set up either a VPN gateway or act as a VPN client. There are also additional applications in the ports collection, which allow interconnectivity with other VPN peers on non-FreeBSD systems.

Fortunately, there are also plenty of VPN resources, FAQs and configuration tips on the Internet. My favorites include Tina Bird's VPN Information, VPN Labs, and the Virtual Private Network Consortium (VPNC).

Regardless of the VPN software being used, all VPNs share the following behaviors:

  1. Both peers authenticate each other before establishing the tunnel to ensure the encrypted data will be sent to the expected peer
  2. Both peers require a pre-configured policy stating which protocols can be used to encrypt the data and provide data integrity
  3. The peers compare their policies to determine which algorithms will be used; if the peers are unable to agree upon the necessary algorithms, the tunnel is not established
  4. Once the policy is agreed upon, a key is created which will be used by the symmetric algorithm to encrypt and decrypt the data

There are several standards which dictate how the above occurs. You may have heard of some of them: L2TP, PPTP, and IPSec. Since IPSec is the standard that most VPNs support, and it also has the most acronyms to wade through, I'll devote the rest of this article to IPSec.


The IPSec standard was designed to add security to the IP protocol. It does this by using protocols which add additional headers, also called encapsulations, to an IP packet. Since IPSec is an Internet standard, there are RFCs (Request For Comments) for IPSec and the protocols it uses. If you are interested in the inner workings of IPSec, the following RFCs are useful references and can be found at

RFC 2401 	IPSec
RFC 2402 	AH
RFC 2406 	ESP
RFC 2409 	IKE

I'll give a brief overview of each, so you will have the necessary information you'll need to understand the next article when I demonstrate configuring an IPSec VPN using a FreeBSD system. Let's start with the acronyms and then see how they fit together to create an IPSec VPN.

AH is the Authentication Header protocol. It provides integrity by ensuring that none of the bits in the portion of the packet being protected were tampered with during transit. I won't get into the details of which portion of the packet is protected and where in the packet the AH header is added as that depends upon the encryption mode and is covered in detail, with diagrams, in the RFC. Additionally, using AH can be problematic, especially if the packet passes through a NAT device. NAT changes the IP address in an IP packet to allow a private source address to access the Internet. Since the packet changes, it won't match the AH checksum. Also, AH was designed only to provide integrity. It does not provide privacy by encrypting the contents of the packet.

ESP is the Encapsulating Security Protocol and it provides both integrity and privacy. In transport mode, the ESP header is placed between the original, clear text IP header and the TCP or UDP header. In tunnel mode, the ESP header is placed between the new IP header and the original, completely encrypted IP packet.

Related Reading

Virtual Private Networks
By Charlie Scott, Paul Wolfe, Mike Erwin

Since both AH and ESP add an additional header, they have an associated protocol ID so the IP header knows to look for an AH or an ESP header. You may remember from TCP Protocol Layers Explained that each type of header has a number associated with it. For example, TCP is 6 and UDP is 17. If you are using IPSec behind a firewall, you have to remember to create a rule allowing the AH and/or ESP protocol ID. AH has a protocol ID of 51, and ESP has a protocol ID of 50. When making your firewall rule, remember that a protocol ID is not the same as a port number.

The third protocol used by IPSec is IKE, or the Internet Key Exchange protocol. As its name suggests, this protocol is used to manage the exchange of keys between the two VPN peers. While it is possible to generate the key manually, it is better and certainly much more scalable to allow the IKE protocol to do this for you automatically. Remember, you want the keys to change often, and you don't want to rely on yourself to remember to find time to do this manually on a regular basis. However, do remember to create a rule in your firewall to allow UDP port 500 as this is the port used by IKE.

The SA, or Security Association, is the IPSec term for a connection. When a VPN is established, one SA pair is created for each protocol used (i.e. one for AH and one for ESP). SAs are created in pairs as each SA is one directional, so two are required (one for each direction). These SA pairs are stored at each peer. If your peer has SAs, it means that a VPN tunnel has successfully been established.

Since each peer is capable of establishing multiple tunnels with multiple peers, the SAs have a unique number so you can tell which SAs belong to which peers. That number is called an SPI or Security Parameter Index.

SAs are stored in a database which is called, surprisingly enough, the SAD or Security Association Database. We will see this database again when we set up an IPSec VPN.

Each IPSec peer also has a second database, the SPD or Security Policy Database. This database contains the policy you have pre-configured on the peer. Most VPN implementations allow you to create multiple policies so you can choose the algorithm combinations which are suited to each peer with which you want to establish.

What sort of configuration goes into a policy?

  1. The symmetric algorithm(s) you are willing to use to encrypt/decrypt the data
  2. The cryptographic checksum(s) you are willing to use to ensure the integrity of the data
  3. The type of authentication method you are willing to use to authenticate the peer. Pre-shared secrets or RSA digital certificates are the most common forms of authentication.
  4. Whether to use tunnel mode or transport mode
  5. Which Diffie Hellman group to use
  6. How often to re-authenticate the peer
  7. How often to exchange the key used to encrypt/decrypt the data
  8. Whether or not to use PFS
  9. Whether to use AH, ESP, or both

When creating the policy, it is usually possible to create an ordered list of algorithms and Diffie Hellman groups. The first to match on both peers will be used. Remember, it is very important to ensure that everything in the policy will allow for a match on the peer. Even if everything else in the policy matches but one portion does not match, the peers will be unable to negotiate the VPN. If you are configuring a VPN between two different types of systems, you should take the time to research which algorithms each system supports, so you can choose the most secure policy from what is available.

Phase One and Phase Two

Now, let's see how all of this works together. Establishing and maintaining an IPSec VPN tunnel occurs in two phases. In Phase One, the two peers negotiate an authentication method, encryption algorithm, hash algorithm, and Diffie Hellman group. They also authenticate each other. All of this can occur in either three clear text packets (known as aggressive mode) or in six clear text packets (known as main mode). Assuming this is successful, a Phase 1 SA (also called an IKE SA) is created and the peers move on to Phase Two.

Also in FreeBSD Basics:

Fun with Xorg

Sharing Internet Connections

Building a Desktop Firewall

Using DesktopBSD

Using PC-BSD

In Phase Two, the keying material is generated, and the two peers agree upon the policy that will be used. This mode, which is called quick mode, differs from Phase One in that it can't occur until after Phase One is successful and all of the Phase Two packets are encrypted. This makes troubleshooting a little more difficult as it is possible for Phase One to be successful but Phase Two to fail. If Phase Two is successful, it will result in a Phase 2 SA, also called an IPSec SA, and the tunnel will be fully established.

When does all of this occur? The very first time a packet arrives at a peer with a destination in another encryption domain, the peer will initiate Phase One with the peer associated with that other encryption domain. Assuming both peers successfully establish the tunnel, the tunnel will stay up, ready to receive packets. However, the peers will re-authenticate each other and re-compare their policies at a pre-configured time. This time is known as the Phase One or IKE SA lifetime.

The peers will also re-negotiate the key that is used to encrypt and decrypt the data at another pre-configured time known as the Phase Two or IPSec SA lifetime. The Phase Two lifetime is shorter than the Phase One lifetime as you do want to change the key often. A typical Phase Two lifetime is 60 minutes. A typical Phase One lifetime is 24 hours.

It is your job to ensure both peers are configured with the same lifetimes. If they are not, it is possible for the tunnel to be established initially, but then cease to work when one of the mis-matched lifetime periods arrives. You will also encounter strange problems if the Phase One lifetime is shorter than the Phase Two lifetime. If a previously working tunnel appears to hang, one of the first things to check is the two lifetime settings on each peer. Also, if you do change a policy on a peer, it won't take effect until the next time Phase One occurs. If you want your change to take effect immediately, you need to remove the SAs for that tunnel from the SAD. This will force a renegotiation using your new policy parameters.

We now have enough background information to create an IPSec VPN on your FreeBSD box. A demonstration of the required configurations is the subject of the next article.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.