Published on (
 See this if you're having trouble printing code examples

IPSec Certificate Basics

by Mike DeGraw-Bertsch

So you've got IPSec running between your hosts using preshared keys, and you want to move to X.509 certificates? It's a good idea. They're easier to manage than shared keys, and preshared keys may disappear in IPSec version 2.

It's a relatively easy move, but if you're not familiar with OpenSSL, it can get frustrating. This article will ease that frustration by providing a step-by-step guide to deploying a certificate-based authentication scheme. The article assumes you are configuring certificates for use between various hosts and a tunnel server, but the functionality and setup are identical for transport mode as well.

Quick review

Let's just run down the terms we'll be using here. X.509 certificates are based on public/private keypairs. Each certificate contains a public key, along with other information (identifying and not), such as the owner's common name and the certificate's expiration date. The owner keeps her private key in a separate file.

The certs are signed by a Certificate Authority, or CA, and contain information as to which authority signed it. This digitally proves that the certificate is authentic and that the information contained within it is accurate--perfect for verifying the identity of a remote host. The CA's authenticity is verified by its certificate, which is generally available to the public.

For more information on X.509 certificates, click here.


Running certificate-based IPSec authentication requires two things. First off is a recent version of racoon; grab the latest available in /usr/ports/security/racoon (currently 20011215a). Next is OpenSSL, version 0.9.5a or higher. The version supplied with FreeBSD by default does not have the useful script, so you should download and install the latest version, which will put in /usr/local/ssl/misc/ by default.

Creating your own CA

With OpenSSL, you can create your own certificates and even your own Certificate Authority--meaning that you can create, sign, and distribute certificates to the world. While the rest of the world may not recognize your standing as a CA ("Do I want to accept a certificate signed by Joe's Garage and Certificate Authority? No!"), at least you will recognize yourself as one. And that's all you need for hosts connecting to your tunnel server.

You may want to create your CA on the tunnel server, or you may not. It's up to you.

To start, login to whatever host you decide upon as your CA, and create a directory where you'll manage your certificates. You'll then create your CA subdirectory (demoCA) by running:

/usr/local/ssl/misc/ -newca

When prompted for the CA certificate filename, press Enter. You'll then be prompted for a password to protect the CA's private key. It's very important that the password stays safe, or anyone could sign certificates as you, so make it a good one. Next, you will be prompted to enter identifying information about your location, company, common name, and email address. It's all self-explanatory, except for the common name. This is a mandatory bit of uniquely identifying data, such as your host's FQDN or your name.

Virtual Private Networks

Related Reading

Virtual Private Networks
By Charlie Scott, Paul Wolfe, Mike Erwin

After entering the required information, the demoCA subdirectory is created. If you're paranoid, you may want to chmod the private key file (demoCA/private/cakey.pem) so that only root can read it. But this shouldn't be necessary since you used a really good password to protect it, right?

You can ignore most of demoCA's remaining contents, but you'll need to use demoCA/cacert.pem in the near future.

Creating and installing certificates

Now that you're set up to sign certificates, you can begin creating them for all the hosts using IPSec on your network. There are a few simple steps, repeated for each of the remote boxes and the tunnel server.

The first step is to create a public key and certificate request. On the CA server, run:

openssl req -new -nodes -newkey rsa:1024 -sha1 -keyform PEM -keyout privkey.pem -outform PEM -out newreq.pem

Next, the certificate must be signed. It's easiest to do this with the -sign command. Give the CA's password, then respond with y to the prompts, indicating that you want to sign the certificate.

You'll now have three files to copy to the appropriate machine: demoCA/cacert.pem, newcert.pem, and privkey.pem. Put them in /usr/local/etc/racoon/cert on the client you are configuring. Make sure the directory is only readable by root; otherwise, malicious parties could steal your private key and masquerade as you!

The final step is to allow the host to recognize other certificates signed by your CA. Login to the machine and run:

cd /usr/local/etc/racoon/cert
ln -s cacert.pem `openssl x509 -noout -hash -in cacert.pem`.0

Make sure to replace cacert.pem with whatever you named the file. The symbolic link allows OpenSSL to find your CA's certificate based on its hash, thus verifying that the certificate is authentic and acceptable.

Repeat the above steps for all hosts that you want to have connect to your tunnel server.

Racoon configuration

Your last remaining task is to configure racoon to accept the certificates. Edit your racoon configuration file, typically /usr/local/etc/racoon/racoon.conf.

First, make sure racoon knows where to find your certificates:

path certificate "/usr/local/etc/racoon/cert" ;

Next, you'll need to change the remote block(s) to use and accept certificates. The necessary changes are bolded below for IPSec over IPv4 on anonymous connections. The changes are the same for IPv6 and identified connections.

remote anonymous
        exchange_mode main,aggressive;
        doi ipsec_doi;
        situation identity_only;

        my_identifier asn1dn;
        peers_identifier asn1dn;
        verify_identifier on;
        certificate_type x509 "newcert.pem" "privkey.pem";

        nonce_size 16;
        lifetime time 1 min;    # sec,min,hour
        support_mip6 on;
        proposal_check strict;  # obey, strict or claim

        proposal {
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method rsasig ;
                dh_group 2 ;

my_identifier asn1dn tells racoon that the local host should identify itself by using its Distinguished Name (DN) as defined by its certificate (that is, all of the fields in its certificate strung together). peers_identifier asn1dn states that the remote host should identify itself with its certificate's DN. verify_identifier on ensures that the identification the peer sends is the same type that is specified in peers_identifier; in this case, it's asn1dn. The certificate_type line tells racoon to use an X.509 certificate. cert.pem is the name of the local host's certificate file, while key.pem is its private key. These file names should be changed to whatever you named them.

The last line to change, authentication_method rsasig, tells racoon that it should use certificate-based identification of its peers.

Once you've made the necessary changes on each host's racoon.conf, you're ready to go. The next section discusses how to proceed and gives some troubleshooting tips.

Running and troubleshooting

Kill the racoon daemon on your server and client. Restart it, and tail your logfile. Ping one host from the other. You should see lines similar to:

2002-01-27 02:12:46: INFO: isakmp.c:1681:isakmp_post_acquire(): IPsec-SA request for queued due to no phase1 found.
2002-01-27 02:12:46: INFO: isakmp.c:795:isakmp_ph1begin_i(): initiate new phase 1 negotiation:[500]<=>[500]
2002-01-27 02:12:46: INFO: isakmp.c:800:isakmp_ph1begin_i(): begin Identity Protection mode.
2002-01-27 02:12:46: INFO: pfkey.c:1365:pk_recvexpire(): IPsec-SA expired: ESP/Tunnel> spi=122493290(0x74d196a)
2002-01-27 02:12:46: INFO: vendorid.c:128:check_vendorid(): received Vendor ID: KAME/racoon
2002-01-27 02:12:46: INFO: vendorid.c:128:check_vendorid(): received Vendor ID: KAME/racoon
2002-01-27 02:12:46: INFO: isakmp.c:2409:log_ph1established(): ISAKMP-SA established[500]-[500] spi:0dab7579614a1047:50c0edf86b1a
2002-01-27 02:12:47: INFO: isakmp.c:939:isakmp_ph2begin_i(): initiate new phase 2 negotiation:[0]<=>[0]
2002-01-27 02:12:47: INFO: pfkey.c:1107:pk_recvupdate(): IPsec-SA established: ESP/Tunnel> spi=23298201(0x1638099)
2002-01-27 02:12:47: INFO: pfkey.c:1319:pk_recvadd(): IPsec-SA established: ESP/Tunnel> spi=52337518(0x31e9b6e)

If you don't, there are a few things to check. First, make sure your path and file names are specified correctly in racoon.conf. Next, check that you created a symbolic link, mentioned above in "Creating and installing certificates." Double-check your configuration, making sure your identifier and your peer's are set correctly, along with any other lines you changed (certificate related or not.) Finally, check your security policies and make sure they still match.


In this article, you learned all the basics necessary to use X.509 certificates for authentication in IPSec on a FreeBSD box. You're now ready to enjoy the benefits, and you don't have to worry about creating, changing, and remembering all of your shared keys. Further, since the IETF may discontinue preshared keys in the next version of IPSec, it's one less headache if you upgrade (once IPSec version 2 comes out, of course).

Mike DeGraw-Bertsch is a security and Unix system administration consultant in the Boston, Mass. area. When he's not at a job, writing, hacking with Perl, or playing with his wireless network, he can usually be found playing goal in ice hockey.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.