BSD DevCenter
oreilly.comSafari Books Online.Conferences.


Cryptosystems: Debugging IPSec
Pages: 1, 2, 3


This is a common racoon log error message:

2002-12-29 12:48:31: NOTIFY: oakley.c:2040:oakley_skeyid(): 
couldn't find the proper pskey, try to get one by the peer's address.
2002-12-29 12:48:31: ERROR: oakley.c:2054:oakley_skeyid(): couldn't find 
the pskey for B.B.B.B.

Fortunately, this one is fairly straightforward as it indicates a problem with /usr/local/etc/racoon/psk.txt. On both peers, check that:

  • The file contains the external IP address of the other peer
  • The password is the same
  • The permissions are set correctly

You don't have to restart racoon once you've fixed this error. As soon as the typo/permission problem is fixed, Phase One will continue.

If you don't receive any tcpdump output at all, there aren't any IKE packets being sent. This usually means that you forgot to allow UDP port 500 on your firewall, or forgot to tell your firewall about the changes to your ruleset, so your firewall is discarding those packets. However, it could also indicate a routing or policy misconfiguration. Remember, your policy tells racoon when it is required to negotiate a tunnel.

For example, part of my policy in /etc/ipsec.conf states:

spdadd any -P out ipsec

This means that if I try to send packets (from a ping, for example) to a host with an IP address starting with 192.168.1, those packets won't be sent until IKE has negotiated a tunnel. It also means that if I send packets to any other network, IKE does not come into play. If tcpdump shows that IKE isn't being used, double-check that your policy does require ipsec for the remote network.

Sometimes a tcpdump will show your packets going out, but no replies coming back. This usually indicates that the firewall at the peer is discarding the packets. Check the firewall rules at the peer to ensure they allow IKE and ESP.

Sometimes Phase One will fail, and you can't quite tell why from the racoon log. When this happens, I bring out the big guns and use the /usr/ports/net/ethereal utility. (See Using Ethereal) This allows me to analyze the policy negotiations in great detail, as it records each policy parameter as sent by each peer. If Phase One is failing, it will be because of a policy mismatch or a peer authentication failure. If it is a policy mismatch, you should find it in the "remote" section of racoon.conf. However, if the remote sections are identical, you'll really need that packet sniff to pinpoint exactly what is happening.

Advanced Troubleshooting

Things get a lot more interesting if Phase One succeeds but Phase Two fails. Once Phase One is finished, all of the data is encrypted. A packet sniff won't help you in pinpointing which policy parameter is failing to match up. If both peers are running racoon, the culprit will most likely be a typo or mismatch in the "sainfo" section of racoon.conf. However, if the other peer is not running racoon, you may have some detective work ahead of you.

Many commercial IPSec implementations include additional features. It's unfortunate that some of those features are only supported by that particular product and are usually enabled by default. If Phase One works, Phase Two hangs, and you know you've configured both peers with identical parameters, you've most likely stumbled upon such a feature. Scour that product's documentation and vendor's knowledge base looking for default configurations and how to disable them.

Failing that, you may be able to find working configs between racoon and the particular product you are struggling with. I've found the Interoperability Profiles at VPNC to be useful.

The maintainers of racoon also maintain a searchable list of known and solved issues. They also provide prompt answers to queries.

Finally, if your blood, sweat, and tears results in a working tunnel, be kind and post your how-to somewhere on the Internet. You just might make a harried admin's day.

Interactions with Anti-Spoofing Rules

The last scenario I want to discuss in this article deals with VPNs and anti-spoofing rules. Depending upon the firewall(s) you are using to protect the gateways, this may or may not become an issue. This scenario can be the most frustrating as IKE is able to successfully establish the VPN. Unfortunately, all of the packets that enter the resulting tunnel end up being discarded by the firewall's anti-spoof rules.

Related Reading

Virtual Private Networks
By Charlie Scott, Paul Wolfe, Mike Erwin

If you're unfamiliar with anti-spoof rules, they are the set of rules that drop all traffic entering a network from a private address range. Normally, this is useful, as most legitimate traffic coming in from the Internet can't have a source address in the 10.x.x.x, 172.16.x.x-172.31.x.x, or 192.168.x.x ranges as Internet routers drop those addresses. A VPN tunnel operates differently. While in the tunnel, the packets have the Internet routable addresses of the peers' external interfaces. However, the data in the packet itself is destined for an internal address, which is usually within the private address range.

Whether or not this is a problem depends upon the firewall and the order in which packet processing occurs. Several factors are at play here: a packet needs to be encrypted or decrypted, it must be NATed, and it also needs to be compared to a firewall ruleset to see if the packet is allowed to pass. If the NATing occurs in the wrong place, it is quite possible that a legitimate packet will be mistaken for a malicious, spoofed packet.

How do you resolve the situation if your particular firewall's processing order is working against you? Sometimes trial and error in rule order will work. You've already created rules to allow the VPN to be established; those were the IKE and ESP rules. You'll still need rules allowing your users to access the services you wish to allow through the tunnel, otherwise you wouldn't have created a tunnel in the first place. Try placing those rules towards the top of a rulebase. Include the "quick" word in the case of an ipfilter rulebase.

If that doesn't help, it may be time to reconsider the usefulness of the anti-spoof rules in regards to your particular security policy. Balance the need for the VPN against the benefits gained from the anti-spoof rules. If you log those rules for a week or two, you'll have an idea of how much those rules are used and how much protection they offer.

If your security policy requires the anti-spoof rules, there is some information that can be gleaned from the Internet, which might apply to your situation. As firewalls mature, they gain extra features. There may indeed be an extra switch or keyword that you can place in your rulebase that will resolve the anti-spoof problem. That extra knob may be in the works and available in another six months.

I'd like to leave you with some resources, which I have found to be helpful:

I hope you have enjoyed the cryptosystems series. I had a chance over the holidays to go through my "things I want to try out if I ever have more time" list. I'll share some of the results in the next article.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Sponsored by: