ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Preventing Denial of Service Attacks

by Avleen Vig
06/24/2004

The Internet is no longer the cute and fluffy cloud it once was. Protecting your servers, workstations, and networks can only go so far. Attacks that consume your available Internet-facing bandwidth or overpower your router's CPU can still take you offline. This article will help you mitigate the effects of such attacks, guiding you in what to do if you are attacked.

The techniques here apply equally well to FreeBSD 4.x and 5.x.

Different Types of Attacks

Denial of Service (DoS) attacks set out to remove a service from functional use by its clients. Web servers will stop serving web pages, email servers will stop accepting or delivering email, and routers will go dark, taking you off the Internet all together.

Denial of a particular service will come in one of two forms:

Over the last few years, attackers have refined their methods. As developers make software more reliable and more resilient to DoS, the attack vectors have changed to target hard-to-secure parts of a service. We'll discuss the first type of attack and what we can do to protect our services from it.

Make the Most of your Services

Protecting your services from attack is similar to tuning your services for maximum performance. The greater the load you can handle, the more resilient you are. Things change slightly when the attack alters the profile of your service.

Related Reading

BSD Hacks
100 Industrial Tip & Tools
By Dru Lavigne

For example, if you have a web server tuned to transfer large files and the attack forces through a lot of small, short-lived transactions, you could find you run out of network memory buffers very quickly. I would recommend starting by reading the papers on tuning FreeBSD for different applications. The paper describes good ways to start tuning your servers. Also, the tuning(7) man page is an excellent resource on performance improvements.

Analyzing and Blocking Denial of Service

The first step to protecting yourself from an attack is to understand the nature of different types of attacks. As we said earlier, resource-consumption attacks target your system in places that can cause bottlenecks. The most popular targets are network bandwidth, system memory, network stack memory, disk I/O, operating system limitations such as a limit on the number of open file handles, and the CPU. These bottlenecks can be on your systems or in your network hardware.

Attacks on Bandwidth

Attacks against your network bandwidth are difficult to defend. How you deal with them depends heavily on your network topology and how helpful your ISP is. Start by asking the following questions:

We are fortunate today that most attacks are simple in their nature. They choose one or two styles of attack and at most a small number of IP addresses. This makes sense — bandwidth is as hard for attackers to acquire as it is for us to defend. If your Internet-peering bandwidth is not saturated, the accepted approach is to block traffic to the attacked host(s) at your gateway.

It's a good idea to run tcpdump on the attacked servers if you can, to see what kind of attack is taking place. Look for floods of very similar packets — all TCP SYN, UDP, or ICMP. Look for packets all headed for a particular port. If you find the number of source IP addresses is reasonably small, it may be possible to block packets based on source address.

However, if the source addresses are highly volatile in addressing, this can indicate spoofing or forging. When this is the case, you may need to look for other similarities in the attack such as packet size, window size, fragmentation, etc. If you have the ability to block based on these less common criteria you may want to investigate here further.

With modern, multi-gigabit networks, it is not unusual for an Internet connection to have more bandwidth than the local LAN, so it may be possible for you to block the attack at your Internet gateway. More often than not though, this does not apply. Having your Internet bandwidth consumed can be tiring and frustrating. This might be the right time to call your ISP, if they're willing to work with you on these problems.

Before you make the call, try to analyze the attack. This will help your ISP in selectively filtering the attack off your network. If filtering is possible, you'll have one of two common options available: selectively filtering out the attacking systems or dropping all packets to the attacked servers. The latter is easier to manage and is more effective in the event that the attack profile changes against those hosts.

If you run Border Gateway Protocol (BGP) at your Internet gateway to announce your IP space to the Internet, you may have a third option, one that UUNet, C&W, XO, and many ISPs allow users to export routes as small as /32 with a special community string that causes their border routers to drop all incoming data for the route. This is a highly effective method of dropping an attack with the least damage to yourself and your ISP. Of course, this only works well with a small number of hosts under attack and if your ISP offers such functionality. Contact your ISP to find out. The obvious downside of this is that the IP addresses you export in this fashion will lose all connectivity to the Internet.

In general it is a good idea to keep your network clean; only allow the traffic that your services need to operate. Allow TCP to ports 80 and 443 on your servers and allow UDP to your game servers. Allow SSH connections only from trusted hosts. All of these limit the options of the attackers when they come to visit.

Attacks on Systems and Services

If your bandwidth is not saturated, the attack is most likely against your systems and the services they host, rather than your entire network. Again, the remedy depends on the nature of the attack. You may find that any one system exposes multiple targeted bottlenecks. Attacks on systems and their services generally fall into the following categories:

System-targetting attacks are very frustrating, as they're hard to defend against. FreeBSD does have some special defensive magic, however.

By default, each time your network card receives a packet, it generates an interrupt to the CPU along its IRQ. The CPU will catch this and dedicate a small amount of time to fetch this packet from the interface. Under normal operations this can happen several thousand times per second — well within the capabilities of even low end CPUs. It is quite likely with older CPUs that you will start to see performance impacts between 25,000 to 50,000 packets per second. With packet sizes of 1,500 bytes, this works out to around 40Mbytes/sec to 75Mbytes/sec, quite a lot for most older CPUs to serve anyway. Most 1Ghz systems will begin to feel pressure around 75,000 packets per second. Two factors exacerbate the problem:

As we discussed previously, each interrupt request eats up some CPU time. With enough IRQs generated, the CPU will have no time to do anything other than serve the interrupts. Inbound packets stay unprocessed, applications receive no CPU time, and your system is effectively dead in the water. This is known as "Live-lock." Your system is still live, inasmuch as it has not crashed, but it cannot perform any useful functions. Once packets stop coming in to the interface, the CPU starts to process all of the backlogged packets it has already accepted. This can take anything from a few minutes to several hours.

There are several things you can do to prevent or mitigate the effects of a high rate of packets without buying any hardware upgrades. All of these use FreeBSD's sysctl(8) command. Here are the settings you will need to place in /etc/sysctl.conf:

You may find these settings to be either too aggressive or not aggressive enough. Tune them until you receive satisfactory results.

Finally, if you are blessed enough to own one of the following network cards, you can enable a kernel feature call DEVICE_POLLING:

DEVICE_POLLING changed interrupt handling; with DEVICE_POLLING, the kernel does not handle them at all! Instead, at certain times, the CPU will poll the network card to pick up any packets that are waiting for processing. This can significantly reduce the amount of CPU time used in processing inbound traffic. This only works with the above cards, as their drivers must support DEVICE_POLLING.

The FXP cards generally work best with this feature, as their drivers and hardware are very well developed. The hardware design and quality of RL cards is a lot lower — without sufficient CPU (usually around 1Ghz), they have a hard time achieving the full 100MB/s at all. Keep this in mind if you are looking for a new network card.

You can learn more about DEVICE_POLLING at the author's home page. You can also find good installation and tuning instructions there, as well as statistics from comparative tests with DEVICE_POLLING enabled and disabled.

Tracking the Source of the Attack

Attacks can come from inside and outside your network. Obviously one is easier to isolate than the other. Tracking the sources of attacks requires some familiarity with packet-sniffing tools such as tcpdump, ngrep, and ethereal. Unless you have spent several months carefully profiling your network traffic and set up monitoring specifically to alert you of anomalies, the chances of discovering you are under Denial of Service conditions before someone else does are slim. More often than not, complaints such as "The Internet is slow" or "I can't get my email" will lead you to the truth. It is important to realize two things:

What does this mean to you? It means that when you start to look for why your Internet is slow or why people cannot download their email, remember that the source of the problem could be from any machine on your network or the Internet. If there's a denial, it may be accidental.

A good place to start is the point of bottleneck. This could be the CPU on your HTTP proxy or your Internet gateway. If your bottleneck is a system process such as a proxy server, examine the logs for this. Is a single system or small number of systems making an unusually large number of requests or using more resources than normal?

If your bottleneck is your Internet gateway (which we assume is running FreeBSD), you can use the following command to view the IP packets passing through your gateway:

router# tcpdump -n -i <interface> -c 100

This command will display a summary of the first 100 packets (-c 100) it sees on the given <interface> (-i <interface>) and will not resolve the IP addresses to host names (-n), which can take extra time and may itself fail if you are having connectivity issues. An example output line will resemble:

04:59:53.915324 192.168.0.3.2327 > 192.168.0.10.1214:
	S 3199611726:3199611726(0) win 16384 <mss 1460,nop,nop,sackOK> (DF)

Let us look at the first few parts of this output, which can be useful to us.

What you may see during an actual attack is hard to predict, as Denial of Service attacks come in so many shapes and sizes. A typical attack involves flooding a listening port on your server with SYN packets. The idea is to make your system so busy processing the new connections that it cannot do anything else. Here you may see a large number of SYN packets. Normally, you'll see a balance of packets of all types.

References and Credits

This is a list of references I found when writing this article. Most of them contain much more detail on their particular subject than I have provided here. Hopefully they will be as useful to you as they were to me.

Avleen Vig is a Systems Administrator at Google.


Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.