The Internet is no longer the cute and fluffy cloud it once was. Protecting your servers, workstations, and networks can only go so far. Attacks that consume your available Internet-facing bandwidth or overpower your router's CPU can still take you offline. This article will help you mitigate the effects of such attacks, guiding you in what to do if you are attacked.
The techniques here apply equally well to FreeBSD 4.x and 5.x.
Denial of Service (DoS) attacks set out to remove a service from functional use by its clients. Web servers will stop serving web pages, email servers will stop accepting or delivering email, and routers will go dark, taking you off the Internet all together.
Denial of a particular service will come in one of two forms:
Over the last few years, attackers have refined their methods. As developers make software more reliable and more resilient to DoS, the attack vectors have changed to target hard-to-secure parts of a service. We'll discuss the first type of attack and what we can do to protect our services from it.
Protecting your services from attack is similar to tuning your services for maximum performance. The greater the load you can handle, the more resilient you are. Things change slightly when the attack alters the profile of your service.
For example, if you have a web server tuned to transfer large files and the attack forces through a lot of small, short-lived transactions, you could find
you run out of network memory buffers very quickly. I would recommend starting
by reading the papers on tuning FreeBSD for different applications. The paper describes good ways to start tuning your servers. Also, the
man page is an excellent resource on performance improvements.
The first step to protecting yourself from an attack is to understand the nature of different types of attacks. As we said earlier, resource-consumption attacks target your system in places that can cause bottlenecks. The most popular targets are network bandwidth, system memory, network stack memory, disk I/O, operating system limitations such as a limit on the number of open file handles, and the CPU. These bottlenecks can be on your systems or in your network hardware.
Attacks against your network bandwidth are difficult to defend. How you deal with them depends heavily on your network topology and how helpful your ISP is. Start by asking the following questions:
We are fortunate today that most attacks are simple in their nature. They choose one or two styles of attack and at most a small number of IP addresses. This makes sense — bandwidth is as hard for attackers to acquire as it is for us to defend. If your Internet-peering bandwidth is not saturated, the accepted approach is to block traffic to the attacked host(s) at your gateway.
It's a good idea to run tcpdump on the attacked servers if you can, to see what kind of attack is taking place. Look for floods of very similar packets — all TCP SYN, UDP, or ICMP. Look for packets all headed for a particular port. If you find the number of source IP addresses is reasonably small, it may be possible to block packets based on source address.
However, if the source addresses are highly volatile in addressing, this can indicate spoofing or forging. When this is the case, you may need to look for other similarities in the attack such as packet size, window size, fragmentation, etc. If you have the ability to block based on these less common criteria you may want to investigate here further.
With modern, multi-gigabit networks, it is not unusual for an Internet connection to have more bandwidth than the local LAN, so it may be possible for you to block the attack at your Internet gateway. More often than not though, this does not apply. Having your Internet bandwidth consumed can be tiring and frustrating. This might be the right time to call your ISP, if they're willing to work with you on these problems.
Before you make the call, try to analyze the attack. This will help your ISP in selectively filtering the attack off your network. If filtering is possible, you'll have one of two common options available: selectively filtering out the attacking systems or dropping all packets to the attacked servers. The latter is easier to manage and is more effective in the event that the attack profile changes against those hosts.
If you run Border Gateway Protocol (BGP) at your Internet gateway to announce your IP space to the Internet, you may have a third option, one that UUNet, C&W, XO, and many ISPs allow users to export routes as small as /32 with a special community string that causes their border routers to drop all incoming data for the route. This is a highly effective method of dropping an attack with the least damage to yourself and your ISP. Of course, this only works well with a small number of hosts under attack and if your ISP offers such functionality. Contact your ISP to find out. The obvious downside of this is that the IP addresses you export in this fashion will lose all connectivity to the Internet.
In general it is a good idea to keep your network clean; only allow the traffic that your services need to operate. Allow TCP to ports 80 and 443 on your servers and allow UDP to your game servers. Allow SSH connections only from trusted hosts. All of these limit the options of the attackers when they come to visit.
If your bandwidth is not saturated, the attack is most likely against your systems and the services they host, rather than your entire network. Again, the remedy depends on the nature of the attack. You may find that any one system exposes multiple targeted bottlenecks. Attacks on systems and their services generally fall into the following categories:
System-targetting attacks are very frustrating, as they're hard to defend against. FreeBSD does have some special defensive magic, however.
By default, each time your network card receives a packet, it generates an interrupt to the CPU along its IRQ. The CPU will catch this and dedicate a small amount of time to fetch this packet from the interface. Under normal operations this can happen several thousand times per second — well within the capabilities of even low end CPUs. It is quite likely with older CPUs that you will start to see performance impacts between 25,000 to 50,000 packets per second. With packet sizes of 1,500 bytes, this works out to around 40Mbytes/sec to 75Mbytes/sec, quite a lot for most older CPUs to serve anyway. Most 1Ghz systems will begin to feel pressure around 75,000 packets per second. Two factors exacerbate the problem:
TCP SYN packets require full processing before the system can respond with SYN ACK packets back to the source address. TCP and UDP packets to closed ports as well as ICMP packets in general also need similar processing and replies. While not as expensive as SYN processing, this still takes time and consumes outbound bandwidth.
Packet size also plays an important factor. You can fit more small packets in a particular amount of bandwidth than you can large packets. The more packets you take in, the more CPU time you need to process them, no matter what type of packets they are.
As we discussed previously, each interrupt request eats up some CPU time. With enough IRQs generated, the CPU will have no time to do anything other than serve the interrupts. Inbound packets stay unprocessed, applications receive no CPU time, and your system is effectively dead in the water. This is known as "Live-lock." Your system is still live, inasmuch as it has not crashed, but it cannot perform any useful functions. Once packets stop coming in to the interface, the CPU starts to process all of the backlogged packets it has already accepted. This can take anything from a few minutes to several hours.
There are several things you can do to prevent or mitigate the effects of a
high rate of packets without buying any hardware upgrades. All of these use
command. Here are the settings you will need to place in
net.inet.tcp.msl defines the Maximum Segment Life. This is the
maximum amount of time to wait for an ACK in reply to a SYN-ACK or FIN-ACK, in
milliseconds. If the computer does not receive an ACK in this time, it
considers the segment lost and frees the network connection.
This has two implications. When you are trying to close a connection, if the final ACK is lost or delayed, the socket will close more quickly. However, if a client is trying to open a connection to you and their ACK is delayed more than 7,500 ms, the connection will not form. RFC 753 defines the MSL as 120 seconds (120,000 ms). However, this was written in 1979; timing issues have changed slightly since then. Today, FreeBSD's default is 30,000 ms. This is sufficient for most conditions, but for stronger DoS protection you can lower this to 7,500 or less.
net.inet.tcp.blackhole defines what happens when the system
receives a TCP packet on a closed port. When set to
1, SYN packets
arriving on a closed port will be dropped without a RST packet being sent back.
When set to
2, all packets arriving on a closed port are dropped
without an RST being sent back. This saves CPU time, because packets don't need
as much processing, and outbound bandwidth, by not sending out
net.inet.tcp.blackhole in its function. As the UDP protocol does
not have states like TCP, there is only one choice when it comes to dropping
UDP packets. When
system will drop all UDP packets that arrive on a closed port.
net.inet.icmp.icmplim is somewhat misleading. This
controls the maximum number of ICMP "Unreachables" and also TCP RST packets to
return every second. It helps curb the effects of attacks that generate a lot
of reply packets.
kern.ipc.somaxconn limits the maximum number of concurrently
open sockets. The default here is just 128. If an attacker can flood you with a
sufficiently high number of SYN packets in a short enough period of time, he
can use up all of your possible network connections, successfully denying your
users access to the service.
You may find these settings to be either too aggressive or not aggressive enough. Tune them until you receive satisfactory results.
Finally, if you are blessed enough to own one of the following network cards, you can enable a kernel feature call
DEVICE_POLLING changed interrupt handling; with
DEVICE_POLLING, the kernel does not handle them at all! Instead,
at certain times, the CPU will poll the network card to pick up any packets that
are waiting for processing. This can significantly reduce the amount of CPU
time used in processing inbound traffic. This only works with the above cards,
as their drivers must support
The FXP cards generally work best with this feature, as their drivers and hardware are very well developed. The hardware design and quality of RL cards is a lot lower — without sufficient CPU (usually around 1Ghz), they have a hard time achieving the full 100MB/s at all. Keep this in mind if you are looking for a new network card.
You can learn more about
DEVICE_POLLING at the author's home page. You can also find good installation and tuning instructions there, as well as statistics from comparative tests with
DEVICE_POLLING enabled and disabled.
Attacks can come from inside and outside your network. Obviously one is easier to isolate than the other. Tracking the sources of attacks requires some familiarity with packet-sniffing tools such as tcpdump, ngrep, and ethereal. Unless you have spent several months carefully profiling your network traffic and set up monitoring specifically to alert you of anomalies, the chances of discovering you are under Denial of Service conditions before someone else does are slim. More often than not, complaints such as "The Internet is slow" or "I can't get my email" will lead you to the truth. It is important to realize two things:
What does this mean to you? It means that when you start to look for why your Internet is slow or why people cannot download their email, remember that the source of the problem could be from any machine on your network or the Internet. If there's a denial, it may be accidental.
A good place to start is the point of bottleneck. This could be the CPU on your HTTP proxy or your Internet gateway. If your bottleneck is a system process such as a proxy server, examine the logs for this. Is a single system or small number of systems making an unusually large number of requests or using more resources than normal?
If your bottleneck is your Internet gateway (which we assume is running FreeBSD), you can use the following command to view the IP packets passing through your gateway:
router# tcpdump -n -i <interface> -c 100
This command will display a summary of the first 100 packets (
100) it sees on the given <interface> (
<interface>) and will not resolve the IP addresses to
host names (
-n), which can take extra time and may itself fail if
you are having connectivity issues. An example output line will resemble:
04:59:53.915324 192.168.0.3.2327 > 192.168.0.10.1214: S 3199611726:3199611726(0) win 16384 <mss 1460,nop,nop,sackOK> (DF)
Let us look at the first few parts of this output, which can be useful to us.
This is the timestamp of when the packet was processed.
This is the source IP address. The numbers after the last octet, 2327, indicate the packet's source port number.
This is the destination IP address. The numbers after the last octet, 1214, indicate the destination port number.
This indicated the type of packet, in this case a SYN packet. See Daryl's TCP/IP Primer to learn about the process and life of a TCP connection and the types of packets you would see here.
What you may see during an actual attack is hard to predict, as Denial of Service attacks come in so many shapes and sizes. A typical attack involves flooding a listening port on your server with SYN packets. The idea is to make your system so busy processing the new connections that it cannot do anything else. Here you may see a large number of SYN packets. Normally, you'll see a balance of packets of all types.
This is a list of references I found when writing this article. Most of them contain much more detail on their particular subject than I have provided here. Hopefully they will be as useful to you as they were to me.
The FreeBSD handbook. A great source of information about FreeBSD. Have a read here if you want to learn something new about FreeBSD.
Google Groups. I used this extensively to search USENET for articles others had posted in the past. You should too!
HTMLHelp.com is made by the Web Design Group to "promote the creation of non-browser specific, non-resolution specific, creative, and informative sites that are accessible to all users worldwide." It was valuable in the creation of this site.
Avleen Vig is a Systems Administrator at Google.
Return to the BSD DevCenter.
Copyright © 2009 O'Reilly Media, Inc.