Published on (
 See this if you're having trouble printing code examples

Big Scary Daemons

Controlling Bandwidth


A buddy over at my favorite small ISP called me the other day: "Our T1 is running really, really slow!" I tried a traceroute from the outside world. While I hit the backbone's border router in 50 milliseconds, my packets took over 2 seconds to cross the T1 and hit the ISP's router.

The network monitor showed that his T1 was actually doing just fine. After all, it was pushing about 1.544 megabits out and about 20 megabits in -- excellent throughput for a circuit that was running at capacity. The traffic had been averaging about half a T1 just the day before -- something had obviously changed.

Because every web server on the network is monitored, it was trivial to track down the offender. One of the web servers accounted for all that traffic. Five minutes with WebTrends gave him everything we needed, and we checked the site in question.

Nope, not porn. No MP3s. None of the usual suspects, just a small company with the typical half-dozen page site, a guestbook CGI, and a banner ad.

The banner ad was consuming a full megabit a second. The ultra-high-end FreeBSD server -- Celeron 433-MHz processor, 128 Mbytes of RAM, 20-Gbyte IDE hard disk -- wasn't even noticing the strain, but the T1 was buckling and every other site was slower than Microsoft admitting wrongdoing in the antitrust suit.

These days, a small Web company stays in business by maintaining a high level of customer service. My friend called the user. The client appreciated advance warning of the outrageous bandwidth charge he was going to get, and asked that we help trim the bandwidth used to a reasonable level.

And herein lies the problem. How do you control bandwidth usage? One thought rattled at the back of my brain, but I pushed it aside in favor of something that wouldn't touch the kernel.

We first checked in Apache itself. Apache has a module to control bandwidth consumption, mod_throttle (/usr/ports/www/mod_throttle). You can easily install it just like any other port.

The mod_throttle configuration is very simple. First, edit httpd.conf to include the mod_throttle configuration file.

Include /usr/local/etc/apache/throttle.conf

The throttle.conf file starts off by telling Apache to load the throttle module, and to add it to the system. Reading the mod_throttle documentation, we quickly arrive at a basic setup.

LoadModule throttle_module    libexec/apache/
AddModule mod_throttle.c

ThrottleUser websiteusername speed 300 60 m

This worked, but was rather jerky. Pages would sometimes load quickly, sometimes slowly, and frequently be flat-out denied. It would be better to enable a smooth bandwidth restriction -- say, hook up this web site with a 128-Kbps bottleneck and let Apache behave normally. I've done this before by connecting an entire server to the network via PPP over a serial cable, but you can't disconnect one virtual domain from the rest of the server.

Also in Big Scary Daemons:

Running Commercial Linux Software on FreeBSD

Building Detailed Network Reports with Netflow

Visualizing Network Traffic with Netflow and FlowScan

Monitoring Network Traffic with Netflow

Information Security with Colin Percival

It's quite possible that mod_throttle could have been tweaked to perform better. I also suspect that on a large installation, the law of large numbers would have taken over. Circuit bandwidth would not have been irregular, because every throttled site would be behaving similarly.

Also, this Apache daemon was already heavily patched -- mod_ssl, FrontPage, and php4. I'm no Apache guru, and adding yet another module into the mix isn't in my comfort zone. The simpler we keep the systems, the better.

The next thought was to try shaping the traffic. The ISP has a Cisco router, after all. Ciscos can do this sort of thing. That's what you buy that expensive service contract for.

It turned out that we could have done this exactly as we liked on the Cisco -- if we upgraded to a more recent IOS. And added memory. Oh, you can't add more memory to a Cisco 2501? Would you like to speak to a salesman?

This is when that niggling thought in the back of my head stood up and screamed, "Dummynet!"

Dummynet was originally invented to simulate poor or lossy links, so the author could test network protocols in bad conditions. However, it also allows you to control bandwidth usage. One example the developer uses on his web page is using dummynet to simulate an ADSL link to the moon! We don't want to throttle our client that badly, but it's nice to know it's possible.

Articles about IPFW

BSD Firewalls: IPFW

BSD Firewalls: IPFW Rulesets

IPFW Logging

Monitoring IPFW Logs

One nice thing about dummynet is that it works on arbitrary ports, IP addresses, and protocols. If you want to restrict the bandwidth usage of IPSec tunnels, Sendmail, or anything, you can do it.

Dummynet is part of IPFW. You can control all sorts of packet flows with dummynet. One of the most basic controls is the amount of traffic through a given IP address.

The IPFW integration was my problem. I'm loathe to muck with the kernel of a working production system. It's just not a good idea. Things have only gone wrong a couple of times, but those failures have been pretty spectacular. New kernel problems might only take a couple of hours to fix, but ISPs don't get downtime. Customers don't like outage explanations like "rebuilding the kernel." Their eyes glaze over, then and they find another ISP that doesn't use those "kernel" thingies.

You must have IPFW in your kernel to use dummynet. I added the following configuration to the system kernel.

options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         DUMMYNET

Technically I didn't need the IPFIREWALL_VERBOSE statement, but I like logs. They give me a warm fuzzy feeling. I've been known to keep logs for 30 months at a time (generally when I thought I was rotating them daily, but was actually rotating them once a month).

Because we're not using IPFW for packet filtering, but for bandwidth control, I'm using the default accept mode. If you're doing packet filtering, you probably don't want this.

We can use two different types of dummynet filters -- pipes and queues. For this example, we'll use pipes. We'll look at queues some other time.

Dummynet expects to have some outside agency forward packets to it. IPFW includes this functionality. For one dummynet operation, you need at least two IPFW configuration statements. If you aren't familiar with IPFW commands, check out the FreeBSD Basics IPFW series.

You configure dummynet, like all IPFW subsystems, with the ipfw(8) command. First, redirect the packets from the IP address of the site ( to a dummynet pipe.

ipfw add 100 pipe 1 ip from to any

The syntax is pretty straightforward. We're adding a rule, number 100, and giving it a pipe. Packets that match ip from to any -- packets from this web site -- will be sent to pipe number one.

Our pipe configuration is really very basic, too.

ipfw pipe 1 config bw 128Kbit/s

The pipe is created and labeled. The config statement tells IPFW that this is a configuration statement. The bw means that this is a bandwidth control and 128 Kbps is the bandwidth allowed.

The most disconcerting thing was when I checked the IPFW rule list.

# ipfw list
00100 pipe 1 ip from to any
65535 allow ip from any to any

Where's my pipe?

It turns out that the pipes are stored in a separate list. To view your pipes, do ipfw pipe list.

# ipfw pipe list
00001: 128.000 Kbit/s    0 ms   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  128050681 35518324182  0    0 50486587 #

Wow, impressive! We can see where traffic is coming from, where it's going to, and how many packets have been passed and dropped.

Our rules covered the straightforward dummynet configuration. There's many more options that we'll look at another time. If you're interested in experimenting with network congestion, dummynet is your friend.

I put this rule in and the bandwidth instantly dropped. My friend went away happy, his client didn't have to pay for a T1 for a month, and I got back to changing the rodent cages, watering the fish, and cleaning the garage.

One of the nice things about FreeBSD is that you get your software for free. You can use terribly inexpensive hardware. The one irremediable expense is bandwidth. FreeBSD lets you control your bandwidth usage as well, keeping your costs in check.

Michael W. Lucas

Read more Big Scary Daemons columns.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.