ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Monitoring Network Traffic with Netflow

by Michael W. Lucas
08/18/2005

As far as I'm concerned, network management is about knowledge. If you know what your network and servers are doing, making decisions is easy. I make heavy use of SNMP and MRTG to provide information I consider necessary on my network. Getting a new hard drive from management is much easier when you can say "Our fileserver usage grows by 2GB a month, so we need a new disk installed within six months," as opposed to "Our fileserver ran out of space; we need a new disk now!"

This helps, but MRTG doesn't provide any detail about what the network is actually doing. So your T1 is 90% full? What hosts are responsible, and what services are they using or providing? If you have MRTGified your network down to the switch level you can tell which host is using that traffic, but that's all. You then have to fire up tcpdump or investigate that host in a more painstaking way.

Netflow provides a session-level view of network traffic, recording information about every TCP/IP transaction that occurs over your network. It might not be as complete a record of network traffic as tcpdump can provide, but it's also much easier to manage and much more informative when aggregated.

By session data I mean "which host talked to which other host, on which ports, and how much data did they exchange?" One session is a flow. Netflow can easily tell you if that 1.5Mb/s to your Web server is honest Web traffic, or if your system has become part of a botnet and is providing vital assistance in the destruction of ONLamp.com. Once you have Netflow running, you'll wonder how you ever survived without it.

The downside of this visibility is that Netflow is more complicated to set up than MRTG or other SNMP-based monitoring software. The vast array of obsolete and irrelevant documentation still available on abandoned Web sites and ancient mailing list archives only confuses the issue. The setup is more than worth it, however.

Netflow Architecture

A Netflow system has three major components: a sensor, a collector, and some sort of reporting system. The sensor (also known as a probe) is a daemon that listens to the network and captures your session data. Just as with Snort or any other IDS system, the collector needs to connect to a hub, "mirrored" switch port, or other device where it can see all the network traffic. If you're running a BSD or Linux firewall, this is an excellent place to run a Netflow collector--all the traffic has to pass through this device anyway! This sensor bundles up the session information and tosses it at the collector.

The collector is a second daemon that listens on a UDP port of your choice for reports from the sensor and dumps them into a file for later evaluation. Different collectors store their data in different file formats.

Finally, the reporting system reads the files generated by the collector and produces human-readable reports. The reporting system must be able to read the format used by the collector.

Because there are a wide variety of sensor software, collector daemon, and reporting systems available, you can forgive a newcomer for thinking that Netflow is too complicated to begin to set up. I'd like to sidestep this problem by choosing a single specific scenario that will fit 90% of network environments immediately, and that can work for most of the remainder with only minor changes. Assume that:

Mastering FreeBSD and OpenBSD Security

Related Reading

Mastering FreeBSD and OpenBSD Security
By Yanek Korff, Paco Hope, Bruce Potter

Sensor Setup

The easiest software to install is the sensor. First, make sure that your sensor hardware can listen to all network traffic. Once you have that you can install sensor software. I recommend softflowd(8). Sortflowd runs out-of-the-box with a simple make all install clean on BSD and Linux, requiring only libpcap. (FreeBSD includes the ng_netflow netflow system, but as I have sensors running on Linux as well, I don't use it. I prefer to use a single piece of software on every operating system whenever possible.) Once you have softflowd installed, you only need the interface name you want to monitor and the IP address and UDP port where your collector is listening. For example, to listen on the em0 interface and send the collected data to 172.16.13.5, port 8818, run:

# softflowd -i em0 -n 172.16.13.5:8818

The sensor will immediately begin listening to the network and sending session information to the collector. Make sure that this program starts at boot!

Softflowd includes a control program, softflowctl(8), that allows you to issue commands to a running softflowd. To make sure that the software is actually working, check the softflow statistics after softflowd has been running for a few moments.

# softflowctl statistics
softflowd[40475]: Accumulated statistics:
Number of active flows: 2298
Packets processed: 268086
Fragments: 0
Ignored packets: 867 (867 non-IP, 0 too short)
Flows expired: 3103 (0 forced)
Flows exported: 6206 in 214 packets (0 failures)
...

The important output here is the second line, which tells you how many flows are active at the moment, and the exported line, which tells you how many flows softflowd has exported to the collector.

If you search, you can find a wide variety of sensors. Cisco routers can export Netflow data--at the expense of your router's precious CPU time. If you have a complicated router setup, or if you have a very low-end router, this can overload your router. Cisco would be very happy to sell you a router upgrade so that you could properly export Netflow, mind you, but generally a Unix-like box is more cost-effective. Many other devices also speak Netflow; check your documentation if you're interested.

If you have Ethereal or tcpdump installed, I recommend using it at this point to confirm that you are actually receiving Netflow data at your collector. If not, confirm that softflowd is running and perhaps try the -D (debug) flag to see if there are any problems with your setup.

Softflowd only sends flow information to the collector once the flow has ended--for example, when the FTP session ends, when the Web page has been delivered, and so on. This means that at any moment, softflowd will have a cache of connections in progress. When you stop softflowd, run softflowctl shutdown so that softflowd will expire those flows and send them to your collector immediately. Simply shutting down the server running softflowd will cause you to lose the active-but-incomplete flows. You're going to lose some information anyway if you reboot your sensor, but you might as well keep that loss as minimal as possible.

Collector Setup

Your collector gathers the data exported by the sensor and stores it on disk for long-term reference. If possible, install your collector on a Web server; it will make reporting much nicer and easier. I recommend flow-capture, a very popular Netflow collector included in the flow-tools package. On FreeBSD, flow-tools is in the Ports tree at /usr/ports/net-mgmt/flow-tools. Install it with the usual "make all install." Do not type make clean! You might have to rebuild some components by hand. For the same reason, don't use a precompiled flow-tools package.

Make a directory for flow-capture to keep its records. I usually use /var/log/netflow, but anywhere you have space works. On a multimegabit network, Netflow files can fill several GB of disk over a few weeks. I also recommend creating a saved subdirectory under your log directory, for the reporting system to use.

Now you need a startup script so flow-capture will run automatically at boot. A command like the following works nicely:

# /usr/local/bin/flow-capture -p /var/run/flow-capture.pid -n 287 \
    -N 0 -w /var/log/netflows/ -S 5 0/0/8818

Most of this you can use unchanged. The -w flag tells flow-capture where to place its files. The final argument tells flow-capture which local IP to listen to, which remote IP to listen to, and which UDP port it should listen on. In this case, 0/0/8818, the collector listens on all local IP addresses, for requests from any remote IP address, on port 8818. If you can receive random Internet traffic on your collector, specify a particular sensor IP in the middle value. (My collector is behind a firewall, and anyone who can sneak past the firewall would have no trouble deceiving flow-capture as well.) Flow-capture needs the -n 287, -N, and -S 5 arguments to interoperate with the reporting package, so leave them alone.

Once you start flow-capture, flow files will appear in your log directory. The names of these files come from the version of Netflow data they're gathering and the date and time at which this data began. For example, the filename tmp-v05.2005-04-28.201001-0400 indicates a temporary file, containing Netflow version 5 data, collected on April 28, 2005, starting at 20:10:01 (or 1 second after 8:10 p.m.), at -4 hours from GMT. Every five minutes, flow-capture moves the temporary file to a permanent location and starts a new temporary file. Permanent files begin with ft instead of tmp, but otherwise the names are exactly the same.

To confirm that your flow-capture install is actually collecting something, see if the temporary file grows. This should happen quickly, within a few minutes on a busy network.

The information in these files is in a binary format requiring special tools to read. Many of those tools use Cflow.pm.

Cflow.pm Setup

Many different Netflow reporting tools use the Cflow.pm perl module to read Netflow files. This includes a library and command-line tool for viewing and manipulating flow files. The hard part is that each collector has its own storage format. While the original purpose of Cflow.pm was to process cflowd(8) files, Cflow.pm can support other formats if properly installed.

This part is where most people give up on Netflow. Follow the directions carefully. Be sure to verify your work when your Cflow.pm install is complete.

On some recent versions of FreeBSD, /usr/ports/net-mgmt/p5-Cflow automatically detects the presence of the flow-tools libraries. Cflow links this library as -lnsl, and if the build process doesn't find it during the configure process you'll see a warning like:

Note (probably harmless): No library found for -lnsl

This warning is not harmless; it means that this Cflow build will not work for you. If you don't see this line, just install Cflow and see if it works. Cflow includes flowdumper(1), a program to read flow files on the command line. Check the largest flow file you have, so that you can be sure the record includes something to view.

#flowdumper -s ft-v05.2005-04-28.201501-0400 | more
2005/04/28 19:14:01 172.16.30.247.80 -> 216.98.200.250.63647 6(SYN|ACK) 3 144
2005/04/28 19:14:01 216.98.200.250.63647 -> 172.16.30.247.80 6(SYN) 1 48
2005/04/28 19:14:01 172.16.30.247.80 -> 216.98.200.250.63648 6(SYN|ACK) 3 144
2005/04/28 19:14:01 216.98.200.250.63648 -> 172.16.30.247.80 6(SYN) 1 48
...

Each line is a flow. This records the source and destination IPs of assorted TCP/IP transactions. You might notice that this particular snippet of four lines is actually only two TCP/IP sessions. The first line indicates that traffic is coming from 172.16.30.247, port 80, to the host 216.98.200.25. The next line shows traffic from the second host going to the first.

If your Cflow install is faulty, flowdumper will return either silence or an error. You cannot proceed until you resolve this error--at least, you can't proceed if you want your reporting tools to work! Uninstall your current p5-Cflow package and build it another way.

Remember how I said to not clean the flow-tools port? Go back to the port directory, cd to the work subdirectory, and go to the source code directory. There is another Cflow tarball in a subdirectory named contrib. Extract it.

# cd /usr/ports/net-mgmt/flow-tools/work/flow-tools-0.67/contrib
# tar -xzvf Cflow-1.051.tar.gz
# 

Cflow frequently picks up the proper library when installed from this location under a compiled flow-tools package. (This means that you have to have a built flow-tools in the directory above you; this is why I told you not to do a make clean.) Just follow the usual Perl module building process.

# perl Makefile.PL
# make
# make install

Try flowdumper again, and it should work.

On occasion, I've had even this fail. In that case, use brute force. Flow-tools installs libft.a under /usr/local/lib. Edit the section of Cflow.pm's Makefile.PL where it checks for the flow-tools library:

sub find_flow_tools {
   my($ver, $dir);
   my($libdir, $incdir);
   if (-f '../../lib/libft.a') {
      $dir = '../../lib';
      $incdir = "-I$dir -I$dir/..";
      $libdir = "-L$dir";
   }

Edit the line that reads

   if (-f '../../lib/libft.a') {

to read

   if (-f '/usr/local/lib/libft.a') {

If this fails, there's something seriously wrong with your Perl install. Now, run make and make install. You now have a flow-tools aware flowdumper, which indicates that the Cflow.pm Perl module underlying it works correctly with your collector.

You can probably easily imagine a whole slew of Perl scripts that would take this data and generate pretty graphs and reports on usage, or identify peak bandwidth consumers. Other people have already done the heavy lifting on this one, however. My next article will look at creating pretty pictures from Netflow data.

Michael W. Lucas


Return to the Sysadmin DevCenter

Copyright © 2009 O'Reilly Media, Inc.