ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Visualizing Network Traffic with Netflow and FlowScan

by Michael W. Lucas
09/15/2005

My previous article explained how to gather session-level network information with Netflow. This article shows how to convert all that nifty data into pretty multicolored pictures on a web server. If you found this article by a web search, you really do need to read the previous article. I will discuss the details of flowscan as appropriate for the setup discussed in that article; if you are not using softflowd and flow-tools to collect your flows, and if you don't have Cflow.pm properly installed, this may or may not work for you.

FlowScan is a Perl script that parses Netflow records and stores them in RRD database files. Many network monitoring tools use RRD to show recent activity in great detail but increasingly aggregate older records. This allows retention of long-term trends without using excessive disk space. Other programs generate reports with these RRD files. FlowScan includes hooks to allow third-party modules to take advantage of FlowScan's processes to generate custom reports. I will focus on a particular FlowScan module, CUFlow.

FlowScan Setup

Start by installing the basic flowscan program from /usr/ports/net-mgmt/flowscan. This installs several other Perl modules as dependencies, then installs the main FlowScan tools in /usr/local/var/db/flows/bin. FlowScan will not work out of the box, however, so don't start it yet!

First, you need to install an updated FlowScan module. The official FlowScan distribution hasn't had an official update for some time, and doesn't handle flow-capture records. The author has posted an updated flowscan.pm module, but hasn't integrated it into the distribution. Fetch the updated FlowScan.pm version 1.6. Copy it into /usr/local/var/db/flows/bin, overwriting version 1.5 of that module.

This same directory contains a sample FlowScan config file, flowscan.cf.sample. Copy this to flowscan.cf and edit the copy. First, tell FlowScan where to find the flow files. The sample assumes that you are logging to the FlowScan directory, whereas you probably have a directory elsewhere. FlowScan will try to process every file in that directory unless you give it a regular expression that renders correctly, including flow-capture's temporary file and the saved directory, unless you also give it a regex that describes the completed files. The following example assumes that you're logging to /var/log/netflows, and will capture any completed flow-capture files.

FlowFileGlob /var/log/netflows/ft-v*[0-9]

ReportClasses lists all of the report modules that you're using. FlowScan comes with two modules, CampusIO and SubNetIO. While they were great when they came out, I find CUFlow more useful than either on a day-to-day basis. List all the report classes you want to use.

ReportClasses CUFlow

WaitSeconds is the number of seconds between FlowScan's attempts to process the directory. Many add-on tools assume this five-minute rollover and will break if you choose another time. While you can customize those tools and tweak flow-capture to match them, for right now just take the defaults.

WaitSeconds 300

Finally, verbose logging can be very useful during setup. You might choose to remove this once the whole system is up and running correctly.

Verbose 1

FlowScan is configured, but you still need to configure the report module, so that FlowScan records information appropriately for that report.

Mastering FreeBSD and OpenBSD Security

Related Reading

Mastering FreeBSD and OpenBSD Security
By Yanek Korff, Paco Hope, Bruce Potter

Configuring CUFlow

Grab CUFlow, extract the tarball, and copy CUFlow.pm and CUFlow.cf to /usr/local/var/db/flows/bin. You can leave the Perl module unchanged, but you must edit cuflow.cf to fit your environment.

The Subnet statement indicates which networks belong to you. CUFlow will use this variable to decide if traffic is inbound or outbound.

Subnet 192.168.2/23

Network statements include things that you want to process separately. You can have any number of Network statements. Each Network statement will show up as an option in a drop-down CGI script. As you see from the example here, Network statements can overlap.

Network 192.168.2.3,192.168.2.5,192.168.3.80      webservers
Network 192.168.2.9,192.168.3.1                   mailservers
Network 192.168.2.0/25                            infrastructure
Network 192.168.2.128/25              dmz
Network 192.168.3.0/25                            administration
Network 192.168.3.128/25              development

Tell CUFlow where to store its records with the OutputDir directive. Do not store your records in a web-accessible location or in your flow-capture log directory.

OutputDir /var/log/cuflow

CUFlow also computes the most active sites and presents a "scoreboard" of the IP addresses that have passed the most traffic in the previous five minutes. Control this with the Scoreboard option. Scoreboard takes three arguments: the number of IPs in your "most active" list, a directory to store old "most active" lists, and the filename of the current list. The following line tells CUFlow to make a Top 10 list, storing the records under /usr/local/www/data/scoreboard, and that /usr/local/www/data/scoreboard/topten.html should always contain the current list.

Scoreboard 10 /usr/local/www/data/scoreboard \
    /usr/local/www/data/scoreboard/topten.html

While the biggest talkers in a given five-minute period is useful for troubleshooting purposes, it's also nice to be able to grant an award for Most Busy Host Overall. The AggregateScore option allows you to do that. AggregateScore also takes three arguments: the number of hosts in your list, a log file to store the accumulated data, and a location to post the overall Top Talker list.

AggregateScore 10 /var/log/cuflow/agg.dat /usr/local/www/data/overall.html

If you have a complicated network, you might have multiple Netflow sensors. CUFlow can separate data from separate sensors so that you can get a handle on where different sorts of traffic are going. Each router listed becomes a separate drop-down item in the CUFlow CGI.

Router 192.168.2.1    fred
Router 192.168.3.1    barney

Use Services statements to define TCP/IP ports that you wish to track separately. The Services statement allows you to make definitive statements such as "80 percent of our internet traffic is outbound web browsing." As Netflow tracks each service, increasing the number of services increases the amount of processing power Netflow requires. Don't just copy /etc/services here! I always eliminate unnecessary protocols; for example, nobody on my hosting network can use Gnutella, so I don't bother trying to track it. Here are some example Service statements:

Service 20-21/tcp ftp
Service 22/tcp ssh
Service 23/tcp telnet

The Protocol statement is very similar to Services, except for Layer 3 instead of Layer 4. I recommend tracking Protocol 1 (ICMP), Protocol 6 (UDP), and Protocol 17 (TCP), at a bare minimum. If you have lots of VPN users, you might wish to track IPSec and GRE as well.

Protocol 1 icmp
Protocol 6 tcp
Protocol 17 udp

As Netflow originated with Cisco, it's not surprising that many Netflow sensors include BGP information. CUFlow can report on traffic to and from various AS numbers using the ASNumber option, but softflowd doesn't provide that information and most of the readers out there have only a single internet feed. Comment out the ASNumber option when using softflowd.

Saving Netflow Records from FlowScan

By default, FlowScan deletes flow files that it has processed. I suggest you retain those files for a few months, or as long as disk space allows. Create a saved subdirectory under your Netflow log directory and FlowScan will automatically move processed logs to this subdirectory.

Even if you don't want to retain records as ongoing practice, I recommend keeping them until you know FlowScan is working correctly. If your FlowScan configuration is broken, it will destroy the data you've already gathered without recording it properly. This is vastly annoying when troubleshooting.

Starting FlowScan

In theory, you have everything configured now. Cross your fingers and start FlowScan.

# /usr/local/var/db/flows/bin/flowscan

FlowScan will start spewing out all sorts of messages.

2004/09/02 11:35:17 working on file /var/log/netflows/ft-v01.2004-08-31.142629-0400...
2004/09/02 11:35:18 flowscan-1.020 CUFlow: Cflow::find took  1 wallclock secs ( 0.60 usr +  0.02 sys =  0.62 CPU) for 43011 flow file bytes, flow hit ratio: 2759/2760
2004/09/02 11:35:18 flowscan-1.020 CUFlow: report took  0 wallclock secs ( 0.15 usr  0.19 sys +  0.02 cusr  0.09 csys =  0.44 CPU)

FlowScan is parsing all the old flow files. This can take quite a while, depending on how many flows you've accumulated between implementing your collector and starting FlowScan. One interesting thing to look for here is the "flow hit ratio," or how many flows FlowScan found described in the configuration file. This particular flow file had a hit ratio of 2759 out of 2760; one flow out of 2760 didn't fit FlowScan's expectations. That's pretty good. If you have a hit ratio of 0, you probably messed up your FlowScan install or your Subnet statement.

If FlowScan complains about an "Invalid index in cflowd flow file," you probably didn't install the newest Flowscan.pm module. This is perhaps the most common error people make with FlowScan. If you have this problem, go get the appropriate version of the module as described earlier.

When FlowScan finishes parsing all your old flow files, it will print out "sleep 300...", wait for five minutes, and check your log directory for new flow files. You can Ctrl-C out of FlowScan.

You probably want FlowScan to start automatically at boot rather than taking over your terminal, so go under /usr/local/etc/rc.d and copy the sample startup script to flowscan.sh. This file works unedited, but I usually change the logfile to /var/log/flowscan.log simply because I like all my logs in one place.

Generating Graphs

"This is nice, but where are my graphs? You promised me pretty pictures!"

Fortunately, getting graphs out of the RRD files is trivial. CUFlow includes a CGI script, CUGrapher.pl. Copy this to your web server's cgi-bin directory. You only need to set two variables: $rrddir and $organization.

The $rrddir variable contains the directory where CUFlow stores the RRD files.

my $rrddir = "/var/log/cuflow";

To print your company's name at the top of the page, be sure to set the $organization variable.

my $organization = "LogicaCMG US IDT development area";

Now browse to the URL for this script and select, say, a network. You'll see an array of drop-down menus. Choose some item--say, a network, or a protocol--and hit "Generate graph."

Congratulations! You have better bandwidth graphs than MRTG alone provides.

One drawback with CUFlow is that it doesn't break down traffic by network and service. For example, if you choose "Dev network" and "http," you'll get entries for the amount of traffic to and from the dev network added to the amount of HTTP traffic the whole network sees. This isn't exactly useful. To generate more fine-grained reports than this, you'll have to write some custom Netflow reports. I'll explain that in a future article.

Michael W. Lucas


Return to the Sysadmin DevCenter

Copyright © 2009 O'Reilly Media, Inc.