My previous article explained how to gather session-level network information with Netflow. This article shows how to convert all that nifty data into pretty multicolored pictures on a web server. If you found this article by a web search, you really do need to read the previous article. I will discuss the details of
flowscan as appropriate for the setup discussed in that article; if you are not using
flow-tools to collect your flows, and if you don't have Cflow.pm properly installed, this may or may not work for you.
FlowScan is a Perl script that parses Netflow records and stores them in RRD database files. Many network monitoring tools use RRD to show recent activity in great detail but increasingly aggregate older records. This allows retention of long-term trends without using excessive disk space. Other programs generate reports with these RRD files. FlowScan includes hooks to allow third-party modules to take advantage of FlowScan's processes to generate custom reports. I will focus on a particular FlowScan module,
Start by installing the basic
flowscan program from /usr/ports/net-mgmt/flowscan. This installs several other Perl modules as dependencies, then installs the main FlowScan tools in /usr/local/var/db/flows/bin. FlowScan will not work out of the box, however, so don't start it yet!
First, you need to install an updated FlowScan module. The official FlowScan distribution hasn't had an official update for some time, and doesn't handle flow-capture records. The author has posted an updated flowscan.pm module, but hasn't integrated it into the distribution. Fetch the updated FlowScan.pm version 1.6. Copy it into /usr/local/var/db/flows/bin, overwriting version 1.5 of that module.
This same directory contains a sample FlowScan config file, flowscan.cf.sample. Copy this to flowscan.cf and edit the copy. First, tell FlowScan where to find the flow files. The sample assumes that you are logging to the FlowScan directory, whereas you probably have a directory elsewhere. FlowScan will try to process every file in that directory unless you give it a regular expression that renders correctly, including flow-capture's temporary file and the saved directory, unless you also give it a regex that describes the completed files. The following example assumes that you're logging to /var/log/netflows, and will capture any completed flow-capture files.
ReportClasses lists all of the report modules that you're using. FlowScan comes with two modules,
SubNetIO. While they were great when they came out, I find
CUFlow more useful than either on a day-to-day basis. List all the report classes you want to use.
WaitSeconds is the number of seconds between FlowScan's attempts to process the directory. Many add-on tools assume this five-minute rollover and will break if you choose another time. While you can customize those tools and tweak
flow-capture to match them, for right now just take the defaults.
Finally, verbose logging can be very useful during setup. You might choose to remove this once the whole system is up and running correctly.
FlowScan is configured, but you still need to configure the report module, so that FlowScan records information appropriately for that report.
CUFlow, extract the tarball, and copy CUFlow.pm and CUFlow.cf to /usr/local/var/db/flows/bin. You can leave the Perl module unchanged, but you must edit cuflow.cf to fit your environment.
Subnet statement indicates which networks belong to you.
CUFlow will use this variable to decide if traffic is inbound or outbound.
Network statements include things that you want to process separately. You can have any number of
Network statements. Each
Network statement will show up as an option in a drop-down CGI script. As you see from the example here,
Network statements can overlap.
Network 192.168.2.3,192.168.2.5,192.168.3.80 webservers Network 192.168.2.9,192.168.3.1 mailservers Network 192.168.2.0/25 infrastructure Network 192.168.2.128/25 dmz Network 192.168.3.0/25 administration Network 192.168.3.128/25 development
CUFlow where to store its records with the
OutputDir directive. Do not store your records in a web-accessible location or in your flow-capture log directory.
CUFlow also computes the most active sites and presents a "scoreboard" of the IP addresses that have passed the most traffic in the previous five minutes. Control this with the
Scoreboard takes three arguments: the number of IPs in your "most active" list, a directory to store old "most active" lists, and the filename of the current list. The following line tells
CUFlow to make a Top 10 list, storing the records under /usr/local/www/data/scoreboard, and that /usr/local/www/data/scoreboard/topten.html should always contain the current list.
Scoreboard 10 /usr/local/www/data/scoreboard \ /usr/local/www/data/scoreboard/topten.html
While the biggest talkers in a given five-minute period is useful for troubleshooting purposes, it's also nice to be able to grant an award for Most Busy Host Overall. The
AggregateScore option allows you to do that.
AggregateScore also takes three arguments: the number of hosts in your list, a log file to store the accumulated data, and a location to post the overall Top Talker list.
AggregateScore 10 /var/log/cuflow/agg.dat /usr/local/www/data/overall.html
If you have a complicated network, you might have multiple Netflow sensors.
CUFlow can separate data from separate sensors so that you can get a handle on where different sorts of traffic are going. Each router listed becomes a separate drop-down item in the
Router 192.168.2.1 fred Router 192.168.3.1 barney
Services statements to define TCP/IP ports that you wish to track separately. The
Services statement allows you to make definitive statements such as "80 percent of our internet traffic is outbound web browsing." As Netflow tracks each service, increasing the number of services increases the amount of processing power Netflow requires. Don't just copy /etc/services here! I always eliminate unnecessary protocols; for example, nobody on my hosting network can use Gnutella, so I don't bother trying to track it. Here are some example
Service 20-21/tcp ftp Service 22/tcp ssh Service 23/tcp telnet
Protocol statement is very similar to
Services, except for Layer 3 instead of Layer 4. I recommend tracking
Protocol 1 (ICMP),
Protocol 6 (UDP), and
Protocol 17 (TCP), at a bare minimum. If you have lots of VPN users, you might wish to track IPSec and GRE as well.
Protocol 1 icmp Protocol 6 tcp Protocol 17 udp
As Netflow originated with Cisco, it's not surprising that many Netflow sensors include BGP information.
CUFlow can report on traffic to and from various AS numbers using the ASNumber option, but
softflowd doesn't provide that information and most of the readers out there have only a single internet feed. Comment out the
ASNumber option when using
By default, FlowScan deletes flow files that it has processed. I suggest you retain those files for a few months, or as long as disk space allows. Create a saved subdirectory under your Netflow log directory and FlowScan will automatically move processed logs to this subdirectory.
Even if you don't want to retain records as ongoing practice, I recommend keeping them until you know FlowScan is working correctly. If your FlowScan configuration is broken, it will destroy the data you've already gathered without recording it properly. This is vastly annoying when troubleshooting.
In theory, you have everything configured now. Cross your fingers and start FlowScan.
FlowScan will start spewing out all sorts of messages.
2004/09/02 11:35:17 working on file /var/log/netflows/ft-v01.2004-08-31.142629-0400... 2004/09/02 11:35:18 flowscan-1.020 CUFlow: Cflow::find took 1 wallclock secs ( 0.60 usr + 0.02 sys = 0.62 CPU) for 43011 flow file bytes, flow hit ratio: 2759/2760 2004/09/02 11:35:18 flowscan-1.020 CUFlow: report took 0 wallclock secs ( 0.15 usr 0.19 sys + 0.02 cusr 0.09 csys = 0.44 CPU)
FlowScan is parsing all the old flow files. This can take quite a while, depending on how many flows you've accumulated between implementing your collector and starting FlowScan. One interesting thing to look for here is the "flow hit ratio," or how many flows FlowScan found described in the configuration file. This particular flow file had a hit ratio of 2759 out of 2760; one flow out of 2760 didn't fit FlowScan's expectations. That's pretty good. If you have a hit ratio of 0, you probably messed up your FlowScan install or your
If FlowScan complains about an "Invalid index in cflowd flow file," you probably didn't install the newest Flowscan.pm module. This is perhaps the most common error people make with FlowScan. If you have this problem, go get the appropriate version of the module as described earlier.
When FlowScan finishes parsing all your old flow files, it will print out "sleep 300...", wait for five minutes, and check your log directory for new flow files. You can
C out of FlowScan.
You probably want FlowScan to start automatically at boot rather than taking over your terminal, so go under /usr/local/etc/rc.d and copy the sample startup script to flowscan.sh. This file works unedited, but I usually change the logfile to /var/log/flowscan.log simply because I like all my logs in one place.
"This is nice, but where are my graphs? You promised me pretty pictures!"
Fortunately, getting graphs out of the RRD files is trivial.
CUFlow includes a CGI script, CUGrapher.pl. Copy this to your web server's cgi-bin directory. You only need to set two variables:
$rrddir variable contains the directory where
CUFlow stores the RRD files.
my $rrddir = "/var/log/cuflow";
To print your company's name at the top of the page, be sure to set the
my $organization = "LogicaCMG US IDT development area";
Now browse to the URL for this script and select, say, a network. You'll see an array of drop-down menus. Choose some item--say, a network, or a protocol--and hit "Generate graph."
Congratulations! You have better bandwidth graphs than MRTG alone provides.
One drawback with
CUFlow is that it doesn't break down traffic by network and service. For example, if you choose "Dev network" and "http," you'll get entries for the amount of traffic to and from the dev network added to the amount of HTTP traffic the whole network sees. This isn't exactly useful. To generate more fine-grained reports than this, you'll have to write some custom Netflow reports. I'll explain that in a future article.
Michael W. Lucas
Return to the Sysadmin DevCenter
Copyright © 2009 O'Reilly Media, Inc.