BSD DevCenter
oreilly.comSafari Books Online.Conferences.


Building Detailed Network Reports with Netflow
Pages: 1, 2, 3, 4

Now that you have a set of primitives, you can define filters using them. Filters start with the filter-definition keyword and a name, then have a series of match statements. As with primitives, flow-nfilter supports all kinds of statements to match any characteristic of traffic. The match statements I find most useful include src-ip-addr, dst-ip-addr, src-ip-port, dst-ip-port, and ip-protocol.

Use a match statement with a primitive only for the traffic type you want to match. For example, if you want to match an IP address, you must list an IP address primitive in the filter definition. You can't match an IP address with a protocol primitive, sensibly enough. You can get a full list of matching types in flow-nfilter(1), or check the sample filter.cfg file distributed with flow-tools.

To write a filter to catch a certain type of traffic, you must understand what makes that traffic unique. For example, consider a rule to match all email traffic to a SMTP server. What characteristics does an inbound SMTP exchange have? Well, it's on port 25, runs over TCP, and has a destination IP of your mail server. That's enough to write a filter definition.

filter-definition inboundmail
  match dst-ip-addr local_mailservers
  match dst-ip-port smtpports
  match ip-protocol tcp

An implicit AND joins these terms. Any flow must match all the primitives for this filter to pick it out.

Similarly, you can write a filter definition that will catch all email originating from your mail server and heading out to other servers. The only difference here is that the destination IP is now the source IP:

filter-definition outboundmail
  match src-ip-addr local_mailservers
  match dst-ip-port smtpports
  match ip-protocol tcp

Using these two filters, you can easily run two reports that pick out all traffic to and from your mail server. Using the or keyword, you can combine those two reports into a single report:

filter-definition allmail
  match dst-ip-addr local_mailservers
  match dst-ip-port smtpports
  match ip-protocol tcp
  match src-ip-addr local_mailservers
  match dst-ip-port smtpports
  match ip-protocol tcp

This is just a "superreport" that contains all traffic that meets either of the one-way mail definitions, joined by the or keyword.

One sort of traffic that I'm particularly interested in is stuff that shouldn't be on my network at all. Nobody should be trying to use telnet(1) to reach my servers, and nobody should be using MS SQL network connections inbound, either. Anyone who tries to telnet into my servers is by definition a troublemaker. I want to know about them. I've previously defined a redflags primitive listing TCP/IP ports that I consider bad:

filter-definition redflags
  match dst-ip-port redflags
  match dst-ip-addr hostingnet
  match ip-protocol tcp

Similarly, anyone who tries to send email to an IP that isn't actually a mail server is a troublemaker--at best, he's infected with a spam-relay virus; at worst, he's an intruder. I want to know about him. Specifically, I want my firewall team to know about him so they can block his IP address!

filter-definition inboundmail_false
  match dst-ip-addr local_not_mailservers
  match dst-ip-port smtpports
  match ip-protocol tcp

I can combine the last two reports to create a "bad guys" report, and then run it via cron(1) on a regular basis and mail the results to the appropriate network minion.

To run flow-nfilter, use flow-print to dump the binary format from the flow file into a (non-human-readable) format that the other tools can read more easily. Pipe that into flow-nfilter, which will strip out the flows that don't match your desired flow. Flow-nfilter requires two arguments: -F and the filter definition, and then -f with the filename of filter configuration you're using. Finally, flow-print translates the binary flow format into human-readable text:

# flow-cat -p flowfiles | flow-nfilter -F reportname \
    -f /usr/local/etc/flow-tools/filter.cfg | flow-print

For example, here's how to check the flow file ft-v05.2005-07-06.125500-0400 for all inbound email connections. The filename shows that this checks the flows during the 5-minute period between 12:55 and 13 p.m. on July 6, 2005.

# flow-cat -p /var/log/netflows/saved/ft-v05.2005-07-06.125500-0400 \
    | flow-nfilter -F inboundmail -f /usr/local/etc/flow-tools/filter.cfg \
    | flow-print
srcIP            dstIP            prot  srcPort  dstPort  octets      packets      6     4983     25       497         9      6     2379     25       509         9      6     3012     25       232         5   

Arrange this in any order you like with sort(1).

If a particular connection strikes your interest, you can go into further detail on it with a customized flowdumper expression, or use one of the reports built into flow-print(1). One day, your email administrator will thank you on bended knee for this data--if it occurs to him that it might possibly be available, that is. Fortunately, you can provide this data for any service provided by your network! Once you implement Netflow, you will wonder how you ever solved any problems without it.

Michael W. Lucas

Return to the BSD DevCenter.

Sponsored by: