ONLamp.com: System Logging

Published on ONLamp.com (http://www.onlamp.com/)
See this if you're having trouble printing code examples

System Logging

05/17/2001

The syslog system is one of the most delightful things about Unix. Unlike some operating systems that force you to use the limited range of logs that they condescend to provide, Unix allows you to log almost anything, at almost any level of detail. While system logging hooks are provided for the most common Unix resources, administrators can choose a logging configuration that meets their needs. My networks usually have a single logging host that handles not only the FreeBSD boxes, but Cisco routers, switches, and any other syslog-speaking systems.

The system logger is actually fairly straightforward. Programs send log entries to the system logging daemon, syslogd. Syslogd compares each submission to the entries in /etc/syslog.conf. When it finds a matching entry, it processes the log entry in the manner described.

/etc/syslog.conf has two columns. The first is the system providing the information to be logged. The second is the action to be taken when a log message matches. The most confusing part is understanding exactly how to specify a logging information source.

The standard method of specifying a logging source is by facility and level. A facility is a log entry source, or a program that sends messages to syslogd. These facilities are described below.

`auth`	Anything having to do with user authorization, such as `login` and `su`.
`authpriv`	This is identical to `auth`, except it logs to a file that only selected users can read.
`console`	Messages that are normally printed to the system console can be captured by using the console facility.
`cron`	Messages from the system scheduler.
`daemon`	This is a catch-all for all system daemons that don't have other explicit handlers.
`ftp`	You can configure your FTP daemon to log its transfers. See `/etc/inetd.conf`.
`kern`	This is for messages from the kernel.
`lpr`	Messages from the printing system
`mail`	Messages from the mail system
`mark`	This isn't an actual log from a system; instead, the `mark` facility puts a notification in your log every 20 minutes. This is useful combined with some other log.
`news`	Messages from the Internet News daemons.
`ntp`	Messages from Network Time Protocol
`security`	Messages from various security systems, such as `ipf(8)` and `ipfw(8)`.
`syslog`	Yes, the log service can log to itself. Just don't log when you log logs from the log system, or you'll make yourself dizzy.
`user`	The catch-all messages facility. If you don't specify a logging facility for user programs, they'll use this.
`uucp`	Logs from the Unix-to-Unix Copy Protocol. This is a piece of Unix history you'll probably never encounter.
`local0 through local7`	These are reserved for administrator use. Many programs have an option to set a logging facility; choose one of these if at all possible.

Most systems don't log everything their programs send to syslog; rather, they discard trivial messages and only record the important stuff. One man's trivial is another's vital data. This is where the level comes in.

Also in Big Scary Daemons:

Running Commercial Linux Software on FreeBSD

Building Detailed Network Reports with Netflow

Visualizing Network Traffic with Netflow and FlowScan

Monitoring Network Traffic with Netflow

Information Security with Colin Percival

FreeBSD offers eight levels of syslog importance. You can use these levels to tell syslog what to record and what to discard. The levels follow, in order from most to least important.

emerg -- System panic. Messages are flashed on every terminal. The system is basically hosed, or at best horribly, horribly unstable.
alert -- This is bad, but not as bad as the emerg level. The system can continue to operate, but this error should be attended to immediately.
crit -- These are critical errors, such as hardware problems or serious software issues. If your hard drive has bad blocks, they'll show up as critical errors. You can continue running, if you're brave.
err -- Miscellaneous errors. They're bad, and should be fixed, but aren't going to destroy the system.
warning -- Miscellaneous warnings.
notice -- General information that should be logged, in case you need it, but probably doesn't really require action on your part.
info -- General system information.
debug -- This level is usually only of use to programmers, and occasionally to system admininistrators who are trying to figure out just why some program behaves in this way. Debugging logs can contain whatever information the programmer felt necessary to debug the code; this might include information that will violate this users' privacy.
none -- This special level means "don't log anything from this facility here." It's most commonly used when excluding information from wildcard entries.

Information sources include both a facility and a level, separated by a period. When you specify a level, the system defaults to recording messages of that level or greater. For example, look at this entry from /etc/syslog.conf.

mail.info     /var/log/maillog

Messages from the mail system, with a level equal to or above info, are logged to /var/log/maillog.

If you like, you can use wildcards in your information source. To log absolutely all messages from the mail system, you would use this.

mail.*    /var/log/maillog

To log everything from everywhere, uncomment the all.log entry.

*.*      /var/log/all.log

This works, but it's far too full of information to be of any real use; you'll find yourself building complex grep commands just to find what you want.

Also, this would include all sorts of private information, thanks to the debug level. You probably don't want to record that sort of thing. You can exclude authentication information with the authpriv facility and the none level. The semicolon allows you to combine entries on a single line.

*.*;authpriv.none   /var/log/all.log

You can also use comparison operators in /etc/syslog.conf. The valid operators are < (less than), = (equals), and > (greater than). You might want to have a log for mail traffic, and another for mail debugging.

mail.info   /var/log/maillog
mail.=debug /var/log/maillog.debug

This way, you don't have to sort through debugging information to learn what your mail server thinks it's doing.

Similarly, you might have a program that wants to log to local3. You can set this up as such.

local3.*  /var/log/whatever

You can also use a program name as an information source. If a program supports a facility, use it. If you're out of facilities, however, or if your program simply doesn't support syslogd, you can use the name.

An entry for a name requires at least two lines. The first is the program name with a leading exclamation point. The second is the logging information. For example, look at the sample entry for logging PPP.

!ppp
*.*   /var/log/ppp.log

It starts by specifying the program name, and then tells syslogd to append absolutely everything to a file. You can't be certain a random third-party program will have reasonable logging facilities, so it's safest to record everything.

Finally, we have the log message destination. The most common destination is a log file, specified by full path name, but there are other destinations.

You can send log messages to another host with the @ symbol. The following example would dump everything your syslog receives to the logging host on my network.

*.*   @loghost.blackhelicopters.org

The /etc/syslog.conf on the loghost will be used to send messages to their final destinations.

You can list user names, separated by commas. If they're logged in when the log message arrives, the system will write the message on its terminal.

If you want the messages to be written to all users' terminals, use a destination of "*".

Finally, if you want another program to handle the logs, you can use a pipe symbol to redirect the messages to that program.

mail.*   |/usr/local/bin/mailstats.pl

Now that you have logging running, all you have to worry about is your logs eventually filling your disk. We'll cover that next time, when we discuss newsyslog.

Michael W. Lucas

Read more Big Scary Daemons columns.

Return to the BSD DevCenter.