ONLamp.com: Procmail Basics

Published on ONLamp.com (http://www.onlamp.com/)
http://www.onlamp.com/pub/a/bsd/2001/12/06/FreeBSD_Basics.html
See this if you're having trouble printing code examples

Procmail Basics

12/06/2001

If you receive more than a few email messages a day, you've probably discovered that it becomes increasingly difficult to sort and prioritize your email. Messages you want to read immediately can get lost in a sea of less-important messages. Worse, your inbox can become cluttered with spam, virus-infected messages, and other disagreeables. Fortunately, the procmail program has been designed to help you sort through this mess.

By creating your own customized "recipes," you can organize the messages you do want to receive and deal with the messages you don't want to receive.

In today's article, I'll build and configure procmail and get you started on a few basic recipes. In next week's article, I'll continue with some more complicated recipes and look at procmail's logging features.

To install procmail, become the superuser and build the port:

su Password: cd /usr/ports/mail/procmail make install clean

When you build this port, it will give you some informative messages and require you to press enter to continue the build. The first message is regarding a locking test; I just pressed enter to accept the default directories:

========================================================= In order for the kernel-locking tests to work as intended I have to be able to test-lock files on as many semantically different filesystems as possible (for more information about this, READ PARAGRAPH TWO in INSTALL). To suppress this prompt you can set the LOCKINGTEST variable in the Makefile.

Please add writable directories to the list. You should only add directories that reside on filesystems that have unique characteristics. E.g. if you have several remote NFS partitions, pick some unique client-server pairs, there is little use in picking the same server twice from the same client-machine. An excellent candidate would be a remotely mounted mail spool directory.

I will temporarily use a testdirectory named _locktest in the following directories:

/tmp .

If you would like to add any, please specify them below, press return to continue: =========================================================

The build also reminds you that you can integrate procmail into your mail delivery system, such as "sendmail." Since I'm building procmail on a single-user system I won't bother. If you'd like to investigate this possiblility, read the file /usr/local/share/examples/procmail/advanced once you've finished your build.

========================================================= If you are a system administrator you should consider integrating procmail into the mail-delivery system -- for advanced functionality, speed AND SECURITY --. For more information about this topic you should look in the examples/advanced file. ========================================================= ===> SECURITY NOTE: This port has installed the following binaries which execute with increased privileges. 1142851 132 -rwsr-xr-x 1 root wheel 66612 Nov 27 13:27 /usr/local/bin/procmail If there are vulnerabilities in these programs there may be a security risk to the system. FreeBSD makes no guarantee about the security of ports included in the Ports Collection. Please type 'make deinstall' to deinstall the port if this is a concern. For more information, and contact details about the security status of this software, see the following webpage: http://www.procmail.org/ =========================================================

And finally, we receive a security note, as procmail is installed with setuid permissions. You'll be reminded of this when you read your security output tomorrow (see Understanding the Automatons Part 2). We should also check to see if there are any SAs on procmail due to the setuid, and there is one: FreeBSD-SA-01:60.procmail.asc. If you read that SA, you'll see that you shouldn't run a procmail version lower than 3.20, as that version addressed the vulnerability. I'll double-check what version of procmail I just built:

So far, so good. Let's take a quick look at what we can expect out of this port we just built:

more /usr/local/share/doc/procmail/README

Can be used to create mail-servers, mailing lists, sort your incoming mail into separate folders/files (real convenient when subscribing to one or more mailing lists or for prioritising your mail), preprocess your mail, start any programs upon mail arrival (e.g. to generate different chimes on your workstation for different types of mail) or selectively forward certain incoming mail automatically to someone.

Procmail can be used: - and installed by an unprivileged user (for himself only). - as a drop in replacement for the local delivery agent /bin/mail (with biff/comsat support). - as a general mailfilter for whole groups of messages (e.g. when called from within sendmail.cf rules).

The accompanying formail program enables you to generate autoreplies, split up digests/mailboxes into the original messages, do some very simple header-munging/extraction, or force mail into mail-format (with leading From line).

----------------------

Let's start off simple and configure procmail for one user. Once you've finished building the actual port, you can leave the superuser account. I'll be working as the regular user "genisis" and will set up that user's procmail configuration. I'll start by entering my home directory and copying over the example configuration files:

cd
cp /usr/local/share/examples/procmail/forward ~/.forward
cp /usr/local/share/examples/procmail/1procmailrc ~/.procmailrc

I'll then reset the permissions on those files:

chmod 644 ~/.forward
chmod 644 ~/.procmailrc

Since I use the fetchmail program to regularly poll my ISP's mail server to check for new mail, I'll kill the fetchmail process until I'm finished configuring procmail:

killall fetchmail

The procmail utility reads a configuration file known as procmailrc. The superuser can create a global configuration file in /usr/local/etc/procmail that will affect all users on that system. However, it is recommended that each user instead create their own .procmailrc in their home directory. This way, users can create their own filtering recipes without affecting any other users.

The .procmailrc file has two sections. The first section contains your path and environment variables; the second section contains your filtering recipes. Once you've copied the default .procmailrc into your home directory, immediately edit the first section before you try to download any email. I'll open up my .procmailrc with my favorite text editor to demonstrate:

# Please check if all the paths in PATH are reachable, # remove the ones that are not. PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.

FreeBSD doesn't have a /usr/ucb, so remove that bit, along with the extra colon, so the line looks like this:

PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin:. MAILDIR=$HOME/Mail # You'd better make sure it exists

Because I use pine, my mail goes to a directory called mail, so I'll change that capital M to a small m. If you don't use pine as your email reader, do an ls -F of your home directory. Somewhere in the output you should have a directory where you store your email; it is usually called mail or Mail. If you do an ls of that directory, you should be able to recognize your mail folders (e.g. sent, saved, etc.) It is very important that your MAILDIR line reflects the correct directory.

The next three environment variables can be left as is. The DEFAULT variable tells procmail to create a directory called mbox, where it will store all messages that don't match any of your recipes. The LOGFILE variable creates a directory called from, where your procmail logs will be stored.

DEFAULT=$MAILDIR/mbox
LOGFILE=$MAILDIR/from
LOCKFILE=$HOME/.lockmail

After the environment variables is the section containing some sample recipes:

:0                  # Anything from thf
* ^From.*thf@somewhere.someplace
todd                # will go to $MAILDIR/todd

:0                  # Anything from people at uunet
* ^From.*@uunet
uunetbox            # will go to $MAILDIR/uunetbox

:0                  # Anything from Henry
* ^From.*henry
henries             # will go to $MAILDIR/henries

# Anything that has not been delivered by now will go
# to $DEFAULT using LOCKFILE=$DEFAULT$LOCKEXT

Once you've edited your path and environment variables, the procmail utility has enough information to run correctly; until you customize those recipes, it will simply place all of your email in a folder called mbox. For now, I'll remove the sample recipes. I'll also test to make sure that I haven't broken anything by restarting fetchmail and opening up pine. I notice that I have a new folder in my folder list called mbox, and after a few moments, there's some new mail in it. So far so good.

Now the fun part begins, as I devise recipes to organize my email. I'll start by taking a look at the mailing lists I'm subscribed to, as they represent the biggest chunk of my email. I'm subscribed to freebsd questions, several mailing lists from securityfocus.com, and the Internet drafts list. So I'll start by making three recipes to sort those three general topics.

Every recipe has the same basic syntax and requires a minimum of three lines. The recipe can be as simple or as complicated as you want to make it; there is no right or wrong way to create a recipe, as long as it follows the correct syntax and gets the job done. Let's start by looking at that three line syntax:

:0              #first line
*               #second line
file            #third line

You'll note that comments are allowed (and recommended, so you remember why you created that recipe) and are indicated by a #.

The first line indicates the beginning of a recipe and is easy for me to remember, as it sort of looks like my expression when I'm in the kitchen. You'll sometimes see more characters in the first line if someone is doing something a bit more complicated, but we'll save complicated for later.

The second line is the condition. In simple recipes, it will start with an * and go on from there; this is the part of the recipe that can be as complicated as you want. You can also make as many conditions as you want in a recipe.

The third line is the action, which usually means the folder in which you want procmail to put the email message that met your condition.

Now, let's translate all this into three recipes to organize my mailing lists. Recipes to sort mailing lists are the simplest to create, as we can use procmail's built-in ^TO_ expression. This expression will scan the To: and Cc: lines of an email message's header. This is ideal for finding mailing lists, as a person either sends a message "to" the mailing list, or they reply to the original posting, which will send a "cc" back to the mailing list.

Before I create the actual recipes, it's helpful to know what procmail does when it reads my recipes. By default, it will scan the headers of incoming email messages for the conditions I've created in my recipes. And, by default, it will stop reading when it finds a matching condition. This means that order is important for two reasons. One, procmail will work much faster if I put the recipes that will generate the most matches at the top of the recipe section; for example, most of my email comes from the freebsd questions mailing list, so that recipe should be near the top. Second, if I want a recipe that scans for virii, it should be placed before any other recipes, so it will scan every message as it arrives.

One last thing before creating those recipes: since procmail scans headers by default and most of your recipes will reflect this, it is helpful to understand what an email header is and what type of information one would expect to see in the various header fields. If you're rusty on this subject, there is an excellent tutorial here.

Okay, on with the first three recipes. Since most of my messages come from freebsd questions, followed by the security lists, followed by the Internet drafts, my recipes look like this:

:0                        # Anything from questions@freebsd.org
* ^TO_*questions@freebsd.org
questions                       # will go to $MAILDIR/questions

:0                      # Anything from a security mailing list
* ^TO_*security
security                         # will go to $MAILDIR/security
 
:0                         # Anything regarding Internet drafts
* ^TO_*ietf
drafts                             # will go to $MAILDIR/drafts

I've used the same logic in each of the recipes. The only semi-complicated bit is the syntax of that expression in the condition line; it should look like this:

* ^TO_*

and be immediately followed by the expression you want procmail to search for in the To: or Cc: fields of a message's header. When you're creating your own recipes, take a look at those fields in your email messages and look for the common expression. For example, if I had told the first recipe to search for the expression "freebsd" instead of "questions@freebsd.org," it would catch all messages from any freebsd mailing list (e.g. newbies, advocacy, security, etc.). I was a bit more particular and specified that I just wanted it to catch messages from the questions list at freebsd.org.

I kept my second recipe more general. It actually catches every message from "securityfocus.com," "security@freebsd.org," and even my daily security output, since they all share the common expression "security." This way any email that is security-related will be placed into a common folder, which is what I wanted.

For the Internet drafts, I noticed that they always came from ietf.org. Since the expression ietf is fairly unique, I used that as my condition.

Each action specifies a folder for the messages to be placed in. The procmail utility automagically created the folders questions, security, and drafts for me the first time it came across a message that required that action. Any message that didn't match those three recipes was put in the default folder of mbox. This is handy when you're creating your recipes, as you'll be able to see which messages don't match a recipe and see if there is a need to lump them together and create a recipe to do that for you.

There're a few other recipes you may want to include to get you started. The first is out of man procmailex and deals with duplicate messages. It should be the very first recipe, and looks like this:

:0 Whc: msgid.lock
| formail -D 8192 msgid.cache   #prevent duplicate messages

:0 a:
duplicates         #but store them instead of deleting them

This variant of the recipe will send all duplicate messages to a folder called duplicates. That manpage also gives an alternate recipe that will instead delete any duplicate messages, but also warns that you might lose other email messages if you have a scripting issue in a complicated recipe. You may want to stick with the above recipe until you reach procmail guru status.

The last recipe I'll demonstrate today is an example of a virus-scanning recipe. The week I wrote this article, the "Badtrans" virus was rampant and was becoming a bit of an irritation. I borrowed the following recipe from the freebsd questions mailing list and placed it right below my duplicate recipe:

# Stupid BadTrans virus
:0:
* ^From: .+ \<_.*\>$
garbage

:0:
* ^From: _.+ \(.+\)$
garbage

This is an example of a more complicated expression. Don't despair if you don't understand the expressions in those condition lines; the important thing is that this recipe works. Remember, when you want to create a recipe, you don't have to reinvent the wheel. There's a good chance that someone else has already created a recipe that does what you want to do. This is especially true when new virii are released, as procmail users will post their recipes to the procmail mailing list. You can search for those recipes here.

If I do a search for "badtrans," I'll come across several recipes which vary in their complexity, but which all get the job done.

This should get you started on using procmail. In the next article, we'll look at some more complicated expressions you can use in your recipes and also take a look at procmail's logging statistics.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Read more FreeBSD Basics columns.

Return to the BSD DevCenter.