Weekly Tip #3: Categorizing E-Mails with maildrop

I will be breaking the football-themed tips this week — I know, I know… bust out the handkerchief and sob a bit. It’s time to move on to bigger and better things. This week we will be looking at sorting e-mail as it arrives on the server. Sorting upon delivery has two advantages over sorting in your e-mail client (Outlook, Thunderbird, Mail.app, etc.): (1) access from multiple e-mail clients will have the same e-mail layout and (2) mail is filtered as soon as it’s delivered; there is no need to perform a batch sort first thing in the morning.

Filtering on the server is handled by maildrop, which acts as the local delivery agent. maildrop is also responsible for handing the message off to SpamAssassin for spam checking. Filter rules are stored in the file named .mailfilter in each user’s home directory. There are two very important caveats that you need to know before hacking your filter to bits.

1. maildrop expects Unix-style end-of-line markers (“\n”). If you edit the script on a Windows/Mac PC either in the browser via File Manager in the control panel or a separate text editor, then watch out! EOL markers are generally converted over to their respective OS’ style. On Windows, this means “\r\n” would silently be used to mark the end of a line and on Mac it would be “\r”. Both of these EOL markers will result in unexpected behavior in the filter.EOL markers may be converted either in your editor or in the File Manager. Conversion in text editors vary from editor to editor; for example in Komodo right-click file tab -> Properties and Settings -> Line Endings -> UNIX \n, untick Preserve existing line endings; Vim is simpler with “set ff=unix”; and if you are using ed… then, well, you may be a lost cause.
2. maildrop also requires very restrictive permissions. Only the owner may have read/write access to the file. Permissions may be changed by either bringing up the file options in the File Manager by clicking on the icon to the left of the file name, through your FTP client (SITE CHMOD command), or inside the shell with chmod. maildrop will refuse delivery if the .mailfilter is anything but 0600 (-rw-------).

Keep those two requirements in mind. These are the two most popular problems that I see with users who embark on maildrop modifications.

Syntax is very simple with maildrop.

if (/Subject: Fantasy Dungeons and Dragons/D)
{
   to "Mail/.Mailbox/"
}
if (/(To|Cc):.+msaladna@apisnetworks.com/ && /From:.+sports-fantasy-[^@]+@yahoo-inc.com/)
{
   if (/Subject: Fantasy Football/D)
   {
      cc "!insider-analysis@apisnetworks.com"
      to "Mail/.Mailing Lists.Fantasy Football/"
   }
   if (/Subject: Fantasy Baseball/D)
   {
      to "/dev/null"
   }
   cc "Mail/.Important Copies/"
}

Let’s step through each line.

Line 1: examine the “Subject” line of an e-mail. If the subject, what appears in the Subject line of an e-mail message, explicitly matches, case-sensitive (D flag at the end) “Fantasy Dungeons and Dragons” [aside: I’m stoked for the 4th edition rules] then process the following rules between lines 2 and 4 [aside: line 3 for those mathematically challenged folk such as myself] enclosed by braces ({…}).

Line 2,4: maildrop isn’t a fan of K&R braces, so you will need to place the opening/closing braces, which are required, on their own lines.

Line 3: deliver it to a Maildir-style mailbox named “Mailbox”. This would appear in the e-mail client under Inbox -> Mailbox. Recall that e-mail is stored in $HOME/Mail/. maildrop begins in your home directory, so using a relative path (e.g. Mail/…) would work fine. An absolute path (/Mail/…) would fail; however, an absolute path prefixed with the home directory, stored in the HOME variable, would be OK ($HOME/Mail/…). If there is one thing to learn, learn that to begin mailbox delivery locations with “Mail/”. to instructs maildrop to deliver the message to this directory/e-mail address and finish processing the message.

Line 5: slightly more complex example of header matching. This time the e-mail must have not matched any previous blocks which ended in a to (terminates message processing); think of it falling through to the next case. Programmatically this could be represented as if (…) { … } else { … } except maildrop doesn’t understand else statements. If my address, msalandna@apisnetworks.com, is provided in either the To or Cc field, i.e. I wasn’t addressed in a Bcc, and (&&) the message is from sports-fantasy-<anything>@yahoo-inc.com, then process the next chunk of code. Note the [^@]+, which may very well look alien to you. This is a regular expression, which is a more flexible process of wildcard matching. You’ve used “ca*” to match “cat” and “car” before, right? Same principle, but this offers more flexibility and, incidentally, complexity. .+ is a close analog, which matches one or more of any character and this is used to ensure the target e-mail address, subject — or more generally, string — appears in the line. If you want the true analog to “*“, then use the regular expression .*, which means to match zero or more. [aside: I had to add that otherwise my inbox would be flooded with angry nerds upset over labeling AD&D as “fantasy” and misrepresenting regex patterns in the morning.] Note the absence of D in either pattern. This means both matches disregard case.

Line 7: if the subject begins with “Fantasy Football”, and again case doesn’t matter with the D, then process the next section enclosed by braces.

Line 9: make a copy of the message (cc) and forward it to insider-analysis@apisnetworks.com. Note the placement of the exclamation mark, !. ! informs maildrop that the text immediately to the right is an e-mail address. If it read ‘cc “insider-analysis@apisnetworks.com”‘, guess where the message would be copied? If you guessed to a file called $HOME/insider-analysis@apisnetworks.com, then you would be absolutely correct. Don’t forget the ! if you intend of forwarding a message. Replacing cc with to would forward a copy of the message to insider-analysis@apisnetworks.com and terminate execution. Because execution is not terminated with the cc directive, we continue onto the next line.

Line 10: terminate execution and deliver the message to “Mail/.Lists.Fantasy Football”. Note the double-quotes surrounding the destination for each directive. It’s good form to do this for various reasons, which you’ll find out the hard way if you specify an IMAP folder with a space in it or try forwarding to an external e-mail account (it won’t work correctly). This IMAP folder would be represented in the e-mail client as Inbox -> Lists -> Fantasy Football.

Line 12: this line is left as an exercise to the reader to ensure you have a pulse. See Line 7‘s explanation if you get stuck.

Line 14: just like before, we’re delivering the message to a file, but /dev/null is a special file on the server designed to swallow whatever it is fed. Think of it like a black hole. Because my fantasy baseball team is in shambles this year the message is deleted, never to reach my inbox and prolong my misery.

Line 16: finally, if the message makes it this far it has to be important. Store a copy of the message in “Mail/.Important Copies”, which would appear in the e-mail client as Inbox -> Important Copies.

Line 17: if the message hasn’t triggered a to directive yet, then fall through out of the special processing and deliver to the default mailbox. If a message matched line 16, then it would also fall through to the end, because cc was used and not to. Implicitly to “Mail/” appears at the end of the .mailfilter file; that is another way of looking at this. If nothing else terminates message delivery, then execute to “Mail/” and terminate delivery.

That was fairly easy, right? I hope so, because you’re going to be quizzed now. But before we do that, make sure the IMAP directories exist on the server. You can accomplish this with the maildirmake program or by creating four directories with the permission set 0700.

mkdir $HOME/Mail/.MyDir/
mkdir $HOME/Mail/.MyDir/{cur,new,tmp}
chmod -R 0700 $HOME/Mail/.MyDir/

Let’s wrap up this week’s tip with a quick exercise to make sure you know what you’re doing. Refer to the headers below as a guide to answer the following questions. If you get them correct, then you are ready to write filters on your own.

Return-Path: <bounce-live-964777629-46771286@ezinedirector.net>
X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on
assmule.apisnetworks.com
X-Spam-Level: *
X-Spam-Status: No, score=1.3 required=5.0 tests=AWL,BAYES_60,
DK_POLICY_SIGNSOME,HTML_MESSAGE autolearn=no version=3.2.0
X-Original-To: football@apisnetworks.com
Delivered-To: msaladna@apisnetworks.com
Received: from mx11.ezinedirector.net (mx11.ezinedirector.net [65.207.215.17])
by assmule.apisnetworks.com (Postfix) with ESMTP id 6FB60274719
for <msaladna@apisnetworks.com>; Fri, 31 Aug 2007 07:57:07 -0400 (EDT)
Message-ID: <31063140@964777629.ezinedirector.net>
X-Subscriber: 46771286
Subject: FF Today News: Week 1 Right Around The Corner
From: <list@fftoday.com>
To: football@apisnetworks.com
X-Campaign: 964777629
Reply-To: <bounce-live-964777629-46771286@ezinedirector.net>
Errors-To: <>
Date: Fri, 31 Aug 2007 06:32:16 -0500
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=”_NextPart_000_1188541936_CFX_iMSMail_1017171156″
Content-Transfer-Encoding: 8bit

1. If the e-mail was delivered in August of this year, then store it in Inbox -> Football -> August
2. If the message starts with “FF Today:”, then save it under Inbox -> FF Today
3. If the message is from “list@fftoday.com” and contains the subject “FF Today:”, then store it under Inbox -> Lists -> FF Today -> Announce, otherwise if the subject doesn’t match, then save it under Inbox -> Lists -> FF Today -> Misc
4. Finally, for the smart-asses out there, if the message isn’t labeled as spam, addressed to football@apisnetworks.com, a multipart MIME message, and less than 30 KB in size, save in Inbox -> FF Today -> Small otherwise if the size if greater than 40 KB forward it to football-big@apisnetworks.com, which just so happens to be an alias to football@apisnetworks.com. Hint: use reformail and avoid forwarding loops.

Answers will be posted on Wednesday. Anyone wishing to venture forth bravely to answer question 4 feel free to post it here and I’ll let you know whether you’re warm or cold ;). Also, it looks like we had a football-themed tip in the end. That’s a three week streak.

Wednesday Update
As promised, here are the answers.  Problem 4, despite having a real answer, was a joke problem.  If you have seriously attempted it and can’t make sense of the solution, then let me know and I can walk you through it.

Problem 1:

if (/^Date:.+Aug 2007/)
{
	to "Mail/.Football.August/"
}

Problem 2:

if (/^Subject: FF Today/)
{
	to "Mail/.FF Today/"
}

Problem 3:

if (/^From:.+list@fftoday.com/)
{
	if (/^Subject: FF Today:/)
	{
		to "Mail/.Lists.FF Today.Announce/"
	}
	to "Mail/.Lists.FF Today.Misc/"
}

Problem 4:

if (/^X-Spam-Status: No/ && /^(?:To|Cc|X-Loop):.+football@apisnetworks.com/ && /^MIME-Version: /)
{
   if ($SIZE < 30720)
   {
      to "Mail/.FF Today.Small/"
   }
   if ($SIZE > 40960 && !/X-Loop: football@apisnetworks.com/)
   {
      xfilter "reformail -A 'X-Loop: football@apisnetworks.com'"
      to "!football-big@apisnetworks.com"
   }
   # Messages fall through to the default mailbox between 30 KB - 40 KB
}