UNIX mail format annoyances

For many years now I’ve been dealing with an ongoing issue which still to this day has no real solution: classic UNIX mailboxes (called mbox) comparing the files’ mtime to its atime to determine if there’s new mail inside of the mailbox (if the mtime is greater than the atime, there’s new mail. If the mtime is smaller than the atime, new mail has been read/there is no new mail). “Classic mail spools” (e.g. /var/mail or /var/spool/mail) are mbox.

Why is this a problem? Because those of us who use mutt/alpine/etc. on our UNIX machines, who also do backups using things like tar/cp/rsync (more on rsync in a moment) end up with mailboxes with a lost/clobbered atime after the backup takes place. The end result: our mail clients no longer tell us there’s new mail in that mailbox, which can be detrimental in many respects.

The most common rebuttal is “shut up and use Maildir“. What Maildir advocates don’t care to acknowledge is that there are many problems with the Maildir concept, particularly when used on a filesystem like ZFS. With classic mbox, your multi-megabyte mailboxes loads quickly — but with Maildir, since it uses a single file per mail, the end result is a mail client that takes forever to load due to the one-file-per-mail concept. ZFS does not perform well when it comes to massive numbers of small/terse files.

UFS/UFS2, ext2fs/ext3fs, and other filesystems don’t have this problem, but let’s pull our heads out for a moment (since tunnel vision/ostrich syndrome is what got us here in the first place!) — we’re entering year 2010 and ZFS is already being used heavily by Solaris/OpenSolaris and FreeBSD users across the globe; ZFS is here to stay, end of discussion. There are some proposed solutions such as making use of ZFS’s semi-new L2ARC to add an additional layer of caching using dedicated low-latency devices (specifically SSDs), but there’s been no actual evidence this improves things with Maildir. And besides, who in their right mind is going to go out and drop hundreds of dollars on an Intel X25-M per machine just to solve this problem? Seriously.

And let’s not forget administrators who mount their filesystems with the noatime mount flag for added performance benefits, especially on a journalled filesystem.

One workaround proposed for mutt users involves recompiling mutt to use Oracle/SleepyCat DB, GDBM, or Tokyo Cabinet to maintain a cache of mailbox headers (using the header_cache directive), thus speeding up the process. Does this help? Yes, there’s a decent improvement, but anyone who uses this method (such as me) can tell you that it’s still no where near as fast as classic mbox, especially when you’ve got a mailbox with a couple hundred new mails in it.

Does the saga end here? Not even close.

There’s a new mailbox format, called MIX, which is being used within alpine. This format is more or less a combination of mbox and Maildir, and performs much better than Maildir. Sounds great, right? Except those of us who use mutt are out of luck — unsupported, and there’s been absolutely no discussion of it since February 2007. Even the author of mutt, Michael Elkins, had nothing useful to say other than snide comments. Oh, and MIX isn’t supported in procmail or Sieve either — double whammy. But MIX does sound like the way to go — too bad it isn’t getting the attention it should.

Some administrators using ZFS are using ZFS snapshots to do their backups instead of something like rsync, which is great except that they’re hit-or-miss (reliability-wise) on FreeBSD — or at least that’s what I last read 6-9 months ago — while rsync is filesystem-independent. Most folks I know who run into snapshot problems revert back to rsync.

So what now? With all the above in mind,I decided to poke at rsync, because there’s been many discussions in the past on the mailing lists about getting rsync to preserve file atime. rsync out-of-the-box will preserve ctime and mtime when using the --times flag. However, there’s a patch called atimes.diff which comes with the rsync-patches tarball that provides a --atimes flag that supposedly solves this. Sounds great… except there’s one problem…

The flag does cause the atimes of the source file to be copied to the destination, but the atimes of the source file are lost! And here’s a more recent confirmation.

If that’s not enough, here’s final confirmation. Note that I’m using non-zero-byte files intentionally; rsync behaves differently when the files are zero bytes.

rsync -a:
$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:27:05 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:05 2009
$ rsync -a source dest
$ stat -x source
Access: Wed Oct 28 06:27:29 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:05 2009
$ stat -x dest
Access: Wed Oct 28 06:27:29 2009
Modify: Wed Oct 28 06:27:05 2009
Change: Wed Oct 28 06:27:29 2009
$ rm source dest

Above, we see that after the rsync, the atime in the source file is lost, and the ctime in the destination file does not match that of the source — only the mtime is retained.

rsync -a --atimes:
$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:32:50 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:32:50 2009
$ rsync -a --atimes source dest
$ stat -x source
Access: Wed Oct 28 06:34:06 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:32:50 2009
$ stat -x dest
Access: Wed Oct 28 06:32:50 2009
Modify: Wed Oct 28 06:32:50 2009
Change: Wed Oct 28 06:34:06 2009

Above, we see the atime and the mtime in the source file is retained in the destination. However, again, the atime in the source file is lost and the ctime doesn’t match that of the source.

cp -p:
$ echo "hello" > source
$ stat -x source
Access: Wed Oct 28 06:37:56 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:37:56 2009
$ cp -p source dest
$ stat -x source
Access: Wed Oct 28 06:38:27 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:37:56 2009
$ stat -x dest
Access: Wed Oct 28 06:37:56 2009
Modify: Wed Oct 28 06:37:56 2009
Change: Wed Oct 28 06:38:27 2009

With cp -p, we see identical behaviour to that of rsync -a --atimes.

Some may be wondering: “is it even possible to solve this problem?” Of course it is. The logic flow should be pretty obvious at this point:

  1. stat(2) or fstat(3) the source file and save (in memory) the atime, mtime, and ctime. Neither call modifies the atime
  2. Copy the source file to the destination file
  3. Set the atime, mtime, and ctime of the destination file using utimes(3) with the previously-obtained values
  4. Set the atime and mtime of the source file using utimes(3) with the previously-obtained values

You can accomplish the same thing with touch.

And let’s not forget that FreeBSD lacks the O_NOATIME GNU extension for open(2), which was proposed in 1998.

So is there a solution to all of this? As far as I’ve been able to tell, no, there isn’t. Using filesystem-level snapshots appears to be the only way to “solve” this issue. I’d be much happier if the --atimes patch for rsync did what it was supposed to… but it’s 23KB, and I’m not familiar with the rsync code (it’s not as black-and-white as one may think).

We UNIX folks should be ashamed of this whole débâcle. There isn’t a better way to say it: what a clusterfuck.