Super beefy machine at Internap to be primary MX. Backup, less impressive machine on home DSL to just gather mail to relay to beefy one if it dies or becomes unavailable. Backup one also does regular encrypted backups from primary one, efficiently, only fetching changed content. (I'm not making this item a huge concern because all the other points are more imporant and I know from experience that high-availability and distributed systems are a bitch.... I don't want this design consideration eating into time working on the rest.... In my many years of email, I've only had outage problems a few short times. Data backup is more important (and easier) than 100% availablity)
Free Software, Good Docs
You'll do it entirely with free software. OS must be Debian. MTA can be Postfix or Courier (or others... Cyrus?) but not Sendmail or qmail. Must include docs on how to setup it up again, and overview doc explaining all components.
IMAP and POP
Both IMAP and POP support. And good, complete IMAP support. Both over SSL only. Will pay to get a signed cert to clients don't bitch.
SMTP-AUTH over SSL
I want to be able to use it as my outgoing mail server from any location, using SMTP AUTH.
Efficient data store/indexes
Not mbox. Database or database format okay. A little reluctant yet curious about Maildir. Would probably need to be on ReiserFS?
Optional quotas on people's accounts
Should work with multiple domains, wildcard domains, etc.
Auth via LDAP, MySQL, PAM, other...
SpamAssassin + support for its Bayesian filtering by letting users drag messages into a "spam" folder. Server picks up on this move (either immediately or every 'n' minutes) and adds its tokens to that user's spam tokens database. Server automatically deletes things in the spam folder after a configurable amount of time (week or month) and/or size (100,000 spams). Server delivers messages to "inbox", "spam", or "spam-maybe" (or other rules, for mailing lists). System must also learn ham messages as whatever users move to non-spam/non-inbox folders, and what users reply to. (integration of SMTP and In-Reply-To with spam system and mail retrieval system!) Messages delivered to inbox and deleted without a reply or move should be ignored totally: can't infer spam or non-spam from that action, unless user is on a whitelist. (And whitelists should be more than just email addresses.... should be the tuple (email address, originating server))
Obviously anybody willing to take this project on knows their shit, so I'll keep an open mind about modifications to any of this.