Log in

No account? Create an account
plan: Extensible IMAP Server - brad's life [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

plan: Extensible IMAP Server [Aug. 3rd, 2006|04:22 pm]
Brad Fitzpatrick

I've decided my next fun project will be a modular IMAP server. dovecot's better than most IMAP servers, but it still sucks in a lot of ways. Maildir doesn't have the indexes I need for searching, no vfolders.....

I'm sick of client-side virtual folders. I use 3 or 4 machines, usually ssh to pine, which I'm fast at, but each has its own GUI IMAP client (typically Thunderbird), with different client-side virtual folders, and also my cell phone, whose INBOX I want to be filtered to very particular messages.

Things to abstract and other misc features:

-- underlying store (I'll probably only implement some MySQL schema that I care about and Maildir). don't worry, I'll force it to fail on startup if you're using MyISAM and not a reliable storage engine.
-- folders / vfolders (saved searches, tags, etc...)..
-- delete policies. what does delete command do? delete, or move to archive folder?
-- views per-port. so my phone's INBOX view is a different vfolder than other client's INBOX.
-- if client disconnects, a long-running disk-bound search will actually terminate and not continue to thrash the disk/db.
-- my store will support multiply linked emails in different folders w/ shared flags.
-- integrate well w/ MTAs/qpsmtpd/greylisters/virus scanners where previous mail setups have annoyed me. a lot of this is actually me hacking on qpsmtpd more probably.

The above sounds big/complicated, but I intend for this to be very quick/hacky. Unlike djabberd, I don't intend or care for/if anybody to use it. This will be me scratching my own itch. Everybody else should use gmail, but it's not for me.

[User Picture]From: andrewducker
2006-08-03 11:52 pm (UTC)
Something I've always thought IMAP should do:

Rather than use SMTP to send email, you should be able to dump a fully formed email into an Outbox folder, which would send it and then move it to the Sent Items folder. No more having to use SMTP - no more SMTP insecurity!
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 11:56 pm (UTC)
That's not hard, but useless without clients supporting it, which none do.

But what's wrong with SMTP w/ SSL and SMTP-AUTH? What if you used SMTP+SSL+AUTH and had that inject into your Sent Items folder for you?
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: quappa
2006-08-03 11:56 pm (UTC)
Looks like a real feat in planning :)
Beware, IMAP is hard. And it's usually hard to implement part of IMAP.
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 11:58 pm (UTC)
People have been scaring me away from IMAP for ~5 years now. I think I'm finally ready to tackle this thing. :-)
(Reply) (Parent) (Thread)
[User Picture]From: christowang
2006-08-03 11:58 pm (UTC)
Just out of curiousity, why isn't Gmail for you?
(Reply) (Thread)
[User Picture]From: askbjoernhansen
2006-08-04 03:03 am (UTC)

Too much mail

I suspect Brad has the same problem I'd have using gmail.

1) Too much mail.
2) Too simple filtering language.

- ask
(Reply) (Parent) (Thread)
From: jamesd
2006-08-04 12:51 am (UTC)
Worth alerting if the binary log isn't on as well. Inevitably some people will have disk systems that don't flush properly. Probably most people.
(Reply) (Thread)
[User Picture]From: robbat2
2006-08-04 01:34 am (UTC)
Couple of comments on this, from when I considered something similar.
- Storage of the mails themselves with 1 folder=1 Maildir, same as existing setups, allows good base setup.
- vfolders based on the base folder name - come up with some encoding, eg INBOX_Delivired-To_EMAILADDRESS_AND_From_EMAILADDRESS. This applies said filters to the INBOX. Accessible with stupid IMAP clients.
- IMAP login usernames eg brad-cellphone, brad-thunderbird using for your per-port views, for stupid clients that don't support different INBOX locations.
- All of the tags and basic message headers stored in MySQL. Use a storage layout like the LJ userprops.
(Reply) (Thread)
[User Picture]From: brad
2006-08-04 02:47 am (UTC)
Good ideas.

Maybe even a hybid store approach: Maildir, but MySQL for indexing. Keep raw data on filesystem in Maildir. (if FS doesn't suck too much... large directories)
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: taral
2006-08-04 03:05 am (UTC)
If you do this, I will steal your code. :P
(Reply) (Thread)
[User Picture]From: dan_lane
2006-08-04 03:31 am (UTC)
Sounds great... be sure to include IMAP IDLE for those creating the poor man's BlackBerry :)
(Reply) (Thread)
[User Picture]From: cjcollier
2006-08-04 03:37 am (UTC)
http://dbmail.org/ is the new Jesus.
(Reply) (Thread)
[User Picture]From: brad
2006-08-04 04:49 am (UTC)
Hm. Worth a look. Doesn't seem to address my vfolder needs, though, which is my main itch.
(Reply) (Parent) (Thread)
[User Picture]From: octal
2006-08-04 04:41 am (UTC)
Could you write a dns daemon after this?
(Reply) (Thread)
[User Picture]From: brad
2006-08-04 04:49 am (UTC)
I've never been frustrated with DNS enough to write code. My needs are small.
(Reply) (Parent) (Thread)
[User Picture]From: mart
2006-08-04 06:58 am (UTC)

If you keep on like this I'm going to have nothing left on my own personal todo list. I've been meaning to write a decent (for me) IMAP server for years, and your requirements seem to match up with mine quite well.

My plan was not to support "real" folders at all, though. I would just implement them as a virtual folder with a label named after its name and a search that matches that label. Much easier/better to keep everything in one "mailbox" and filter than to deal with both vfolders and real folders at the same time, both from a programming point of view and a usability/organisation point of view. The virtual folders just need some magic attached to them so that they do the right thing when you MOVE or COPY into them, I guess. But who really wants to be sitting around shuffling mail into mailboxes? It should sort itself out!

(Reply) (Thread)
[User Picture]From: fanf
2006-08-04 01:09 pm (UTC)
Can I stongly encourage you to implement support for the binary extension and the (still in flux) UTF8 addresses + headers extension? Escape from the 7 bit legacy braindamage!

Also for efficient email from a cell phone you want to look at the Lemonade extensions. Dave Cridland is a helpful person to talk to about this - http://dave.cridland.net/
(Reply) (Thread)
From: (Anonymous)
2006-08-04 05:16 pm (UTC)

Dovecot author's reply

Full text search indexes are coming to Dovecot's unstable code branch very soon now. It'll support multiple backends, initially Lucene and Dovecot's own Cyrus squat-like backend. The reason for the squat-like backend is that it supports substring searches, which IMAP protocol requires from the SEARCH command. Lucene will be used only if specifically requested by the client with a new IMAP extension (and possibly also with imap_search_fast_but_broken=yes setting or something).

I've also thought about implementing virtual folders. I don't think it would be too difficult to implement with Dovecot's code base, but since it's a pretty huge task I doubt I'll have time to look into it anytime soon.

Implementing a new IMAP server from scratch won't be easy to get right. I guess getting it mostly usable for a couple of clients could be done somewhat quickly (took 2 months for me), but dealing with all the weirdness of some commands properly, such as LIST's reference parameter, could take forever. I just noticed a couple of days ago yet another bug with my LIST implementation, which has been rewritten at least 3 times already.

Maildir is also way much harder to deal with it than it sounds, at least if you don't want random problems to occur with simultaneous access. Although I suppose those problems aren't all that common since Courier is widely used and it can be easily made to break with making lots of clients modify the same mailbox at the same time.

Anyway, just to let you know that it might be easier to implement virtual folders to Dovecot than to write everything from scratch ;)
(Reply) (Thread)
From: (Anonymous)
2006-08-05 03:20 am (UTC)



Archiveopteryx is a Postgres-based IMAP (and POP, and webmail) server that does support virtual folders (based on IMAP searches) and a number of useful IMAP extensions, and will soon support delete policies and SIEVE.

We prioritise development based on what features people ask for, so feedback is most welcome.

-- Abhijit Menon-Sen <ams@oryx.com>
(Reply) (Thread)
[User Picture]From: brad
2006-08-05 04:12 pm (UTC)

Re: Archiveopteryx

(Reply) (Parent) (Thread)