Log in

No account? Create an account
brad's life [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

dpopperd [Sep. 8th, 2006|11:45 pm]
Brad Fitzpatrick

Some people fix up old cars.
Some people collect old guns.
Some people trace their family lineage back hundreds of generations.

I wrote a POP3 server.

It's so refreshing how simple specs used to be just 10-20 years ago. Ever read the GIF spec, for instance?

"Modern specs aren't complex," you say, "they just have lots of dependencies, standing on the shoulders of giants!" Right. Except the giants more often than not have died already so the modern specs are instead carefully balancing on untold numbers of decomposing corpses.

I've wanted to graph this.

Imagine a scatterplot of RFCs. X axis is time. Y axis is number of words in the spec, inclusive all referenced dependency specs. (lazyweb?)

Anyway, I wrote a POP server because it was fun. I got to abstract out DJabberd's SSL code in the process into something I could reuse for this and Perlbal, which somebody on the mailing list keeps asking for. Not to mention cleaning up DJabberd. But I haven't done that yet. That'd be too productive.

Okay, I actually wrote it so people could pull down ESN's message center into their mail client, and stop bitching about mail not making it to them because the spam wars have broken SMTP. (well, it was broken before)

Internet Mail 2000, baby. Or something similar. Meng keeps saying "RSS Email" to totally confuse people, but I think he means IM2000. RSS's got the buzz tho, so maybe I see his angle.

Perlbal now supports concatenated GET URLs. Because the middle of the Internet is presumably broken, browsers don't enable HTTP pipelining by default, so if you're 80 milliseconds away from LiveJournal and we have 20 javascript files and 30 images... that's 50 * 160 milliseconds until you're getting your page. That's a long time. Europeans and Russians are probably used to it. So fuck it all, and throw away cachability, and do it by hand. Look at the front page of livejournal and view source. see those funky URLs? Later we should analyze common groups of dependent resources and serve them in isolated chunks for cachability. Or just serve all our JavaScript on all pages and have one cacheable resource.... that's just 500k or so. But then UAs have to parse it each time probably because they likely suck and don't cache a parallel parsed verson.

Back to the subject. dpopperd (uniquely named POP server) has a pluggable model. (model = "What is an inbox? What is a message?") At least that's the theory. Got sidetracked before I finished implementing it. Need to abstract out more of djabberd so I don't feel like I'm duplicating too much code. But presumably getting your friends page over POP is possible too. Not sure how useful that'd be. :-)

Okay, back to doing nothing useful.

(Deleted comment)
[User Picture]From: brad
2006-09-09 07:30 am (UTC)

Re: Plotting

Oh, I fetched them all awhile ago. I stopped at parsing them for dependencies and moved on to something else.

I want more than just:
$ wc -l *.txt | sort -n
(Reply) (Parent) (Thread)
[User Picture]From: xlerb
2006-09-09 07:50 am (UTC)
You sound like someone who's tried to read the IMAP spec.
(Reply) (Thread)
[User Picture]From: scsi
2006-09-09 09:13 am (UTC)
Woah, are you suggesting a concept (IM2000) thought of by DJB?
(Reply) (Thread)
[User Picture]From: ghewgill
2006-09-09 10:23 am (UTC)
I think he is. Unfortunately IM2000 was stillborn due to lack of actual implementations. I started to write one once, but stopped when I realized that it has a whole new set of vulnerabilities for spammers to exploit.

djb's IM2000 is centered around the idea that the sender should be responsible for storage of the message until delivery instead of an intermediate receiver closer to the final recipient. This used to be plausible before (a) really cheap storage, and (b) botnets. Spam messages haven't got terribly much larger, but the cost of storage has dropped by orders of magnitude. And with botnets, there's nothing stopping a spammer from using a botnet to store the undelivered spam messages. Still no extra cost for the spammer, and no net win in the fight against spam. With astronomical adoption and transition costs, it's just not worthwhile.

I'm optimistic about DKIM. Ideally, I will be able to digitally sign my outgoing mail, and declare that anything unsigned is a forgery. By not messing with the transport layer, this avoids all the problems SPF has with forwarders.
(Reply) (Parent) (Thread)
[User Picture]From: fanf
2006-09-10 11:24 pm (UTC)
There are more fundamental reasons that IM2000 doesn't solve the spam problem. Instead of being spammed with messages, you would get spammed with message notifications. Spammers would try to put the main point of the spam into the notification instead of into the message. Because notifications are small, there's less information in them for anti-spam filters to work on.

I have not yet seen any "replacement" for SMTP that is better, let alone beter enough to be worth deploying.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-09-10 11:36 pm (UTC)
To be clear, the main part of IM2000 I like is when you get those notifications, at least you know the real person (well, hostname) that the mail is supposed to be from, so you can choose to get it or not. Who stores the message doesn't matter so much. I suppose one of these dozen SMTP identity extensions will gain traction sometime here.
(Reply) (Parent) (Thread)
[User Picture]From: fanf
2006-09-11 10:49 am (UTC)
Is that really an advantage? Most people use webmail of one form or another or their employer's centralized email system, so for most legitimate email the notifications will just tell you the sender's ESP or their shared office firewall. So for the majority of cases the message's path is no different from what we have with SMTP - you see the server, not the client. In cases where people use full-featured email software, you still have the problem of telling the difference between 0wned and legit sources on the end of home broadband connections, but you can no longer reasonably use DUL-alike blacklists.

One thing that has changed since IM2000 was thought up is that many more people are using mobile occasionally-connected computers, which cannot be expected to implement the notification retry part of the protocol themselves. They will have to offload that as well as the message storage to some central always-online server, which again would have exactly the same role as SMTP's message submission servers or smart hosts.
(Reply) (Parent) (Thread)
[User Picture]From: ghewgill
2006-09-09 11:43 am (UTC)
Oh, and the IMAP spec is horridly complex. At my last job it took me a few weeks to get a server working acceptably, and I was even using an existing framework that took care of most of the low-level nonsense.

I have a rule of thumb about protocols - if they have "Simple" in the acronym expansion, it means it's complex. eg. SMTP, SNMP, SOAP, SNTP.
(Reply) (Thread)
From: baudehlo
2006-09-09 03:50 pm (UTC)
I admit I've only been looking at IMAP from the client side, but I don't really (yet) see what's so complicated about IMAP. There's admittedly some stupid stuff (like why every query has to include a unique id when only the final response line includes that id is beyond me), but everything seems fairly sane to me apart from that.
(Reply) (Parent) (Thread)
[User Picture]From: askbjoernhansen
2006-09-11 07:10 pm (UTC)

uh, ...

But SMTP and SNTP *are* simple.

- ask
(Reply) (Parent) (Thread)
[User Picture]From: byron
2006-09-09 05:01 pm (UTC)
Dude...you need a hobby. :) And I actually do the first two things you suggest in between my computer hobbies. :)

Only two of my guns are newer than 1948. I also play with a '62 Thunderbird.
(Reply) (Thread)
[User Picture]From: smitty1e
2006-09-09 06:47 pm (UTC)

Some people

"Some people call me the space cowboy,
Some call me the gangster of love,
Some people call me Maurice..."

Your intro just brought on a Steve Miller moment. ;)
(Reply) (Thread)
(Deleted comment)
[User Picture]From: bsdguru
2006-09-09 08:33 pm (UTC)
Aye eye sir :P
(Reply) (Parent) (Thread)
From: jamesd
2006-09-09 11:55 pm (UTC)
Users in Europe don't like it at all but at least parallel requests work. Wikipedia has squids in Netherlands and Korea to help those not logged in. We're thinking of adding database slaves and page builders to the set so logged in people get the benefit as well.
(Reply) (Thread)
[User Picture]From: smackfu
2006-09-10 12:31 am (UTC)
That's pretty clever. Most people would write an SMTP client and then send the messages to a dropbox email address.
(Reply) (Thread)
[User Picture]From: ydna
2006-09-11 07:09 pm (UTC)
I'm starting a collection to buy you a rename token so you can change your handle to [info]dbradderd (the gangsta o' code).
(Reply) (Thread)