OSCON, ... Persist-RSS, ..., Solaris 10

brad's life - OSCON, ... Persist-RSS, ..., Solaris 10

Brad Fitzpatrick

[	website	\|	bradfitz.com	]
[	userinfo	\|	livejournal userinfo	]
[	archive	\|	journal archive	]

[Jul. 30th, 2004|12:24 am]

[

Tags

perl, tech

]

Wow, haven't updated in a while.

I've been spending all week at OSCON. On both Monday and Tuesday I attended two 4 hour sessions (4 total). Notably these included spam fighting (Matt Sergeant) and Perl6::Rules (Damian Conway).

Damian's keynote the following day (Tuesday night, actually) was amazing. The game of life, Perl, linguistics, Klingon, Perl 6, thermodynamics, .... then a Perl game of life program to demonstrate a property of therodynamics, written in Klingon. (wtf!?) We were just dying. That guy never gives a dull presentation.

My presentation on LiveJournal went wonderfully. Got a bunch of good questions, feedback, and complements afterwards. We did a BoF that night on memcached and load balancing related stuff.

After that (still wednesday night) we did the big Stonehenge party. Open Bar. Lots of cool people. Great times. Thanks, Stonehenge!

Thursday morning, little hungover, skipped the keynote. Good talks about qpsmtpd (the mod_perl of smtpd servers!), the kernel, parrot, etc.

Tonight was especially cool. Did an RSS BoF that Jeremy Zawodny put on about scaling RSS distribution, etc. Some good ideas came out of that. (The main problems are doing non-centralized distribution and efficient update notifications (not client polling)). Ignoring the peer-peer distribution problem (which would require signing, etc), I want to take Perlbal ideas and provide a specialized HTTP service for all users where you could do a GET on http://persist-feed.livejournal.com/users/brad/{rss|atom} and you'd get an XML doc (Atom or RSS) from the time you specified in If-Modified-Since, then the HTTP connection would just stay open until there's a new feed, in which case you'd get a new HTTP chunk (well, a true "chunk" if HTTP/1.1 or just more data in 1.0). Then internally we'd keep track of who's subscribed and send messages to Perlbal when people post new stuff, which would send them new HTTP messages. That'd basically solve the notification problems.... just get everybody's desktop news aggregators to hold upon connections forever and use SAX (event-based) XML parsers.

But what fucking blew my mind tonight was the Solaris 10 BoF. Wow. I can't wait to play with it. I doubt we'll use it in production, but just using DTrace to do performance analysis would be wonderful. Their filesystem (ZFS) and zoning looks incredible too.

Link

Comments:

From:

whitaker
2004-07-30 12:52 am (UTC)

(Link)

I wanted to go to the SOlaris thing... but my brain wasn't q

(Reply) (Thread)

From:

whitaker
2004-07-30 12:53 am (UTC)

(Link)

...uite functional.

(as you can see by this little slip up)

(Reply) (Parent) (Thread)

From:

caladri
2004-07-30 03:08 am (UTC)

(Link)

you guys are so cute, what with the insanity.

(Reply) (Parent) (Thread)

From:

ex_tawneelyn258
2004-07-30 12:55 am (UTC)

(Link)

you have a beautiful mind...

(Reply) (Thread)

From:

avatraxiom
2004-07-30 01:13 am (UTC)

(Link)

Wow, I keep hearing more and more about OSCON. It seems like it's been pretty successful. Seems like they certainly managed to attract some good speakers. :-)

brad: ...and you'd get an XML doc (Atom or RSS) from the time you specified in If-Modified-Since, then the HTTP connection would just stay open until there's a new feed, in which case you'd get a new HTTP chunk...

Wow, that would be pretty cool. Of course, wouldn't you run into IMAP-server-like problems with zillions of open sockets?

brad: But what fucking blew my mind tonight was the Solaris 10 BoF.

Also, I'm impressed that Solaris can still be impressive. I'd been pretty much convinced that they were out of the game. DTrace does sound really cool, though...

-Max

(Reply) (Thread)

From: (Anonymous)
2004-07-30 01:47 am (UTC)

tools

(Link)

i don't know too much about dtrace..

i'm curious if you've seen any of the mac os x CHUD tools? every year at WWDC, i'm blown away by how powerful they are.

(Reply) (Thread)

From:

mart
2004-07-30 05:55 am (UTC)

(Link)

Be careful of overloading If-Modified-Since like that. You should read the discussion that came out of PossibleHTTPExtensionForEfficientFeedTransfer (it had a much shorter name before, but it was refactored) over at the Atom wiki. It has some reasons why overloading If-Modified-Since isn't great and also why adding new HTTP headers probably isn't the best plan. (I'm not sure I agree with all of them, by the way.)

I've since thought more about my X-Last-Polled proposal and think it'd be better if the server was to return an opaque token which clients then send back. Times are troublesome because several things can happen in a single instant, but there'd be nothing stopping a server using time if it was easier. The proviso would just be to ignore any unchanged duplicates you get back, to account for possible overlap.

Your persistant connection thing sounds interesting but scary. You want clients to just parse half an XML document and then wait for the rest? That's not exactly in the spirit of HTTP nor how most of the clients out there work.

See also AggregateFeeds; Special mention of LiveJournal in there. I personally think that the proxy aggregator, modelled slightly on how NNTP works, is the way to go for syndication scalability without moving to full peer-to-peer. An RSS extension could make it work there, too. LiveJournal providing Atom feeds of friends pages would be a good start, although I don't think core Atom is currently set up for this so it'd have to be done with extensions. (plus the caveat noted in the wiki that LJ doesn't currently retain enough information to reconstruct entries pulled from other sources)

A combination of the above (feed aggregation and delta-feeds in one) would be killer.

(Reply) (Thread)

From:

evan
2004-07-30 08:24 am (UTC)

(Link)

The persistent connection thing reminds me of Jabber works. You send an opening XML tag at the beginning of the communication and then you just keep sending subelements without ever closing it.

Any push system would require special aggregators, which makes me think non-standard behavior isn't as much of a problem; in a sense, it's the whole point.

(Reply) (Parent) (Thread)

From:

mart
2004-07-30 09:09 am (UTC)

(Link)

Sure, but working off If-Modified-Since will break “standard” clients trying to be polite, since they'll never see the end of the document, and (as noted in the wiki page I linked) feed consumers which don't keep the data in any structured fashion will lose the data at the next poll.

I suppose there's no harm in using a completely non-standard HTTP header to trigger this non-standard behavior, but I'm sure it'll leave a bad taste in some mouths, like those of the people who didn't like the X-Last-Polled proposal.

(Reply) (Parent) (Thread)

From:

iamo
2004-07-30 09:33 am (UTC)

(Link)

What about using the Accept request header to indicate a preference for a RSS feed delta or persistent RSS feed connection. Like:
Accept: application/xml, application/xml+rss-delta, application/xml+rss-persist
and then the other end knows that a feed reader capable of dealing with a delta is reading. And then combine it with the Modified-Since or whatever header to indicate the dynamic nature of it. Dynamic feed generators would know the client is capable of dealing with just a subset of an rss document, or a persistent connection, and could put out a no-cache statement to discourage proxies from holding those mime types.

Or would this still screw with unassuming proxies?

(Reply) (Parent) (Thread)

From:

snej
2004-07-30 10:00 am (UTC)

Jabber can already do this

(Link)

Pushing XML stuff over a persistent socket connection is, basically, a solved problem: it's Jabber, aka XMPP. Lord knows Jabber isn't ideal, and having implemented a client for it in the past I've done my share of griping, but it works and there are a number of implementations of it and it's better not to re-invent the wheel.

There is a Jabber publish/subscribe extension that would be ideal for pushing journal posts. PubSub.com has been adapting it for pushing news feeds.

None of this stuff is quite 1.0 yet, AFAIK, but it looks very promising. In particular, it scales well because the topology is like SMTP. When someone's feed is updated you don't have to shove data at a zillion socket connections for all the listeners; you just push data to the Jabber servers those listeners are logged in through, and those fan the messages out to their users. (PubSub.com isn't doing this yet, but they should and hopefully will.) And if someone is offline, their Jabber server will queue up the notifications until they go online.

I know you're good at making stuff up from scratch, Brad, but in this case I think it would be a lot better to go along with what others are already working on. Feel free to AIM me if you want to talk about this in more detail...

(Reply) (Parent) (Thread)

From:

mart
2004-07-30 01:00 pm (UTC)

Re: Jabber can already do this

(Link)

Heh… this seems to be getting dangerously close to starting up the “LiveJournal-Jabber integration” discussion again.

(Reply) (Parent) (Thread)

From:

x-f [x-f.lv]
2006-08-13 01:27 pm (UTC)

Re: Jabber can already do this

(Link)

..and look at LiveJournal+Jabber now. :)

(Reply) (Parent) (Thread)

From:

livemerlyn
2004-07-30 06:26 am (UTC)

Stonehenge party

(Link)

You're most welcome. As they say, "A good time was had by all".

(Reply) (Thread)

From:

edm
2004-07-30 06:28 am (UTC)

DTrace

(Link)

I trust you realise that the DTrace tools are available for testing via the Solaris Express programme which provides for (free IIRC, very low cost) non-commercial use or a relatively low cost commercial use yearly subscription. It's in the versions available since November 2003 IIRC. Available in both x86 and Sparc versions AFAIK.

I've been tempted to have a play with it, as DTrace looks very cool, but haven't yet had the time to look further.

Ewen

(Reply) (Thread)

From:

evan
2004-07-30 08:24 am (UTC)

(Link)

Is qpsmtpd the server you've always wanted to write?

(Reply) (Thread)

From:

brad
2004-07-30 11:39 am (UTC)

(Link)

Half of it. (the SMTP plug-in system land)

The other half is the message store / delivery (MDA), but there are good enough solutions for that already.

I'm so excited to write qpsmtpd plug-ins.

(Reply) (Parent) (Thread)

From:

torgo_x
2004-07-30 08:57 am (UTC)

Pictures

(Link)

I took pictures! Because I'm really that guy in Memento. Except I never remember to scribble "THIS MAN CAN HELP YOU" or "DO NOT TRUST THIS MAN" on anything.

I rarely do indoor photography, and I never use a flash, so only a few of my shots turned out. But everyone looks terribly thoughtful (almost angst-ridden) in them.

(Reply) (Thread)

From: (Anonymous)
2004-07-30 11:53 am (UTC)

Damian's Keynote?

(Link)

Which session was Damian's thing? From the schedule looks like he only talked in the morning on tuesday - sorry I missed it. Are his slides up anywhere?

(Reply) (Thread)

From:

brad
2004-07-30 01:20 pm (UTC)

Re: Damian's Keynote?

(Link)

No clue. You could probably mail him.

(Reply) (Parent) (Thread)

From: (Anonymous)
2004-07-30 01:49 pm (UTC)

Solaris BOF

(Link)

[TrackBack] http://blogs.sun.com/roller/page/ahl/20040730#linux_solaris_and_open_source

This past week at OSCON I've spent my time trying to understand open source processes, talking about Solaris, and trying to figure out what OpenSolaris is going to look like...

(Reply) (Thread)