Coolest.
Thing.
Ever.

It's entertaining to watch, at a minimum.

I want a way to stream just my friends page!

Wow, you could completely add a "Wankery Per Minute" counter using this data.

Oh no! It's an endless stream of angst!

sorryTooSlow is the best XML tag ever.

haha i second that. :D

Hmm. I don't understand how but somehow when I try this on my Windows box here it starts printing out gibberish and items from my command history going back weeks. Weird stuff. On my linux box it messes up my terminal. I guess there are some control characters in there causing weirdness. Good thing that telnet isn't the recommended way to access this. ;)

Hopefully this stuff will start getting PubSubbed at some point, assuming that someone can keep up with it well enough to produce decent PubSub output.

It's utf-8 data. The Russians are fucking with you.

An increasing number of companies (large and small) are really insistent that we ping them with all blog updates, for reasons I won't rant about.

Uhh, do they have rocks in their heads? At ~6 posts a second, thats like asking to take a shower via a firehose.

I'd just get something in writing just to confirm they really want to do this. Then when you open the proverbial floodgates upon them they cant nail you or 6A for blowing the servers off of the face of the planet.

In their defense its probably some marketing/management drone pressing for you to do this, while the admins are already getting their white flags and upstream blocks in place.

It's not such a BFD --- in the 4 minutes 6 seconds I listed, I got about 1.3 megabytes, comprising 733 posts. That's only about 5 kB/sec, which you could handle on a 56kbps modem, and a little under 3 posts per second. (Presumably this is the public half of LJ.) That's, like, half a gigabyte a day, or a few dimes worth of bandwidth. The junky but relatively optimized full-text indexer I use on my email can reindex a gigabyte in about ten minutes on my old 600MHz laptop. (see kragen-hacks archives for details.)

I am not really qualified to speculate on why LJ needs a big server farm to handle 6 posts per second (although bradfitz's talk at OSCON was really great), but I'm guessing that it includes items from the following list: pageviews, comments, authorization, usericons, reliability, friends pages. All of these are required to actually run LJ, but you don't need any of them to slurp from this particular firehose.

FWIW I think this is really excellent. Thanks Brad! Yay perlbal!

The two obvious ways of parsing it would be to either use a SAX API parser (ie, process the start/stop tags as they come in), or do some high-level parsing to spot start/end of Atom entries and then wrap those in enough surrounding junk to make a DOM parser happy. Neither seems especially difficult to do.

I, too, love the <sorryTooSlow/> tag. And marvel at the people who ever thought their architecture could take 6+ incoming posts a second and do useful things with them, without a LJ sized infrastructure.

Ewen

Bloody brilliant. You're like that kid at school. The one in the school yard. With the firehose. Going full tilt. A kid behind you says, "Brad?," and you turn casually then watch as he skids on his ass across the blacktop into a cinder block wall. "Brad, could you?" You turn again.

Won't somebody think of the children?!!?!

"You get to drink from the firehose!!!! Are you READY?"

you guys should be like google and get a giant LCD screen for your office, and just have this scrolling past it all day.

it's the live angstweb!

i hesitate between spawn of satanor saviour of mankind :)

neat... http://www.knams.wikimedia.org/~kate/lj/words.html

Nice. Of course the next thing that occurred to me was, "What about usernames?" And apparently it was to you too... http://www.knams.wikimedia.org/~kate/lj/users.html

C:\Documents and Settings\Thomas>nc danga.com 8081
GET /atom-stream.xml HTTP/1.0


C:\Documents and Settings\Thomas>

Ideas? About the only thing I can think of is netcat only sends 0x0a, not 0x0d 0x0a.

You need 0x0d.

hilarious! thank you.

Wow, Safari doesn't like that at all, if you turn it into a URL. Presumably because the atomStream tag is never terminated.

Must be pretty quiet... I have no trouble keeping up. :)

Is this public entries only?

Of course.

Now all I need is a machine and link fast enough to download, parse and forward all of my friends entries to a jabber client.

Best Telnet session ever.

I can see it now. Someone monitoring that and producing the LJ sextalk sub-feed containing every sex-related post. Or the LJ censored images feed comparing this to the lj images feed and reporting differences.

I suppose a few places could keep up; those doing a billion queries per day already have probably got a handle on what keeping up means.

So, um... is it unreasonable to ask what "an increasing number of companies" want with a real-time feed of all public posts? Sounds a bit paranoia-inducing.

Everybody loves this blogging thing lately and wants to link/index/aggregate/analyze the data. Nothing scary... just people trying to compete on making blog data the most interesting.

Sweet!

<link href='http://www.livejournal.com/users/buttonfeind/16132.html' />
<content type='html'>
Hey, does anyone know the address for grant hall? I have to get something sent there and it's impossible to find it on the website. That website is crap by the way, complete crap.

The post is "friends only" for possible stalker purposes, by the way.

</content>

Even if there are no bugs (likely), from a user-happiness perspective it might be nice to only push these posts a few minutes after they've been up and the security is still public.

The data comes from the same place as the recently updated data on the front page of the site, so if there are bugs about security, they're ancient bugs. And also, my code does a redundant filtering pass checking doing:

foreach my $p (@$recent) {
next unless $p->{security} eq 'public';

Also, the existing front-page data isn't artificially lagged. Though it might not be a bad idea... just kidna lame.

Fat content was the big motivator here.

This is essentially a changes.xml format. Why not just implement that format instead of inventing your own?

Not that changes is all THAT but a lot of people already support it.

Kevin

Fat content.