Log in

No account? Create an account
Never ending feed of Atom feeds - Never ending feed of Atom feeds - brad's life Page 3 — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Never ending feed of Atom feeds [Aug. 16th, 2005|12:58 pm]
Brad Fitzpatrick
[Tags|, , , ]

An increasing number of companies (large and small) are really insistent that we ping them with all blog updates, for reasons I won't rant about.

Just to prove a point, I flooded a couple of them and found that sure enough, nobody can really keep up. It's even more annoying when they don't even support persistent HTTP connections.

So --- I decided to turn things on their head and make them get data from us. If they can't keep up, it's their loss.

Prototype: (not its final home)

$ telnet danga.com 8081
GET /atom-stream.xml HTTP/1.0<enter>

And enjoy the never ending XML stream of Atom feeds, each containing one entry. And if you get more than 256k behind (not including your TCP window size), then we start dropping entries to you and you see:

<sorryTooSlow youMissed="23" />

I think soon we'll get TypePad and perhaps MovableType blogs all being sent through this. The final home will probably be on a subdomain of sixapart.com somewhere, including documentation better than this blog entry.

And yes, I'm sure my Atom syntax is bogus or something. I spent a good 2 minutes on that part of it.

Page 3 of 3
<<[1] [2] [3] >>
[User Picture]From: mendel
2005-08-19 02:19 pm (UTC)
Aw, where'd it go?
(Reply) (Thread)
From: bobwyman
2005-08-21 11:52 pm (UTC)

Just stream Atom files. You don't need to define a new format.

As you know, since I asked for this over a year ago, I think your proposal to push streams of LiveJournal updates is wonderful! At PubSub, we currently read the LiveJournal latest.bml file as frequently as once a minute, if not more frequently. The result has been that we're able to keep up to date on LiveJournal posts with massively less bandwidth and processing cost than with any other blogging system. Thanks for moving in the direction of the streaming the updates.

While I like what you're doing, I must admit that I am not terribly excited by the *way* you are proposing to do it. I believe that what you want to do can be done by streaming a feed of atom:entry's which contain atom:source elements carrying the feed metadata. The atom:source element was explicitly designed for exactly the kind of application you are proposing. (i.e. the generation of an aggregate feed). I've discussed your proposal on both the FeedMesh and Atom-Syntax lists. Please take a look at the alternatives I propose in these messages:


Thanks again for considering push feeds of the LiveJournal updates. My hope is that other blog hosting platforms will follow your lead and implement similar feeds as soon as possible. Hosters who stream feeds in this manner will, I think, be seen to be serving the needs of the users much better since it will be much more likely that user's posts will get into search and/or monitoring systems. Also, the bandwidth and processing impact of having services consume the stream of updates will be much less than required if they are all fetching raw RSS/Atom files. This means that a greater proportion of the bandwidth and processing power you consume can be dedicated to providing your users with new and/or faster services. A win for everyone!

bob wyman
CTO, PubSub.com
(Reply) (Thread)
From: bobwyman
2005-08-22 12:13 am (UTC)

Less White-Space please...

It would be very nice if you were to remove some of the unnecessary white-space from the feed. A feed like this can only be usefully read by a machine I think. We'd all save a great deal of bandwidth if we didn't have to pay for the white-space. (Yes, this is a very small point...)

bob wyman
(Reply) (Thread)
Page 3 of 3
<<[1] [2] [3] >>