brad's life - On PubSubHubbub and rssCloud ... [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

On PubSubHubbub and rssCloud ... [Sep. 15th, 2009|11:17 am]
Previous Entry Share Next Entry
[Tags|]

Are you confused about the difference between PubSubHubbub and rssCloud?  You're not alone.

Here's how the confusion came about:

Dave Winer invented rssCloud way back in the day.  It only distributed lite pings, the callback endpoint was the IP address that you subscribed from, and nobody really ever implemented it, so you probably never heard of it.  We sure hadn't.

Fast forward 5 or 6 years.  Brett Slatkin and I want to fix the polling problem and we're annoyed that all companies seem to have an internal pubsub system, but none of them work on the Internet, and XEP-0060 just isn't getting adopted, probably because XMPP weirds people out.  We start sketching out PubSubHubbub.  From day 1 we do all development in the open, on pubsubhubbub.googlecode.com.  Initially we only target Atom for simplicity until we got a prototype working.  We work on PSHB for about a year, during which time we hear about rssCloud and are impressed at the foresight but reject it as not satisfying our design goals.   After a year of working on PSHB, we demo it at the "TechCrunch Real-time Stream Crunch-Up" event.  We add RSS support a few days before the event because it's trivial at that point.  It's unfortunate that both Atom and RSS exist, but that's reality, so we support both.

Right after we present, we get a request to call Dave Winer.  He wants to "do voice", so Brett and I hop on our cellphones and the three of us have a conference call, with Brett and I outside on the street trying to find places to stand with less road noise.  Over the next 15-30 minutes, we slowly walk him through PubSubHubbub, repeatedly, explaining why webhooks and fat pings are important (no thundering herds DoSing publishers!), explain all of our design goals (pushing complexity to the hub, keeping publishing simple, decentralized, using HTTP, etc, etc...)

Dave Winer writes an article praising PubSubHubbub.  Great!

Dave Winer reads the PSHB spec and notices it still says "Atom only, not RSS".  Shit.  We forgot to update the spec after we added RSS support.

Perhaps due to our RSS documentation omission, or perhaps because he realized pubsub was finally in vogue, he's now gone and dusted-off and augmented his old rssCloud protocol that's RSS-centric.

The arguments in favor of rssCloud go something like this:  'we can't have BigCo control the spec.  We should have an independent spec!'  Or, in his words: "Google sux".  To reply to that specifically: This isn't even a Google-initiated project --- it's Brett & my 20% (or 5%?) time project, trying to fix something we find annoying on the web. We've been transparently working on this in the open from day 1.  Yes, we happen to be Google employees.  We have no internal docs or project plan on this.  If Dave wants something different, he's just as welcome on our mailing list as everybody else (many individuals and companies, working together to build consensus....).  Instead, he's heavily promoting the largely-unchanged rssCloud and not wanting feedback. Seems silly, but that's that.

Unlike rssCloud, which Winer says is frozen and a "done deal", the Hubbub protocol isn't frozen.  It's in development so far as we'll make changes and additions that are good and useful, and try hard not to break backwards compatibility (especially on pinging).  We have a few major things yet mostly untackled (including distribution of private content).  The rssCloud mailing list says "This is a mail list not a standards body".  If you'd like to work on a standard, join the PSHB mailing list.

Some of the good articles on the technical differences between the two protocols:
Anyway, I apologize for all the confusion.  I feel like had we only promoted RSS more heavily in the 0.1 draft of the spec, I wouldn't be writing this blog post today.

Hopefully this is the last I'm going to say on this topic.  Back to doing productive things....
LinkReply

Comments:
[User Picture]From: signe
2009-09-15 06:56 pm (UTC)

(Link)

I've had to have this exact discussion a number of times already, and I'm probably getting as tired of it as you are.
[User Picture]From: signe
2009-09-15 07:55 pm (UTC)

(Link)

One of the reasons that PSHB continues to lead in my mind, is that rssCloud is... well, only for RSS. It's a separate system, and has nothing to do with the format that it's being used in. (Element namespaces aside...)
From: (Anonymous)
2009-09-15 07:05 pm (UTC)

RSS's cloud was in fact implemented

(Link)

Hi Brad -- Dave Winer here.

We implemented the cloud element in Radio and Manila, two leading blog platforms, in 2001 and 2002. So it's not true that no one implemented it. And it wasn't hiding, it was right there in the RSS 2.0 spec, which is a very widely implemented format.
[User Picture]From: brad
2009-09-15 07:10 pm (UTC)

Re: RSS's cloud was in fact implemented

(Link)

Sorry, I meant widely implemented. 2 implementations isn't very many.
From: blasdelf
2009-09-15 08:27 pm (UTC)

Re: RSS's cloud was in fact implemented

(Link)

[User Picture]From: signe
2009-09-15 07:52 pm (UTC)

Re: RSS's cloud was in fact implemented

(Link)

One could make the argument that since you integrated it and nobody else ran with it, that it was really more of an example or prototype.

If I write it, and I'm the only user, it's a hard sell to say that it was implemented. IMO, of course.
From: (Anonymous)
2009-09-15 08:02 pm (UTC)

Re: RSS's cloud was in fact implemented

(Link)

Remember that this is Dave Winer we are talking about - he's pretty much the expert when it comes to these sort of things. He basically is the RSS spec.
From: blasdelf
2009-09-15 08:25 pm (UTC)

Re: RSS's cloud was in fact implemented

(Link)

Dave Winer is the RSS 'expert' because he makes it up as he goes along — specifications are meaningless to him, he'll just edit the canonical version live without bumping the version number, much less telling anyone.
From: sherm
2009-09-15 07:42 pm (UTC)

(Link)

I would take it as a compliment that he's complaining about something you're doing.
From: (Anonymous)
2009-09-15 08:53 pm (UTC)

Fusion

(Link)

Brad. Kudos for Pubsubhubbub: this is a key piece of technology. I hope that you and Dave can found a middle ground: the competition is not an alternative spec but the proprietary micro messaging platforms. May be the solution is to: get pubsubhubbub, make RSS a first class citizen, add a section explaining how it is an evolution of RSS Cloud) and even use the RSS cloud name (in my opinion pubsubhubbub is too complex of name for it to become mainstream).

Can we try to set up a time for you and Dave to sit down and discuss this?

-Edwin Khodabakchian (edwink@devhd.com, co-founder of feedly)
From: (Anonymous)
2009-09-15 10:21 pm (UTC)

Re: Fusion

(Link)

Edwin: Could you elaborate on what you mean by first-class citizen? RSS is fully supported (0.92, 1.0, 2.0) by the reference implementation and documented here (http://code.google.com/p/pubsubhubbub/wiki/RssFeeds).
-Brett
[User Picture]From: erik
2009-09-15 10:24 pm (UTC)

Re: Fusion

(Link)

Heh, this is starting to sound pretty epic. Like Israel and Palestine sitting down to talk peace negotiations.
[User Picture]From: ydna
2009-09-16 01:08 am (UTC)

Re: Fusion

(Link)

Too complex a name? This is just the next step in Brad's effort to see how far he can push the naming insanity. We keep slurping up his technology that he spews out and the names just get weirder. What's in a name? The name means nothing.
From: rcade [typekey.com]
2009-09-15 09:36 pm (UTC)

I'm Glad PSHB has a Public Process

(Link)

You're right to tout the benefits of how PubSubHubbub is being created.

As a member of the RSS Advisory Board, I can't stress enough how important it is for specs to be written as a public process. The RSSCloud Interface should not be controlled by one person, as I argue today on my blog:

http://workbench.cadenhead.org/news/3559 (http://workbench.cadenhead.org/news/3559)
[User Picture]From: kragen
2009-09-16 01:49 am (UTC)

(Link)

As usual, interacting with Dave Winer leads to no good. You should have known better than to try to engage with him. Best just to ignore him, stay far, far away, and don't let him get involved in anything you care about.
[User Picture]From: anildash
2009-09-16 04:26 am (UTC)

(Link)

Kragen, fwiw, I've been interacting a lot with Dave lately, and a bit with Brad and Brett on this, to try and help explain the benefits of all this work to outsiders. Here's the real-world perspective: All three are on the side of the angels. They're doing good work, iterating in their own way, and are gonna get good results. And frankly, they've all been magnanimous and open to feedback, even though they're obvious coming at this from slightly divergent points of origin.

Do I wish things were a little less complicated? Sure, but ultimately, the distinctions between these systems will filter out, and we'll have one or two (but less than three!) systems that enable a valuable new ecosystem. Let the shippers ship, and don't be one of those guys who's flaming one side or the other. If you have an opinion, then commit code or make something useful, but I care too much about the overall success of the effort to have decentralized realtime infrastructure to let this turn into another stupid internet shitstorm that distracts from the work at hand.

In short, don't be That Guy. I don't think you wanna be That Guy. :)
[User Picture]From: brad
2009-09-16 05:10 am (UTC)

(Link)

Yes!

An rssCloud <-> PSHB proxy would be trivial. I'm just happy we're making data realtime. The only sad part is that developers might be confused for awhile.
From: (Anonymous)
2009-09-16 06:16 am (UTC)

Dave Winer is the Kanye West of Tech

(Link)

Let's be honest here and put what everyone knows to be true on the table. Dave Winer is a jerk. That's been amply demonstrated on innumerable occasions by Dave Winer himself. He is also a grumpy old narcissist who wants his name associated with everything. Any reasonable person looking at pubsub vs. RSS cloud would conclude that pubsub is a better solution - but Dave can't handle that so he's going to bitch his way into the spotlight. He "could have invented Twitter" - but he didn't, so he's bitter about that. RSS cloud "could have been big" - but it wasn't, so now he's bitter about that. What would truly be great is if everyone would just ignore Dave Winer and let RSS cloud go the way of several of his other dumb ideas that never went anywhere. (Flickrfan, anyone?) Dave Winer is the Kanye West of tech.
From: blasdelf
2009-09-17 12:27 am (UTC)

Re: Dave Winer is the Kanye West of Tech

(Link)

That's an unfair comparison — Kanye West has taste, talent, produces original work that is massively successful on its own merits, and most importantly he does great collaborative work.

Dave Winer has none of those qualities.
[User Picture]From: daveman692
2009-09-16 07:10 am (UTC)

(Link)

And just makings hubs do the hard part of understanding various types of incoming "pings", but then normalizing their outgoing notifications.
From: edge_walker
2009-09-16 02:57 pm (UTC)

(Link)

I don’t see how Brad ignoring Winer, as Kragen suggests, can possibly impede anyone from shipping anything. If it can do anything, it is to speed things up, and prevent pain further down the road. (I hope Brad and Brett haven’t lost that much time already.) So Kragen’s advice is perfectly compatible with your cause.

[User Picture]From: anildash
2009-09-16 03:24 pm (UTC)

(Link)

Point taken, but the larger idea I was trying to get across is that insulting *any* of the guys hacking on this stuff is just going to distract them from doing good work, because we're all human and want to defend ourselves or our friends or whatever. I don't worry about Brad or Brett or Dave getting code shipped -- I worry about everybody else wanting to see the spectacle of a stupid flamewar.
From: edge_walker
2009-09-16 04:27 pm (UTC)

(Link)

Is a track record fact or insult?

I don’t hate Winer… I don’t even know him. I’ve just seen too much of his spectacle already – thus: don’t worry; and, I’d rather people studiously ignore him.

[User Picture]From: kragen
2009-09-16 03:14 pm (UTC)

(Link)

Anil,

You've been around long enough to know that although Dave's heart is apparently in the right place, his mental health issues create a high risk of things falling apart when he touches them. In this case, Dave is That Guy: he wrote a spec ten years ago and implemented some prototypes, and now that other people are doing the work of making something similar into reality, he wants to be involved and get credit. This is a bad idea. Dave Winer is not an asset. He is a liability. Every hour you spend interacting with him is an hour you're going to regret for the rest of your life when he viciously turns on you.

Here's a short list of some of the projects Dave has attempted to derail over the years: RSS 1.0, Atom, JSON (!), WDDX, Bronco/Casbah — all because they did or intended to do something similar to some project of Dave's, but not the way he wanted. The best case is something like JSON, where people basically ignore the guy and he goes away. The worst case is Casbah, where the project fades away because everybody loses interest.

So what makes you so optimistic that he's changed?
From: (Anonymous)
2009-09-16 07:55 pm (UTC)

Let's look at WordPress...

(Link)

We could have had one protocol instead of two if WordPress implemented PSHH instead of RSSCloud. Is it too late to convince them to switch?
[User Picture]From: mart
2009-09-16 06:43 am (UTC)

(Link)

This reminds me of the early days of OpenID when there were all the other random OpenID-like systems that did mostly the same thing but with different bits on the wire. I think just as with OpenID vs. TypeKey (for example) it won't be too hard to map between these things so that one can be implemented in terms of the other.

While we're on the subject of bridging things, it'd be neat to have a hubbub hub that can use weblogs.com-style pings as an optional second notification mechanism so that folks running software that already supports such pings don't need to install anything new to get started.

From: onebigfluke.com
2009-09-16 07:17 am (UTC)

(Link)

I already did this in July, but we haven't turned it on:

http://code.google.com/p/pubsubhubbub/source/browse/trunk/hub/main.py#1762

The issue is that XML-RPC pings only take the channel URL, so you have to parse HTML to rediscover the actual feed URLs, or maintain a relation of feeds to channel links (the latter is easier, actually).

Otherwise, I still think that light pinging is going to hinder everyone long-term. The performance characteristics are bad. I'm starting to expound on why here:

http://code.google.com/p/pubsubhubbub/wiki/ComparingProtocols

Feedback would be appreciated!
From: evan
2009-09-16 07:53 am (UTC)

(Link)

Comments on some posts I read indicate they think only Google can run a hub, which is where the "bigco sux" stuff comes from. Can you preemptively respond to that? (Perhaps by amending your post in the paragraph where you talk about 20% time.)
From: evan
2009-09-16 07:56 am (UTC)

(Link)

PS they just checked libjingle into Chrome since it's used for bookmarks sync. If you have any grand plans involving XMPP bridges into JS I'd be interested in hacking them out.
From: onebigfluke.com
2009-09-16 02:44 pm (UTC)

(Link)

So far, I did a presentation on how to build a hub here:
http://code.google.com/p/pubsubhubbub/wiki/BuildingAHub

But more documentation (and a more thorough compliance test suite than we already have) would be great.
[User Picture]From: mirukux
2009-09-16 11:20 am (UTC)

(Link)

and XEP-0060 just isn't getting adopted, probably because XMPP weirds people out

what about GTalk and Wave? and how will PubSubHubbub communicate with these? XMPP-0060, some other gateway or, ugh, a bot?

what do you think of the Superfeedr architecture?

why can't gateways and clients start using XEP-0004 more like XEP-0060 asks? what if Google had bridged this chicken-and-egg gap.. (possibly with the help their own handrolled protocol a la VOIP *wonders how long until someone implements Jingle in Wave..*)?
(Deleted comment)
[User Picture]From: kragen
2009-09-16 04:35 pm (UTC)

(Link)

That's completely unfair. Ted Nelson actually invented a lot of cool stuff, wrote a couple of important books, and funded a lot of research.
From: blasdelf
2009-09-17 12:32 am (UTC)

(Link)

And the problem with Nelson's ideas is their utopian idealism -- Winer's ideas are simply terrible.