Its good to be back in Portland for my favorite geek convention: O'Reilly's Open Source Conference. The overcast sky in Portland is making it a little easier this year to focus on the plethora of excellent speakers and sessions. The first session to really grip and and speak to me was Rabble and Kellan's "Beyond REST? Building Data Services with XMPP PubSub" presentation.
They started out their presentation stating that they were not "Jabber Heads", but that they were in the business of building web sites. For Rabble and Kellan, Jabber presents one more tool in their huge tool-chest to build web sites. Jabber wasn't designed to be a part of a functioning web site, but they insist that it works great for building social web sites that require many people to be notified of updates.
For example, Kellan talked about FriendFeed, a site that lets their users know when their friends share new items. In this example, Kellan pointed out that FriendFeed polls Flickr 2.9 million times in order to check on updates for 45 thousand users. And of those 45 thousand users, only 6.7 thousand are logged in at any one time. This of course, its a poor way of checking for changed content. Kellan says: "Polling sucks!"
To solve this problem its key to leave standard REST web services behind and find a way to use message passing, which is a direct communication way of notifying users of changed content. The open and mature infrastructure that Rabble and Kellan found to use for this service is Jabber. Jabber has 10 years of experience of passing messages around the internet and has been embraced by many companies including Google.
XMPP, Jabber's protocol, works well for message passing and does not have many of the problems/limitations of HTTP:
- XMPP works over persistent connections
- It it stateful (SSL becomes cheap)
- Designed as an event stream protocol
- Natively federated and asynchronous
- Identity, security and presence are built in.
- Jabber servers are built and deployed to do this stuff.
Given this, Kellan and Rabble decided to piggy-back a notification system on Jabber by sending XML fragments using a PubSub paradigm. In this context, PubSub is a simple method for passing XMPP pubsub stanzas via Jabber. PubSub is nothing more than a convention for how to send XML via Jabber, including a method for embedding Atom fragments in the XML.
Rabble presented using XMPP for FireEagle, Yahoo!'s new personal geolocation service that allows users to provide their current location to other users. For a few users and a few updates you can paginate the data stream into RSS/atom feeds. But once you have more than a few users and frequent updates a paginated stream cannot keep up. What if a user publishes more updates than can an RSS feed can capture? Updates get lost -- and for applications using FireEagle missing an update presents a critical flaw. Using a system like XMPP, FireEagle can rely on Jabber to deliver all the updates -- exactly what Jabber was meant to do.
Kellan also applied XMPP/PubSub to Flickr and how a Flickr update "Firehose" might work. If Flickr sends a ~2k an atom enriched packet for each new public picture posted at a rate of 60 updates a second, it would take roughly a megabit of traffic. Even a normal DSL line can handle one mbit of traffic, so the network effects are manageable on this level, compared to the polling system that FriendFeed uses. (Kellan also points out that FriendFeed is not doing anything wrong at all -- the current web service centric model is simply insufficient for this type of service.)
To deploy your own message passing service based on XMPP/PubSub, you'll need to follow these 4 easy steps:
- Get a Jabber client library. There are many available for all the popular languages.
- Set up a Jabber server -- again there are many available to choose from. Turn off the features you won't be needing. (e.g. creating new accounts)
- Build a component (according to Jabber XEP-0114)
- Integrate the message passing system in your own site.
Pretty simple, overall! The beauty of this approach comes from the fact that all off-the-shelf components were used to build this new notification system. No new magic technology is being created to enable this system, which is a personal metric of mine for determining the likelihood that a new system will succeed.
It's clear that REST web-services provide the heavy lifting for many Web 2.0 sites, but its also clear that REST and its inherent polling mechanism isn't the best way of building a user notification system. With social networking sites not about to fade away, we're going to see an increasing need for capable message passing sites. And since Jabber is a well established and supported system, it only makes sense to piggyback on this great technology. Thanks for the awesome presentation Rabble and Kellan!
Update: Rabble posted the slides for this talk.
(And a big thanks to Tim O'Reilly for letting me guest blog OSCON here!)
