rss

Yesterday Development Seed was kind enough to give Chris & me a rundown of how the Drupal community is organizing its participation in the Google Summer of Code program. Along the way I got a chance to chat with Alex and Ian about FeedAPI and FeedAPI Mapper, two excellent projects that DevSeed ushered into being through last year's Summer of Code and now continues to maintain and extend.

I've just begun using FeedAPI for the first time in a project destined for production, and so far I'm very pleased with it. It offers a more fully-considered alternative to aggregator.module — and with the addition of the optional Mapper module it becomes simple to turn aggregated RSS items into Drupal nodes, with the items' attributes stuck in whatever CCK fields you care to create. It's really slick.

In my case I'm using it as an integration point for Blip.tv. Our client needs video capabilities, but I saw no reason why we should mess around with transcoding, customizing an FLV player and all the rest of the headaches that come with web video (been there, done that). Blip does all of that stuff very well, and has social features baked in, too. I'd rather just have the client upload their videos there, then count on FeedAPI to turn them into nodes that can be exposed through Views. Any configuration that we can't get from the Blip RSS feed can be manually handled by an editor — Workflow-NG fires off a "please come edit and publish me!" email whenever a new video node is created.

It's been a while since I gushed about Yahoo Pipes, hasn't it? Well, the RSS-combining service remains insanely powerful. This morning I decided to try using it with one of my other favorite web technologies: jQuery. When combined, the two can take the place of many aggregation tasks that previously required a server.

Here's the problem I set out to solve: a bunch of my friends write for more than one blog. They usually have a personal site and contribute to a number of other multi-author blogs. Sometimes they keep a list of their posts in a del.icio.us sidebar block. But maintaining that can be kind of a pain — they have to remember to create a new del.icio.us entry every time they post. Clients like Pukka make using del.icio.us a breeze, but this workflow is still not as automated as it could be (in that it exists at all).

As you know, Pipes makes it easy to combine and filter RSS feeds. Here's a pipe I threw together containing feeds from a couple of EchoDitto clients and our main blog. Naturally we love all of our clients equally, but for my friends' purposes a simple Pipes filter block could be applied to check the dc:author field, ensuring that no coauthors' posts are retrieved. The result will be an aggregated feed of the author's entries.

But how to retrieve them? jQuery 1.2 added in support for JSONP, an unbelievably slick technology that allows Javascript to load JSON data from different domains. There are two things you need to know to get this to work with Pipes: how to get the service to spit out JSON, and how to pass it a JSONP callback (which safely assigns the JSON to a given variable name — without this, loading cross-domain data would be a huge security hole). Both things are accomplished by adding parameters to your pipe's querystring.

There's a pretty popular blog with extremely active comment threads that I like to participate in. The threads are so active that they're pretty much the blog's entire raison d'etre — posts are conversation-starters more than anything else. An average thread runs for at least a hundred comments, and there are several of them a day.

The conversation is fun, but it can be a real pain to keep up with it. Participants are mostly reduced to constantly-refreshing their tiny comment pop-up windows. If you've got a real job to do, it becomes nearly impossible to be a part of the thread.

I wasn't too happy about that, so I cooked up a bot to solve the problem. Instead of constantly-querying threads I can simply tell the bot to pay attention to a particular discussion and IM me when new comments in it show up in the comment RSS feed. I was pleasantly surprised by how well this ended up working, so I've generalized the code a (very) little and thought I'd post it here.

A few things, though. First, the bot uses the AIM network. I realize that the cool kids are all playing with Jabber. But I only use GTalk and AIM, and my mostly-uninformed impression is that Google's implementation of Jabber may not be as open as the protocol's designers intended. Besides, AIM bots are a pretty well-developed technology.

Second, the bot was designed with a very specific purpose and platform in mind. It's likely that you'll have to mess with some Perl to adapt it to whatever RSS-and-IM-involving application you have in mind. But it should at least serve as a good starting point.

With that said, let me know if you have a use for it or need some help bending it to your will.

• echoditto-rssbot.tar.gz

The popular Freakonomics Blog has recently moved to the New York Times, and along the way it's dropped its full-text RSS feed. Stephen Dubner wrote a thoughtful explanation of the reasons for the switch — basically, advertisers aren't comfortable with RSS — but judging from the comment section the blog's readers are still upset.

Well, let me offer a gentle reminder that our very own full text RSS tool continues to work, and was designed for exactly this purpose. I've tested it with the new, truncated Freakonomics feed and it works great. Why not give it a try and help push advertisers just a little bit closer to grappling with the internet?

Freakonomics full-text feed

UPDATE: Alas! It looks like the Freakonomics authors have adopted the intermittent and irritating habit of writing descriptions in the RSS description field rather than including excerpts of the actual text. That confuses our general-purpose algorithm.

However, it won't stop a dedicated Freakonomics fan from creating a blog-specific script to provide full feeds. Here, I'll even get them started:

m/<div\s+class="post\-content">(.*?)\-+>/i
s/<\/?div[^>]*>//igx;

I'm a little shocked to see that we haven't talked about Yahoo Pipes around these parts. It's an awfully cool service, and one that we use around the office for various non-mission-critical functions — getting larger photos from a Flickr stream for display on the monitors in the lobby, for one thing.

For those unaware, Pipes lets you mix, translate, annotate and otherwise alchemically modify feeds of all sorts. Want to add a (possibly) relevant photo to each entry on your blog? Or only get posts from a particular author on a group blog? Or get filtered Craiglist real estate listings translated into German and sent to your mobile phone? Pipes can do all of this, and does so via a terminally-slick GUI.

But recently a friend tipped me off to Pipes' most significant capability. The folks at Yahoo! have unveiled the Holy Grail of feed-manipulation components: a JSON-powered interface to external web services. If Pipes doesn't offer some functionality you need, you can now write a web service that does and connect it to the system.

Their example code is in Java, but it didn't take much effort for me to get one working in PHP. Just install the JSON PECL Package and use code like the following:

Partial-text RSS feeds are a pet peeve of mine. I'm not alone: I've read about Dave Winer and Steve Rubel's dislike of the practice. I'm sure there are a lot of other RSS users who are similarly irked by it.

So, after having a post-workout algorithmic epiphany (it's the best time for them), I started work on a little project to fix this annoyance — and ended up quite pleased with the result. You might find it useful, too: it's a little script that creates full-text RSS feeds from partial feeds. Just enter the URL of a partial feed in the box below and hit submit. You'll be directed to a URL that will (hopefully) provide a full-text version of the feed you specified.

I've been through a few different versions of the algorithm, but this one seems to be fairly universal and stable. It won't work for every partial-text feed, but it seems to work for a lot of them. I'm sure it could be better, which tempts me to open source the algorithm and invite people to improve upon it. But I won't — not yet, anyway.

I'm sensitive to the pressures that make bloggers use partial text feeds — some of my friends depend on selling advertising to support their sites. Unfortunately, RSS simply isn't respected by marketers and their clients. Offering a full text feed means fewer page views, which means less revenue — I've been told this bluntly by a friend who wanted to offer full text, did so, then noticed his revenues were shrinking. It's hard to fault him for returning to partial-text feeds.