EchoDitto Labs - Using Blip.tv With FeedAPI - Comments

That empty array index is

Ryan Parman — Sat, 03 May 2008 12:48:42 -0700

That empty array index is actually a namespace URL... for RSS 2.0. Since RSS 2.0 doesn't have a namespace, and SimplePie does a lot of indexing with namespaces, that's just how it ended up coming out. :(

Thanks for the thoughtful

Tom — Mon, 24 Mar 2008 09:07:31 -0700

Thanks for the thoughtful reply, Ryan. I don't mean to just bitch about SimplePie, although I'm sure that's how it came off -- SP really is a great project. And in fact the sanitization I complained about is not happening in SimplePie: it's definitely Drupal that's killing the data somewhere along the way.

It's interesting to hear that the custom tag data is already available in the [data] object of... the parser? I definitely did a print_r on the item objects, and thought I probably would've done the same for the parser. Perhaps I forgot to. It's definitely not available in the item object, though, which seems like the logical place for it.

In any case, I do have to stick by my complaint about the variability of the data structures returned by the custom-tag-grabbing routine, particularly when retrieving an item with nested subelements. Check out how you access an FLV included by the widely-supported MediaRSS namespace:

$item[0]['child']['http://search.yahoo.com/mrss/']['content'][0]['attribs']['']['url']

Using an empty string to index an array is pretty awful. Ick.

"This is SimplePie's fault.

Ryan Parman — Sun, 23 Mar 2008 21:37:09 -0700

"This is SimplePie's fault. Rather than simply walking the XML structure of the feed and throwing it all into data structures, SimplePie decides it's smarter than the feed author. If you want an item attribute from a custom RSS namespace, you have to ask for it."

Or, you could walk the data structure, which is stored in the SimplePie::data property. The API methods are just a simplified way of accessing more commonly used/requested data.

"...the data structures returned by SimplePie vary pretty wildly by tag."

I could be mistaken, but if you're talking about the data structures that map to the XML, we have a consistent format for noting the XML attributes and child data. In particular, Media RSS is not as clear a spec as I'd hoped, the spec was broken for over a year, and there are numerous sites implementing both correct and broken Media RSS all over the web. We've done our best to normalize this data so that our users don't have to do as much hacking. Some of the things you're trying to do might be simpler if you used SimplePie's enclosure-related methods which normalize data between RSS enclosures, Media RSS, and iTunes RSS.

"Worse, somewhere along the line ... the data is sanitized and some tags are stripped, making it necessary to urlencode() fields like the video embed tag."

By default SimplePie does sanitize data and strip out potentially "dangerous" tags and attributes, as per suggestions from Mark Pilgrim on how to consume RSS safely . This sanitization can be turned off pretty simply using a SimplePie config option, and I believe that the guys who do Drupal's SimpleFeed module do this and use Drupal's internal sanitizer instead.

In the end, we have a support forum/list as well as an IRC chat available for asking these very kinds of questions, and complete API documentation in our wiki. We probably could have answered your questions relatively quickly and saved you some headache. :)

Using Blip.tv With FeedAPI

Tom — Fri, 21 Mar 2008 08:42:22 -0700

Yesterday Development Seed was kind enough to give Chris & me a rundown of how the Drupal community is organizing its participation in the Google Summer of Code program. Along the way I got a chance to chat with Alex and Ian about FeedAPI and FeedAPI Mapper, two excellent projects that DevSeed ushered into being through last year's Summer of Code and now continues to maintain and extend.

I've just begun using FeedAPI for the first time in a project destined for production, and so far I'm very pleased with it. It offers a more fully-considered alternative to aggregator.module — and with the addition of the optional Mapper module it becomes simple to turn aggregated RSS items into Drupal nodes, with the items' attributes stuck in whatever CCK fields you care to create. It's really slick.

In my case I'm using it as an integration point for Blip.tv. Our client needs video capabilities, but I saw no reason why we should mess around with transcoding, customizing an FLV player and all the rest of the headaches that come with web video (been there, done that). Blip does all of that stuff very well, and has social features baked in, too. I'd rather just have the client upload their videos there, then count on FeedAPI to turn them into nodes that can be exposed through Views. Any configuration that we can't get from the Blip RSS feed can be manually handled by an editor — Workflow-NG fires off a "please come edit and publish me!" email whenever a new video node is created.

But there's an awful lot of data available in the Blip RSS feed. In addition to things like tags and titles there's the source FLV, the embed code for the flash player, the video's runtime and the thumbnail graphic URL (if any). The problem was that the SimplePie parser module included with FeedAPI wasn't picking any of these attributes up.

This is SimplePie's fault. Rather than simply walking the XML structure of the feed and throwing it all into data structures, SimplePie decides it's smarter than the feed author. If you want an item attribute from a custom RSS namespace, you have to ask for it.

Right now, there's not a good way to do this. The good news, though, is that the DevSeed guys are aware of this limitation and are thinking hard about how future versions of SimplePie should be designed to accommodate custom feed item attributes. I'm hopeful that I'll be able to contribute to that effort, but unfortunately it's one that involves larger questions about FeedAPI's architecture and place in the Drupal project. It's not something that's going to be fixed right away, in other words.

But I was on a deadline, so a stopgap was in order. What to do? I could try to add to the hooks exposed by the parser_simplepie submodule. But that would be adding a module system to a module system employed by a module. That's a pretty deep rabbit hole. Besides, it's a lot of work that would probably be discarded in favor of something smarter by the bright folks maintaining the project. And this is just a one-off requirement.

Next I considered adding a configuration page to the parser_simplepie module, where a user could specify the namespaces and tags that they'd like retrieved for each feed item. But this is a no-go: the data structures returned by SimplePie vary pretty wildly by tag. Worse, somewhere along the line (CCK? I haven't looked) the data is sanitized and some tags are stripped, making it necessary to urlencode() fields like the video embed tag. Yeah, you could include a block for eval'ed PHP on the aforementioned settings page. But at that point, what are you really buying yourself? You'll still need expert knowledge to use it. At that point the difference between putting the code in an admin page or in the module basically winds up being that, in the case of the former, a curious administrator can accidentally break things.

So, reluctantly, I ended up forking parser_simplepie and adding the following custom code to _parser_simplepie_feedapi_parse(). If you're trying to use Blip, maybe you'll find it useful (I've also included a patch). It grabs the video embed code, source FLV, graphic, runtime and description (the standard RSS description field gets the player stuffed into it, too; this way you don't have to strip it out yourself). In the case of the graphic it actually downloads the file if it hasn't already — handy if you're going to have to make thumbnails out of it with imagecache. Of course for performance reasons this also means that if you need to update the graphic you'll need to make sure the new version you send to Blip has a unique filename — a little hacky, I know. A better solution would be to patch imagecache so that it can handle remote files, periodically redownloading and examining their md5's on the cron to determine if the local copy of the source file needs to be refreshed. But that's a topic for another blog post.

// BEGIN blip.tv-specific attribute-grabbing

// flv
if($tag_media = $simplepie_item->get_item_tags(SIMPLEPIE_NAMESPACE_MEDIARSS,'group'))
	$curr_item->options->mediarss_flv = $tag_media[0]['child']['http://search.yahoo.com/mrss/']['content'][0]['attribs']['']['url'];		

// player code
if($tag_media = $simplepie_item->get_item_tags(SIMPLEPIE_NAMESPACE_MEDIARSS,'player'))
	$curr_item->options->mediarss_player = urlencode($tag_media[0]['data']);

// runtime
if($tag_media = $simplepie_item->get_item_tags('http://blip.tv/dtd/blip/1.0','runtime'))
	$curr_item->options->bliptv_runtime = intval($tag_media[0]['data']);

// pure description
if($tag_media = $simplepie_item->get_item_tags('http://blip.tv/dtd/blip/1.0','puredescription'))
	$curr_item->options->bliptv_puredescription = intval($tag_media[0]['data']);
	
// thumbnail
if($tag_media = $simplepie_item->get_item_tags(SIMPLEPIE_NAMESPACE_MEDIARSS,'thumbnail'))
{
	$curr_item->options->mediarss_thumbnail_remote = $tag_media[0]['attribs']['']['url'];
	$file_dst = preg_replace('/[^\w\.\-]/','_',str_replace('http://','',$curr_item->options->mediarss_thumbnail_remote));
	$d = file_directory_path() . '/feedapi';
	if(!file_exists($d))
		mkdir($d);
	if(file_check_directory($d))
	{
		$file_dst = file_create_filename($file_dst,$d);
		if(!file_exists($file_dst))
			if($file_data = file_get_contents($curr_item->options->mediarss_thumbnail_remote))
				$file_dst = file_save_data($file_data,$file_dst,FILE_EXISTS_RENAME);
	}
	if(file_exists($file_dst))
		$curr_item->options->mediarss_thumbnail_local = $file_dst;
}		

// END blip.tv