Tom's blog

...look no further than the comments on this TechCrunch post. Reacting to the scoop that Twitter is dumping Ruby on Rails (which almost immediately turned out to be wrong), more than 150 people decided to chime in about what Twitter's scaling problems are and how their own preferred web framework never would have encountered them. A whopping one person managed to refer to Twitter's custom message queueing software by name.

The rest presumably just like the (drag & drop HTML creation in Visual Studio|error detection in Zend|Google-enabled buzz surrounding Python), and have read "Rails doesn't scale" somewhere. I suppose their prattling is still slightly more meaningful than the yammerings of the web's legions of widget evangelists, social network triumphalists and self-proclaimed "SEO experts". But not by much.

At any rate, I think this is a good reminder of the signal:noise ratio facing our clients as they attempt to find technical help in a crowded marketplace. If you're not a technical person it's genuinely hard to tell the difference between someone who knows what they're talking about and someone who's simply regurgitating buzzwords. My rule of thumb when encountering bold pronouncements like the ones in that TC thread: ask "why?", then count the number of unexplained acronyms and buzzwords in the answer.

One of the things that I've learned over my years at EchoDitto — that's been hammered home again and again — is the importance of email. Twitter's cool, social networking is great, but your online strategy absolutely has to account for email. It's not glamorous, but it's important.

The same is true on the tech side of things. It's easy to forget: on a day-to-day basis, my wranglings with email generally include dealing with spam blacklists, ensuring that scripts don't function as open relays, and writing templating systems that'll be used to send mail. All of it's pretty boring. But it's worth keeping in mind that email, when piped to a script, can serve as the infrastructure for some pretty neat services, too.

I did this two years ago for SXSW, building an SMS app on the cheap by counting on the mobile carriers' SMS-to-email functionality. It worked pretty well, although it was a rat's nest of Perl scripts.

This week I took a pass at another application in Ruby (with a PHP frontend), and this time the resulting code is a bit less cringeworthy. The idea is simple: set your Twitter "new follower" email notifications to go to a custom email address. They'll be piped to a script, disassembled and the new follower's statistics analyzed. If it looks like the new guy is a spammer or bot, they'll automatically be blocked. If not, they won't be. Either way you'll get an RSS notification about it. You can try it out here, if you'd like.

It's not exactly going to set the world on fire, particularly since Twitter is expected to release similar functionality soon. But the project does serve as a pretty good template for how a piped email service can work.

A number of people have emailed me asking for the Full-text RSS project code. I appreciate the interest! And I'm happy to oblige so long as they send me an assurance that they won't be using it for commercial purposes — e.g. processing and selling ads on someone else's feed.

A number of people have sent me those assurances, but I've been slow to send out tarballs. I apologize! If you're waiting on one and haven't heard from me by the end of the day, please email me again: tom (at) echoditto (dot) com.

Remember this guy? Yeah, I barely do, too. But way back in 2006 I put some of the lingering knowledge I had during an earlier job together with a newfound passion for writing horrible rats' nests of Perl code. The result was a set of scripts that could scrape the rather lame HTML interface to House Information Service's zipcode+9-to-congressional-database matching service.

I'm sure my scripts haven't aged all that well, but Aaron Swartz was kind enough to take up the torch and share a set of Python scripts he coded to accomplish the same thing. Thanks, Aaron!

Those of you looking for zipcode-to-congressional-district matching would do well to check those scripts out. You might also want to follow the threads from this tweet, in which exoDitto Tim Jones asks the LazyWeb for other solutions to the problem.

I'm going to be presenting at tomorrow's Dorkbot DC meeting — anyone who's interested should come on by. I'm going to be talking about making a Fonera router talk to an Arduino, a subject I first blogged about right here.

And of course the DC nerd community also shouldn't forget about Wednesday's Drupal Meetup at Affinity Lab, which is once again being hosted by our friend Mike McCaffrey. You can find details here. I think it's safe to say you'll be able to find some of us at each of these events.

Yesterday Development Seed was kind enough to give Chris & me a rundown of how the Drupal community is organizing its participation in the Google Summer of Code program. Along the way I got a chance to chat with Alex and Ian about FeedAPI and FeedAPI Mapper, two excellent projects that DevSeed ushered into being through last year's Summer of Code and now continues to maintain and extend.

I've just begun using FeedAPI for the first time in a project destined for production, and so far I'm very pleased with it. It offers a more fully-considered alternative to aggregator.module — and with the addition of the optional Mapper module it becomes simple to turn aggregated RSS items into Drupal nodes, with the items' attributes stuck in whatever CCK fields you care to create. It's really slick.

In my case I'm using it as an integration point for Blip.tv. Our client needs video capabilities, but I saw no reason why we should mess around with transcoding, customizing an FLV player and all the rest of the headaches that come with web video (been there, done that). Blip does all of that stuff very well, and has social features baked in, too. I'd rather just have the client upload their videos there, then count on FeedAPI to turn them into nodes that can be exposed through Views. Any configuration that we can't get from the Blip RSS feed can be manually handled by an editor — Workflow-NG fires off a "please come edit and publish me!" email whenever a new video node is created.

Yesterday Ethan, Chris and I found ourselves needing to move some data into Drupal — we're building an existing client a new site and they need some blog posts moved over from their old one. It sure sounds simple. But as anyone who's actually done data migration from one blogging platform to another (or just from one version to another) can attest, it's rarely that easy. There are a few options, though, which I present here in increasing order of their likelihood to unexpectedly spiral into a huge, awful mess:

So, as Phil mentioned, we all got iPhones as a holiday bonus. And they're pretty great. JP and I held off until Macworld, lest we miss out on a new 3G handset (a longshot, I know). When that didn't happen I immediately scurried over to Pentagon City and bought a beautiful internet lozenge.

Since then I've been figuring out all of the amazing things it can do for me. On Thursday I had a stroke of inspiration. I've been looking longingly at Bluetooth presence detection setups for a while. I like the idea of having my hardware respond to my proximity, but I've got some reservations about wasting a Bluetooth dongle (and server, and Bluetooth-on-Linux configuration time) on the effort. The iPhone is the first mobile I've owned that does wifi. This seems like an opportunity to do presence detection on the cheap.

My approach is pretty simple: ping the iPhone. If it's there, I probably am, too. But how to ping it? The iPhone uses DHCP to get an address. At home I've got a router running the SveaSoft firmware. It's simple enough to configure DNSMasq to always dole out the same IP address to my iPhone's MAC.

Then I wrote a bash script to send three pings, check the number of successful replies, and log the result to a text file. I set it up to run on a minute cron and let it go overnight:

PING_COUNT=`ping -c 3 192.168.2.21 | grep "bytes from" | wc -l`
test $PING_COUNT -eq 3 && RESULT="here" || RESULT="not here"
RIGHT_NOW=`date`
echo "$RIGHT_NOW - $RESULT" >> status.txt

Here are the results from the first 24 hours:

On Wednesday Ben and I headed over to the monthly meeting of Dorkbot DC and soldered our little hearts out. I've only been to one Dorkbot before (well, unless you count the one at SXSW — but that was mostly an opportunity to buy t-shirts and drink free beer). My previous experience had been enjoyable, but maybe a little dry. It's inevitable that the speakers won't pique everyone's interest every time, I suppose.

This meeting was a lot more fun, and a lot more hands-on. The organizers did a fantastic job, preparing instruction material, assembling kits and even pre-drilling jigs for the rest of us in an effort to introduce the extremely large crowd to soldering by way of MAKE Magazine's LED cube weekend project, slightly modified to work with an Arduino.

Sure, it's a simple project, but we've never claimed to be electronics gurus. Besides, it was a great opportunity to refine our soldering skills — something I'm in sore need of after nearly trashing my Wii during a botched modchip installation.

It looks like Bug Labs has set some prices for its open hardware initiative. And frankly, I'm pleasantly surprised. Yeah, $350 for the base unit isn't cheap, nor is the $100ish average price of the add-on modules.

But after reacquainting myself with this video, it all seems pretty reasonable. The base unit runs Linux, has a wired ethernet connection and storage capabilities. The add-on LCD is the same unit as ipods ship with, and has touchscreen capabilities thrown in. All in all it seems to be a very high-quality product. I don't think I'm going to rush out to buy one — I'd rather clog my breadboard with torn-out hair and call it a "learning experience", masochistic though it may be. But I certainly won't complain if Nicco shows up with one in the office.

As you might've noticed, this blog went offline for while yesterday. Sorry about that! Santa stopped by the EchoDitto offices a little early this year and brought us a shiny new load balancer (we've been very good). We were trying out a few things, using this domain as a guinea pig, and it all took a bit longer than expected. Everything should be fine now.

Wow. For the first time in my life I've undertaken an electronics project and had it immediately work. I'm still a little shaken by the experience.

As you might recall, in my last post I discussed loading the custom, Linux-based DD-WRT firmware onto the Fonera router. I left off with some thoughts about using that environment to achieve serial communication with an Arduino.

Getting the Fonera to talk to my Arduino turned out to be shockingly easy. The 2200 Fonera model has a four-pin header exposed. The pin closest to the ethernet port is ground; next is serial receive; then comes serial transmit. I scrounged up a female pin header connector from a USB break-out bezel I had lying around (already half-cannibalized — last year I'd used the USB end to make the cable necessary for softmodding the office Xbox). That makes the photo to the right look a bit messier than it needs to. In truth I simply connected ground to ground, then the Fonera transmit pin to the Arduino receive pin.

The Arduino software environment's serial monitor only displays transmitted data, so I whipped up a quick program to read incoming data and echo it back out on the transmit pin (which wasn't connected to anything, but the Arduino's serial output is picked up by my macbook through the USB connection). You can see the program in the picture below. You can also see the result I got when booting up the Fonera:

It worked! First you see the bootloader, then a bunch of garbage. That's nothing to worry about, though. The bootloader runs at 9600 baud, which is what I had the Arduino serial link configured to expect. Once the bootloader is done, the DD-WRT environment takes over, and it ramps the serial port's speed up to 115200 baud, producing the cloud of gibberish that follows the bootloader.

Fortunately, that's easy to fix with the stty command:

All of us at EchoDitto are big fans of the FON project — we run a FON access point here at the office, and I run one at home even though my primary router is a much-more-capable WRT54G running an old version of the Sveasoft Linux firmware. I got my Fonera router from Phil, so I assume he's running a FON AP, too. We're trying to support the project — honest!

I say all this out of guilt. Last night I hacked a Fonera. I know, I know — they sell these things for next to nothing only so that they'll get distributed and the project will grow. But I don't have any more WAN pipes available to share! So I hope the project's sponsors will forgive me for succumbing to the siren song of $10 Fonera routers on eBay. I couldn't help myself from checking out what a spare unit can do.

"Kind of a lot" turns out to be the answer. The La Fonera is based on the Atheros chipset, which is found in a number of other commercial routers and supported by both the DD-WRT and OpenWRT firmware projects.

Sveasoft kicked off the custom router firmware scene, but the author's controversial attitude toward the GPL made them fall out of favor rather quickly. Besides, truly open efforts quickly outpaced the project. OpenWRT is the king of functionality, but also the most intimidating — they seem to have a general "GUIs are for chumps" sort of attitude. While I aspire to similar levels of snobbery, I'm more comfortable sticking with projects that have support forums filled with posts that are comprehensible to someone just getting started with the project.

Besides, I've had great luck with DD-WRT in the past, using it to successfully repeat a neighbor's faint wifi signal through my girlfriend's apartment (with permission, of course — but without having to do anything to his router, i.e. no WDS).

Via Slashdot I see that folks are finding new things to dislike about Facebook's privacy-threatening Beacon initiative. This PC World article relates that Beacon transmits data from Epicurious back to Mr. Zuckerberg & friends even when the user is logged out of Facebook.

My first thought was that this objection might be silly: if Epicurious is serving the Javascript that sends the AJAX request (via some sort of XSS technique) there should be no way for Epicurious to detect whether a user is logged into FB or not. Doing so would violate all sorts of other, more important security principles — the sorts of ones that prevent me from writing some Javascript to, say, have this page read that open Gmail session one tab over and report its contents back to my hidden lair.

But sure enough, the Beacon Javascript is being served from Facebook.com, which means that your session cookies are perfectly available to it. Epicurious pulls the Beacon code from the following URL:

http://facebook.com/beacon/beacon.js.php?source=5194643289

And yeah, it serves exactly the same code regardless of whether my browser is logged into FB or not.

So this story's legitimate: Facebook could turn off Beacon for logged-out users with something as simple as two lines in .htaccess. Based on this example, here's a snippet that would check for Facebook's h_user cookie, if they wanted to:

It's been a while since I gushed about Yahoo Pipes, hasn't it? Well, the RSS-combining service remains insanely powerful. This morning I decided to try using it with one of my other favorite web technologies: jQuery. When combined, the two can take the place of many aggregation tasks that previously required a server.

Here's the problem I set out to solve: a bunch of my friends write for more than one blog. They usually have a personal site and contribute to a number of other multi-author blogs. Sometimes they keep a list of their posts in a del.icio.us sidebar block. But maintaining that can be kind of a pain — they have to remember to create a new del.icio.us entry every time they post. Clients like Pukka make using del.icio.us a breeze, but this workflow is still not as automated as it could be (in that it exists at all).

As you know, Pipes makes it easy to combine and filter RSS feeds. Here's a pipe I threw together containing feeds from a couple of EchoDitto clients and our main blog. Naturally we love all of our clients equally, but for my friends' purposes a simple Pipes filter block could be applied to check the dc:author field, ensuring that no coauthors' posts are retrieved. The result will be an aggregated feed of the author's entries.

But how to retrieve them? jQuery 1.2 added in support for JSONP, an unbelievably slick technology that allows Javascript to load JSON data from different domains. There are two things you need to know to get this to work with Pipes: how to get the service to spit out JSON, and how to pass it a JSONP callback (which safely assigns the JSON to a given variable name — without this, loading cross-domain data would be a huge security hole). Both things are accomplished by adding parameters to your pipe's querystring.