...look no further than the comments on this TechCrunch post. Reacting to the scoop that Twitter is dumping Ruby on Rails (which almost immediately turned out to be wrong), more than 150 people decided to chime in about what Twitter's scaling problems are and how their own preferred web framework never would have encountered them. A whopping one person managed to refer to Twitter's custom message queueing software by name.
The rest presumably just like the (drag & drop HTML creation in Visual Studio|error detection in Zend|Google-enabled buzz surrounding Python), and have read "Rails doesn't scale" somewhere. I suppose their prattling is still slightly more meaningful than the yammerings of the web's legions of widget evangelists, social network triumphalists and self-proclaimed "SEO experts". But not by much.
At any rate, I think this is a good reminder of the signal:noise ratio facing our clients as they attempt to find technical help in a crowded marketplace. If you're not a technical person it's genuinely hard to tell the difference between someone who knows what they're talking about and someone who's simply regurgitating buzzwords. My rule of thumb when encountering bold pronouncements like the ones in that TC thread: ask "why?", then count the number of unexplained acronyms and buzzwords in the answer.
One of the things that I've learned over my years at EchoDitto — that's been hammered home again and again — is the importance of email. Twitter's cool, social networking is great, but your online strategy absolutely has to account for email. It's not glamorous, but it's important.
The same is true on the tech side of things. It's easy to forget: on a day-to-day basis, my wranglings with email generally include dealing with spam blacklists, ensuring that scripts don't function as open relays, and writing templating systems that'll be used to send mail. All of it's pretty boring. But it's worth keeping in mind that email, when piped to a script, can serve as the infrastructure for some pretty neat services, too.
I did this two years ago for SXSW, building an SMS app on the cheap by counting on the mobile carriers' SMS-to-email functionality. It worked pretty well, although it was a rat's nest of Perl scripts.
This week I took a pass at another application in Ruby (with a PHP frontend), and this time the resulting code is a bit less cringeworthy. The idea is simple: set your Twitter "new follower" email notifications to go to a custom email address. They'll be piped to a script, disassembled and the new follower's statistics analyzed. If it looks like the new guy is a spammer or bot, they'll automatically be blocked. If not, they won't be. Either way you'll get an RSS notification about it. You can try it out here, if you'd like.
It's not exactly going to set the world on fire, particularly since Twitter is expected to release similar functionality soon. But the project does serve as a pretty good template for how a piped email service can work.
Let me start by saying I’m not proud of this, it is one of those awful awful hacks which I should be ashamed of and appalled that I would even consider, one of those hideous violations of every coding standard known to good coders throughout the world.
What is this horrid, monstrous bit of code hackery I’m referring to? It’s a simple hack to the theme function which provides a very rudimentary version of some of the theme-level information so elegantly and profoundly usefully exposed by Moshe Weitzman’s Devel Themer module. For various reasons, this hack actually isn’t that egregious; for one it is ONLY FOR USE ON DEVELOPMENT SITES, and therefore does not all into that more loathsome category of coding dark secrets which are injected into codebases in the 11th hour to make Drupal conform to non-Drupal-tuned functional specifications. In addition, the hack allows custom processing of theme output to be included in the info dump for themed blocks, and therefore makes incredibly arduous tasks like finding the source of a missing </div> tag a great deal less awful. Let’s face it (with relish): sometimes hacking core is just the best way to debug difficult problems.
So, enough stalling, just what is this horrible, awful, shameful hack I’ve devised. Well, it’s anticlimactically simple, actually. Just insert the following code into the function theme() declaration in theme.inc just above the return call_user_func_array($functions[$function], $args) call:
A few folks have recently asked me about our process for the actual launch of web sites. Aside from doing lots of development work, we also maintain our own server cluster, and do our own hosting - so transitioning, launching, and re-launching sites is something we've gotten pretty good at. Despite the uniqueness of some of our sites, in general, launch process on LAMP systems can generally be pretty straight forward. Here's what we generally do:
At least one week before a site goes live:
About a week before go-live:
A number of people have emailed me asking for the Full-text RSS project code. I appreciate the interest! And I'm happy to oblige so long as they send me an assurance that they won't be using it for commercial purposes — e.g. processing and selling ads on someone else's feed.
A number of people have sent me those assurances, but I've been slow to send out tarballs. I apologize! If you're waiting on one and haven't heard from me by the end of the day, please email me again: tom (at) echoditto (dot) com.
Remember this guy? Yeah, I barely do, too. But way back in 2006 I put some of the lingering knowledge I had during an earlier job together with a newfound passion for writing horrible rats' nests of Perl code. The result was a set of scripts that could scrape the rather lame HTML interface to House Information Service's zipcode+9-to-congressional-database matching service.
I'm sure my scripts haven't aged all that well, but Aaron Swartz was kind enough to take up the torch and share a set of Python scripts he coded to accomplish the same thing. Thanks, Aaron!
Those of you looking for zipcode-to-congressional-district matching would do well to check those scripts out. You might also want to follow the threads from this tweet, in which exoDitto Tim Jones asks the LazyWeb for other solutions to the problem.
I'm going to be presenting at tomorrow's Dorkbot DC meeting — anyone who's interested should come on by. I'm going to be talking about making a Fonera router talk to an Arduino, a subject I first blogged about right here.
And of course the DC nerd community also shouldn't forget about Wednesday's Drupal Meetup at Affinity Lab, which is once again being hosted by our friend Mike McCaffrey. You can find details here. I think it's safe to say you'll be able to find some of us at each of these events.
Yesterday Development Seed was kind enough to give Chris & me a rundown of how the Drupal community is organizing its participation in the Google Summer of Code program. Along the way I got a chance to chat with Alex and Ian about FeedAPI and FeedAPI Mapper, two excellent projects that DevSeed ushered into being through last year's Summer of Code and now continues to maintain and extend.
I've just begun using FeedAPI for the first time in a project destined for production, and so far I'm very pleased with it. It offers a more fully-considered alternative to aggregator.module — and with the addition of the optional Mapper module it becomes simple to turn aggregated RSS items into Drupal nodes, with the items' attributes stuck in whatever CCK fields you care to create. It's really slick.
In my case I'm using it as an integration point for Blip.tv. Our client needs video capabilities, but I saw no reason why we should mess around with transcoding, customizing an FLV player and all the rest of the headaches that come with web video (been there, done that). Blip does all of that stuff very well, and has social features baked in, too. I'd rather just have the client upload their videos there, then count on FeedAPI to turn them into nodes that can be exposed through Views. Any configuration that we can't get from the Blip RSS feed can be manually handled by an editor — Workflow-NG fires off a "please come edit and publish me!" email whenever a new video node is created.
I thought I'd share some lessons learned in the node creation wild as a follow-up to Tom's recent post surveying the options available to prospective content importers who need to get a slew of nodes into Drupal from some far off place such as the land of flat HTML, custom CMSes or even the fabled Drupal 4.6.
For a very exciting upcoming site we had to import a few hundred old blog posts from a Drupal 4.6 install, along with 30 authors and around 2000 taxonomy associations. While these numbers are nowhere near as big as content migration figures can crawl, the complexity of the data structure was multiplied by the specific requirements of the various node importing schemes available in Drupal. This multiplicative factor yielded a somewhat dizzying array of alternatives that had to be evaluated, with the vast majority of them leading down paths that involved exporting nodes, importing with old legacy data attached then iteratively importing the author and taxonomy data one piece at a time using custom scripts and translation tables.
Scary.
Mercury-bound space craft, adding MIDI to everyday objects, and the software running ludicrously-complicated automated painting machines. Dorkbot DC doesn't fail to bring interesting stuff. Tom and I had a great time seeing these projects.
It started off with a talk about the MESSENGER probe. Katie Bechtold is a software developer who works on the X-Ray Spectrometer, some other sensors I can't remember the name of, and some day to day behind-the-wheel space probe driving. She brought along a 1/10 scale model of the probe and did her best to point out where the various sensors and crucial parts are. The model was probably two feet tall, if you want an idea of how big it is.
Her explanation of everything was great for those lacking a background in astrophysics (hint: me). I won't say much about the actual workings of the probe, as I don't think it's appropriate to do so until Echoditto is in the business of slinging extremely complicated computers off to distant planets. Here's what was really neat to me:
I really liked that movie Sunshine. Mercury is pretty darn close to the sun, and MESSENGER has to be protected from it at all times. That means there's a big sunshade the majority of the craft lives behind. It isn't even to Mercury yet and it already hits 700 degrees Fahrenheit on the surface of the shade! Bonkers.
Yesterday Ethan, Chris and I found ourselves needing to move some data into Drupal — we're building an existing client a new site and they need some blog posts moved over from their old one. It sure sounds simple. But as anyone who's actually done data migration from one blogging platform to another (or just from one version to another) can attest, it's rarely that easy. There are a few options, though, which I present here in increasing order of their likelihood to unexpectedly spiral into a huge, awful mess:
Hello, World.
That seems appropriate as an opener for my first post on Labs. Usually I find myself on the less-technical, more client-management focused side of things here at EchoDitto, but I was encouraged to join the discussion here, so here I am!
So I was tasked yesterday with adding a graphic button to a client’s site and they wanted the image to change when you mouse over it. A rollover image. Simple enough. Except that I didn’t want to use Javascript. It would have been possible, sure, but I would have had to snag one of my friends on the tech team and it would have been a hassle. So I wanted to see if I could make it work without Javascript.
Luckily, the rollover image the client wanted to use simply involved changing the color of a particular part of the original image. In a flash of resourcefulness, I thought: CSS can do that! Five minutes later, I had what appeared to be a working rollover image.
First, I edited the original image so that the part that needed to change colors was transparent. Then, placed the image in my blog post on a white background. Finally, created a CSS class such that the background changes color on hover.
Voila! Instant fake rollover image.
Maybe it’ll help you next time you can’t, or don’t want to, bother with Javascript.
As the last EchoDitto holdout, I now finally find myself the proud owner of Apple's new tiny PC.
Yes, that's right, I call it a PC. Let's face it, it's a computer, not a phone. (How did Macs stop being PCs anyway? They're still personal computers. Perhaps Apple vanity prevents us from comparing their products to anything existing. After all, when was the last time you heard someone refer to an iPod as an MP3 player? Its usually the other way around. But I digress.)
The iPhone is still a pretty nifty device -- even when you don't consider the enormous and sophisticated 'underground' software movement. The browser is great, the email works fine, although lacks some obvious features. Junk mail or cut & paste, anyone?
And the iPod features -- well, I have an iPod Nano for that.
But what really got me to switch over to AT&T was the geolocation. I've been on the GPS bandwagon for awhile now, having recently purchased a TomTom GO, and the idea of having geolocation and directions in my pocket is almost too much to bear.
But the real story isn't the famed cell-tower geolocation, its the the WiFi geolocation. While (ahem) exploring some of the 3rd party options out there for the iPhone, I found myself temporarily without cell service. Much to my surprise, geolocation was still working marvelously.
Later that day, I disabled my WiFi to save some battery (it didn't--the iPhone turns off the WiFi when 'sleeping' anyway) and I tried geolocation again. It was much, much inferior. (Go ahead, try it!)
Turn the WiFi on, and -- bam! -- back within 100 feet.
Now, I don't know about you, but I'm floored. How did Apple find a way to find me within 100 feet using WiFi alone? Granted, I'm in the DC area, so your mileage may vary, but either my Comcast IP address pinpoints me as 20 feet in front of my door or Apple is something awesome.
But we all knew that.
So, as Phil mentioned, we all got iPhones as a holiday bonus. And they're pretty great. JP and I held off until Macworld, lest we miss out on a new 3G handset (a longshot, I know). When that didn't happen I immediately scurried over to Pentagon City and bought a beautiful internet lozenge.
Since then I've been figuring out all of the amazing things it can do for me. On Thursday I had a stroke of inspiration. I've been looking longingly at Bluetooth presence detection setups for a while. I like the idea of having my hardware respond to my proximity, but I've got some reservations about wasting a Bluetooth dongle (and server, and Bluetooth-on-Linux configuration time) on the effort. The iPhone is the first mobile I've owned that does wifi. This seems like an opportunity to do presence detection on the cheap.
My approach is pretty simple: ping the iPhone. If it's there, I probably am, too. But how to ping it? The iPhone uses DHCP to get an address. At home I've got a router running the SveaSoft firmware. It's simple enough to configure DNSMasq to always dole out the same IP address to my iPhone's MAC.
Then I wrote a bash script to send three pings, check the number of successful replies, and log the result to a text file. I set it up to run on a minute cron and let it go overnight:
PING_COUNT=`ping -c 3 192.168.2.21 | grep "bytes from" | wc -l` test $PING_COUNT -eq 3 && RESULT="here" || RESULT="not here" RIGHT_NOW=`date` echo "$RIGHT_NOW - $RESULT" >> status.txt
Here are the results from the first 24 hours:
On Wednesday Ben and I headed over to the monthly meeting of Dorkbot DC and soldered our little hearts out. I've only been to one Dorkbot before (well, unless you count the one at SXSW — but that was mostly an opportunity to buy t-shirts and drink free beer). My previous experience had been enjoyable, but maybe a little dry. It's inevitable that the speakers won't pique everyone's interest every time, I suppose.
This meeting was a lot more fun, and a lot more hands-on. The organizers did a fantastic job, preparing instruction material, assembling kits and even pre-drilling jigs for the rest of us in an effort to introduce the extremely large crowd to soldering by way of MAKE Magazine's LED cube weekend project, slightly modified to work with an Arduino.
Sure, it's a simple project, but we've never claimed to be electronics gurus. Besides, it was a great opportunity to refine our soldering skills — something I'm in sore need of after nearly trashing my Wii during a botched modchip installation.