curl

Sometimes I wonder why I ever bothered to compile wget for OS X. I keep finding clever new things that curl can do — even some that make me wonder why I bother with Perl and Ruby's Mechanize module/gem.

For instance: today I found myself needing to export a friend's Typepad blog. He's got a lot of content, and I wanted to pull it down on a speedy connection and then gzip it before pulling it onto my laptop over my sluggish DSL (Typepad unhelpfully delivers its exported content in an uncompressed non-ML format).

Typepad provides a URL for the export, but it only works when you've authenticated through their website. Curl makes working around this pretty simple, though, as I found out from this Ask Metafilter thread. Here's how it works:

curl -k -d "__mode=redir&next=http://www.typepad.com/t/app&username=user%27s+name&password=their%20password" -c cookie.txt https://www.typepad.com/t/app

You simply use -k to turn off SSL requirements, -d to pass a series of key/value form pairs (properly url-encoded) to the URL to which the form posts, and -c to output the returned cookie to a file.

Then you pass in the session-containing cookie file, allowing you to stay authenticated on subsequent requests:

curl -b cookie.txt -o complete-export.txt "http://www.typepad.com/t/app/weblog/post?__mode=export&blog_id=YOUR_BLOG_ID"

Naturally, you'll want to replace the URL in that last request with the one corresponding to your own export URL.

It all worked perfectly — well, up until the point at which I discovered that Typepad exports seem to inevitably fail once they top 100 megs. Seriously guys: file compression. Try it, you'll like it.