Navigating the tar(1) Pit on an R710

Posted on Tue 15 March 2016 in misc

In honor of Pi Day, I've decided to do something completely irrational: use my Dell R710's tape drive for backups of SmartOS's ZFS datasets. The Dell RD1000 is a bit of a weird beast - it's an internal tape drive that presents as a SATA USB disk. I can only assume this tortured firmware kabuki is an attempt to make it easier for Windows admins to do tape backups, lacking Unix's proper tape handling tools like tar and mt, while making it more complicated for Unix users. This is a running theme with Dell servers.

SmartOS in particular is a pain in the behind to get working with the RD1000 because half of the usual Solaris/Illumos tools are missing and the other half don't always match the Solaris documentation. Here's a walkthrough. All commands must be run from the Global Zone as root.

  1. Identify the RD1000 with diskinfo

  2. Wipe the disk by running format -e, selecting the RD1000 from the list, and typing format at the prompt.

Laptops and Licenses

Posted on Thu 28 January 2016 in misc

Today has been a most satisfyingly productive day. I tested how well Haiku OS runs on a tiny 16GB USB drive that barely sticks out of the port on my Thinkpad T530 (spoiler alert: it works fucking GREAT), and used it as a developer workstation for porting unix applications to Haiku. I got complete, 100% working ports of the Mutt mail client and OfflineIMAP method of retrieving IMAP as Unix Maildir. Along with the excellent terminal emulator, decent hardware support, package of TeX Live, and surprisingly OK browser, I can now use Haiku OS as a quiet little operating system for writing, blogging, and sysadmin work. The attractive UI and insane boot times - faster from USB 2.0 than some OSes boot from an SSD on equivalent hardware - certainly don't hurt either.

I also got fairly close to porting C-Kermit, although that's more of a personal challenge to prove I can wrestle the 35+ year old codebase with at least 200 possible combinations of conditional compilation. God, what a fucking mess it is. It's the exact sort of insane makefile-target and #ifdef fuckery that GNU Autotools was designed to avoid, only that's not an option because some of the target systems never GOT an Autotools port, because they haven't been actively developed for at ALL since the 1970s. Haiku's sort-of-but-not-quite-FreeBSD headers and libraries for porting Unix applications make finding the right set of CFLAG definitions extra challenging. Oy. OpenSSH, telnet, and SerialConnect all do the actual work of communicating with remote systems I'd need out of C-Kermit, which means I'm not in a huge rush even by the standards of my Haiku usage. I can get it to do most things except connect to a system over SSH so far.

That was all before I put pants on this morning.

Later on I finally bumped into limitations of what Haiku can do, so I booted back into Linux to retrieve some documentation from the byzantine depths of a utility company's website. Between that and locating my birth certificate I finally have the rest of the documentation I need to renew my driver's license, hence the other half of the post title. Working with computers all day really reminds you of how slow and unwieldy government can be... but that's preferable to them becoming too efficient at doing things you don't like before you have a chance to stop it.

Oh well, at least my desktop looks pretty as hell now.

image found on 4chan, no idea of original source

Another Milestone Passed

Posted on Sun 10 January 2016 in misc

For the first time in my life, I have gone an entire calendar year plus a few days on each end without changing jobs or moving. I suppose this marks the crossing of the Rubicon into stable adulthood. I'm beginning to reconsider my decision to move to the Seattle area after walking around in the ~10°F weather and snow today. I had forgotten how much I enjoyed the cold weather, and Seattle barely ever drops below freezing thanks to the massive water heatsink of Puget Sound and the humidity and rain trapped between the Olympics and the Cascades. On the other hand, when I want cold weather I can always go find it, and some of the mountains around the Seattle area are snowcapped year round. Food for thought.

On a more technical note, I've been setting up CJDNS on all my personal systems in various physical locations, so that I have encrypted point to point tunnels between them for all traffic. This blog, for example, is available for cjdns users at This is conceptually similar to more traditional VPNs, but makes it easier to connect servers and PCs in half a dozen physical locations without a lot of needless round trips. Adding an additional layer of encryption for traffic between my mail server and laptop was of particular value for privacy. CJDNS can handle both TCP and UDP traffic, so things like Bittorrent and IPFS work transparently.

Oh the Weather Outside is Frightful

Posted on Tue 29 December 2015 in misc

Thanks Obama Climate Change

The weather is officially psychotic this year. It was over 60 degrees in Boston on Christmas, and there were tornadoes in the Chicago area. Fast forward half a week and there's howling wind and ice pellets hammering the windows, fading into snow overnight.

Automating Let's Encrypt Certificates for Nginx

Posted on Mon 07 December 2015 in misc

The Let's Encrypt public beta is here, which means you can get free 90-day SSL certificates for as many domains as you'd like, absolutely free. The 90 day certs are designed to encourage automation, and the client even includes support for updating your certs non-interactively from the command line. Basic configuration of virtual hosts with Let's Encrypt generated certificates and nginx is covered well here.

Free auto-renewing SSL certificates are awesome, but right now the only httpd supported by the letsencrypt tool itself is Apache. The letsencrypt client's webroot option (verifying the certificate by writing to the domain's root directory) works for nginx, but that requires the root of the virtual host be a place where static files are served. This doesn't work if you're using nginx as a reverse proxy for an application server, like an IPFS gateway. Instead, we need to use the standalone method which actually binds to ports 80 and 443.

Unfortunately, the need to stop and start the webserver manually pretty much precludes running letsencrypt as a non-root user, unless you feel like giving another user the ability to run init scripts as sudo without a password. This means that we need to write a script to run as root and put it in root's crontab. To avoid cluttering up /root, I suggest cloning the Let's Encrypt github repository to /usr/src. Here's a script to automatically update the certificates for any Linux distribution with systemd. If you care enough about init systems to use something else, I'm sure you can adapt it yourself.

systemctl stop nginx
sleep 5
/usr/src/letsencrypt/letsencrypt-auto certonly --renew-by-default --standalone \
 --email -d > /dev/null
systemctl start nginx

The --email flag is the email that Let's Encrypt will send messages to, and the -d flag is the domain for which you're requesting a certificate. sleep 5 is just a precaution to allow running nginx workers to finish.

Once that script is saved to root's $PATH and set executable, add it to crontab at some interval less than 90 days to avoid renewal problems. I have it set to run on the first day of every other month, for example.

Powerleveling in the Game of Tech

Posted on Mon 02 November 2015 in misc

I Will Not Be Growing Old In the Midwest

image courtesy Wikipedia

Three years ago this week, I sold some childhood posessions, left a letter on the table, and hopped a train for Seattle. The Emerald City has held a place in my heart and my imagination for nigh on a decade now. I daydreamed about living there with CC when I was a 16 year old with more hormones than sense. Seattle was my "next year in Jerusalem" while I slaved away at a keyboard on my CS homework in college. It wasn't until 2012 that I was able to make my first attempt at living there, and only job hunting struggles combined with a death in the family drew me back home to New England.

On that fateful journey across the continent, I was stuck in Chicago for half a day. My oversized backpack and luggage, containing all my worldly possessions, didn't make for easy tourism, so I mostly hung around Union Station and killed time until my train came in. The minute I was able to, I dashed over to the platform and found myself a seat for the ride into the sunset.

In a very real sense, those three days on Amtrak were a microcosm of what my life was to be for the next two years. I struck out westward on my own once again two years ago, this time in Chicago to live. I have put a good face on it for my relatives, but the truth is, I've never really cared about putting down roots here. These last two years have been nothing but an extended layover.

That's where powerleveling comes in. Powerleveling in a video game means dedicating a large chunk of time in the game doing the most tedious, boring stuff ad infinitum, to ease your character's way through the next difficult stage in the game. Victory Road in Pokemon, or collecting heart pieces before the final boss fight in a Zelda game, are classic examples of powerleveling.

That's what I've been doing here in Chicago. I vowed after my last attempt at Seattle to never move to a new city without finding a job again. Living in a hostel and working minimum wage was an interesting experience, but never again. I went from unemployed to on the high side of the median personal income, got a nice set of wheels, and found a job that lets me telecommute from anywhere in the country that has a decent internet connection. Every step of this has been in service of my ultimate goal in getting the hell out of Chicago and back to Seattle. I expected to spend three or four years here before I got to this level, but I'm certainly not going to look a gift horse in the mouth.

This is why my apartment still looks like a temporary crash pad. This is why I never tried too hard to date or enmesh myself in a social network here. This is why I resisted getting Illinois documentation and proof of residency until the last possible moment. The day my lease runs out this spring, I intend to be ensconced in my new place in Seattle.

One Moment of Perfect Beauty

Posted on Sun 01 November 2015 in misc

A few weeks ago, before the howling wind and rain hit and before I ended up working 11 hours on Halloween night, I had the brilliant idea to get the heck out of Dodge for the day. I have fond memories of harvest season activities back home, especially the Columbus Day fair in Sandwich, NH, so I decided to find something similar to do on Columbus Day this year. I left early in the morning, driving through foliage at the absolute peak of reds and golds. Lakeshore Drive on a fall morning is amazing - like something out of a movie or a painting. It was enough to make me forget the ugly reality of living in this city for a while. Northwestern Indiana, by contrast, is the armpit of the entire midwest. There's a reason the skyway bridge extracts a $5 toll each way but people keep using it. Metaphorically fast forwarding over the sprawl like a Gibson protagonist, I found myself on US-6 in the countryside. I had almost forgotten US-6 even existed this far west - I more normally associate it with memories of vacations on Cape Cod, or cutting across Connecticut twenty years ago in a rusted out Hyundai. The memories made me smile more than the actual sights along the road, which was just generic Indiana - dull and white bread to the nth degree. I took one last turn and found myself where I'd been aiming for since morning: County Line Orchard. I took a few pictures, but they're all fuzzy crap compared to the ones on the website. My next phone will have a much better camera.

It seems like half the population of Cook County had the same idea to get out of the city that day - there were enough other cars in the grassy parking field with Illinois plates that I found another black Camaro of similar vintage. What are the odds?

I spent hours wandering around the orchard, picking apples, listening to bluegrass music, and listening to the wind in the trees and absence of engine noise other than a couple of tractors towing the lazier tourists around the fields. Of course there was a corn maze - what's a plot of land in Indiana without corn? - but there a few too many teenage couples seeking isolated spots for it to be really enjoyable. The clouds overhead thickened and spilled a few drops of rain, so I made a tactical retreat back to the gift shop and picked up a pumpkin to leave outside for Halloween.

As I worked my way back towards Illinois, the clouds thinned, until I was south of Chicago and the sky was the color of a television tuned to a dead channel - a clear, sharp, unmarred blue. I opened the windows and sunroof and cut across to US-41 (which becomes Lakeshore Drive) a bit early. As I passed through the projects on the south side, I learned a lesson that photographers have learned sicne the first days of color film: a clear autumn afternoon with fall foliage can make even a bullet-scarred, impoverished hellhole like the south side seem like paradise. As I approached downtown, the sunlight shining on the skyscrapers seemed to welcome me back home, if only for a little while.

(To anyone who caught the Babylon 5 reference, no Vorlons or Pak'ma'ra were involved in the telling of this story).

Week of Hell

Posted on Mon 20 April 2015 in misc

What a week.

This has easily been the worst week of 2015 thus far. If I believed in karma, I would think this past week was some sort of karmic balance, making up for getting a Camaro last week. If I were the type of Jew to take the Torah literally, I would think the plagues of Egypt were being visited upon my cramped little Chicago slum apartment. So far I've been visited by the plagues of disease, bad luck, and malicious landlords.

I've been sick as a dog for most of the past week. Chills, coughing up round after round of junk from my respiratory tract, and the occasional asthma attack have made it an especially shitty sickness. I suspect I brought it upon myself by letting my apartment get nasty, but it could also be something I picked up on the south side of town - the car dealership was full of snot-nosed kids, and who knows how many of their parents were anti-vaxer loons.

The "plague of bad luck" happened on Wednesday at about 1am. I stepped out into the hall to pick up a food delivery, barefoot and with only my Grandpa's old swiss army knife in my pocket... and reflexively closed and locked the door behind me. My landlords claim to have a 24 hour emergency hotline for building maintenance, so I borrowed the delivery guy's cell phone and left a message. An hour later, I realized they weren't coming. Now, I might have been content to catch a few Zs in the hall and wait for morning if not for one very important detail: I was technically "at work" until 8am and had to be available to handle support tickets! Fear of getting fired overrode my fear of damaging my own apartment; I pulled out the swiss army knife and got to work. I spent the next four hours or so cutting a hole in the door big enough to stick my hand through and unlock it from the inside. By the time I got back in I was sore, sneezing, covered in sawdust, and thoroughly pissed off. My coworkers were very understanding, and were outraged on my behalf when I told them what had happened.


This leads directly into the plague of scummy landlords. A full day after my little adventure, I finally get a call from the management company. This wasn't checking in on me, making sure I was OK, or anything, but an idiotic "Durr, did you know somebody drilled a hole in your door?" call. I really let them have it at that point, giving their representative on the phone a verbal lashing. (On a side note, it boosted my masculine pride a bit to think I'd done enough damage with a pocket knife for them to think power tools were involved.) The next morning I found a note outside my door notifying the tenants of a "routine inspection" that would be taking place next week. This is noteworthy because I've been here coming up on a full year without any such inspections taking place - it smells like a cheap pressure tactic.

So, that has been my miserable week. On the plus side, I got this fancy new set of wheels and there's lovely spring weather outside.

Vroom vroom

C-Kermit the Frog goes lpribbit

Posted on Sat 04 April 2015 in misc


Earlier this week, when I was staring at my work machine bored out of my skull, watching paint dry a long rsync between servers run, I decided to poke around the options menu of MinTTY, the Cygwin terminal emulator I use when I'm on Windows. Under the Terminal settings menu, I found a curious setting called "Printer." I thought this would be something to do with text output, fonts, and the like, so imagine my surprise when I saw a drop-down menu with my network printer and the default Windows XPS writer in there. I was initially confused as all hell, since this didn't seem to have much of a connection to CUPS, lpd(8), or any other printing system that could reasonably be expected to talk to a program running in an xterm.

Way back in the Palaeozoic era when VAXen roamed the earth, many people connected to their Unix machines over serial terminals made by DEC and Wyse. These terminals had secondary ports on them, often labeled "AUX" or "PRINT". On this VT520, the printer port is the third large port from the left, with an icon that looks either like a desktop printer or a box of tissues under it.

What an ugly mess.  Aren't you glad we have USB now?

This port could be connected to a printer (serial or parallel, depending on the terminal), which would intercept input from the terminal if a certain ANSI escape sequence was received, and stop receiving input when another certain ANSI escape sequence was received. These are the same sort of escape sequences that let you have colors, bold, reverse video, blinking, and set the terminal window title. If you want to see a whole bunch of them in use at once, ask an Arch Linux user to show you his bash $PS1 or his zsh $PROMPT. This is a hideous kludge that doesn't leave any room for error correction, formatting, or binary data like images or PostScript, so it was of limited use in the long run.

Time passed, the Reagan administration ended, VAXen in the big office gave way to PCs on your desk, and slowly the world forgot that time-sharing systems ever existed.

Enter Linux VPSes, and the Raspberry Pi. Now suddenly, everyone and his dog has their own cheap little Unix system of reasonable power that they can do their work on, but don't have a sane, simple way of printing from them. You have to open another, separate connection to download an output file, save it to the computer you're sitting in front of, open THAT copy in a program that can read PDF files, and send it to the printer. That's not going into the complexities of and pitfalls of managing your printers, but enough bile has been spewed about that over the years to skeletonize a large whale, so we'll leave it alone.

That's not a whole lot of fun. It would be much nicer if you could just type a word or two inside your SSH session on the remote machine, wait briefly for a file download, and have it automatically sent to your printer so you could get back to whatever else you wanted to do in your system besides babysit printers.

As it turns out, you can. This is where C-Kermit comes in. C-Kermit is the last actively developed branch of the Kermit project, which started at Columbia University in NYC in the 1980s as a way to get IBM mainframes and CP/M computers to talk to each other nicely. Kermit has since been ported to everything from MS-DOS to VMS, and every variety of Unix ever made unless you count Cygwin (which you shouldn't, it's terrible). C-Kermit is the Unix branch of the family tree. In its long life, it has evolved from a simple error-correcting serial communication and file transfer program to a fairly complete scripting language. It gained support for internet protocols like HTTP, FTP, and telnet, SSL encryption, Kerberos authentication, and most relevant for our purposes, SSH.

At this point, if you use Windows, you're out of luck, and have to rely on kludgy raw terminal passthrough with MinTTY or similar. Kermit 95 has been out of development for over a decade, which means its SSH implementation is only moderately more secure than mailing your actual credit card to the Russian mob. Dilbert said it best.

I suppose these days it would be a Debian install CD instead of a nickel.

If you're using a Mac, Linux, or some other unix-like operating system, read on. You will need a system on which you have already configured printing, a package manager, and a text editor. If you're on OS X, I recommend Homebrew or MacPorts, otherwise just use whatever package manager came with your OS.

Setup Instructions

  1. Install C-Kermit. This is usually packaged as kermit or ckermit. RHEL, CentOS, etc. will need to install the EPEL repository.

  2. Open ~/.kermrc in your preferred editor and append the following line:

    SET PRINTER |lpr

    This means that kermit-downloaded files destined for a printer will be piped to lpr. If you have flags you need to pass to lpr or use a different printing command, substitute them as needed.

  3. Start Kermit. Your terminal should display output that looks something like this:

    C-Kermit 9.0.302 OPEN SOURCE:, 20 Aug 2011, for Linux+SSL+KRB5 (64-bit)
     Copyright (C) 1985, 2011,
       Trustees of Columbia University in the City of New York.
      Type ? or HELP for help.
    (/home/tidux/) C-Kermit>
  4. Connect to your remote system inside Kermit. This basically works like an SSH connection from the shell, since Kermit is calling your system's ssh command directly.

    (/home/tidux/) C-Kermit>ssh
  5. Now we need to install C-Kermit on the remote system, just like in step 1 above.

  6. Still on the remote system, open a file called "kermit-print" somewhere in your $PATH that you have write access. This is usually /usr/local/bin or $HOME/bin. Put the following lines in it:

    #!/usr/bin/env kermit 
    SEND /PRINT: \%1

    Save it, and set the file executable. Rehash your path if you're using (t)csh for some godawful reason.

    Kermit syntax is pretty simple. The file is opened with Kermit as the interpreter, and the first argument to the script is read as a filename, and then sent to the printer we configured in step 2 on your local machine. The EXIT line tells kermit to return you to the remote machine's shell rather than keep running kermit inside of kermit.

  7. Now you've got it all configured, so test it to make sure everything works as expected.$ kermit-print war-and-peace-complete.pdf

    You should see Kermit take over the screen and present a nice pseudographical display of the file download, and when completed be returned to your shell on the remote system. If you just want to save the file to your local machine and not print it, kermit's built in -s flag is what you want.$ kermit -s war-and-peace-complete.pdf

Final Thoughts

If you want instant PDF preview of remote files, you could make your default printer destination in ~/.kermrc a PDF printer or a document viewing program that accepts files on stdin.

And So It Begins

Posted on Thu 16 October 2014 in misc

Autumn has finally come to Chicago. The cold, gray weather and shrinking days encourage a transition to more indoor pursuits, which brings us to this blog. It's been a few years, but I want to share my thoughts with the universe again. The past year in particular, I've done a lot of observing and reacting to things in my life, but not a whole lot of writing about it beyond occasional emails to the family back east.

The "Uncanny Valley" is that space where a recreation of a human looks just lifelike enough to be scarier than something obviously fake. It's why people are afraid of zombies, and why we hate those stilted robotic voices from phone touchtone menu systems. This month, I've noticed a similar effect for climate. Chicagoland is just enough like New England that I keep expecting things to happen that don't, like nighttime choruses of spring peepers in May, or the trees turning in September to mid October. Instead of a nocturnal amphibian chorus in the spring, we get cicadas beat-boxing all summer through September. Instead of the trees turning to living flame by mid-October, we're only now getting around to seeing the earlier leaves turn yellow. Other places I've lived, like the Southwest or the Pacific Northwest are so obviously different than New England that there's no expectation that the seasons will act the same.

On the other hand, it could just be homesickness filling in the blanks. Thirteen months in to living in this big, crazy city I've only made a handful of friends, and that's mostly been through work. Night shifts are an absolute bitch as a long term thing. Your diet suffers because all the healthy food places are only open when you're asleep. Your social life withers because everyone else is asleep or at work when you've got free time. It even makes simple things like getting packages delivered to your place needlessly difficult. Between the isolation in the midst of millions and my unease about the weather it's taken me this long to put into words, I don't think I've ever been totally comfortable here. Even my happiest moments here have been tinged with loneliness and wishing I was somewhere else. I could be having a relaxing summer day at the beach, and all I could think about was how it felt like I had a cold because I couldn't smell sea salt wafting off the water.

I think the biggest issue is that I am not, at heart, a city person. A lot of my friends and family would be happy to live in a major metropolis for the rest of their lives - one of my friends has spent his entire life here in the windy city and has no plans to leave - but not me. I start to feel caged in if I go too long without being out in nature, especially without being able to see the stars at night. I went up to the Upper Peninsula of Michigan last month for a few days of backpacking through swamps and over mountains. I almost cried the first night, when I could see thousands of stars shining through the trees. There are less than ten stars visible in the entire sky at night in Chicago.

No, ultimately I'd be much happier if I flipped my hours around to day shift and I could telecommute from a house in the forests of Colorado, New England, or the Pacific Northwest. That holds true even if I was telecommuting to this same support monkey job, managing overpriced, underpowered hardware for people who probably shouldn't be allowed to own scissors, let alone servers.