mike.sh

The Long Road of the Advanced Beginner

2010-03-20T00:00:00-06:00

This is the story of an Arduino electronics project I built. When you hear “Arduino,” you may think you know where this story is going, but you don’t. There are no schematics, no breadboard layouts, no code snippets. This is the project that carried my electronics skills from Advanced Beginner to Competent in the Dreyfus model of skill acquisition (with a nod to Merlin), and I’m disappointed to say it took me two years to do so.

I’m not going to describe the finished project, the code for that is available on GitHub. What I’m going to describe is the failed predecessor. It’s far more interesting, and it consumed 80% of the last two years. Briefly, though, the final project’s homepage is LanternHook.com, and it’s four Arduinos with WiShields and BlinkM multi-color LEDs that make a persistent socket connection to a Python Twisted service. Each lantern is available via a URL and functions as a physical webhook. The code explains how it really works.

Here is the auspicious beginning of my failed LanternHook predecessor. This photo was taken on January 20, 2008:

Why, in this supposed Arduino project, are the most recognizable components on the desk Foneras? I’ll tell you in a moment. First, I was lucky with this project because the idea came to me quickly and clearly. I wanted a computer that didn’t look like a computer, and—like the Ambient Orb—I wanted something pleasing to look at that conveyed a little information. I also wanted to build four computers to live in my parents’, brother’s, sister’s, any my houses, and have them connected in some way. Having them interconnected meant WiFi to me, and as Rands would say that small but crucial decision early on had repercussions I was not prepared for. At the time, there were no Arduino WiFi options that were cheap and easy, but I found this post describing how easy it is to connect a Fonera to an Arduino over serial. Great, I thought, and quickly created a system that worked like this:

Fonera → Arduino → 2 BlinkMs

The Fonera runs Linux, so I setup a scratchbox in a virtual machine (requisite self-link) to build the packages I needed. It was snappy, too. The Fonera connected to a beanstalkd server, which meant it blocked on a socket that returned right away when there was a new command (color) for the lantern. Simple. If you’re an advanced beginner, I highly recommend emulating what I did up to about this point. Tie together some pieces you understand, and keep your momentum up by building a project in iterations. Demo it to people whose opinion you value. I demoed the lanterns in the summer of 2008 (with an iPhone app), and it worked great and looked great. Then why did this project take over two years instead of six months? Just when I could have written up the project and called it done, I decided to take one more design iteration. I had a good reason: The purpose of this project was to learn about electronics, but it was so simple that I hadn’t learned that much.

The hardware worked, but there was too much of it. The whole thing seemed too complicated. With the Arduino, it was more than a bit crowded it the top. Looked to eliminate some parts, and it seemed like a good idea to cut out the middleman, so I cut out the Arduino. This was the smartest and dumbest thing I did. It was smart because the Arduino wasn’t really doing anything except as as a serial-to-I2C converter. It was dumb because I should have asked myself why the Arduino wasn’t doing anything. This was an advanced beginner electronics project, where Arduinos are as common as the electrons themselves. Instead, observe how I ignored the warning signs I was plowing into a 18-month run:

First clue: It’s possible, but by no means easy, to get I2C working on the Fonera. You use these two GPIO pins next to the Ethernet connector:

Don’t ask me which one is for data and which one for clock, because you shouldn’t be considering doing this. I love installing and configuring Linux boxes, and even I got bored with all the work required to set up I2C. Even after I got all the software working—because I’m a software expert—it still didn’t work—because I’m an electronics advanced beginner. I needed 4.7k pull-up resistors to on the data and clock lines. Without them, no signal. I actually knew I needed pull-ups, but I used 10k resistors. I used the only guidance my advanced beginner skill-set contained, which was, “10k is usually a good resistor to add.” I know this is dumb, but it’s all I had, and it’s comforting to advanced beginners to have rules to cling to.

Second clue, and this is a big one: I started blowing BlinkMs. Meaning, the lights would stop working. I ignored the problem at first, thinking it was ESD. Then I figured out it was the BlinkM’s ATtiny micro that was blowing, and I ignored the problem again. Then the problem got to the point where I couldn’t ignore it:

That’s what failure looks like. 9 blown BlinkMs.

I sought advice on how to protect the lines, but the advice I got was still over my advanced beginner head. I tried crazy solution with regulators. Didn’t work. In hindsight, I may have been blowing the micro via the digital lines, not the power lines. I’ve sense learned the digital lines have even less protection. It didn’t matter, though, because I gave up. In fact, I almost gave up on the project entirely.

This is the point in the story where the advanced beginner needs a miracle. Mine came in the form of the WiShield. (By the way, the WiShield in the Wattvision box. It was kinda amusing to see when I opened the case.) The WiShield solved the problem because it meant I could give up and still move forward. It’s important to do both (give up and move forward) when you’re learning, but before the WiShield I couldn’t move forward if I gave up on the Foneras.

Still not done.

The project works so well now, and it’s wired so conventionally, that it’s boring. One hiccup though, is that the socket connection can get lost. I tried to add code to reinitiate connection, but it doesn’t always work. My current plan is to use a watchdog to reset the Arduino. Maybe add an external 555 timer as a watchdog, like the Practical Arduino blog describes.

The project is still not done in the big picture, either. Why does it require a Python Twisted service running on a server? It should be replaced with something more standard, but for now a socket connection from the Arduino was the best.

Here’s one of the finished lanterns.

Virtual Machines as Mitochondria

2010-02-15T00:00:00-07:00

Here’s the idea: Use Linux virtual machines (VMs) hosted on Mac OS X for all types of one-off, console- or server-related tasks. That’s it. Depending on your background, that may be all you need to read. If you’re looking for tips, you’re done.

And this one even comes with a metaphor: Think of the VM like a mitochondrion; not necessarily the energy-producing-powerhouse, but the specialized-cell-that-evolved-to-live-in-another-cell aspect. If you lack the technical skill to implement this idea, I’m sorry but you’re also done. A lone article written by me in February 2010 isn’t the right venue to impart this skill. If you’re still here and you’d like to explore how far you can take this idea, and hear a one-sided argument for it, read on.

Let Linux be Linux

This is a setup I’ve been using for a long time. I tweak it when I think of ways to improve it, which happens a lot. It stems from questions like these:

You need to run an NFS server. How would you do it?
You need to cross-compile a whole stack of code. How quickly can you gather the tools?

I use Mac OS X as my everyday machine, but I’m painfully aware of how easy Linux can make these two tasks, and a slew of related ones. So I let Linux play to its strengths and just run them in a VM. I’m coming way late this argument:

Installing MySQL on Ubuntu (the NSFW way)

but Mark Pilgrim’s characteristically good article solidified my answer to the question, “How do you setup MySQL for local webapp development?” and it’s not, “Compile it from the source and run it on your Mac.” I like to apt-get install as much as anyone, but I’m willing to accept lock-in that Pilgirm will not. My setup is more in line with Alex Payne’s Rules for Computing Happiness:

http://al3x.net/2008/09/08/al3xs-rules-for-computing-happiness.html

i.e., “Do not use anything other than a Mac at home and Linux/BSD on the server.” (But see here to read how the tide is also turning for him.) The virtual machines I run all happen to be Ubuntu server, and I think of them as if I’m carrying along a few extra servers with my laptop. Because I run the server versions, I ssh into them and otherwise interact with them as if they were remote servers. They just have great ping times.

One parting shot at Linux: It’s come a long way since I used it as my everyday, desktop operating system. For example, I never had a Linux machine with reliably-working WiFi. So while this doesn’t cause current users as much grief, now it causes me and my VM no grief. The VM thinks it got its IP address from a very friendly DHCP server over eth0.

Keep the Mac clean

In a nutshell, anything that’s moderately server related, I put in a VM. For example, even though my Mac is capable and eager to serve files from my ~/Sites directory, I spin up an Ubuntu 8.04 machine running Apache to do that work. Yes, the same Apache project (and much of the codebase) that my Mac would have used. And that’s just for Apache, which ships with the Mac. I’ve stopped using Fink and MacPorts altogether, and that’s because you have to be ruthless about what you keep on your Mac. If it doesn’t install by dragging into the /Applications folder, there has to be a compelling reason to install it. I’ve been using Mac OS X since 10.0 in 2001. One of my favorite features is the User Migration Manager. Upgrades are painless. As Kottke observed, they’re painless almost to a fault. But with upgrade after upgrade, the footprint of software I carried with me in /sw (Fink) and /opt/local (MacPorts) became a burden. Worse than a burden, in fact, because system upgrades would render the software incompatible.

I have no loyalty to any of the VMs. Whenever I think I need a clean slate I grab the latest version of Ubuntu. I delete VMs at whim without grabbing any data from them. If something was important, it would have already been on my Mac or in the cloud. But despite my lack of loyalty, the insular nature of the VMs ensures that they will run fine no matter the host system.

Trade-offs

This system doesn’t optimize for:

Cost. Mac OS X and VMware Fusion cost money. Using Mac OS X allows me to run other great, expensive software.
Freedom. It’s important to me, but I don’t optimize for it.
Disk space. My biggest regret about this system is that Time Machine chews up the virtual disk images. I read that CrashPlan is better in this regard, but I haven’t tried it.
Performance. The Ubuntu server VMs I run require 256 MB of resident memory. If that’s significant, buy a new machine.

I try to optimize for:

Flow. Even though “it’s all the same machine,” I find lots of situations where there’s a natural division of labor between the Mac frontend and Linux backend. The ability to focus on one or the other helps, and it helps the most when the VM is a local development mirror of a remote Linux production server. Need to edit your config file? You do it in the same place on the local VM as the real webserver.
Security. The fewer services I run on my Mac, the better. The VMs are generally NAT’d from the rest of the network.
Google-ability of problems. Forget for a moment that google-ability isn’t a word, and recognize a great disparity. Searching for “Ubuntu <name of some service or command line program or error message>” almost always leads to the right answer. There are just so many other people in the same boat. At the time of writing, the top hit for “mac os x nfs server” has screenshots full of pinstripes. (It’s a Tiger tutorial from 2005.)

Tools

Here are five things I’ve done to make interface between my Mac and Linux VMs even better:

SSH keys. I shouldn’t even need to state this. If you don’t have these properly working, I’ve failed you in letting you read this far. You should be reading more introductory material.
Prefer Bonjour over editing /etc/hosts. Run Avahi on the Linux VM. It’s sweet, and it saves you from having to note the VMs IP address.
Matching user ids. This is a pain, and somewhat ridiculous, but just suck it up and make your UID in the VM the same as the one on your Mac. It will pay off when you install the…
VMWare HGFS module to mount the Mac’s home directory. The Linux VM can read the Mac’s files directly with this module. I edit webpages in Coda on my Mac, and then have Apache in the VM serve them back to me.
Expandrive to view Linux files. It’s rare that I need to grab a lot of files from the VM (I’ll use scp for small jobs), but when I do Expandrive handles the job well.

Specialization and trade

If you didn’t like the junior-high-biology-class metaphor from the opening, I’ll close with an economic one: Specialization and trade promote higher output and prosperity. Two countries can find ways to capitalize on each other’s strengths, so I challenge you to do the same with these UNIX offspring.

Exercises left to the reader

I’ve left out details of how to replicate my setup, partly because that information has a shelf-life of about five minutes. Also, this is an idea you should take and personalize. Maybe you’ll use Dropbox, rsync, sshfs, WebDAV, or git to move files around. Maybe you’ll try out the latest versions of the ejabberd or Prosody XMPP servers in a VM instead of littering your Mac with files. The ideas and the metaphors are more important than my details.