Tools and emergent complexity: exonerating Twitter and Rails

Twitter has had substantial downtime over the last several days, and this has prompted no end of commentary and analysis. Ruby on Rails was initially blamed for the problems a year ago, then exonerated, then blamed again (and exonerated again). But blaming the hammer for improperly driving a screw is not very illuminating; blaming a screwdriver for how it drives a nail even less so, and although using a hammer and screwdriver combination to drive a large number of finishing nails probably isn't the best solution, until a better machine is invented you wouldn't necessarily know that.

The reason Twitter is having difficulties is that it truly is a novel application. The rules are deceptively simple on the surface, but the emergent complexity is profound, especially as you start to build a massive database of users (which Twitter certainly is now doing). The sort of many-to-many relationships embodied in the way people follow one another, coupled with the different options on what sorts of tweets you want to see, and the different ways of interfacing – the website, instant messaging, text messaging, a raft of third party applications (Twhirl, gTwitter, FriendFeed, et cetera, etc, &c, ...), the ability to track specific terms...

All of this adds up into an extremely complex system that gets exponentially harder to manage as the user base grows. The telephone systems' switching rules are simple by comparison: they are simple, one-to-one connections that connect, persist a short time, and go away, leaving nothing but possibly a billing record (and definitely an entry in an NSA database). A tweet goes onto a user's own list, their friends lists, possibly the lists of friends-of-friends, the list of anyone who is tracking that term, sends it out via SMS, instant messenger and the API, AND persists the message forever; if the user then decides to delete it or make it private then it is removed from all of those lists. Simple, huh? Oh yeah, and it has to do all of that in realtime.

Twitter is built on Ruby on Rails, which came from a simple project management application. Obviously a simple project management application isn't designed to robustly handle the type of complex operations outlined above. It turns out nothing is, which is why Twitter has no easy solutions at hand. Their difficulties in scaling would have likely happened with any existing platform, as not even airline reservation and telephone switching systems handle such a flood of interrelated and interdependent traffic coming from so many different sources – traffic that doubles in two months.

Evan Williams and company invented something new, and they shouldn't be blamed for not initially understanding the true potential and nature of the beast. Although it isn't profitable, it continues to attract investors; anything with this kind of growth and engagement is interesting to businesspeople. NTT invested for a reason, and it's not just because it is popular (and profitable) in Japan. This is an example of how next-generation communication is working: modern switching rules, attention-based networking – a step beyond instant messaging, a step beyond SMS and a step sideways from the phone system. The right tools for the job probably don't exist yet; maybe Erlang is a step in the right direction.

Lastly, I don't blame the Twitter staff for doing experiments on the site during the day. They live in the United States and there's no reason they should have to stay up all night. Besides, we should face the sobering conclusion that Japan's market and the rest of Asia might be more important to Twitter than the depressed, aging, and troubled North American market. From that standpoint, the US is a cheap, talented labour pool crafting clever mercantile goods to send to Asia in exchange for hard currency. Oh, how the worm turns.

Squeezebox mash note

A couple of months ago I ordered a new home music player, the Squeezebox Duet. I was a little reluctant because it wasn't cheap (nearly $500 by the time you include shipping, import duties, etc) and it requires server software, but I liked the look of it and I liked the fact that the server software was open source. I didn't really look too deeply: I just thought "shiny!" and dug out my credit card. Besides, my bro-in-law loves his old Squeezebox, so I figured it'd be good.

But zOMG, I was totally unprepared for what I got: more than meets the eye. This thing is incredible – a real example of what a great product a company can build when it totally understands how to use open source software to build a platform and build a committed community of users and developers. Everything about it is cool. Yes, I knew I was essentially buying an iPod-style remote control for my stereo which would hook to my network and suck all of my music off my computer (actually, my NAS). But it really is a platform.

First, the hardware is very slick. It is well designed, feels nice, looks very chic, and has tons of room to grow. They built in so many extras that the platform can grow for a long time just catching up with the hardware: 3-axis accelerometer, USB adapter and SD flash slot on the remote, WiFi all over the place... simply spectacular. You could build a bunch of cool things on this platform.

Second, the software is quite capable. It was easy to install (on Ubuntu, I just had to add their repository and the package). Unfortunately, the latest version of the server software, SqueezeCenter, has not yet been adapted to run on the NSLU2 I purchased for the express purpose, but it is doing okay on my four-year-old laptop. Besides, I might be able to put it directly on my NAS200 soon.

Third, they're working quite actively on the software, and they walk the talk when it comes to full disclosure: the whole stack is open. They not only release all of their code, they give you a real toolchain and support in making changes. Unlike other devices which manufacturers churn out and abandon, they've recently released an update to the remote control firmware that addresses issues with seeking through music lists, making it much more friendly and responsive. The remote control software also builds and runs on desktop Linux, OSX, and even ghetto Windows (a true tour de force) which makes it easy to build and debug your add-in modules (in Lua: how very nerd chic) or your own custom firmware. It also just gives you a nice remote control to run on your laptop. The open source firmware makes it possible for a community of hackers to come up with endless cool applications for all of that tasty overengineering that went into the remote.

But here's the clincher: I had an issue with my album images not showing up. I've been compulsively tagging my music collection and applying album images to make the lists look pretty, but the images weren't showing up properly – even worse, it was sending corrupt images to the browser, throwing the format and making it all look weird. So I started looking at it, and reported the problem on their bugtrack system. One of their engineers fixed it fourteen hours later; I grabbed the in-development build the next day and my issue was fixed. Twenty-four hour turnaround.

That is not only great service, it helps them build and take advantage of a community of expert users. Unlike some companies which persecute people who fix their problems, at least one part of Logitech has it figured out.

Nortel's continued sad decline

Once-proud Nortel is back in the news, this time with a funny story about how they have a hard time keeping their former subsidiaries as customers. This is especially funny since Nortel's big successes in the go-go 90s were selling switches to Baby Bells who defected from Western Electric when they needed more switches.

For a while there it seemed like half the people I knew went to work at BNR/Northern Telecom/Nortel, and then just as quickly, none of them worked there anymore. The telecom boom died, and everybody had more than enough expensive circuit-switched almost-obsolete equipment depreciating noisily in their expensively airconditioned telecom equipment rooms – around the same time that people really started using blackberry and VOIP in a serious way. The Bay Networks acquisition never fared well against Cisco. Nortel never managed to come out with anything that resonated in the marketplace again, and their financials reflect that.

The telecom industry has become very rapidly commodified, and Nortel's half-cousin and arch-enemy Avaya has become the standard for awful, expensive local PBX solutions, while Asterisk-based solutions are ruining the party for all of the lumbering giants. Nortel could have ridden the wave of open source to become a new low-price leader, but instead seems intent on circling the wagons and riding its customer base down the drain.

And back to the earliest item: they apparently don't have any competent public relations staff. That's pathetic.

LinkedIn

This was recommended to me today, and I was somewhat dubious. Ah, another Friendster clone, I thought. Well, yes, in fact, it is -- but built for business. This is the sort of thing that, if done right, could wipe out ACT! and take over the contact management market. Hell, if it synced to my cell phones I'd even shell out actual money for it. What really surprised me was that in my first pass through my address book 34 people I knew were already members. Some of them actually even seem to have used it (though there were only about 8 of them with more than 2 contacts). An old friend even found me, though he has ~400 contacts and probably takes it pretty seriously.