Let me start with this year, 2009. What was good? Hm; my prosthetic brain units at iTunes and last.fm tell me that it’s:

• Dysrhythmia, "Psychic Maps" [jaw-dropping instrumental math-metal, will have you banging your head in 7/13 time.]

• Isis, "Wavering Radiant" [post-metal? huge lowercase-’p’ progressive epics. so good I’m willing to overlook the cookie-monster vocals.]

• Pelican, "Ephemeral" EP [is it post-rock with big riffs? or restrained brainy instrumental metal?]

• Apricot Rail, "Apricot Rail" [ok, this is instrumental-post-rock for sure, but atypically cheerful, kind of like Do Make Say Think. they’ve even got oboes omg!]

• The Happy Hollows, "Spells" [this is the token recognizably-indie-rock album on the list. are they secretly Deerhoof covering the Pixies? Or Lush covering Interpol? also, they are hella cute and I can’t wait to see them live]

• Dirty Projectors, "Bitte Orca" [I still can’t figure this out, with its Afrobeat guitar lines, intricate interlocking vocals, weird rhythms and weirder lyrics. ok, that description makes it sound like "Remain In Light", which it isn’t at all like, but maybe is somehow.]

• Elisa Luu, "Chromatic Sigh" [can I say "eclectic" with a straight face? really interesting electronic music, clearly song-like, ranging between poles of rock and ambient.]

• Anduin + Jasper TX, "The Bending Of Light" [guitar-based drone. majestic.]

• Brock van Wey, "White Clouds Drift On And On" [exactly what it sounds like. it comes with or without beats, your choice.]

• Eluder, "Drift" [exactly what it sounds like, too. careful with this one, some people have had trouble returning to earth afterwards.]

• Jónsi & Alex, "Riceboy Sleeps" [Sigur Rós guitarist and his boyfriend do Stars Of The Lid. delicious in small doses.]

• Victoire, "A Door Into The Dark" EP [a string ensemble that embraces electronics as a natural part of their sound]

• Sub, "Id" EP [nothing new, but basically the best Photek tracks I’ve heard in ten years.]

PS: there might be some last minute additions to this list if Kate Simko’s "Sounds Of The Atom Smashers" and Concern’s "Truth And Distance" live up to their potential.
PPS: ZOMG I forgot Sunn O)))’s stark and forbidding "Monoliths And Dimensions", whose final track "Alice" is deserving of a space up there.
PPPS: Yes, a mix of this stuff is forthcoming…

* It’s not like I’ve been having trouble finding great new music, or that I resent other people for all picking different music than me; but it’s a little sad to not be part of the zeitgeist. Long ago it was really important to me whether punk and new wave would break dinosaur rock’s lock on the mainstream, or whether little-known underground bands like the Cure or the Pixies would get the recognition they deserved. I think the peak of my with-it-ness was circa 1990-91 when I could rattle off the name of every shoegazer band that mattered and I treated every issue of Melody Maker as a shopping list to take with me to the import bin at Tower to find the next Cranes or Chapterhouse.

Problems worthy of attack
Prove their worth by fighting back.

I just pushed my latest changes up to bitbucket.org. What’s new?

Variable-sized top-level index. Previously, the hash data structure used a top level array of 256 hashtables. I’ve now made that ‘256’ a variable that scales with the number of records. This saves a lot of room for smaller datasets. (Sounds easy, right? I thought so too. But it ended up triggering significant changes to the file format and algorithms and took a lot of debugging.)

The return of the trees! part un. I started this as a project to learn about B+trees, but when I got to the point of implementing the append-only multiversion stuff I did it with hashtables first because it’s more straightforward that way, and the tree code went into the freezer. Well, it’s back now. There is a “new” Tree class that can be used to read and write persistent B+trees. It’s not yet integrated into the top-level Ottoman class, though, so you can only use it on its own, not intermixed with hash-tables, and without the API for tracking versions.

The next task is of course to integrate the tree API into the Ottoman class. But more than that, it will be entwined with the hashtable such that trees will serve as searchable indexes into the data store. So the hashtable will provide extremely fast but unordered access via a unique “record ID”, but you can build as many indexes as you like that will use ordered secondary keys (of your choice) to look up hash values.

At that point, I believe Ottoman will be ready to serve as a substrate for HTML local data storage, for implementing a CouchDB-compatible database (a la JSONDB), or for other fun databasey purposes.

“ZSync is an open source syncing library designed to allow easy syncing of data between an iPhone/iPod Touch and the OS X Desktop.
ZSync utilizes the BLIP library and Apple’s Sync Services to allow easy and seamless syncing of data.”

It’s still in early development though, with a first public release expected in January:

Right now the code is in a private GitHub repository while the initial framework and protocols are fleshed out. This is expected to go public in January of 2010. Until then we are keeping the development team very small so that we can flesh out the design without a lot of overhead.

This looks like it’ll be super useful for iPhone apps that want to integrate with their Mac siblings, especially since their design won’t require you to have the Mac app running while you sync.

The trouble is, everything you type into Wave is transmitted live, in real time—every keystroke was getting sent to Zach just as I hit it. This made me too self-conscious to get my thoughts across.

… Maybe I should just delete what I’d written and say, “Twitter works because it’s simple.” But I couldn’t do that, because Zach was watching me. He could see me struggling right now—he could see that I’d gotten myself stuck in a textual cul-de-sac and that I was desperately searching for a way out without looking foolish. Now I saw Zach beginning to type: “Don’t let the live-typing get you down!” The game was up; what was the point of making a point now? I ended my thought clumsily and then resolved never to attempt to say anything very deep on Wave.

The same thing happened seven years ago with the live-typing feature that I implemented in iChat 1.0 (which was only supported for Bonjour chats.) I thought it was an awesome idea, and I’d wanted to have it in a chat program since about 1997. But it turned out that, in actual use, people hated it, for exactly the reasons Manjoo describes: it makes you self-conscious. We took it out in the next release.

(Interestingly, I hate video chat for a very similar reason. Somehow, the fact that my picture is being shown in real time to the remote person makes me horrifically self-conscious, even though it wouldn’t bother me at all to talk face-to-face with that person. I don’t know whether it’s the little preview on my screen, or the fact that the person is spookily both present and not-present, but the few times I’ve tried video-chat have been really unpleasant.)

I’m usually on the side of more technology. I believe that our online communications tools are still horribly primitive and have only scratched the surface of what’s possible. But this was a case where more technology was bad.

The low-tech alternative that lots of people use in IM,
is to write in short fragments,
each a separate message,
so the other person can see them one by one
without waiting for you to finish the whole sentence.

The difference is that you’re in control over when to send a partial message, and the other person knows you’re in control, and so on. I still think it might be possible to do this in a higher-tech way, like using a hot-key to send a partial message on demand without having the funky line-breaks, but the current approach isn’t so bad as long as you’re not embarrassed about unintentional free verse.

I could have told the Wave people about what I’d learned, except I didn’t know Wave existed until April (shortly before the public announcement), and even then I was just some guy lost in the crowd at the demos….

Part of the problem, in both cases, is that live typing is one of those Cool Demo Features that looks really awesome when showing off the app. Features like that can be dangerous because they are legitimately very useful during the app’s gestation, when exciting demos are a key survival trait; but then they can’t be removed later on because they’re so well-known, even if they turn out to be useless. Sometimes these features aren’t actually harmful to the user experience, they just make the code more complex and harder to maintain. Instant typing is both, unfortunately. (The clever sync algorithms and rapid-fire network messages Wave uses would be needed even without live typing, but the fact that they have to run on every few keystrokes, not just every minute or so, pushes those things so much harder.)

Ottoman is a lightweight, reliable key-value store with multi-version concurrency control.

A key-value store is a persistent on-disk dictionary that maps arbitrary keys (usually short text strings) to arbitrary values (binary data of any length). There’s already a large and venerable family of libraries that do this, often referred to by the name “db”. Notable members include dbm, Berkeley DB, cdb and Tokyo Cabinet. What makes Ottoman different?

1. It tries to provide extremely reliable storage, immune to file corruption caused by operating system crashes or power failure while writing.
2. It stores past versions of the database, as a useful side-effect of the way it works. Applications can use this to provide features like snapshots, timelines or persistent undo. (You can prune old versions, for space or privacy reasons, by writing a fresh copy of the file.)
3. It has very low memory overhead. The file and its contents are memory-mapped, so even large values can be accessed without having to copy them into the heap. (Opening a database only mallocs a few hundred bytes!)
4. It allows concurrent read/write access by multiple processes. Optimistic concurrency — as used by version-control systems like Subversion — keeps file locking to a minimum: readers are //never// blocked at all, and writers are only blocked briefly (clients can’t lock the database for arbitrary periods, only for as long as it takes the accumulated changes to be written to disk.)

Why is it named after a footstool? First, as an homage to CouchDB. Secondly, it turns out ottomans can be used for storage. Third, because I have a mild fascination with Asian and Islamic history.

_…Read more at the project home page!_

Almost everything should work more or less the same. I’ve taken the opportunity to move to a subdomain—jens.mooseyard.com—but the old URLs redirect. The theme is different, because my hacked-up, customized old theme probably wouldn’t have survived the upgrade intact. I can’t say I like this one better, but it’ll do.

The only data loss I know of is the email subscriptions. If you were subscribed to the comments of any prior post, that’s gone now. If you still have something new to say, you can post it and check the “Subscribe” checkbox, and then you’ll get emails again.

And also because I’m a big fan of CouchDB, and really admire its elegant storage model. It’s an on-disk B-tree—no surprises there—but the file is append-only, which both makes it impervious to crash-related corruption, provides nearly lockless concurrency, and makes it easy to access earlier revisions.

[In a nutshell: Updated data values or tree nodes are appended to the file instead of overwriting the earlier versions. Since updating a node changes its location, its parent node needs to be updated too to point to the new location. This recurses up the tree, meaning any change ends up with a new root node written at the very end of the file. In fact, when you open the file you find the root by looking at the very end. Since no data is ever changed, once you open the file you’re impervious to changes made by other writers since they don’t affect anything you’re looking at.]

I’d love to use something like that for various projects, but as CouchDB is implemented in the exotic functional language Erlang, I can’t really use its storage layer as-is. So: could I implement something like it in C++?

Thus far I have a working in-memory B+tree implementation. Inserts and deletes work, and I’m working on the iterator. Even this much was harder to get working than it should have been, or so it feels. But that always seems to be true—algorithms sound straightforward when you read about them, but putting them into practice exposes you to all the details and subtleties inherent in hand-waving like “now merge the node with a neighbor”.

Actually I haven’t implemented a straight B+tree, rather a ‘top-down’ variant described by Ohad Rodeh that’s better suited to this type of application because it changes fewer numbers of interior nodes during an update.

What’s next?

• Support for string keys (so far it just handles ints)
• Serializing nodes to/from disk
• Keeping track of which nodes are touched during an operation, and appending those to the file
• Writing a trailer to the file to mark a successful update (and to link back to the previous trailer, for historical purposes.)

Sounds straightforward, but of course the devil’s in the details.

“The social networking integration that we reported iTunes 9 would have seems to be part of a bigger social networking push by Apple,” the report states. “We’ve been informed that Apple has plans to tie iTunes 9 into a “Social” application that they plan to release in the future.”

This sounds like the kind of app (though separate from iTunes) that Jessica Kahn and I kept trying in vain to get Apple to build, circa 2003-2005. Maybe they’ll get some use out of our abandoned prototypes.

The report goes on to say that the new application would allow users to share their listening habits with friends [and] send music to friends”

Mike Estee and I had actually prototyped this in iChat in 2003, but the feature never got approved since there were so many more important things to add, like 3-way video conferencing. (Plus the fact that Apple execs turned white as a sheet if you said the words “send music” near them.)

Anyway, personal bitterness aside, I think it’s really amusing that Apple keeps shoving the kitchen sink into iTunes, since that has to be the single nastiest, hardest-to-extend codebase they have — it’s their last remaining Carbon app, with a foundation that dates back to Casady & Greene’s SoundJam, circa 1998.

I decided on Google Sites, which I knew and liked from its days as JotSpot, a hosted wiki with some powerful features. It ended up not being the right place for me, for a couple of reasons:

• Currently, Sites’ priorities are in website publishing, as a replacement for Google Page Creator (which is being phased out soon.) It’s quite good at it, but I’m less interested in that than in collaboration features.
• Google’s server-side infrastructure is really, really, really huge and complex. There is an endless landscape of internal technologies and tools—the few that have been described in public (MapReduce, BigTable, Chubby, etc.) are just the tip of the iceberg. I have discovered that I am not very interested in this kind of stuff, and I quickly became frustrated by the deluge of technologies I needed to learn to get things done.
• Running a large web service is like running a nuclear power plant or an electric power grid. It requires 24/7/365 monitoring, and at that scale, anything that can go wrong will go wrong, and frequently does. I do not have the right temperament for working with this, especially not when it comes to taking turns carrying a pager that wakes me up at 4AM because some service’s latency has gone above 300ms.
• I’ve been vocal about my frustration with centralized systems; Google’s websites are kind of the ultimate in that (even though, ironically, they’re implemented as P2P-like networks internally, as I believe all large web operations are today.)

The good news is that Google encourages transfers between teams, and makes it easy to do so. The near-total transparancy inside the company makes it easy to find out everything that’s going on, and there’s a well-designed website for engineering transfers that helps you find teams that need people. I even went to an internal job fair.

I’ve now ended up working on Chrome, Google’s web browser. The team I’m on is responsible for implementing HTML 5 features, as well as designing and implementing other new features (for standardization) that will help web apps become as powerful as native apps. Much of what we do will go into the WebKit source tree, where it will also directly benefit Safari, Android, the Palm Pre, and other WebKit-based browsers.

In fact, everything I work on (more or less) is going to be open source. Both the WebKit and Chromium source trees are public. You can view, if you care to, the one patch I’ve contributed so far and the one that’s currently out for review. That’s kind of mind-blowing, in a good way, to me, steeped as I am in the secrecy of Apple.

I’m pretty excited by this. There are quite a lot of things I’m interested in working on—client-side storage, local apps, drag-and-drop, better font support, menus, even far-out stuff like peer-to-peer networking. Forward in all directions!

Speaking of “developers”, I’ve been really pleased with the reaction to my initial source-dump to Bitbucket! So far, in nine days:

• 79 people have subscribed to change notifications;
• 10 people have forked the repository so they can modify the code;
• 8 people have contributed to the code, fixing bugs and adding some new features;
• 39 changesets have been committed to the repository;
• 29 bug reports and feature suggestions have been filed,
• 12 of which have been fixed.

I’m especially happy about the patches I’ve received; I’ve gotten some in the past, but never this many or so quickly. Part of this is thanks to Bitbucket, which (like its inspiration, Github) makes it so easy to share code and manage contributions. I’ve been getting more contributions to MYNetwork as well, since hosting it there.

My initial motivation was like Tom Sawyer whitewashing the fence—I was hoping someone else would take over the project, since I’d run out of steam and hadn’t done any work on it in a while. But what happened instead is that the attention got me enthused to work on it again, so I’ve made a lot of progress on it this past week. That’s a good thing!, just not what I expected.

Now I think it’s about time to put out an actual easy-to-download binary release of Murky, as some of the main bugs have been squashed and I have more confidence that it works for other people besides me. I’ll probably do that this weekend, and mention it here as well as on the new mailing list (don’t forget to subscribe.)