No again, I will not show you what’s under the bandage on my arm. I won’t even look myself, anymore; it’s gotten too disturbing. I mean, the wound hasn’t changed, but every time I look at it it bothers me more, takes me longer to stop shivering. I keep wanting to touch it.

Listen: Did you ever play Shock City? I’m not trying to change the subject; hear me out. I played the hell out of that game when I was twelve, and I always wondered what was inside all those buildings you couldn’t get into. There were only eight buildings with working doors in the whole game, with all kinds of things to explore and enemies to stalk in each one; so if you could somehow get into the other hundred or so buildings, what would be inside them? Could you play them too?

I know, I was pretty dumb back then. I didn’t understand the way the world works.

Finally someone told me about a hack to get into any building. It exploited a bug in the hit detection: you had to run at a corner and jump right at it. If you got the position and angle just right, you’d slip through the join between the two walls and be inside. People were using this as a cheat to get into the higher levels early, but all I could think about was the “secret” buildings. By then I had already covered my bedroom walls with hand-drawn maps of what I thought their insides might be like. I booted up the game and ran straight for the Library — I knew it was full of maze-like miles of dimly-lit dusty stacks and cavernous reading rooms, through which I could pursue my quarry of mutant beasts.

Instead I hit the corner and bounced off, of course. The instructions said to be patient, so I backed up and tried again. All the good skills took practice. I got into my best gamer trance state of endless repetition and fine-tuning of reflexes. Finally after an hour I made it: instead of the bounce I got a split second of mangled polygons warping across the screen, and then … nothing. The inside of the building didn’t exist, and neither did the insides of its walls, so it was as though there were nothing there. I could see the streets and buildings on the other side. But the building’s floor didn’t exist either, and there was no ground under it, just a yawning abyss of pure electric video blue that I fell into like a rock. I had about ten seconds to turn and look up: the universe was a void with nothing in it but a few square blocks of street-plan dotted with empty building-shaped holes. It receded into the distance, and then I reached the outer boundary of the world and snapped into the death screen.

Yes, I do have a point. The real world is like this too. The things we think are solid are fakes; there’s nothing inside them. The things we can’t open up are hollow, infinitely-thin shells around nauseating blue.

I know what you’re about to do. Everyone does this when I explain things. “I refute it thus!” as you hit a wall with a rock. You’ve got a hammer in that backpack? Even better.

Yeah, that support pillar looks pretty solid right there were you decided to hit it. With realistic bits of fractured concrete inside it, right there. Or over there, yes. Don’t you get it? You were meant to break through there. Just like I was meant to go into those dozen buildings but no others. You’re playing the game the way you were meant to, and it’s a very good game so the limitations are fewer, the glitches are harder to exploit.

But with the right skills, the right mind-set, I can find the places that weren’t meant to open up and make them open up anyway. And now that I’ve learned, I can’t un-learn it. Everything I do is wrong.

I can’t find my way back to the right doors. I don’t want to live out here in the cold underneath a fucking highway overpass with brain-damaged junkies. Present company excepted. I had a job and a place to live and friends. But I can’t get back to them: they’re part of the painted-on scenery now. They’re not real objects, they’re texture-maps. The street level doors open onto blue abysses. So do the windows. I’ve tried them all; then I gave up. I almost fell into them, any number of times, had to grab hold of texture that looked just like brick or paint or wood on one side and didn’t exist on the other. Vertigo made me sick and I threw up into that upside-down sky and watched it fall out of sight and vanish.

The former friends just repeat the same list of canned lines in random order, like any good NPC. Mostly “You look pretty bad, man” and “You really need to get some help”. It’s pathetic.

The last time I tried to get inside, I ran into a corner. Yeah, you’d think I would have tried that trick earlier, but you try ramming into the corner of a building in “real life”. Explain to your hindbrain about cheat codes when it sees a wedge of brick looming up. Even after psyching myself up I’d always swerve aside at the last second.

But I finally managed it. What do you think happened? I fucking knocked myself out on the bricks, that’s what. I came to a while later — this was in a back alley so no one had seen me. My face was covered in drying blood from this big gouge on my scalp, and I’d knocked this tooth out, but no serious head injury.

So I lay there a minute, feeling like shit, gathering up strength to move. Then I looked at how I was lying, and saw that my right arm went right into the wall and disappeared at the elbow. A clipping error. I could still feel my hand, but it wasn’t touching anything. I sat up slowly, the arm came loose with no resistance, but it just … ended there, in a smooth flat cut at the elbow. And I pulled it toward me and looked at it and there wasn’t anything inside me! Just an oval hole into a blue void.

I was too afraid to touch it with my fingers. Still am. I found a curl of rusty wire, unrolled it and stuck the end of it in. It just went straight into my arm without resistance, no matter how I wiggled it around inside. On the other side of that hole I didn’t exist. I got all four feet of wire into there, then lost my grip on the other end and it fell all the way in and disappeared … I didn’t feel it hit anything on the way down.

Of course you don’t believe me. Ha! No, whatever I’m full of, it isn’t shit. Alright, I’ll take the bandage off the stump and show you. It’s not paint. Sure, try sticking your fingers in and wiggling them. Fun, huh? Not so fun, huh?

You know about phantom limbs? I can still feel my forearm and hand. They say it’s because the brain centers that controlled them are still there, and want something to do, so they make up sensations. But like I said, I couldn’t feel them touching anything, just touch the fingers together and to the palm, make a fist and feel that. … Only now, just this morning, I can feel something else.

It feels like hard plastic. In my mind’s eye it’s shiny black. Its contours fit my grip perfectly. It’s got buttons on it, and a joystick. I’m going to try moving it now.

This is a sequel to my story Ozone. I’d wanted to write one for years, but didn’t have any inspiration about what happened next. Then last week the ideas in here came to me, and I realized that they fit neatly into that story-world. But the ending only came to me today, after I’d started writing the story down.

I got these ideas while watching my son explore glitches in the notoriously buggy game Pokémon Blue. I would love to have him read this story, but I know it would give him nightmares for weeks…

Twitter: Charming in brief doses, he tells you little one-liner jokes, then wanders off after two sentences to go talk at somebody else. He absolutely will not shut up for an instant, and namedrops shamelessly about his famous friends. When he’s outworn his welcome he passes out drunk on the floor and has to be dragged home.

MySpace: Who? Oh, right, this anorexic high-school girl who threw herself at you at a party once in 2005. She kept bragging about all the bands she knew (and which you could overhear on the tinny earbuds she wore.) After one too many Jägermeister Jell-O shots she barfed Day-Glo all over your shoes. Last you’ve heard, she’s found some 80-year-old media mogul to be her sugar daddy.

Facebook: You vaguely remember him from high school. He was a nonentity then and he’s equally uninteresting now, but he’s somehow infiltrated your circle of friends and shows up at every social event you go to, telling boring anecdotes about last night’s game and what he bought at Wal*Mart. Worse, it seems he’s joined some cult and wants you to join too so he can go up a level.

Tumblr: She’s got impeccable taste, a lovely apartment and fascinating stories, but after a while you realize she only talks about what other people have done; she doesn’t have an original thought in her head. She won’t carry on a conversation, either, so the only way to get her to pay attention to you is to repeat back something she’s already told you.

Soup.io: Similar to Tumblr, but with a cute Austrian accent. She’s more conversational, but on the downside she sometimes insists on talking to you in German.

LiveJournal: A mysterious Goth chick you were introduced to at a club. After you strike up a friendship with her, she starts telling you all her innermost secrets whenever you see her. This is terribly alluring at first, enough so that you can overlook her appallingly bad fanfic, but after a while you begin to realize how seriously disturbed she is. Around then she abruptly stops showing up, and you’re never sure whether she killed herself or just moved to a more elite social circle. You never learn her real name.

[Update: I’ve changed two of them to male. I wanted to be consistent in the personification, but in retrospect that leaves me open to charges of sexism, which absolutely wasn’t intended. They all have male counterparts, of course, whom I’d love to hear about if you want to write about them.]

I’ve never figured out whether I prefer Python or Ruby, so I’ve written things in both languages. Sometimes I start in one, then change my mind and decide I’d rather use the other. Unfortunately, changing over is painful, even though both have fairly similar syntax. For instance, converting to Ruby means inserting zillions of “end” statements!

Having a need to do this recently, I lazily looked around for a script that would do the grunt-work of Python-to-Ruby translation. I couldn’t find one, so I ended up writing one myself. And I’ve uploaded it for the benefit of others who might have the same need, and who might even improve it.

So here it is: py2.rb. Use it wisely. In particular, pay attention to the caveat found in the file’s header comments:

This script just does the obvious, easy transformations, giving you more time to work on the harder ones :) It is NOT a real parser, just a bunch of kludgy regex operations, so it can’t do anything fancy. It may get some things wrong, and won’t even attempt some other things that it’s very likely to get wrong. The output will definitely have to be hand edited by someone familiar with both languages, before it can be expected to compile as Ruby, much less run correctly. The goal is simply to require less hand editing, and less mechanical replacing, than you would have had to do without this script.

It’s in the public domain. “Do what thou wilt” shall be the whole of the law.

Note: Please don’t comment with Python advocacy or, worse, anti-Ruby flames. That’s not the point. I like Python fine; I just prefer to remain promiscuous. And I think it’s ultimately in everyone’s interest for ideas to be able to flow from one form of expression to another.

Brent “NetNewsWire” Simmons raises the idea of an open protocol for syncing RSS/Atom subscriptions, that is, a way of keeping multiple local newsreader apps (like on a Mac and an iPhone) in sync with each other, so that they share the same set of subscribed feeds, and remember which articles have already been read. You can think of it as “IMAP for RSS”.

NetNewsWire already does this using Google Reader as an intermediary, and Apple’s PubSub framework (which is what Safari and Mail use) shares the read/unread state using MobileMe. But it would be nice to have an open protocol.

I have some experience with this, having implemented the sync system used by PubSub. It’s an interesting problem—you might think I would have just used Apple’s SyncServices, and it’s true that it would have worked great for the subscription list, but it doesn’t scale well to huge numbers of rapidly-changing “read/unread” flags.

I have two suggestions (which I would have made on Brent’s blog, except he doesn’t allow comments anymore.)

CouchDB

CouchDB is an awesome web-centric database engine. It doesn’t use SQL; instead, it’s a glorified key-value store whose values are arbitrary JSON objects, and which uses map-reduce for efficient querying. The basic API is pure REST, though glue libraries for many languages exist.

CouchDB natively supports syncing data through distributed groups of servers. It’s sort of like the way distributed version-control systems like Git or Mercurial work: multiple CouchDB instances each store a replica of the same data set, but can “pull” changes from each other over HTTP to stay in sync.

CouchDB is pretty lightweight and is already being used on the desktop by client apps: GNOME has been integrating it into the Linux desktop to use as a shared store for user data like contacts and bookmarks. It plays a similar role to SyncServices on Mac OS, but it’s all open source and any two instances can sync with each other instead of requiring a proprietary server. I hear this is already shipping in the latest Ubuntu releases.

It doesn’t look as though anyone’s designed a schema for storing RSS subscriptions this way, but it would be pretty easy to define one. You then need a local agent running CouchDB (it can be stripped down to be pretty small), a client library for Cocoa apps, and an upstream CouchDB server to sync to.

REST-Logging

This protocol is similar to what I came up with for PubSub. It’s a simple extension of REST, but I haven’t heard of it being used elsewhere. The idea is that you model an append-only log file as an HTTP resource. The items that are logged are ‘events’ describing changes in the data model, in this case the subscriptions and articles.

The sync algorithm looks like this:

1. Download all the data that’s been added to the remote log file since your last sync. Remember the file’s ETag.
2. Parse that data into a sequence of log entries, and process them in order. Each entry names a model object (feed or article) and an action (subscribe, unsubscribe, mark read, mark unread). Apply those changes to the local data store.
3. Query your local data store to find all the changes that have been made since your last sync. Ignore the remote changes you just applied in the previous step, and also any earlier local changes that duplicate a remote change (like marking the same article as read.)
4. Generate a series of log entries for those changes and concatenate them into a data blob.
5. Upload that blob, appending it to the remote log file. Remember the resulting ETag. In case of a conflict (someone else has changed the remote file since step 1), toss out the blob and return to step 1.

You can think of the log file as a queue or message stream that’s being collaboratively read and written by all of the clients. This sounds like something you’d need a fancy web-app to manage, but it turns out that all it takes is a typical HTTP 1.1 server and a trivial server-side script.

The download is a conditional GET, as used for fetching feeds themselves. The difference is that you use a “Range:” header to request only the bytes past the last known EOF. For example, if the last time you read the log it was 123456 bytes long, you add the header “Range: 123456-” to the request. This ensures that you only get back the new bytes that were added to the end. (And since this is a conditional GET, if the file hasn’t changed at all you just get back an empty 304 response.)

That’s all you need to do to track changes. Since the file is append-only, the only bytes you need to read are the ones added to the end. This request efficiently sends you just those bytes.

What’s cool is that this require no server-side software. If the log is a static file, any regular HTTP server like Apache will automatically handle GET requests for it, even byte-range ones. (Ranges are already used by browsers to resume interrupted downloads.) And it sends the response at high speed, since the server’s just streaming from a file, without multiple back-and-forth requests and without expensive database queries.

How about writing? Ideally you’d use the same approach, with a byte-range PUT that specifies that the request body should go at the end of the file. Unfortunately most servers don’t support this for static files, even though it’s basically just HTTP 1.1. But it’s really easy to implement. Any PHP crufter should be able to whip up a one-page script that simply responds to a POST by reading the request body and appending it to a local file (while doing the necessary ETag and range verification.) The great thing is that this script doesn’t have to know anything at all about RSS or subscriptions or unread counts; it’s completely generic. You can upgrade the data model without having to touch the script, and you could use the same script to sync anything, not just RSS.

(Yes, there is a semi-obvious drawback to this protocol: the file grows without limit. Surprisingly, this is not a problem most of the time, since clients only upload or download new data; the only real limit is the maximum file size or disk quota allowed by the server. But it does present a problem for a new client, whose first-time sync would download the entire file. This can be worked around by having new clients ignore very old data (only download the latest 10MB, say) or by periodically writing a compact subscription list to a separate URL.)

Every year the Albums Of The Year lists seem more and more removed from my experience. (Most of the time I haven’t heard a single album on the list.) Worse, we’re now getting into the Of The Decade lists, making me realize how long this has been going on*. If you ask me the top albums of the ‘80s or ‘90s, I don’t have too much trouble rattling off a bunch of names. But this decade? I get confused and have to start thinking hard and looking through the back covers of my mix CDs. Why is that? [Ed.: it’s because you’re getting old. Duh.]

Let me start with this year, 2009. What was good? Hm; my prosthetic brain units at iTunes and last.fm tell me that it’s:

• Dysrhythmia, "Psychic Maps" [jaw-dropping instrumental math-metal, will have you banging your head in 7/13 time.]

• Isis, "Wavering Radiant" [post-metal? huge lowercase-’p’ progressive epics. so good I’m willing to overlook the cookie-monster vocals.]

• Pelican, "Ephemeral" EP [is it post-rock with big riffs? or restrained brainy instrumental metal?]

• Apricot Rail, "Apricot Rail" [ok, this is instrumental-post-rock for sure, but atypically cheerful, kind of like Do Make Say Think. they’ve even got oboes omg!]

• The Happy Hollows, "Spells" [this is the token recognizably-indie-rock album on the list. are they secretly Deerhoof covering the Pixies? Or Lush covering Interpol? also, they are hella cute and I can’t wait to see them live]

• Dirty Projectors, "Bitte Orca" [I still can’t figure this out, with its Afrobeat guitar lines, intricate interlocking vocals, weird rhythms and weirder lyrics. ok, that description makes it sound like "Remain In Light", which it isn’t at all like, but maybe is somehow.]

• Elisa Luu, "Chromatic Sigh" [can I say "eclectic" with a straight face? really interesting electronic music, clearly song-like, ranging between poles of rock and ambient.]

• Anduin + Jasper TX, "The Bending Of Light" [guitar-based drone. majestic.]

• Brock van Wey, "White Clouds Drift On And On" [exactly what it sounds like. it comes with or without beats, your choice.]

• Eluder, "Drift" [exactly what it sounds like, too. careful with this one, some people have had trouble returning to earth afterwards.]

• Jónsi & Alex, "Riceboy Sleeps" [Sigur Rós guitarist and his boyfriend do Stars Of The Lid. delicious in small doses.]

• Victoire, "A Door Into The Dark" EP [a string ensemble that embraces electronics as a natural part of their sound]

• Sub, "Id" EP [nothing new, but basically the best Photek tracks I’ve heard in ten years.]

PS: there might be some last minute additions to this list if Kate Simko’s "Sounds Of The Atom Smashers" and Concern’s "Truth And Distance" live up to their potential.
PPS: ZOMG I forgot Sunn O)))’s stark and forbidding "Monoliths And Dimensions", whose final track "Alice" is deserving of a space up there.
PPPS: Yes, a mix of this stuff is forthcoming…

* It’s not like I’ve been having trouble finding great new music, or that I resent other people for all picking different music than me; but it’s a little sad to not be part of the zeitgeist. Long ago it was really important to me whether punk and new wave would break dinosaur rock’s lock on the mainstream, or whether little-known underground bands like the Cure or the Pixies would get the recognition they deserved. I think the peak of my with-it-ness was circa 1990-91 when I could rattle off the name of every shoegazer band that mattered and I treated every issue of Melody Maker as a shopping list to take with me to the import bin at Tower to find the next Cranes or Chapterhouse.

Yes, I’m still working on Ottoman (my append-only multiversion-concurrent storage library). As the code grows in size and complexity, so it grows in its resistance to being changed, but as Piet Hein said and I never tire of quoting:

Problems worthy of attack
Prove their worth by fighting back.

I just pushed my latest changes up to bitbucket.org. What’s new?

Variable-sized top-level index. Previously, the hash data structure used a top level array of 256 hashtables. I’ve now made that ‘256’ a variable that scales with the number of records. This saves a lot of room for smaller datasets. (Sounds easy, right? I thought so too. But it ended up triggering significant changes to the file format and algorithms and took a lot of debugging.)

The return of the trees! part un. I started this as a project to learn about B+trees, but when I got to the point of implementing the append-only multiversion stuff I did it with hashtables first because it’s more straightforward that way, and the tree code went into the freezer. Well, it’s back now. There is a “new” Tree class that can be used to read and write persistent B+trees. It’s not yet integrated into the top-level Ottoman class, though, so you can only use it on its own, not intermixed with hash-tables, and without the API for tracking versions.

The next task is of course to integrate the tree API into the Ottoman class. But more than that, it will be entwined with the hashtable such that trees will serve as searchable indexes into the data store. So the hashtable will provide extremely fast but unordered access via a unique “record ID”, but you can build as many indexes as you like that will use ordered secondary keys (of your choice) to look up hash values.

At that point, I believe Ottoman will be ready to serve as a substrate for HTML local data storage, for implementing a CouchDB-compatible database (a la JSONDB), or for other fun databasey purposes.

ZSync is a new Mac/iPhone library that uses my BLIP P2P networking protocol:

“ZSync is an open source syncing library designed to allow easy syncing of data between an iPhone/iPod Touch and the OS X Desktop.
ZSync utilizes the BLIP library and Apple’s Sync Services to allow easy and seamless syncing of data.”

It’s still in early development though, with a first public release expected in January:

Right now the code is in a private GitHub repository while the initial framework and protocols are fleshed out. This is expected to go public in January of 2010. Until then we are keeping the development team very small so that we can flesh out the design without a lot of overhead.

This looks like it’ll be super useful for iPhone apps that want to integrate with their Mac siblings, especially since their design won’t require you to have the Mac app running while you sync.

As everyone knows who works in the pet-food industry (or computer software for that matter), it can be hard to start eating your own dogfood. Case in point: I just this week set Chrome to be my default browser, though I’ve been working on it for four months now.

Partly that’s because when I started in July the Mac version of Chrome was too immature; and partly it’s because a web browser is something you need to have running and working all the time—especially since the Chrome project’s bug tracker and code-review tool are web-based.

But Mac Chrome is quite stable enough to use now, and as I haven’t been doing much Chrome development on this MacBook Pro lately (it takes too long to compile compared to my souped-up Mac Pro) I’ve installed the latest dev-channel build and replaced Safari with it in my Dock and as my default browser.

It’s hard to get used to a new browser, after all these years. I remember that I dropped IE 5 like a hot rock as soon as Safari became useable, but that’s because IE sucked so badly on OS X (as you youngsters may not remember.) But Safari is a great browser. Chrome’s great too, but in different ways, and the Mac version’s not finished yet so there are some missing bits.

I should note that I generally don’t work on the user-visible parts of Chrome, rather the underlying WebKit engine; so I haven’t been focusing on the UI much, or noticing the features being added, until experiencing them as an end-user.

In Chrome’s favor:

• It definitely feels faster. At work I’ve been obsessing over some micro-benchmarks, but in regular use what I notice is simply that pages come up more quickly. I think a lot of this has to do with DNS pre-fetching, because DNS query times here at home can often be slow.
• The UI of the tabs is better than Safari’s. I like that they take up less room (though without eliminating the title bar the way the Safari 4 beta did), and the transition animations when opening/closing/detaching tabs are extremely slick, definitely Apple-quality.
• The downloads shelf at the bottom of the window is more visible than Safari’s little Downloads window, which always gets hidden behind the browser.
• The list of recently-closed tabs at the bottom of the new-tab page is very thoughtful and handy. In general I find the new-tab page more useful than Safari’s, if less flashy.

Rough edges (remember, this is a pre-beta build):

• PDF display doesn’t work well. There’s no way to switch pages yet (making multi-page documents unusable), there’s no zooming, the Save command doesn’t work, and often the renderer process starts redrawing constantly and eating up CPU time.
• The only indication you have that a page is loading is the little 16×16-pixel spinning circle that replaces the tab’s icon. I like the visual design of this, but it’s too hard to find. I still miss the big progress bar in the address bar from Safari 1-3. (Cursor feedback would be nice—wasn’t there a browser once that put a little wristwatch badge on the arrow cursor?)
• The bookmarking UI is still unfinished and unpolished. In particular, there’s no way to edit/move/delete bookmarks yet. I love the idea of being able to quickly bookmark by clicking a Google-y star icon, but in practice it pops up a floating panel for choosing a folder, which makes me think about where to put it just like Safari’s bookmark sheet does. I’d rather the star button just added it to an “Unsorted” folder that I could open and organize later.
• Autofill doesn’t seem to work as well as in Safari. For example, it won’t autofill login forms until I type in the username; I think this may be a security feature, but I find it annoying.

I’ve been impressed by Chrome’s stability too, for a pre-beta development build. The app hasn’t crashed once, and I haven’t even gotten the “Oh, snap!” page that shows that a renderer process crashed. (I’ve seen plugins crash a few times, but that’s probably Flash’s fault, and as in Safari on 10.6, this doesn’t affect the browser or even the rest of the page.)

One thing I’m really looking forward to is extensions. Safari’s a closed system, and I’ve long been envious of the plethora of cool plug-ins available for Firefox. I’m looking forward to using, and maybe developing, extensions for Chrome. (In the current dev channel Mac release, extensions can be installed, but the ones I’ve tried don’t do anything yet.)

I’ve pushed a new update of my Ottoman storage library.

What’s new is the option to have individual additions to the dictionary written directly to disk, instead of being buffered in memory until you save a new version. This helps performance if you’re writing a ton of data all at once. And there’s a simple demo called PackDir that writes a ton of data all at once: it packs the contents of an entire directory tree into an Ottoman file, with keys being filenames and values being file contents.

To implement this I had to violate the lockless semantics a bit, because having multiple clients appending key/value pairs to the file at the same time would make it a lot harder to keep their revisions straight. So before being able to do any direct puts, you have to enter an [ugh] ExclusiveTransaction.

(Of course, the issues with locking don’t matter if the file is only being used by one process. Or even if there are multiple readers but only one writer, since the locks don’t interrupt readers.)

Call the roller of big dice,
The long-haired one, and bid him whip
On kitchen tables consecutive 18’s.
Let the fighters dawdle in such armor
As they are used to wear, and let the mages swap
Delicious spells from last month’s Dragon.
Let a fumble be finale of its caster:
The only emperor is the dungeon master.

Take from the manual of monsters
Painted with three crude beasts, that sheet
On which I enumerated his stats once,
And spread it so as to cover his face.
If his bag remains, rifle his hoard
To see who gets his precious +6 sword.
Light the lamp to run away faster.
The only emperor is the dungeon master.

{ after Wallace Stevens }