The trouble is, everything you type into Wave is transmitted live, in real time—every keystroke was getting sent to Zach just as I hit it. This made me too self-conscious to get my thoughts across.

… Maybe I should just delete what I’d written and say, “Twitter works because it’s simple.” But I couldn’t do that, because Zach was watching me. He could see me struggling right now—he could see that I’d gotten myself stuck in a textual cul-de-sac and that I was desperately searching for a way out without looking foolish. Now I saw Zach beginning to type: “Don’t let the live-typing get you down!” The game was up; what was the point of making a point now? I ended my thought clumsily and then resolved never to attempt to say anything very deep on Wave.

The same thing happened seven years ago with the live-typing feature that I implemented in iChat 1.0 (which was only supported for Bonjour chats.) I thought it was an awesome idea, and I’d wanted to have it in a chat program since about 1997. But it turned out that, in actual use, people hated it, for exactly the reasons Manjoo describes: it makes you self-conscious. We took it out in the next release.

(Interestingly, I hate video chat for a very similar reason. Somehow, the fact that my picture is being shown in real time to the remote person makes me horrifically self-conscious, even though it wouldn’t bother me at all to talk face-to-face with that person. I don’t know whether it’s the little preview on my screen, or the fact that the person is spookily both present and not-present, but the few times I’ve tried video-chat have been really unpleasant.)

I’m usually on the side of more technology. I believe that our online communications tools are still horribly primitive and have only scratched the surface of what’s possible. But this was a case where more technology was bad.

The low-tech alternative that lots of people use in IM,
is to write in short fragments,
each a separate message,
so the other person can see them one by one
without waiting for you to finish the whole sentence.

The difference is that you’re in control over when to send a partial message, and the other person knows you’re in control, and so on. I still think it might be possible to do this in a higher-tech way, like using a hot-key to send a partial message on demand without having the funky line-breaks, but the current approach isn’t so bad as long as you’re not embarrassed about unintentional free verse.

I could have told the Wave people about what I’d learned, except I didn’t know Wave existed until April (shortly before the public announcement), and even then I was just some guy lost in the crowd at the demos….

Part of the problem, in both cases, is that live typing is one of those Cool Demo Features that looks really awesome when showing off the app. Features like that can be dangerous because they are legitimately very useful during the app’s gestation, when exciting demos are a key survival trait; but then they can’t be removed later on because they’re so well-known, even if they turn out to be useless. Sometimes these features aren’t actually harmful to the user experience, they just make the code more complex and harder to maintain. Instant typing is both, unfortunately. (The clever sync algorithms and rapid-fire network messages Wave uses would be needed even without live typing, but the fact that they have to run on every few keystrokes, not just every minute or so, pushes those things so much harder.)

“Lakitu”?

I’ve since heard about another unrelated project nicknamed Cloudy; and the whole term “cloud” has gotten so debased in the past year that it now stands for outsourcing to giant hidden server farms, which is the antithesis of what I stand for. So I’ve decided to use the name Lakitu instead. Nintendo fans will recognize Lakitu as a bit character in the Mario games—he’s a goggled turtle who rides a little one-seater cloud. This makes him an appropriate mascot for P2P technologies, I think.

[I’m sure Nintendo has a trademark on the character, but they don’t appear to have copyrighted the word “Lakitu”. He’s not even known by that name in Japan, where he’s called “ジュゲム” or “Jugem”. I have been unable to find out what “Lakitu” means or why they decided to use it in the English translation. I could also note threateningly that I have some intellectual-property issues of my own with Nintendo’s depiction of Lakitu’s smiling cloud, which is clearly infringing on my son’s comic-strip character Cloudy. So let’s call it a draw, Iwata-san?]

My last Cloudy post was about verifying people’s identities, and the next one was going to be about gossip. I’ve become unhappy about the rather kludgy way I designed gossip in Cloudy, so yesterday I started designing a new protocol for it, which I’m going to write about.

“Gossip”?

A gossip protocol is a means of broadcasting information in a distributed system. Pairs of computers periodically connect and swap new bits of information with each other; the result is that the information gets dispersed through the whole network (provided it’s a connected graph.) The tricky part is avoiding infinite loops and combinatorial explosions, and optimizing the way pairs of computers swap messages so it scales well.

I started defining a protocol, based on stuff I’ve been thinking about for a while. I don’t think it’s as advanced as what’s reported in research papers, but I’m hoping it will work well enough when used in a socially-driven network—one where the connections between machines are driven by the social connections between their users. Social networks have short horizons, so any particular participant only “sees” a constrained number of near-neighbors even though the entire network may be huge.

I’m making this protocol agnostic as to the type of messaging being used. BLIP will work well, but it ought to be possible to use Jabber or even email; anything that can send messages between two participants. It’s also agnostic as to message content, beyond a few simple assumptions that a message has an author, a timestamp, and some arbitrary “topic” tags.

For example, it ought to work fine at distributing tweet-like micro-blog posts.

Right now I have the protocol written down as an outline in Notebook. I’ll flatten it out, expand it and post it here in a day or two.

Leaving aside the issue of whether Apple has any business deciding what constitutes obscenity (a task that’s driven grown Supreme Court justices to drink)—
And leaving aside also the fact that Apple’s censors have three times now been too dim to comprehend that the application does not contain any books, obscene or otherwise, but downloads them from the Internet much like Safari—
No, the really outrageous issue is that the supposed obscenity here consists of a text-only English translation of the Kama Sutra. Apple specifically called out some pages of steamy advice for “when a man wishes to enlarge his lingam“. (No, really.) [1]

Now, Richard Burton had to get his 1883 translation of this ancient text printed privately when no publishers would accept it, but that was in the Victorian era. The current authoritative translation is nowadays published by that infamous smut peddler, Oxford University Press. Much harder-core fare like Ulysses and Lady Chatterley’s Lover—books which go so far as to use recognizable English names of the naughty bits—were judged after some controversy to be free of obscenity in the 1930s. By 1970 the obscenity statutes had been lifted from nearly all printed material, and nowadays anything goes—take a look at some of the e-books available for sale on the iTunes store.

Montgomerie has now, humiliatingly, been driven to self-censorship: his latest message to Apple states “I have now submitted a new version that specifically blocks access to the Kama Sutra book you identified. Is this what you mean?”

[Update, May 24: Mongomerie reports that on the 23rd he “received a phone call from an Apple representative. He was very complimentary about Eucalyptus.” The whole matter was, of course, resolved, and Eucalyptus is now available for purchase. A happy ending, certainly, and congratulations on the release! ...But I don’t believe it invalidates my argument below. After all, this isn’t the first time an outrageous rejection has been reversed after mass humiliation of Apple. It’s the overall default policies and behaviors, and their chilling effects, that I’m complaining about, and there’s still no sign of those changing.]

The “E” word.

I don’t think anyone but a card-carrying member of the Christian Coalition or Taliban would disagree that this was a stupid decision on Apple’s part. (And it was a considered decision, not a ‘glitch in the approval process’, given that Apple repeated it twice.)

I feel the need to step up to a stronger word. How does evil sound?

Hear me out. I’m not talking “evil” as in killing babies or nuclear blackmail, rather in the sense it’s meant in Google’s corny motto “Don’t Be Evil”, or alluded to in the older proverb “With great power comes great responsibility”. But yes, I do mean “evil” as malign, the opposite of good, etc. etc.

Last year Apple put itself into an ethically very delicate situation with the App Store, by creating a market in which it has the sole power to make 3rd party software available (or to take it away). As has been amply discussed before, iPhone developers have no choice (if they want to be iPhone developers) but to put tremendous effort into developing the product, only finding out at the very end whether or not Apple will let it be sold.[2]

There are definitely some good reasons for such a model, primarily that it helps keep the platform secure from malware, and that by preventing piracy it allows developers to collect a lot more revenues per user, allowing them to set prices far lower than those in other software markets.

But in return Apple had the obligation to be very, very careful to be ethical, upright and transparent in its dealings with developers and the public, to minimize the dangers (of censorship, of conflicts of interest, of stifling innovation) inherent in its position.

And being Apple, it completely and utterly fucked it up. Because it’s in Apple’s genetic code to be about as transparent as a lead brick. This has always annoyed the press, and it has frequently enraged developers, who suffer from the consequences of blank silence from Apple in between carefully-scripted WWDC keynotes and PR-scrubbed announcements. But in the context of the App Store, Apple’s inscrutability and arbitrariness has become actively malign.

Evil is as evil does.

I’m not saying that Apple is evil; it isn’t run by bluestockings or monopolists or cackling supervillains. But evil is a result of what you do [3], and actions are not excused by good intentions; in the real world those who do evil (excepting psychopaths) uniformly believe they’re working for the good.

Apple’s App Store approval process has, over the past year, shown that the company is:

• acting like a Victorian-era book censor;
• quashing competition by blocking apps that improve on Apple’s products;
• blocking innovation by denying 3rd party apps access to the user’s legally-owned data (such as MP3s);
• attempting to deny end-users the freedom to do what they want with the hardware they bought and paid for (viz. its current efforts to have jailbreaking declared illegal);
• and causing undue hardship to small developers by arbitrarily withholding their ability to sell the apps they’ve developed.

Nor has Apple engaged in the slightest bit of dialog with its developers and users to work through any of these issues. The best that’s happened is that, after much public ridicule, Apple has without comment released some apps that it had previously blocked.

Maybe you think “evil” is too strong or melodramatic or exaggerated a word for this. Then feel free to substitute something that has fewer loaded connotations to you—“unethical” or “anticompetitive” or whatever. But if you’re one of those who, like me, has applied the “e” word to the past actions of Microsoft, or to groups that try to ban books from libraries, then I think there’s really no option but to use the same blunt language here and now.

[3] This semantic distinction is one I’m not sure Google gets either. “Don’t Do Evil” would have been a better motto. But in its defense, Google does in practice seem to understand its responsibilities and is admirably open about its actions.

Now, Chrome is open source. (Technically the open source project is a separate thing called “Chromium”, but that’s mostly an organizational detail; the code is the same.) So when I compared this controversy to the rather different attitude of the many programmers who’ve gladly contributed to Chrome and WebKit (and thus also Safari) without pay … I went “hmm”.

There’s a long comment thread about this on the LiveJournal of the talented cartoonist Rebecca Clements. The craziest comment I saw was by one “Anonymous”:

“Good will never paid a bill or put shoes on my child. I admit there should be some caring people in the world. But Google being kind …so we all should be kind like them and give our work away? Come on give me a break. Someone in their executive chain of management has thought this through enough to realize it as a Public Relations “cool” thing to do to get more attention and drive people to Google as a source. It wasn’t because they feel like being kind and giving something away.” *

Maybe I’m especially pissed off at this attitude because I just spent most of my weekend being kind and giving my work away (as well as the work of several others). And I’m sure much of the software these complaining artists use was created for free by open-source programmers. Do they draw with The Gimp or Inkscape? Use Firefox or Chrome or Safari? Run Linux or BSD? Host their blogs with WordPress or LiveJournal? (Let’s just hope they’re not using pirated copies of Photoshop or CorelDraw…)

So is there a double standard here, or am I missing something?

Abstract.

Numerous studies have been conducted upon patients in terminal paresis (GPI), placing the author J.G. Ballard in a series of simulated auto crashes, e.g. multiple pileups, head-on collisions, motorcade attacks (fantasies of Presidential assassinations remained a continuing preoccupation, subjects showing a marked polymorphic fixation on windshields and rear trunk assemblies). Powerful erotic fantasies of an anal-sadistic nature surrounded the image of the award-winning novelist.

J.G. Ballard And The Conceptual Auto-Disaster.

J.G. Ballard died yesterday in his last car-crash. During his life he had rehearsed his death in many crashes, but this was his only true accident. Driven on a collision course towards the royal limousine, his car jumped the rails of the London Airport flyover and plunged through the roof of a bus filled with airline passengers. The crushed bodies of package tourists, like a hæmorrhage of the sun, still lay across the vinyl seats an hour later. Holding the arm of her chauffeur, the Princess Diana, with whom Ballard had dreamed of dying for so many months, stood alone under the revolving ambulance lights, a gloved hand to her throat.

Could she see, in Ballard’s posture, the formula of the death he had devised for her? During the last weeks of his life Ballard had thought of nothing else but her death, a coronation of wounds he had staged with the devotion of an Earl Marshal. The walls of his apartment near the film studios at Shepperton were covered with the photographs he had taken with his zoom lens each morning as she left her hotel in London, from the pedestrian bridges above the westbound motorways, and from the roof of the multi-storey car-park at the studios. The magnified details of her knees and hands, of the inner surface of her thighs and the left apex of her mouth, he matched at his apartment with the photographs of grotesque wounds in a textbook of plastic surgery.

Yesterday his body lay under the police arc-lights at the foot of the flyover, veiled by a delicate lacework of blood. The broken postures of his legs and arms, the bloody geometry of his face, seemed to parody the photographs of crash injuries that covered the walls of his apartment. Twenty yards away, illuminated by the revolving lamps, the princess hovered on the arm of her chauffeur. Ballard had dreamed of dying at the moment of her orgasm.

Before his death Ballard had taken part in many crashes. As I think of Ballard I see him in the stolen cars he drove and damaged, the surfaces of deformed metal and plastic that for ever embraced him.

The Voices Of Time.

You’re not alone, Ballard, don’t think you are. These are the voices of time, and they’re all saying goodbye to you.
Every particle in your body, every grain of sand, every galaxy carries the same signature.
You know what the time is now,
so what does the rest matter?

References.

• Ballard, J.G. “The Voices Of Time” [1962]
• Ballard, J.G. “Why I Want To Fuck Ronald Reagan” [1967]
• Ballard, J.J. The Atrocity Exhibition [1969]
• Ballard, J.G. Crash [1973]

[For the perplexed or appalled: This is a pastiche assembled out of bits of Ballard’s best-known works, with the names changed around. I claim no ownership of these words nor personal identification with their opinions.]

“What the guild is asserting is that authors have a right to a fair share of the value that audio adds to Kindle 2’s version of books.”

And that assertion makes absolutely no sense. The creator of an item does not have a right to impose an arbitrary tax on anyone who adds value to the item. Otherwise we’d be open to all sorts of nonsensical scenarios, like:

It doesn’t help that so much of the music I like is so inward-focused: the guitarist gazing (not at shoes) at effect pedals, the producer sliding waveforms around a timeline, the listener bracketed in headphones like my picture above.

Everyone wants their experience of music to be shared. To play an instrument or sing for others, to blast the song from car speakers. To identify with music meant to shock, and use it to shock others. To attend a concert and know that those around you are hearing and feeling the same thing you are, right then: sitting following the intricacies of Bach, or exploding in a mosh pit. Drugs of many kinds help to collapse this emotional waveform, unite a group into a Bose-ian condensate, whether it’s alcohol to bring out desire and anger, or Ecstasy to turn arpeggios and filter sweeps into spinal shivers and universal love.

Nowadays I experience music by myself, mostly on headphones, like right now on the bus to work. Little circuits inside them are actively removing the signs of the outside world, leaving a weird sibilant hush that I fill with the noise I choose. My eyes are on a screen where I sort my music into playlists, or read what other people say about the music they love, and sometimes reach out and listen to their music myself.

I take little piles of songs and chain them together, finding the little hooks at the ends where they fit, making daisy chains to share with people. Is this creation? I didn’t create the songs, but joining them can make something new, like a work of collage. I would love to use the songs as words given to me, and string them together to tell a story. In some of the mixes I’ve made I can hear some of that: abstract arcs of emotion flying across 80 minutes, or juxtapositions that change the sounds on either side into something new that reflects the other.

The worst part about creation is finishing, and throwing the end product out the door into the sunlight where it blows away in sheets. Diana tells me the abalone are dying out because they reproduce by scattering their seed into the ocean and hoping the eggs and sperm will meet; and as their numbers decrease, so does the likelihood of this success. In some ways the progression of my musical tastes has been like that. The Internet has been a tremendous boon, but the distance involved always makes me doubt, as the notes and bytes fly away out of sight, that they’ll ever find a mate, or if they do, whether I’ll even know.

There’s really no such single thing as “Web x“, of course. And all predictions are really just wishes. That being said, my wish is that Web 3.0 will be about distributed systems. To oversimplify:

Web 1.0 built up big brand-name websites with their own content—things written by them, or repurposed from the media companies that owned them, or stuff to buy.

Web 2.0 embraced “user-created content” and interaction between users. The content creation has become less centralized, outsourced to whomever wants to register an account and post stuff, but the sites managing, storing and serving the content are still centralized.

Web 3.0, I hope, will take the decentralization to the software, and the storage. Monolithic web apps run by huge server farms—Facebook, Blogger, Twitter, Flickr, etc.—will be at least in part supplanted by apps that users run locally (or at least ‘nearby’) and which share data among each other.

Why is this important?

• Centralization creates concentrations of power, and that’s dangerous. The people who run the servers have total control over your (and everyone’s) data. They can snoop at it (however private it’s supposed to be), they can sell it to advertisers, they can accidentally lose it, they can accidentally expose it to hackers.

• Centralization leads to walled gardens. Your data on each service is intrinsically disjoint. It can be linked together, through hyperlinks and feeds and APIs and mashups, but only to the degree each service allows, and it’s never seamless.

• Centralized services are hard to run. The more popular they get, the heavier the demands on the servers, and the worse the problems of abuse. (I see this every day in my job, even though the service I work on is one of the smallest Google runs.) This acts as a tax on innovation. Modern frameworks like Rails and Django make it easy to create a site, but to take it beyond just you and your friends, you have to get expensive hosting and deal with server clusters, database replication and so on. The early days of 3rd-party Facebook apps were a great example of this: no sooner would an app come online, than it’d drown under the load of its users and have to be resurrected by panicked owners upgrading their servers to more-expensive hosting plans.

• Centralized services are usually closed-source. I’m not an open-source zealot, and I’ve spent my career working on closed-source software, but I don’t think it’s healthy for [nearly] all large web apps to keep their source code locked away. It discourages innovation and it makes it hard for open alternatives to compete (especially when you consider the huge intrinsic network effects that discourage switching.)

What do we need?

Decentralized systems need well-defined protocols and data formats for communicating. We’ve been making headway with that as part of Web 2.0—there’s an arsenal of technologies like REST, Atom, AtomPub, OpenID, OAuth, RDF, JSON and so on—but they’re not well integrated with each other. And we need higher level abstractions.

I’ve been researching CouchDB this week, and I’m getting more and more excited by it the more I learn. It combines data storage, REST-based APIs, scalability and data propagation through replication, and even application hosting. It’s actually a lot like Google’s internal infrastructure, but in an open and modular form.

You can use CouchDB as the back end of a traditional web service, glomming more and more instances of the server together for scalability; that’s the kind of architecture that Google and Amazon use. But you can also run instances independently from each other, and have them pull data from each other, very much like the way distributed version control systems like Git and Mercurial operate. As I’ve said before, once you have a decentralized system, you can easily design centralized systems of any form as special cases.

Since each CouchDB instance also runs as a web server, that means I can run my social network from my machine, and you can run yours from yours, and yet they can be the same social network. But I can keep my private data private, and I can hack on my software if I want, and the load on my server only scales with the size of my friend list, no matter how big the entire global network grows.

These are things I’ve been thinking of for a while (and my unfinished Cloudy app includes some of them), but CouchDB comes closer than any other software platform I’ve seen to making them implementable. It’s still unfinished (nearing version 0.9 right now), and some of the authentication and replication features that would be needed for this aren’t ready yet, but it really sounds like the people developing CouchDB Get It, and are working to make this vision of Web 3.0 come true.

[If this sounds interesting to you, go and read the preliminary draft of the upcoming O’Reilly book on CouchDB. Only the first few chapters exist yet, but they’re well-written and lay out the basics pretty well.]

This could clearly be improved a lot by downloading a delta instead, then using that to patch the current copy of the app. In most cases, using a good algorithm like xdelta3 or zdelta, the data transmitted will be orders of magnitude smaller than the entire app. (Nothing new here; many app updaters already do this, especially for games, and Microsoft apparently has some sophisticated delta-based software update tools in Windows.)

Of course, the delta to be downloaded is specific to which version of the program you already have, as well as which one you’re updating to. This means the server will need to keep a number of deltas on hand, and it and the client need to negotiate which one to use.

An additional problem with using deltas for updating Mac applications is that, on OS X, an app isn’t a single file but a directory tree masquerading as a file. This means that a patch would have to consist of a tree of deltas, one per file.

I started toying with ways of implementing this, but a minute later my brain chimed in with the observation “Hey Jens! This is just like what a version-control system does when updating a working tree from a repository!” That’s what I love about my brain: some fraction of the factoids I cram into it daily will percolate inside and pop out later on at some useful moment. Thanks, brain!

Here’s My Idea.

Release your application in the form of a working tree of a distributed version-control system [DVCS] like Mercurial (or Git, if you must.) That is, the app bundle’s “Contents” directory has a “.hg” subdirectory containing the usual Mercurial metadata. This is sort of unusual in that you’ve checked in the compiled app rather than the source code; but modern DVCSs work fine with binary files.

You also maintain on your server a repository containing all the versions of the application. Whenever you release a new version, you simply check it into that repository as an update of the previous release.

Now, when a copy of the app wants to update itself, it simply does an “hg pull” (or equivalent) of itself from the repository URL. This efficiently determines which files have changed and, for each file, which deltas to download an apply. And it does this without affecting the running app, because the deltas go into the metadata in the “.hg” directory. Then when the pull is complete, the app can prompt the user that it’s time to relaunch, then run “hg update” to patch the actual files and quit and re-launch itself.

This has a number of really nice features:

1. Checking whether an update is present is quick (DVCS’s are optimized for this).
2. Downloading updates uses minimal bandwidth because only compressed deltas are sent.
3. The DVCS automatically handles updating from any old version.
4. Users can even downgrade easily to previous versions. (In fact, switching between previously-downloaded versions can be done offline, since the DVCS metadata retains all the necessary diffs!)
5. This mechanism can easily handle beta versions. By treating a beta as a branch, betas can easily be made “opt-in”, with users able to switch between beta and stable mode.
6. If one user has downloaded an update, another nearby user could efficiently apply the update by pulling directly from that user’s copy of the app over the LAN, instead of having to go back to the original server. (Or alternatively, the DVCS can package the update into a patch-file that can be sent to the other computer and applied there.)

The only drawbacks I can see to this are:

1. The DVCS software needs to be available on every user’s machine. Since none of these ship with the OS, it would probably have to be packaged inside the app as part of the software-update library. That’s a significant amount of code (Mercurial is about 2MB, and I think Git is a lot bigger.)
2. The local app’s revision metadata is a full repository that includes prior versions. You clearly don’t want to ship it that way. I believe DVCSs all support a way of pruning old revisions from a repository, though.

I’m not promising to implement this, but it’s so cool that someone needs to give it a try. And if it’s been done already, I’d be interested to hear about how well it worked.

At the root of Cloudy is the means for creating and establishing identity. A lot of peer-to-peer systems treat the peers mostly as interchangeable anonymous nodes, often deliberately so, but Cloudy is a social system.

Quick Crypto Recap.

The identity and security layers of Cloudy are tightly intertwined, because identity without security is useless. And security is accomplished entirely through cryptography, because the centralized alternatives like locking all of your servers up in a closet don’t apply. Cloudy doesn’t do anything new cryptographically (wisely so), but for the benefit of those who aren’t familiar with it, here’s a superficial overview of the off-the-shelf tools I’m using:

Cryptographic Hashes, or, Digests.

Like any hash algorithm, a cryptographic hash converts a block of data of arbitrary length into a short fixed-length output; the same input always produces the same output; and even the slightest change to the input should produce an entirely different output. Unlike a regular hash, two different inputs should never result in the same hash output. (That’s “never” in the practical sense: collisions are mathematically inevitable, but it should impractically long, ideally millions of years, to find one.) And it should be infeasible to identify anything about the original data given only the hash.

Cryptographic hashes are rather weird and neat. I’ve previously called them “the Dewey Decimal numbers for the Universal Library”. They also remind me of the scene in the old TV cartoon of “The Cat In The Hat”, where the Cat and kids are running around labeling every object in the house with cryptic identifiers like “QW-X12”. Digests are a bit longer than that (SHA-1 outputs 160 bits, i.e. 20 bytes) but it’s still very handy to have a compact label to identify any conceivable chunk of data.

Public/Private Key Pairs, or, Asymmetric Keys

A regular cipher uses what’s called a symmetric key. The sender and receiver choose a single key that they both have to know, but keep secret. The sender inserts the key into the encryption algorithm and feeds the message in, and out comes the encrypted form. The receiver then inserts the same key into the decryption algorithm, feeds the encrypted data into it, and out comes the original message. The point is that they use a single key, and the one who generates the key has to somehow convey it secretly to the other party before they can use it, leading to an obvious chicken-and-egg problem.

Asymmetric encryption algorithms, the best known of which is RSA, use two keys. The keys are generated together in a matched pair. When one key is used by the sender to encrypt data, it takes the other key of the pair to decrypt it.

The genius of this is that you only have to keep one of the keys secret. The other one can be given away freely and is called the “public key”. If someone else has a copy of your public key, s/he can use your public key to encrypt a message to you. Remember, it doesn’t matter how people get your key; you can read it to them over the phone, email it, print it on a billboard. The encrypted message still can’t be read by anyone else, because only you have the matching private key to decrypt it. In other words, it becomes possible to send secure messages without having to share a secret key in advance.

Digital Signatures.

There’s another use for key-pairs. You can use your private key to generate a signature of a message, a small block of data that can be attached to it. Anyone who has your public key can then use it to verify the signature, i.e. prove that you generated the signature of that message with your private key. No one else could have generated the signature. In other words, as the name “signature” implies, this is a way to unforgeably mark a document to show your authorship or approval.

(A digital signature is really just a cryptographic hash of the document, which is then encrypted with the private key. To verify someone’s signature you just decrypt it using their public key, then compute your own hash of the document and compare the two results.)

Digital Certificates.

A certificate is, in general, just a signed document that attests to something. Usually it’s vouching for the identity of the owner of another key; something like “the owner of the public key 3FD8B640 is Joe-Bob Briggs, of Dallas TX, joebob@example.com. Signed, Verisign.com.” This is the standard way that trust gets spread around in a distributed system.

Back To Cloudy: Generating An Identity.

Your Cloudy identity is simply a public key, currently 2048-bit RSA, generated the first time you launch the program. (The matching private key is stored securely in the Mac OS Keychain.) From then on, that public key uniquely identifies you. It’s unique because it’s randomly picked from a space so large that the possibility of collisions is for all practical purposes zero. It identifies you because it can be used by others to verify anything you signed with the matching private key.

[2048 bits (256 bytes) is somewhat bulky to use as an identifier; so a public key can be run through an SHA-1 hash and converted into a 160-bit (20-byte) digest form that’s, for all intents and purposes, equally unique. (2¹⁶⁰ is about 10⁴⁸, nearly the number of atoms in the Earth.) The digest, however, has no cryptographic value to a recipient who doesn’t already have the public key, so it’s not a secure identifier by itself.]

The first thing the new key is used for is to mint an SSL certificate, which will be used for identification when you communicate with other peers over SSL sockets. It’s a “self-signed” certificate because it doesn’t contain a signature from any trusted higher authority (there aren’t any). But that’s OK: when Cloudy peers connect, they only need to make sure of the identities they’re contacting, which are literally just the public keys in the certificates.

Cloudy Certificates.

SSL unfortunately uses an awkward and complex certificate format called X.509, which is one of two evolutionary relics left over from a long-dead overly-ambitious network architecture called X.500. (The other one is the LDAP directory protocol; the fact that the L stands for “Lightweight” gives you an idea of how comparatively elephantine X.500 must have been.) Most cryptographic experts seem to hate X.509: Ferguson and Schneier’s Practical Cryptography flatly recommends avoiding it if possible, and Peter Gutmann’s overview is a masterful takedown, with sarcasm worthy of Doug Piranha.

After spending a week painfully figuring out how to generate a goddamn trivial self-signed cert, even with the help of state-of-the-art system APIs, I could understand what the experts meant. I didn’t want to use X.509 anymore. And it wasn’t flexible enough anyway, since it was designed around the idea of hierarchical authorities. Unfortunately I didn’t have a choice for SSL, but I went with an alternate approach for all the other certs Cloudy peer use when talking to each other.

I really liked the approach taken by SDSI, a distributed identity system from about 10 years ago that never took off. It defined a simple textual syntax for certificates. SDSI used LISP-like S-expressions as the syntax, but the details aren’t important—I took the abstract concepts and went with something I found more readable. I tried JSON first, but found it too limiting, so I ended up using YAML.

[YAML is a data serialization syntax; it’s language-agnostic, but most popular in the Ruby community. Its main advantages over JSON (or OS X property lists) are a richer set of data types, custom typing for collections (i.e. you can say “this array is a Rectangle” or “this dictionary is a Person”), and the ability to represent arbitrary object graphs, not just trees. You can think of it as being like a pretty syntax for Cocoa object archives or Java object serialization.]

All I had to do (aside from writing a good Cocoa wrapper API for YAML) was define a schema for representing things like keys and signatures in YAML. Then I could use those to define my own signed certificate objects.

A YAML Certificate Example.

.  --- !cloudy/CallingCard
.  host: 76.191.199.123
.  prof: 229474364
.  port: 60507
.  stat: 4
.  signature: !cloudy/Signature
.    signed: !binary |-
.      oVCuVVlXPEdRPR+gy1k/UNOXtwvcN7LNpK6xTcA/hmlKh6uIT56E19LxWzA7POxm
.      nhc351NVdoKC9XaUVsaZYDOnp2wWEWLUtdYYA8I++NZZIVlCHOjHCHr7mcfNcceD
.      v+15RE9vguQ/PO1yaOU4DlviYt75y7xKMRs5REbZss6E/mr+0r1KE+f73dpHCVoD
.      SW0azTD43pug2Pyh2Kar0GHXQcS4Iq/Y2nRFv7wyLUUmyVA7XI665a8QjMCiec2w
.      0PqQ32FwGBYkH/iR/cfmaKjuwjAbW/qo7NoTH6WSFQy2ua/PVQs9B+dyjnZ5Z30E
.      rnl9UTCVwjUmCc8J4hoaTQ==
.    digest: !cloudy/Digest pFCzUK7yuO0dWtm0oATB7ag6vj0=
.    date: 2008-04-15 21:55:46.830 -07:00
.    expires: 21600
.    signer: !cloudy/Person
.      nickname: snej
.      publicKey: !cloudy/PublicKey
.        algorithm: 42
.        format: 1
.        bits: 2048
.        data: !binary |-
.          MIIBCgKCAQEApP6/D5aZm7nYfGwSMD3xQCCWw+XeU1NmZE7N/7eHvQlCUHMS8Aac
.          Wh+s/PlPd1o7k+YePhoHnc1vR9uAfWm8iowiUU0RluUNxY0dRkTauRqeYM6//s+5
.          ZXuh27pDDq2BgQYPL6EOp2UtWSQ/ojQjqX2/sGMkZ3k+uYiu1ZGQS2s0xTHPkgtu
.          VI+Kg2TBY/28zAG4H/seUHNAP+frlpX+fizSC2oYNdREpEcVcVacHMQGwrj3mAr7
.          g/LpJTnWgZhiJYvp7c4MkAYfHOIbKIXeXrF8oOz0EwgwSp0ZWkezuIYa4BMAns52
.          WYK3LooQ+GttPIdVhSzzhLlY3psLeOf6nQIDAQAB

This represents a “CallingCard” object, which a peer broadcasts in order to tell other peers that it’s online, and where it is on the network and what it’s current state is. (One of the places a CallingCard appears is in the TXT record of Cloudy’s Bonjour service.)

The syntax is more complex than JSON, but still pretty easy to read:

• The first line says this is a dictionary structure whose higher-level type is “cloudy/CallingCard” (this gets mapped to the CallingCard class in my code.)
• The next four lines describe four key/value pairs in the dictionary. In a CallingCard these represent the IP address, port number, timestamp of the user’s current “profile”, and online status (4 = “Available”). The keys are four letters long just to save some room and because I get nostalgic for OSTypes sometimes.
• Line 6 assigns the “signature” key to a nested dictionary of type “cloudy/Signature”. This dictionary is the digital signature of the enclosing object.
• The “signed” attribute of the signature is the raw RSA signature data.
• The “digest” attribute is the SHA-1 digest of the object being signed, in this case the enclosing CallingCard, ignoring the “signature” attribute.
• The “date” attribute is the timestamp of the moment the signature was generated.
• The “expires” attribute is the lifetime of the signature, in seconds, starting from the “date”. After this interval the signature expires, and the signed object loses its validity and will generally be deleted, or at least not passed on to other peers anymore.
• The “signer” attribute is a cloudy/Person object, the identity who generated the signature.
• “nickname” is a brief human-readable name for this identity. It doesn’t really mean anything; it’s just useful as a default name to display (like an AIM handle in a buddy list) if the local user hasn’t set up a customized name.
• “publicKey” is the identity’s public key, the actual unique identifier.
• “algorithm” identifies the type of key (RSA), “format” identifies the format of the key data (PEM, I think), “bits” is the number of bits in the key, and “data” is the key data itself.

This does look a bit verbose when written out, but of course usually you’d never see this unless you were debugging something. (And it compresses well, by about 50% using gzip.) One space-saving feature that doesn’t show up here is that, if the same object appears more than once, it’s only written out once; after that it appears as a short reference back to the definition. So if I YAML-encode an array of signed objects (which is very common), my cloudy/Person data only appears once.

A Nasty Detail: Canonical Form

I glossed over an important detail: to sign an object you have to compute a digest of it, and to compute a digest you have to be able to express the object as raw data. Clearly the raw data in this case is the YAML encoding of the object, right?

The catch is that in YAML, as with any human-readable syntax, there are many different ways to write out the same object. I can change the indentation, I can change the line breaks, I can list dictionary attributes in a different order. Any of those changes causes the resulting digest to look completely different.

This is a real problem because, if I read a signed object from YAML into a native object and then write it back out to YAML, it’s likely to come out slightly differently. For example, if I write it as part of an array, then as a nested element its lines will be indented. Also, the ways dictionaries are stored in hashtables mean that their keys come out in unpredictable orders when iterated. But if that happens, the digest changes, which invalidates the signature.

So any certificate syntax has to define a single standard (“canonical”) encoding of an object into binary data. In my YAML code I had to enable a “canonical mode” that, when turned on, causes a specific set of spacing rules to be used, dictionary and set entries to be written in alphabetical order, et cetera. This mode isn’t normally used, but it has to be turned on when computing the digest of an object, in order to sign it or to verify a signature.

[Incidentally, one of the reasons that digital signatures aren’t being used much in the various trendy XML-based data formats, like RSS and Atom, is that XML is much more difficult to canonicalize. I don’t understand all of the details, but they looked nasty enough that I was glad enough to rule out using XML.]

Verifying A Signed Object.

When you verify the signature of a block of YAML like the above, you have to do this:

1. Parse the YAML into a graph of native objects.
2. Take the root Signed object, remove the “signature” attribute, and write it back into YAML in “canonical mode”.
3. Compute the digest of that canonical YAML.
4. Compare the digest with the “digest” attribute of the Signature. If it doesn’t match, the object’s been tampered with (or damaged) and should be ignored.
5. Otherwise, write the “Signature” object back into canonical YAML and compute the digest.
6. Encrypt that digest using the public key in the “signer.publicKey” attribute.
7. Compare the result with the “signed” attribute. If it doesn’t match, the signature was forged (or damaged.)
8. Otherwise, the signature is valid and the outer Signed object can be treated as being definitively created by the Person listed in the Signature.

Whew.

OK, so we can create secure identities, encrypt stuff, and sign arbitrary objects. Now what do we do with them? The CallingCard example above should give you some ideas, but I’ll go into more detail in the next ‘thrilling’ installment.

Next: Networking.