I finally made myself a Facebook account, mostly to see what it’s like. Overall, I’m pretty impressed: the UI is nicer than most such sites, particularly the still-antiquated LiveJournal and the disaster that is MySpace. The biggest issue there seems to be that the main profile page absolutely doesn’t scale up to handle the exploding number of apps/widgets people are stuffing into it, so you end up with mile-long profiles containing box after box of junk.
But the most interesting thing I noticed is how the service has no visible identifiers for user identities. Unlike most centralized services, there’s no unique username to pick. I assume that, internally, each account requires a unique email address, but that address plays very little role in the user experience, apart from its use in helping people find their existing contacts’ profiles. The service does assign a unique number to every profile, and this shows up in profiles’ URLs, but it never seems to appear in the page itself. So there’s no obvious way to say “this is my Facebook ID”, other than pasting in the completely non-mnemonic URL of your profile page. And conversely, the visible identifiers you see for other members are simply their real names (plus photos/icons.)
This makes sense, since Facebook is all about the social network. It may have eight hundred million members (assuming the account IDs are assigned serially) but there’s generally no need to know about, identify or refer to some random member, since it’s almost certainly someone you don’t know or care about. The people in your social network can be referred to by real name, because at that scale the number of name conflicts is so much lower, and you or your friends presumably ruled out spoofs before friending that person. Real names work for the same reason that they work in real life social interactions.
This avoids a lot of problems. Unique usernames, while deliciously old-school-geeky, don’t scale well to the modern Internet, as has been apparent since the rise of AOL 15 years ago. A username like “snej” or “JohnWilson” has great mnemonic value, but after a while everything’s already been taken but “fuzzi_bunney_37327” or “JohnWilson9284”, which are far less useful.
Using email addresses as IDs adds more room with a second level of namespace, and removes the need for people to make up new IDs for every site, but in practice it exposes the existing flat-namespace problem of the big ISPs and portals. “fuzzi_bunney_373@gmail.com” isn’t much of an improvement. There’s also the privacy and spam problem of exposing those emails to others.
It turns out there really isn’t any good solution for universal unique personal identifiers. This has been one of the factors blocking the utopian goal of a global Public Key Infrastructure (PKI): to be useful, a certificate has to assure you of its owner’s identity, which means it has to contain some signed identifier that you can associate with a real person (or persona). But, as argued convincingly by Carl Ellison in his paper Improvements on Conventional PKI Wisdom, there isn’t any such identifier that always works. Real names don’t work because they’re not unique enough, not globally or even in any large organization. Ellison calls this the “John Wilson problem”, named after eight Intel employees with that name, who kept getting misdirected email. (I think of it as the “Steve Smith” problem, after an old co-worker of mine at Apple.) Adding middle initials doesn’t help, of course, because most people don’t know their friends’ or co-workers’ middle initials. Ellison concludes:
“Human beings do not use names the way we want them to. … Computer developers … think of names the way we do variable names or path names. That is, a name is some string, unique within its block or directory or context, that unambiguously identifies some object. … Compilers and operating systems may behave this way, but human users do not.”
Of course, the certificate can include more forms of identification. A photo would help a lot, or email address. So would the name of your employer and (for co-workers) what department you work in. So would your hometown and phone number. Or your SSN or driver’s license number. But no one form of ID is sufficient for everyone: I don’t know most of my co-workers’ home phone numbers. I have personal friends whose current employment I’m not sure of. I have online friends who are nonetheless pseudonymous enough that I don’t know what they look like.
And you can’t just put in all (or even most) of those forms of ID, or your certificate becomes a privacy nightmare, a dossier worthy of a police state. The conclusion? There’s really no good way to prove identity using a self-contained certificate. QED.
This quandary has been expressed as Zooko’s Triangle, which in textual form says: No single form of identification can be simultaneously globally unique, decentralized, and human-meaningful. It can only have two of those attributes.
The best approach to identification seems to be to give up on having the identifier try to prove its own relationship to a “principal” (person or entity). Instead, the relationships are given as external statements made by other principals. The identifier itself can then just be a random unique number (such as a public key, which is easy to generate) with no intrinsic mnemonic value.
So for example, if I get a message from F837CA77B6, I have no idea who that is. But if I already know Karen, and Karen makes a signed assertion that F837CA77B6 is her new husband Michael, then I can trust that the message is from Michael. I can add him to my address book as “Michael Jones”, and never have to see that random number again. It doesn’t matter if there are thousands of other Michael Joneses in the world, because I only know one. Even if some of my friends know another Michael Jones, I just have to refer to him as “Michael Jones, my sister Karen’s husband” to disambiguate.
In a nutshell, this approach uses the social network to manage identity, by reducing the size of the problem space by about seven orders of magnitude. It’s perfectly feasible to keep track of the identity of a few hundred people using familiar attributes like names, faces and personal relationships: humans have been doing it for literally millions hundreds of thousands of years. Evolutionary biologists argue that managing social relationships in tribes was a key factor in the development of human language: in other words, gossip was the killer app for the neocortex.
Steve Dohrmann and Carl Ellison’s 2002 paper Public-key Support for Collaborative Groups describes a prototype system that uses such a distributed identity system to implement a peer-to-peer collaborative space that provides access control and trust without needing any centralized PKI.
Marc Stiegler, in his 2005 essay An Introduction To Petname Systems, has coined the term “petname” for such a private mnemonic name each person locally assigns to their contacts; for example, the Michael above might be “Michael” to me, while I’d refer to my friend Michael as “Mikey”. Petnames are (locally) unique and memorable, and make it easy to work with identifiers that are intrinsically just long strings of digits. The use of petnames and raw identifiers together gets around the limitation of Zooko’s Triangle.
Bryan Ford et al, in Persistent Personal Names for Globally Connected Mobile Devices (2007) describe an ongoing project called Unmanaged Internet Architecture that provides a more general naming, discovery and communications protocol for people’s multiple networked devices and those of their friends and colleagues; it’s a bit like Apple’s Bonjour, but it follows your social network, not Ethernet.
Facebook, of course, is centralized. (Well, the add-on apps aren’t, but even they get proxied through facebook.com.) But since a central aspect of its data model — user identities — works in much the same manner as these distributed systems, I believe that implies that a very Facebook-like social network could be built as a distributed architecture that didn’t rely on a central server or organization. It might not even look that different; you’d just notice (or not) that, as you surfed from one friend’s profile to another, the domain name in the address bar changed.
So what? Why does this matter if it doesn’t change the user experience, since existing monolithic sites can already do this, without any need for fancy new protocols?
At this point the elite members of the “blogosphere” [sorry] will be nodding smugly. You don’t need these monolithic services with all their problems if you just set up your own blog on your own website. All you need is a little bit of skill with FTP, and MySQL, and PHP, and HTML, and CSS, and that’s it! You’re free! No more being hassled by The Man.
Standalone blogs just don’t have the social features, though — they’re too standalone. I’ve had this blog for years, and it feels like a soapbox in a big city, where I can occasionally rant about something. I’ve also had accounts on one or two social-network sites for years, and those feel very different: they’re deeply personal, and the presence of friends (and their friends) is always very apparent. They’re like town squares, or big cocktail parties, where I can talk with a few friends or listen in on the buzz going on nearby.
The standalone blogs just don’t have enough protocols to enable the kind of rich interpersonal interaction that centralized social-network sites have. The venerable protocols like RSS/Atom, ping, and trackback are the kind of stuff described as the simplest thing that could possibly work (if you’re charitable) or “something scrawled on the back of a napkin in crayon” (if you’re not.) More recently, OpenID is a big step forward in making identities transportable between sites. Google’s OpenSocial might help; I confess I haven’t yet examined it closely enough to tell if it’s useful for more than just creating yet more Facebook-style widgets.
In any case, there’s more work to be done to bring social networking into the decentralized blogging environment. I think we may be at a tipping point now — people seem to be increasingly aware of the problems that centralization brings, and in their aftermath I keep running across discussions of how it should be possible to do this stuff without pesky corporate overlords messing it up. So there may be enough demand, now, to balance out the intrinsic difficulty of building open standards. I’m hopeful!
It’s a sure sign that wikis are going mainstream when one appears for a video-game console. “ZackAndWiki” has the requisite goofy name (like TikiWiki or WikkaWiki), but once you try it out, you’ll find it approaches its job very differently than you’re probably expecting. Read the rest of this entry »
It was a great relief for Leopard to finally be finished, after more than two years of work. (And if you wonder why it took so long, consider some of the new products that have been released since Tiger shipped in May 2005: the Intel Macs, the Apple TV and the iPhone / iPod Touch. All of these contain system software that absorbed the attentions of significant subsets of the people who work on on OS X.)
And now, a few weeks later, it’s hit the streets. On Tiger Day in 2005 I helped out a bit at the Apple store in Santa Clara; that was fun, but tonight I stayed home because I’m recovering from a bad cold. Still, in between coughing fits, I can ring in the new OS by pointing out yet another little improvement, one that didn’t make it into the official Top 300 list.
You can now choose to leave new articles marked as “unread” until you explicitly mark them as read by clicking on them. This is more like other news-readers, and it’s good if you want to skim through bucketloads of new articles and read a few of them, but still know which ones you skipped over so you can get to them later.
To turn on this mode, go to the “RSS” pane in Safari’s preferences, and look for the pop-up menu labeled “Mark Articles As Read:”. Change the value to “After clicking them.”
You’ll also want to make sure “Highlight unread articles” is checked, so that you can tell the read and unread articles apart. By the way, this highlighting has been improved, too. Instead of just changing the headline text to orange, Safari now draws a pale blue background behind the article, and adds the same blue bullet that Mail uses for unread messages. Click anywhere in an unread article to mark it as read. (There’s also a “Mark All As Read” item under “Actions” in the sidebar.)
If you really want to speed-read through articles in a hurry, drag the article-length slider (in the sidebar at the top right) all the way to the left. In this view, the articles now display in a very compact single-line list view, with just room for the first bit of the text in between the title and the date. You can still click a headline to jump to its web page, or move the slider back to the right to expand the articles. (Clicking the blue bullet at the left marks an article as read without viewing it.)
Nothing earth-shaking, I know, but I’m happy about these tweaks because they bring the Safari RSS experience more into line with the way we first prototyped it in 2004. Most of what my team-mates and I worked on for Leopard is hidden away behind the scenes (the new “PubSub” framework that supports Mail RSS as well as Safari) but it’s nice to have contributed to a bit of more visible functionality too.
(Speaking of the PubSub framework, it has a public API that makes it extremely easy to parse and subscribe to RSS or Atom news feeds. There are some sample applications using it in /Developer/Applications/PubSub, and I’ll also try to post an overview here, now that I can finally talk about these things publicly.)
There’s an interesting kerfluffle going on regarding the scaling woes that Twitter.com is going through, especially since it’s built on Ruby On Rails. Here’s the original interview with one of the Twitter coders, the somewhat evasive reply by the lead Rails architect, and Mark Pilgrim’s cruel-but-funny dissection of the latter.
Ruby is a lovely language and Rails is a lovely framework, but both of them trade performance for aesthetics and convenience. That is, they’re slow. No problem, though — in the magical world of web apps, you just solve performance problems by throwing hardware at the problem, usually in the form of more and more web servers. (If I sound sarcastic, I’m just envious because I write client-side software, where we don’t have that convenient fallback.)
Then, since 99.99% of web apps use a SQL database, the next bottleneck is the database server that all the web servers are talking to. The next step of course is to buy more database servers, but then you have to distribute/replicate the database across those servers, which is even more challenging. (Or if you’re a crazed wunderkind like LiveJournal founder Brad Fitzpatrick, you invent a memory-based distributed hashtable as a cache to put in front of the database.)
This whole thing has me wondering why it is that SQL databases are used as the all-purpose hammer for solving all data storage problems for web apps. To some degree this has to be a historical accident, because I don’t think it’s necessarily obvious. My guess is that (a) a lot of companies already used these databases for storing their, uh, data; (b) in the mid-’90s all these companies were desperate to Get On The Web; so (c ) countless web developers wrote “three-tier” frameworks for gluing those SQL databases onto web servers.
I’ve been using SQL a lot (via sqlite, as a data store for the Syndication framework) and it’s quite nice. But I don’t think SQL — or any sort of query-based database system — is the right tool for every job, in particular these kinds of social-messaging apps like Twitter.
The figure that stands out to me is “…up to 11,000 requests per second”. Jesus Christ, that’s a lot. Where does that come from? Even assuming a million members who each post ten times a day, that’s only about 100 posts per second. If each member runs a client app polling for updates every 15 minutes, 24 hours a day, that’s still only about 1,000 hits/sec. There’s still an order of magnitude unaccounted for. (Maybe the apps poll every 1.5 minutes?)
But think about the polling again. Each of these requests just boils down to the client asking “what messages did I get since the last time I asked?” The SQL way of answering that question is to run a complicated join that looks up all of the members I’m subscribed to, looks up all the messages of those members, and selects the ones whose timestamp is later than the date of the client’s last request. Sure, there are indexes, and databases are optimized for this stuff, but it’s still going on a thousand times a second.
Now, a more sane way to do this (if I may be so bold) is to keep a message queue for every member, and whenever someone posts a new message, copy a pointer to that message into the queue of every member who’s subscribed. Then when someone polls, you just have to get the messages out of their queue and send them along. If that sounds kind of familiar, it’s because that’s exactly how email works — the Twitter posting interface is like a listserv, and the polling interface is like a POP server.
The cool thing is that you don’t even need a database to do this. You can store all of this stuff using very simple file formats, with one file per message queue. Since this is so much simpler than triple-decker SQL joins, one server can handle more requests; and if that’s too much, you can use a networked filesystem (or even a SAN) to distribute it across servers.
And of course you could keep going with this novel “distributed” idea, and make this into an actually-distributed system, where individual users (or groups of them) can run their own Twitter servers that queue their incoming messages and relay their outgoing ones. Imagine!
The trouble is, the idea of centralizing is so entrenched, and there’s so much industry support for “scalability” by means of piling on more servers and moving to bigger data-centers with fatter pipes, that I can see that this is going to go on for some time. (I’ve heard that AIM handles over a billion IMs a day, every one of which streams in and out of an unfathomably fat pipe in a huge AOL data center in Virginia. Shudder.)
In some moods (the same sorts of moods where I fantasize about how $5-per-gallon gas would kick-start desperately needed changes in our transportation system) I think about what would happen if all the DNS servers mysteriously vanished, all the data centers blew up, and we had to re-implement everything in a distributed peer-to-peer fashion. It’s actually really interesting …
There’s an interesting online competition going on called My Dream App. The idea is that a bunch of people pitch their ideas for a Mac application, and the set of ideas gets winnowed down in several rounds of public voting, until one is left. A group of experienced developers have promised to implement the winning idea as a shareware app, whatever it turns out to be.
It’s a fun concept, but it highlights some of the problems of having end users design software. A number of the proposals give me a particular sinking feeling I associate with user-interface design meetings: lots of ideas that sound super-cool as one-sentence pitches, accompanied by irresistibly glitzy faked-up screen shots (all replete with translucency, rounded corners, and this year’s de rigeur reflections). But too often there’s no “there” there. It’s all so vague that I can sense that these people haven’t thought through the difficult bits or worked out a coherent idea of what the app will do. Read the rest of this entry »
I always find this “blog” thing weird because, unlike more-centralized social network services, there’s not an inbuilt sense of community. This is simply a black box into which I type words, and they pop out on a page, and someone might or might not be paying attention. It makes me feel like I’m Helen Keller standing on a soapbox.
I know there are services that help you find out who’s writing about you [_centralized_ services … so much for blogging as a distributed platform.] So I’m getting myself all signed up with those, in anticipation of nervously re-checking my blog rank every 15 minutes.
But it’s not the same as comments. Comments are the spare change of the attention economy. There is no one so rich they don’t get a thrill out of finding a quarter on the street. To continue the strained analogy above, it’s like I’m holding a hat in my hand as I lecture/declaim/narrate/confess from my soapbox, and I feel the little clinks of the coins falling in and it’s like they’re spelling I hear you, you’re interesting into my palm.
Therefore, I am asking of you reading this that you please comment on this post. Just a quick “Hi, I read your blog”, like a wave or passing-in-the-hallway smile. Thanks.
PS: For the occasion, I’ve enabled OpenID authentication for comments. If you have a LiveJournal or TypePad account, or another form of OpenID, you can just enter it in the comment form.
I’d just begun to muse about signing Atom/RSS articles, when Johannes Ernst began blogging about the topic. I had assumed there must be some easy standard way to do it; but the answer turns out to be that there is a standard, but (according to Johannes) it’s far from easy, so much so that it’s nearly unuseable.
(The problem in a nutshell: Digital signatures operate on raw data, so to sign something you have to be able to convert it to a sequence of bytes to stream through the signature algorithm. Crucially, to verify the signature you have to be able to convert the something you received into the exact same sequence of bytes. That’s no problem for JPEGs or HTTP bodies. But XML describes an abstract tree of nodes and attributes, with many possible text representations for the same data. If you parse some XML and then turn those data structures back into XML, the text will probably not be exactly the same. Specifying a canonical way to textualize an XML document turns out to be really hard since it has to take into account namespaces, entities, whitespace, character encodings and more. Yeesh!)
The more I think about this the more worried I get. We are increasingly using XML-based formats for communication—Atom, RSS, Jabber. These formats contain multiple messages in the same document. Distributing these messages may involve copying them from one document into another: for example, when news feeds are aggregated or articles forwarded, and whenever a Jabber message is routed through a server. If we care about the integrity and identifiability of these messages—and the lesson from the current death-throes of email is that we damn well should—we need to sign them, and the original signatures of course need to travel with the messages. But when a XML message/article/entry element is copied from one place to another, its physical manifestation as a byte-sequence will typically change … leading to the XML-signatures quandary.
Johannes’s suggested XML-RSig (“Really Simple Signature”) solution is to avoid transforming the message’s byte sequence. The bytes to sign are the encoded characters from the opening “< " to the closing ">“, and the new sub-element containing the signature is spliced in by character insertion. Anyone copying the message to another XML document has to use an X-Acto knife to cut out the exact message text and insert it into the destination, rather than allowing it to be transformed in any way by an XML processor. (In fact, the destination document even needs to have the same character encoding.)
I have mixed emotions about this. On the one hand, it certainly is clear and simple. (See Johannes’s list of benefits at the bottom of the post.) What bothers me:
I don’t have any good suggestions at this point. I’m writing this as a brain-dump. I’m posting it here because (a) Johanness’s blog doesn’t allow comments, and (b) it seems to be the Blog Way to reply to other people’s posts on your own front page, even if it’ll baffle the rest of your readers…
Much of what I’m consumed with (at work) boils down to a question of: what is the right shape for the small but plentiful bits of writing that we are all creating daily? Here shape means largely visual representation but also sequencing and topology.
It’s a problem of hypertext, primarily. The World Wide Web established one shape for hypertext: individual pages with one-way links in the text, replacing one another in a back-forwards chain. It’s proven to be a pretty good shape, but it’s not the only one, and earlier thinkers like Engelbart and Nelson had lots of other ideas. Read the rest of this entry »
The Simple Mail Transfer Protocol is by no means perfect—its lack of authentication is a prime reason why spam is such a problem—but I think it got one thing right: it has the right topology for building a person-to-person communications system. Read the rest of this entry »
