NetNewsWire already does this using Google Reader as an intermediary, and Apple’s PubSub framework (which is what Safari and Mail use) shares the read/unread state using MobileMe. But it would be nice to have an open protocol.

I have some experience with this, having implemented the sync system used by PubSub. It’s an interesting problem—you might think I would have just used Apple’s SyncServices, and it’s true that it would have worked great for the subscription list, but it doesn’t scale well to huge numbers of rapidly-changing “read/unread” flags.

I have two suggestions (which I would have made on Brent’s blog, except he doesn’t allow comments anymore.)

CouchDB

CouchDB is an awesome web-centric database engine. It doesn’t use SQL; instead, it’s a glorified key-value store whose values are arbitrary JSON objects, and which uses map-reduce for efficient querying. The basic API is pure REST, though glue libraries for many languages exist.

CouchDB natively supports syncing data through distributed groups of servers. It’s sort of like the way distributed version-control systems like Git or Mercurial work: multiple CouchDB instances each store a replica of the same data set, but can “pull” changes from each other over HTTP to stay in sync.

CouchDB is pretty lightweight and is already being used on the desktop by client apps: GNOME has been integrating it into the Linux desktop to use as a shared store for user data like contacts and bookmarks. It plays a similar role to SyncServices on Mac OS, but it’s all open source and any two instances can sync with each other instead of requiring a proprietary server. I hear this is already shipping in the latest Ubuntu releases.

It doesn’t look as though anyone’s designed a schema for storing RSS subscriptions this way, but it would be pretty easy to define one. You then need a local agent running CouchDB (it can be stripped down to be pretty small), a client library for Cocoa apps, and an upstream CouchDB server to sync to.

REST-Logging

This protocol is similar to what I came up with for PubSub. It’s a simple extension of REST, but I haven’t heard of it being used elsewhere. The idea is that you model an append-only log file as an HTTP resource. The items that are logged are ‘events’ describing changes in the data model, in this case the subscriptions and articles.

The sync algorithm looks like this:

1. Download all the data that’s been added to the remote log file since your last sync. Remember the file’s ETag.
2. Parse that data into a sequence of log entries, and process them in order. Each entry names a model object (feed or article) and an action (subscribe, unsubscribe, mark read, mark unread). Apply those changes to the local data store.
3. Query your local data store to find all the changes that have been made since your last sync. Ignore the remote changes you just applied in the previous step, and also any earlier local changes that duplicate a remote change (like marking the same article as read.)
4. Generate a series of log entries for those changes and concatenate them into a data blob.
5. Upload that blob, appending it to the remote log file. Remember the resulting ETag. In case of a conflict (someone else has changed the remote file since step 1), toss out the blob and return to step 1.

You can think of the log file as a queue or message stream that’s being collaboratively read and written by all of the clients. This sounds like something you’d need a fancy web-app to manage, but it turns out that all it takes is a typical HTTP 1.1 server and a trivial server-side script.

The download is a conditional GET, as used for fetching feeds themselves. The difference is that you use a “Range:” header to request only the bytes past the last known EOF. For example, if the last time you read the log it was 123456 bytes long, you add the header “Range: 123456-” to the request. This ensures that you only get back the new bytes that were added to the end. (And since this is a conditional GET, if the file hasn’t changed at all you just get back an empty 304 response.)

That’s all you need to do to track changes. Since the file is append-only, the only bytes you need to read are the ones added to the end. This request efficiently sends you just those bytes.

What’s cool is that this require no server-side software. If the log is a static file, any regular HTTP server like Apache will automatically handle GET requests for it, even byte-range ones. (Ranges are already used by browsers to resume interrupted downloads.) And it sends the response at high speed, since the server’s just streaming from a file, without multiple back-and-forth requests and without expensive database queries.

How about writing? Ideally you’d use the same approach, with a byte-range PUT that specifies that the request body should go at the end of the file. Unfortunately most servers don’t support this for static files, even though it’s basically just HTTP 1.1. But it’s really easy to implement. Any PHP crufter should be able to whip up a one-page script that simply responds to a POST by reading the request body and appending it to a local file (while doing the necessary ETag and range verification.) The great thing is that this script doesn’t have to know anything at all about RSS or subscriptions or unread counts; it’s completely generic. You can upgrade the data model without having to touch the script, and you could use the same script to sync anything, not just RSS.

(Yes, there is a semi-obvious drawback to this protocol: the file grows without limit. Surprisingly, this is not a problem most of the time, since clients only upload or download new data; the only real limit is the maximum file size or disk quota allowed by the server. But it does present a problem for a new client, whose first-time sync would download the entire file. This can be worked around by having new clients ignore very old data (only download the latest 10MB, say) or by periodically writing a compact subscription list to a separate URL.)

Partly that’s because when I started in July the Mac version of Chrome was too immature; and partly it’s because a web browser is something you need to have running and working all the time—especially since the Chrome project’s bug tracker and code-review tool are web-based.

But Mac Chrome is quite stable enough to use now, and as I haven’t been doing much Chrome development on this MacBook Pro lately (it takes too long to compile compared to my souped-up Mac Pro) I’ve installed the latest dev-channel build and replaced Safari with it in my Dock and as my default browser.

It’s hard to get used to a new browser, after all these years. I remember that I dropped IE 5 like a hot rock as soon as Safari became useable, but that’s because IE sucked so badly on OS X (as you youngsters may not remember.) But Safari is a great browser. Chrome’s great too, but in different ways, and the Mac version’s not finished yet so there are some missing bits.

I should note that I generally don’t work on the user-visible parts of Chrome, rather the underlying WebKit engine; so I haven’t been focusing on the UI much, or noticing the features being added, until experiencing them as an end-user.

In Chrome’s favor:

• It definitely feels faster. At work I’ve been obsessing over some micro-benchmarks, but in regular use what I notice is simply that pages come up more quickly. I think a lot of this has to do with DNS pre-fetching, because DNS query times here at home can often be slow.
• The UI of the tabs is better than Safari’s. I like that they take up less room (though without eliminating the title bar the way the Safari 4 beta did), and the transition animations when opening/closing/detaching tabs are extremely slick, definitely Apple-quality.
• The downloads shelf at the bottom of the window is more visible than Safari’s little Downloads window, which always gets hidden behind the browser.
• The list of recently-closed tabs at the bottom of the new-tab page is very thoughtful and handy. In general I find the new-tab page more useful than Safari’s, if less flashy.

Rough edges (remember, this is a pre-beta build):

• PDF display doesn’t work well. There’s no way to switch pages yet (making multi-page documents unusable), there’s no zooming, the Save command doesn’t work, and often the renderer process starts redrawing constantly and eating up CPU time.
• The only indication you have that a page is loading is the little 16×16-pixel spinning circle that replaces the tab’s icon. I like the visual design of this, but it’s too hard to find. I still miss the big progress bar in the address bar from Safari 1-3. (Cursor feedback would be nice—wasn’t there a browser once that put a little wristwatch badge on the arrow cursor?)
• The bookmarking UI is still unfinished and unpolished. In particular, there’s no way to edit/move/delete bookmarks yet. I love the idea of being able to quickly bookmark by clicking a Google-y star icon, but in practice it pops up a floating panel for choosing a folder, which makes me think about where to put it just like Safari’s bookmark sheet does. I’d rather the star button just added it to an “Unsorted” folder that I could open and organize later.
• Autofill doesn’t seem to work as well as in Safari. For example, it won’t autofill login forms until I type in the username; I think this may be a security feature, but I find it annoying.

I’ve been impressed by Chrome’s stability too, for a pre-beta development build. The app hasn’t crashed once, and I haven’t even gotten the “Oh, snap!” page that shows that a renderer process crashed. (I’ve seen plugins crash a few times, but that’s probably Flash’s fault, and as in Safari on 10.6, this doesn’t affect the browser or even the rest of the page.)

One thing I’m really looking forward to is extensions. Safari’s a closed system, and I’ve long been envious of the plethora of cool plug-ins available for Firefox. I’m looking forward to using, and maybe developing, extensions for Chrome. (In the current dev channel Mac release, extensions can be installed, but the ones I’ve tried don’t do anything yet.)

I decided on Google Sites, which I knew and liked from its days as JotSpot, a hosted wiki with some powerful features. It ended up not being the right place for me, for a couple of reasons:

• Currently, Sites’ priorities are in website publishing, as a replacement for Google Page Creator (which is being phased out soon.) It’s quite good at it, but I’m less interested in that than in collaboration features.
• Google’s server-side infrastructure is really, really, really huge and complex. There is an endless landscape of internal technologies and tools—the few that have been described in public (MapReduce, BigTable, Chubby, etc.) are just the tip of the iceberg. I have discovered that I am not very interested in this kind of stuff, and I quickly became frustrated by the deluge of technologies I needed to learn to get things done.
• Running a large web service is like running a nuclear power plant or an electric power grid. It requires 24/7/365 monitoring, and at that scale, anything that can go wrong will go wrong, and frequently does. I do not have the right temperament for working with this, especially not when it comes to taking turns carrying a pager that wakes me up at 4AM because some service’s latency has gone above 300ms.
• I’ve been vocal about my frustration with centralized systems; Google’s websites are kind of the ultimate in that (even though, ironically, they’re implemented as P2P-like networks internally, as I believe all large web operations are today.)

The good news is that Google encourages transfers between teams, and makes it easy to do so. The near-total transparancy inside the company makes it easy to find out everything that’s going on, and there’s a well-designed website for engineering transfers that helps you find teams that need people. I even went to an internal job fair.

I’ve now ended up working on Chrome, Google’s web browser. The team I’m on is responsible for implementing HTML 5 features, as well as designing and implementing other new features (for standardization) that will help web apps become as powerful as native apps. Much of what we do will go into the WebKit source tree, where it will also directly benefit Safari, Android, the Palm Pre, and other WebKit-based browsers.

In fact, everything I work on (more or less) is going to be open source. Both the WebKit and Chromium source trees are public. You can view, if you care to, the one patch I’ve contributed so far and the one that’s currently out for review. That’s kind of mind-blowing, in a good way, to me, steeped as I am in the secrecy of Apple.

I’m pretty excited by this. There are quite a lot of things I’m interested in working on—client-side storage, local apps, drag-and-drop, better font support, menus, even far-out stuff like peer-to-peer networking. Forward in all directions!

There’s really no such single thing as “Web x“, of course. And all predictions are really just wishes. That being said, my wish is that Web 3.0 will be about distributed systems. To oversimplify:

Web 1.0 built up big brand-name websites with their own content—things written by them, or repurposed from the media companies that owned them, or stuff to buy.

Web 2.0 embraced “user-created content” and interaction between users. The content creation has become less centralized, outsourced to whomever wants to register an account and post stuff, but the sites managing, storing and serving the content are still centralized.

Web 3.0, I hope, will take the decentralization to the software, and the storage. Monolithic web apps run by huge server farms—Facebook, Blogger, Twitter, Flickr, etc.—will be at least in part supplanted by apps that users run locally (or at least ‘nearby’) and which share data among each other.

Why is this important?

• Centralization creates concentrations of power, and that’s dangerous. The people who run the servers have total control over your (and everyone’s) data. They can snoop at it (however private it’s supposed to be), they can sell it to advertisers, they can accidentally lose it, they can accidentally expose it to hackers.

• Centralization leads to walled gardens. Your data on each service is intrinsically disjoint. It can be linked together, through hyperlinks and feeds and APIs and mashups, but only to the degree each service allows, and it’s never seamless.

• Centralized services are hard to run. The more popular they get, the heavier the demands on the servers, and the worse the problems of abuse. (I see this every day in my job, even though the service I work on is one of the smallest Google runs.) This acts as a tax on innovation. Modern frameworks like Rails and Django make it easy to create a site, but to take it beyond just you and your friends, you have to get expensive hosting and deal with server clusters, database replication and so on. The early days of 3rd-party Facebook apps were a great example of this: no sooner would an app come online, than it’d drown under the load of its users and have to be resurrected by panicked owners upgrading their servers to more-expensive hosting plans.

• Centralized services are usually closed-source. I’m not an open-source zealot, and I’ve spent my career working on closed-source software, but I don’t think it’s healthy for [nearly] all large web apps to keep their source code locked away. It discourages innovation and it makes it hard for open alternatives to compete (especially when you consider the huge intrinsic network effects that discourage switching.)

What do we need?

Decentralized systems need well-defined protocols and data formats for communicating. We’ve been making headway with that as part of Web 2.0—there’s an arsenal of technologies like REST, Atom, AtomPub, OpenID, OAuth, RDF, JSON and so on—but they’re not well integrated with each other. And we need higher level abstractions.

I’ve been researching CouchDB this week, and I’m getting more and more excited by it the more I learn. It combines data storage, REST-based APIs, scalability and data propagation through replication, and even application hosting. It’s actually a lot like Google’s internal infrastructure, but in an open and modular form.

You can use CouchDB as the back end of a traditional web service, glomming more and more instances of the server together for scalability; that’s the kind of architecture that Google and Amazon use. But you can also run instances independently from each other, and have them pull data from each other, very much like the way distributed version control systems like Git and Mercurial operate. As I’ve said before, once you have a decentralized system, you can easily design centralized systems of any form as special cases.

Since each CouchDB instance also runs as a web server, that means I can run my social network from my machine, and you can run yours from yours, and yet they can be the same social network. But I can keep my private data private, and I can hack on my software if I want, and the load on my server only scales with the size of my friend list, no matter how big the entire global network grows.

These are things I’ve been thinking of for a while (and my unfinished Cloudy app includes some of them), but CouchDB comes closer than any other software platform I’ve seen to making them implementable. It’s still unfinished (nearing version 0.9 right now), and some of the authentication and replication features that would be needed for this aren’t ready yet, but it really sounds like the people developing CouchDB Get It, and are working to make this vision of Web 3.0 come true.

[If this sounds interesting to you, go and read the preliminary draft of the upcoming O’Reilly book on CouchDB. Only the first few chapters exist yet, but they’re well-written and lay out the basics pretty well.]

My current gold standard for templaters is the Python library Genshi (which was inspired by Kid)—what I like about these is that your templates are valid XML: the parameters and control structures are expressed as special XML attributes and tags. This makes it easy to edit your templates in syntax-checking XML editors, and guarantees that your app serves valid XML or HTML.

But as I said, none of the Ruby template engines I’ve seen work this way. They’re all either generic macro systems that intermix markup and Ruby (like ERB), or they’re ruby APIs that output markup (like Builder, HAML and Markaby). I don’t really like either approach.

If you’ve got a good suggestion, I’ve got a gold star ready to stick on your forehead!

I think the problem isn’t that the browser chrome has too much [UI], it’s that the apps inside have too little.

Too little what? What are the web apps lacking? Since you ask:

A menu bar. Despite today’s fashion to get rid of them, menu bars are really useful. They’re a great interface for discovering the capabilities of an app, and for accessing rarely-used but important features. That’s why they were invented. Denied standard menus, web apps have tended either to splatter buttons all over the screen, or implement their own menus out of DHTML (usually doing a terrible job of even basic usability.) Keyboard shortcuts, if present, are divorced from menus so they’re not discoverable; and on the Mac at least, they awkwardly use a different modifier key (Ctrl, vs. Cmd) than “real” shortcuts.

A filing system. Web apps let you manage content, which often takes the form of documents, aka files (word processing, pictures, etc.), but there’s no standard mechanism to manage these files—the whole document model introduced by the Xerox Star and popularized by the Mac’s Finder doesn’t exist. Instead each such web-app has its own incompatible (and mostly lame) way to list, rename, copy or delete documents. If you’re lucky, it will support some modern conveniences like direct manipulation or a trash. What none of them give you is interoperability: if you want to take a document from one web-app, like Flickr, and manipulate it using a different app, like GMail or Photoshop Express, you can’t drag it across, or even use a standard file picker dialog box. Your only options are to copy and paste a long cryptic URL (and deal with cross-site authentication), or to take the document on a scenic detour through your computer’s real filesystem, downloading it from one app and uploading it to the other.

Offline access. There are in fact still times when your computer doesn’t have access to a network. (Even at über-connected Google, the commuter buses are prone to WiFi outages.) Or the network might just be infuriatingly slow or laggy. Losing access to your data in those situations is bad. Particularly in the case of email, I’ve been spoiled by IMAP, and Mail.app’s excellent offline support, long enough that the idea of using webmail is a complete non-starter for me. (This problem, fortunately, is being addressed, both in the form of Google Gears and in the parallel, but hopefully converging, offline-storage functionality being drafted for HTML 5.)

More than four fonts. The last time I had only four fonts available for documents was in 1985, before the LaserWriter Plus arrived. Currently on my computer I have an embarrassing overabundance, several hundred. Too bad web apps can’t actually discover what fonts are installed; instead they fall back on the deathly dull overused set of “safe” fonts like Times and Arial.

I’m sure there’s more to add to this list, but I’ll stop now. Suffice it to say that in too many ways web-apps are still like the Emperor’s new clothes, or like Samuel Johnson’s dog walking on its hind legs (“it is not done well; but you are surprised to find it done at all”), and the enthusiasm for them often seems to be proportional to the shittiness of the native UI of the enthusiast’s OS platform of choice. (I first noticed this during my brief stint at Sun, whose workstations all ran Motif, which made even Web 1.0 UIs look like a lovely opium dream by comparison.)

Don’t get me wrong, some web apps are great, and they all have a lot of potential, but taking away features from the browser wasn’t high on my list of what they chiefly need.

I’m on the Google Sites team. I’ve been interested in wikis for years, and now I get to actually work on one. (Although Sites, née JotSpot, is not a typical wiki.)

I could write a lot about my experience of Google so far. It’s quite an interesting place. Merely learning about how some of their internal systems operate has been jaw-dropping. (Do you have any idea how much hard disk space Google has? Or how many CPUs? Or how many search queries they handle? Unfortunately I don’t think I’m allowed to tell…)

For now I just wanted to say that I’m not in the job market anymore. Also, that I really like all the free food :-d

Data transaction security in MobileMe’s web apps is based upon authenticated handling of JSON data exchanges between the self contained JavaScript client apps and Apple’s cloud, rather than the SSL web page encryption used by HTTPS. The only real web pages MobileMe exchanges with the server are the HTML, JavaScript, and CSS files that make up the application, which have no need for SSL encryption following the initial user authentication. This has caused some unnecessary panic among web users who have equated their browser’s SSL lock icon with web security. And of course, Internet email is not a secured medium anyway once it leaves your server.

If Apple applied SSL encryption in the browser, it would only slow down every data exchange without really improving security, and instead only provide pundits with a false sense of security that distracts from real security threats.

It’s pretty clear to me from this description that (a) McLean doesn’t know much about data security or HTTP, and (b) the system he’s describing would be patently insecure. And unfortunately, the actual system is just as insecure as I was afraid it was.

The most glaring problem is that, since the main page resources (HTML and JavaScript) aren’t loaded over SSL, there’s no way to tell whether they’re genuine. By now everyone ought to be aware of DNS forgery attacks ; if the coffeeshop where you’ve gone online has an infected WiFi router, it would be nice to know whether its DNS record for “me.com” points to Apple’s servers or to a phishing site. But without SSL there’s no way to tell. Obviously, if you’ve loaded a hacked forgery of me.com’s web-app, any assurances made about “authenticated handling of JSON exchanges” are completely pointless, because your JSON exchanges are probably going straight to a pwned server in Uzbekistan.

(You may object that the initial login page is SSL-protected. It is, but unfortunately it’s at a completely different domain, auth.apple.com. A DNS attack on MobileMe would operate by leaving auth.apple.com alone, so you’d be handed off to the fake server after logging in.)

After writing the above, I got curious enough to dissect the traffic myself. So I used HTTPScoop to sniff while I viewed a couple of emails. Not only is there no SSL authentication, but the requests and responses aren’t even encrypted at all. I could easily read the plaintext of the email messages right in HTTPScoop, which means anyone else on my LAN (or any admin at my ISP or workplace) could easily have read them too. Oops.

The takeaway from this is that MobileMe’s web apps (at least email) are quite insecure — they won’t protect you against DNS forgery or phishing attacks, and they leave your email traffic wide open for others to read. I would avoid them if at all possible and access your MobileMe content via good old SSL in real desktop mail and calendar clients (such as Mail.app and iCal), rather than blindly trusting Apple’s pretty-looking web apps.

(You may raise a second objection, that most email traffic is sent across the Internet unencrypted anyway. But even with unencrypted email, it is much, much easier to eavesdrop on (or alter) traffic over a LAN, especially wireless, than it is to intercept Internet backbone traffic between ISPs. It’s the difference between attacking weak, easily accessible infrastructure plugged in by some baristas with little computer experience, vs. hacking into large commercial routers secured in data centers and run by experts.)

[Disclaimer: I am not a security guru. But I’ve spent quite a bit of time over the past year studying data security and cryptography, and implementing some of the techniques; and honestly, it doesn’t take a Bruce Schneier to simply run a packet sniffer and notice plaintext email traffic.]

beautiful snej soup, yum

I’ve got it aggregating stuff from my del.icio.us, flickr and last.fm accounts, as well as this blog. And I’m directly posting some things I’ve run across today, via its very nice bookmarklet.

Part of the reason I got sucked in is that Soup has the single best new-user experience I’ve ever seen on the web. You just click the “try it” button on the home page, and you get your own soup blog. No signup, no registration, just instant gratification. Then you can slide open the control panel (that slider itself is a beautiful piece of UI), import from your other social sites, and fool with the settings, all in privacy. Only after you’re hooked do you need to press the Create button and choose a username and password, whereupon your soup goes live. It’s brilliant — the web equivalent of the “untitled document” UI introduced in the ‘70s by the Xerox Star.

Anyway, please take a look and join me! (It’s not obvious from the untitled-blog experience, but Soup has friends and groups like other social networks.)

This turns JavaScript into a much more useable language, that I could see doing serious development in. I’m well aware that today’s JS isn’t a “toy”, that it’s got an interesting prototype-based object model under the hood; but I’ve found the language so loose as to be difficult to use. This new version adds a full class model, much better support for iteration, a form of generators/coroutines, and more.

Intriguingly, it also offers type-checking, which has been absent from most mainstream dynamic languages. I know it’s trendy to use “duck typing”, but I find that languages with no compile-time type-checking frustrate me because I make so damn many mistakes that aren’t caught until runtime; which means things I could have fixed in two minutes in C++ or Objective-C take half an hour to fix as I run the program over, and over, and over, discovering and fixing one problem at a time.

Given the concurrent developments in Apple’s JavaScript interpreter, I’m getting pretty excited for JavaScript. I hope that in addition to making it blindingly fast, Apple will also add ECMAScript 4 support, and make it easy to access the language outside of a WebView.