I gave a webcast last month for O’Reilly, and it’s up on YouTube now so you can watch it at your leisure. Here’s the abstract:

It can be very liberating to store your Couchbase app’s data as free-form dictionaries, free of the rigid schema of a relational database. But there’s a lot to be said for the syntactic comforts and conveniences of modeling your data as native Cocoa objects with predeclared @properties, the way CoreData does. Don’t fret – the CouchCocoa framework lets you do both, even in the same database. During this webcast I’ll show you how to acquire the glamor of an object model, while still letting your NoSQL freak-flag fly.

A new mix, sequel to Shiver 2, sequel to Shiver. I hope you like it…

[I just posted this to the Couchbase Mobile community mailing list.]

TouchDB is a project I’ve been feverishly working on for a few weeks. It’s an investigation into the feasibility of a CouchDB-compatible database rewritten from the ground up for mobile apps. The comparison I like to make is that “if CouchDB is MySQL, then TouchDB is SQLite”. In fact, it uses SQLite as its underlying storage engine. You can read a longer justification for it on its wiki, as well as an FAQ and design document.

— It speaks CouchDB’s replication protocol. I’m pretty serious about that; I’m even documenting the protocol.
— It also understands a large subset of the REST API, enough so that it works with CouchCocoa. I’ve got a clone of Grocery Sync working as one of the demo apps in the project.
— The current implementation is for iOS. If the investigation pans out we’ll port it to Android, and possibly other platforms.

TouchDB is certainly not ready for prime-time yet, but here are some current statistics to whet your appetite:
• Code size of an ‘empty’ iOS app with nothing in it but TouchDB: ~150k.
• Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm).
• Size of source code: ~4000 lines of Objective-C (plus another ~2500 lines from some existing utility libraries.)

What’s left to do? Probably a lot — that infamous “second 90%”. Prominently:
• Attachments
• Reduce functions and grouping
• Filters for views and replication
• Performance tuning
See the issue tracker for more.

So, what does this mean for Couchbase Mobile? Honestly, we don’t know yet. It may be that TouchDB turns out to be so awesome that it replaces embedded-CouchDB entirely in Couchbase Mobile on iOS and Android. It may be that there are still scenarios where embedded-CouchDB works better and is worth the extra overhead for some developers, in which case we’ll still support it. This is not a product announcement; it’s a technical announcement of something that isn’t a product yet, because we like to do our development in the open. We’d love your feedback or even contributions.

I gave another talk about Couchbase/CouchDB at the Keeping It Realtime conference this week in Portland. This one is titled “_ch_ch_changes: CouchDB/Couchbase Notifications And Replications”, and the slides are now up on slideshare.

I had a great time. The conference itself was pretty exciting, even if some of the content was over my head (I’m not primarily a web developer, server-side isn’t how I roll, and I’ve only just started learning about node.js this week!) Plus: Portland. OMG, I love Portland.

My new employer is doing well:

MOUNTAIN VIEW, Calif. – August 10, 2011 – Couchbase, the leading NoSQL database company, today announced it has secured $14 million in a Series C round of financing led by venture capital firm Ignition Partners with participation from the company’s existing investors Accel Partners, Mayfield Fund, and North Bridge Venture Partners. The company has also reserved an additional $1 million for investment from strategic customers and partners. The new funding will be used to further invest in NoSQL product development, support the adoption and growth of Couchbase in enterprise organizations, and support international expansion.

On the heels of its inaugural CouchConf developer conference, held July 29 in San Francisco with more than 300 attendees from around the world, Couchbase announced a Series C round of financing, bringing the company’s total funding to $30 million.

CouchConf was a great time. I really enjoyed meeting developers, and learning more about Couchbase from the other presentations. The slides from my own talk on Couchbase Mobile for iOS are now online (minus the gratuitous Keynote transitions) if you’d like to take a look.

August in Los Angeles was bone-dry and dusty, but he left it behind in the parking lot as he made his way through the series of three doors, heavy and white, and into the frozen refuge of the ice bar. He was known, there, and the hostess greeted him with a sealskin robe, slipped over his shoulders before he had time to start shivering. The tip of her elegant nose felt icy against his own.

There was room for one more at the bar, and at a nod from the chef he took the seat gratefully. One often had to wait, stamping feet to ward off the cold. The chef slid the amuse-bouche before him as he unfolded his napkin, and it was exquisite in appearance: a translucent carpaccio of walrus blubber sprinkled with snowflakes. The snowflakes were not unique, in fact they came in precisely two shapes, one sprinkled on the left side of the dish, the other on the right. They made not-quite-imperceptibly different crunches as he ate them. It was touches like this that had made the chef’s name when he was but a young man just arrived from Nunavut.

For his first course the customer ordered a cube, thick and meaty. It was peasant fare, but here elevated to fine cuisine. The chef’s assistants trained for three years in the rituals of icemaking. They knew intuitively what temperature of water to use, how to swirl it through the fourteen squares of the traditional whalebone tray, how to tap the sides to dislodge bubbles. One of those assistants now brought the chef a steaming tray, fresh from the freezer, which the master raised overhead and brought down with a single practiced motion onto the stone slab before him, then raised slowly to reveal the unbroken cubes. He then sorted through the cubes with the point of his obsidian knife, whisking the thirteen imperfect ones onto the floor. The remaining one he slid onto a plate and into the toaster oven behind him.

Twenty-five seconds in the oven grilled the outer layer of the cube to perfection, liquefying a thin sheen of pure water across its surface without defacing the deep-frozen insides with cracks. The customer took it in one bite, seasoning it only with a tiny pinch of sea salt, then sucking appreciatively. (He never chewed, of course: you might hear starlets and pop idols crunching their ice at trendy bars on Melrose, but here such behavior would get you ejected permanently.) The ice in his mouth was frictionless, spinning with every slight touch of his tongue, endlessly reconfiguring itself into new shapes as it melted in his heat. The meltwater had the slight mineral tang of true cube-ice, reflecting both its origin in a remote New Zealand spring and the subtle influence of the seasoned whalebone tray, infused during its months-long deep freeze.

While savoring the final drops, he studied the chalkboard for the daily specials. He was startled to see qainngittunga on the list, as this delicacy was only very rarely found, and its availability in summer was practically unheard-of south of the 49th Parallel. Despite the ruinous expense, as a connoisseur he had no choice but to order it. He was not sure he had pronounced the name quite correctly, but merely ordering this challenging dish was enough to ensure the respect of the staff.

With a single grunt of approval the chef knelt and reached into the hidden freezer beneath the bar. His thickly-gloved hands reappeared cradling a cylinder of ice three inches in diameter and a foot long, which he placed on a small polar-bear rug placed on the counter by an assistant. The ice was a pure, intense blue, mostly clear but punctuated by thin dark layers. It was a core sample that had been painstakingly extracted from thousands of meters below the surface of the Greenland ice sheet. (Such drilling is forbidden by international treaty, but there are limited exemptions for scientific research, and some of the resulting cores do find their way out of geology laboratories and into restaurants.)

Using a small handsaw with a diamond-studded wire blade, the chef quickly sliced off a section measuring a perfectly even three millimeters thick and gently lowered it onto a plate of red pumice, then handed it to the customer. He did not garnish it, nor did the customer apply a grain of salt; nothing was needed. The customer merely raised the plate to his mouth and slid the disc onto his waiting tongue.

The experience was indescribable, transcendent, as it had been once before on that memorable evening in Svalbard. His tongue was covered by the freshly-cut side of the ice (this was essential), which had been sealed inside the glacier, untouched, for a hundred thousand years. Through those ages the intense pressure and cold had distilled every essence of the microscopic bits of dust, sand and pollen that had drifted onto the surface so long ago. He tasted a world where mammoths roamed, and saber-toothed cats. He tasted the smoke of his ancestors’ cave fires, the ochre with which they painted the tales of their hunts.

After a long interval he swallowed. Nothing was left now; the ineffable vapors of impossibly distant pasts were already consumed. Before, in Svalbard, Lena was with him and the sensations had lived on in echoed reflections in each others’ eyes — I felt it, did you feel it too? — but that was the past, and on this day he was alone. Two years ago was no less remote than the Ice Age, and Lena was no less dead than the Cro-Magnon cave painters.

After a dish such as that, there was only one thing left to order. He gestured, holding up nine fingers, and the chef bowed deeply in response. A moment later a waitress appeared with a small brass tube the size of a rifle shell. She unscrewed the end, and tipped out a small crystal onto a silver dish. The customer gravely accepted the dish, and she bowed and retreated. In a reciprocal gesture he bowed his own head and gently lowered the tip of his tongue to the dish. As the water of his saliva made contact with the entropically-enhanced synthetic crystal packing of the water molecules in the ice₉ — an arrangement not found in nature, stable enough to remain solid at room temperature — its molecules too attached to the surface and froze, leading to a wave of crystallization that in two seconds had swept through his entire body and frozen it solid.

The staff quickly folded him into his voluminous sealskin robe, lashed it tightly shut with rawhide, and carried the stiff bundle into a remote corner of the basement. (But not before removing his wallet with tongs; the bill he left behind was quite considerable.)

I’ve just released a new open-source project, a small one—Fudge-Cpp, a fast C++ library for reading and writing Fudge messages.

I hadn’t heard of Fudge either, till a few weeks ago, but it’s a type of thing that’s always interested me: a generic structured binary data format. A quick elevator pitch would be “it’s sorta like JSON, except more compact and faster to parse”. (It’s also sorta like Mac property-lists, YAML, etc.) So, it lets you turn collections of scalars, strings, arrays and dictionaries into a standardized blob of data that can be sent over a network or stored on disk or whatever.

From the Fudge website:

“Fudge is primarily useful in situations where you have:

• Data exchanging between nodes in a distributed system; where
• You want to be able to do meaningful work with data without knowing at compile time the precise message format; and
• Performance and message size are important; or
• A requirement to encode data and translate between efficient binary encodings, and more accepted text encodings (XML, JSON) depending on the communications channel.”

The obvious advantage of a binary format for this is that it’s faster to parse. Instead of having to tokenize the input and walk through it counting braces and commas, and converting sequences of digits into numbers, you just read big-endian numbers from the stream and interpret them as types and byte-counts. This can also make the data smaller, if the format is careful to use the smallest number of bytes to represent a number (which Fudge does). It’s especially compact for messages containing payloads of binary data like images, audio, or digital signatures, since those blobs don’t have to be converted to Base64 (which is 25% larger.)

A more subtle advantage is that the library can use less memory. In fact, Fudge-Cpp basically allocates no memory at all from the heap. How does it manage this? When the library returns the caller an object representing a data value from the parsed message (a scalar, string, array, etc.) it doesn’t allocate that object on the heap. Instead, it just returns a pointer to where that value is stored in the binary message itself. This really is a valid C++ object pointer; it’s just that the object’s members have the exact same layout as the corresponding Fudge data.

(Now, there is a bit of impedance mismatch that adds overhead to accessing the data this way. For one thing, on a little-endian CPU every multi-byte numeric value will need to be byte-swapped when accessed. And random access to dictionaries and object arrays is O(N) since they are basically linked lists. But the overhead is low.)

This library isn’t a world-changing project; it was more of a fun diversion for me over a few weekends. I’ve done this sort of thing before—Ottoman uses similar memory-saving pointer tricks, and AEGizmos was literally the first code I ever wrote at Apple back in 1991—and it’s always fun to twiddle bits this way.

I’ve been making little updates to the MYCrypto library for a few months, and after the latest batch I did some housekeeping—fixing iOS and 64-bit build errors, updating the docs—and decided to dignify them with a new version number, 0.5.

Notable improvements:

• More certificate I/O functionality. You can now import and export PKCS12 (.p12) files, which are encrypted archives that contain a private key as well as its cert.
• More certificate trust validation APIs, including read/write access to user trust settings.
• Access to X.509 cert extensions like key-usage.
• Can verify signatures that use algorithms other than SHA-1, and parse certs that use such algorithms.
• MYMockKey, a testing fixture that lets you test code that uses digital signatures without having to generate or use real key-pairs.

If you infer from this list that I’ve been working on an app that manages X.509 certs, I won’t deny it :) Maybe I’ll have more to say about that soon.

The documentation for Rdio’s new API begins:

It’s simple to make requests to Rdio’s REST API. It’s built on widely used standards and conventions so there are libraries for most common web development platforms. All method calls are made as POST requests to http://api.rdio.com/1/. Arguments are sent as application/x-www-form-urlencoded, just like when a browser submits a form. The name of the method is passed as the ‘method’ argument. [Emphasis mine.]

What’s wrong with this? Well, the first bolded point is immediately contradicted by the ones that follow. Specifically, this cannot be a REST API, because it uses only one URL and one HTTP method. Two of the key features of HTTP-based REST are that

1. It’s object-oriented, where objects are identified by URLs. Each request’s URL identifies what object it operates on.
2. The methods to invoke on the objects are primarily indicated by the request’s method (GET, PUT, POST, DELETE). In fact this is why Tim Berners-Lee used the word “method” in the HTTP protocol in the first place.

So in a real REST API there would be a URL representing “my friends”, and a client could GET that URL to retrieve a list of friends, POST to it to add a friend (resulting in a new URL/resource representing that friend), PUT to a friend’s URL to update details, and DELETE to that URL to remove the friend relation. Instead, the actual Rdio API has a couple of dozen ad-hoc verbs including “addFriend” and “removeFriend”. I didn’t see one to get a list of friends or get info about a friend, but there is an Rdio-specific “get” verb that might work for those things. “Get” seems nicely general, but then for some reason there are also sixteen other specific getters ranging from “getActivityStream” to “getTracksInCollection”.

So it’s clear that Rdio’s “REST” API, like many other recent “REST” APIs such as DropBox’s*, isn’t REST at all. It’s more of an ad-hoc RPC scheme, which is ironic because there’s traditionally been a lot of enmity between REST proponents vs. those of RPC protocols like XML-RPC and SOAP. The SOAP boffins must be chortling behind their WSDLs at this.

Maybe we should just give up on the term REST, since it’s become so diluted as to mean nothing more than “HTTP API that’s not as hard to use as SOAP”?

• The hacky DropBox API especially annoys me because there’s already a perfectly good standardized and widely adopted REST API for most of what DropBox does.

I have backed up all the tweets from my Twitter account (@snej) to a local file, and am now mass-deleting all of them. This is a venerable form of protest that goes back to early BBSs like the WELL. Basically, I am no longer willing to donate my ‘valuable’ user-generated content to a centralized service that issues fuck-yous of this magnitude to its developers and users.

I could rant at length about the arrogance, stupidity and just plain creepiness of that message and the policies behind it, but I don’t know that it’s even worth it. Others have already done a pretty good job of deconstructing its marketroid Newspeak. I just can’t resist pointing out that two of the major components of Twitter’s content model—the @-mention and the #hashtag—were invented by early users and app developers, not by Twitter itself, then later integrated directly into the system to make them more useful. That’s a great example of collaborative development. Now, perversely, Twitter sees fit to tell app developers exactly how they can and can’t represent those same features in their UIs.

And yes, this is enforceable, because thanks to OAuth they can and will revoke an app’s access to Twitter at the flick of a switch. They brag about how they “revoke literally hundreds of API tokens / apps a week” [ibid]. I just now realized the implications of this, actually. OAuth may be more secure than traditional HTTP auth in that it doesn’t give apps access to your account password, but the centralization of control that it gives to service providers is really disturbing.

“But Jens”, you say, “you still have accounts on other centralized social networking sites such as Facebook, Tumblr, LiveJournal and flickr, many of which have also shown a similar disregard for users and developers. Why aren’t you deleting those accounts?”

Good question, anonymous readership. It comes down to three factors:

1. These other services feel more like real apps, with idiosyncratic features. Twitter has always seemed more like (and promoted itself as, at least to developers) a general purpose platform. It’s a series of tubes for publishing and subscribing to 140-character blobs. I could ignore its stupid star-shaped topology, centralized control, and frustrating payload limitations … as long as it stuck to being a generic service. Now they’re taking that back. It’s as though the phone company is telling me what color of telephone I’m allowed to plug into their lines and what size the touch-tone buttons have to be. (Think that’s a silly example? The old monopoly AT&T actually did enforce that ludicrous degree of control until court rulings in 1956 and 1968.)
2. Some of these services are, frankly, a lot more important to me than Twitter. I am basically on Twitter mostly to keep up with some friends/acquaintances who post about their daily lives there. (It makes me very sad that some of those friends once used to post far more meaningful content on a regular basis on LiveJournal. I miss those days.) But even though I keep my list of follows really small, my stream still has so many retweets and links to random URLs and unreadably-shorthanded opinions, that it’s often more frustrating than useful.
3. Finally some of those services just haven’t done anything particularly evil yet. Turns out I can put up with innocent everyday failure pretty well when I’m not paying anything for it.

The big question in my mind is what to replace Twitter with. Ironically (and perhaps pathetically) I think I will end up reading Facebook more, because some of my Twitter friends are also there. At least until the next time Facebook does something egregiously evil.

In a larger sense, it should not be rocket science to build some plumbing that does what Twitter does—publish and subscribe small blobs—with an actually-decentralized architecture. There are a lot of smart developers out there, but to some extent we’ve been seduced into suckling at the proprietary API teats of big providers, at the expense of developing the next generation of open protocols.

Yeah, in my current day job I’m as guilty of this as anyone else. But at home I’ve got a garage full of various pieces of half-built tech that attempt to solve that problem in one form or another, if I could ever finish any of them. A lot of the trouble is motivation. Anyone want to help out?