I gave a webcast last month for O’Reilly, and it’s up on YouTube now so you can watch it at your leisure. Here’s the abstract:

It can be very liberating to store your Couchbase app’s data as free-form dictionaries, free of the rigid schema of a relational database. But there’s a lot to be said for the syntactic comforts and conveniences of modeling your data as native Cocoa objects with predeclared @properties, the way CoreData does. Don’t fret – the CouchCocoa framework lets you do both, even in the same database. During this webcast I’ll show you how to acquire the glamor of an object model, while still letting your NoSQL freak-flag fly.

[I just posted this to the Couchbase Mobile community mailing list.]

TouchDB is a project I’ve been feverishly working on for a few weeks. It’s an investigation into the feasibility of a CouchDB-compatible database rewritten from the ground up for mobile apps. The comparison I like to make is that “if CouchDB is MySQL, then TouchDB is SQLite”. In fact, it uses SQLite as its underlying storage engine. You can read a longer justification for it on its wiki, as well as an FAQ and design document.

— It speaks CouchDB’s replication protocol. I’m pretty serious about that; I’m even documenting the protocol.
— It also understands a large subset of the REST API, enough so that it works with CouchCocoa. I’ve got a clone of Grocery Sync working as one of the demo apps in the project.
— The current implementation is for iOS. If the investigation pans out we’ll port it to Android, and possibly other platforms.

TouchDB is certainly not ready for prime-time yet, but here are some current statistics to whet your appetite:
• Code size of an ‘empty’ iOS app with nothing in it but TouchDB: ~150k.
• Time to initialize TouchDB and open a database, on iPad 2: ~100ms (cold) or ~60ms (warm).
• Size of source code: ~4000 lines of Objective-C (plus another ~2500 lines from some existing utility libraries.)

What’s left to do? Probably a lot — that infamous “second 90%”. Prominently:
• Attachments
• Reduce functions and grouping
• Filters for views and replication
• Performance tuning
See the issue tracker for more.

So, what does this mean for Couchbase Mobile? Honestly, we don’t know yet. It may be that TouchDB turns out to be so awesome that it replaces embedded-CouchDB entirely in Couchbase Mobile on iOS and Android. It may be that there are still scenarios where embedded-CouchDB works better and is worth the extra overhead for some developers, in which case we’ll still support it. This is not a product announcement; it’s a technical announcement of something that isn’t a product yet, because we like to do our development in the open. We’d love your feedback or even contributions.

I gave another talk about Couchbase/CouchDB at the Keeping It Realtime conference this week in Portland. This one is titled “_ch_ch_changes: CouchDB/Couchbase Notifications And Replications”, and the slides are now up on slideshare.

I had a great time. The conference itself was pretty exciting, even if some of the content was over my head (I’m not primarily a web developer, server-side isn’t how I roll, and I’ve only just started learning about node.js this week!) Plus: Portland. OMG, I love Portland.

My new employer is doing well:

MOUNTAIN VIEW, Calif. – August 10, 2011 – Couchbase, the leading NoSQL database company, today announced it has secured $14 million in a Series C round of financing led by venture capital firm Ignition Partners with participation from the company’s existing investors Accel Partners, Mayfield Fund, and North Bridge Venture Partners. The company has also reserved an additional $1 million for investment from strategic customers and partners. The new funding will be used to further invest in NoSQL product development, support the adoption and growth of Couchbase in enterprise organizations, and support international expansion.

On the heels of its inaugural CouchConf developer conference, held July 29 in San Francisco with more than 300 attendees from around the world, Couchbase announced a Series C round of financing, bringing the company’s total funding to $30 million.

CouchConf was a great time. I really enjoyed meeting developers, and learning more about Couchbase from the other presentations. The slides from my own talk on Couchbase Mobile for iOS are now online (minus the gratuitous Keynote transitions) if you’d like to take a look.

I’ve just released a new open-source project, a small one—Fudge-Cpp, a fast C++ library for reading and writing Fudge messages.

I hadn’t heard of Fudge either, till a few weeks ago, but it’s a type of thing that’s always interested me: a generic structured binary data format. A quick elevator pitch would be “it’s sorta like JSON, except more compact and faster to parse”. (It’s also sorta like Mac property-lists, YAML, etc.) So, it lets you turn collections of scalars, strings, arrays and dictionaries into a standardized blob of data that can be sent over a network or stored on disk or whatever.

From the Fudge website:

“Fudge is primarily useful in situations where you have:

• Data exchanging between nodes in a distributed system; where
• You want to be able to do meaningful work with data without knowing at compile time the precise message format; and
• Performance and message size are important; or
• A requirement to encode data and translate between efficient binary encodings, and more accepted text encodings (XML, JSON) depending on the communications channel.”

The obvious advantage of a binary format for this is that it’s faster to parse. Instead of having to tokenize the input and walk through it counting braces and commas, and converting sequences of digits into numbers, you just read big-endian numbers from the stream and interpret them as types and byte-counts. This can also make the data smaller, if the format is careful to use the smallest number of bytes to represent a number (which Fudge does). It’s especially compact for messages containing payloads of binary data like images, audio, or digital signatures, since those blobs don’t have to be converted to Base64 (which is 25% larger.)

A more subtle advantage is that the library can use less memory. In fact, Fudge-Cpp basically allocates no memory at all from the heap. How does it manage this? When the library returns the caller an object representing a data value from the parsed message (a scalar, string, array, etc.) it doesn’t allocate that object on the heap. Instead, it just returns a pointer to where that value is stored in the binary message itself. This really is a valid C++ object pointer; it’s just that the object’s members have the exact same layout as the corresponding Fudge data.

(Now, there is a bit of impedance mismatch that adds overhead to accessing the data this way. For one thing, on a little-endian CPU every multi-byte numeric value will need to be byte-swapped when accessed. And random access to dictionaries and object arrays is O(N) since they are basically linked lists. But the overhead is low.)

This library isn’t a world-changing project; it was more of a fun diversion for me over a few weekends. I’ve done this sort of thing before—Ottoman uses similar memory-saving pointer tricks, and AEGizmos was literally the first code I ever wrote at Apple back in 1991—and it’s always fun to twiddle bits this way.

I’ve been making little updates to the MYCrypto library for a few months, and after the latest batch I did some housekeeping—fixing iOS and 64-bit build errors, updating the docs—and decided to dignify them with a new version number, 0.5.

Notable improvements:

• More certificate I/O functionality. You can now import and export PKCS12 (.p12) files, which are encrypted archives that contain a private key as well as its cert.
• More certificate trust validation APIs, including read/write access to user trust settings.
• Access to X.509 cert extensions like key-usage.
• Can verify signatures that use algorithms other than SHA-1, and parse certs that use such algorithms.
• MYMockKey, a testing fixture that lets you test code that uses digital signatures without having to generate or use real key-pairs.

If you infer from this list that I’ve been working on an app that manages X.509 certs, I won’t deny it :) Maybe I’ll have more to say about that soon.

The documentation for Rdio’s new API begins:

It’s simple to make requests to Rdio’s REST API. It’s built on widely used standards and conventions so there are libraries for most common web development platforms. All method calls are made as POST requests to http://api.rdio.com/1/. Arguments are sent as application/x-www-form-urlencoded, just like when a browser submits a form. The name of the method is passed as the ‘method’ argument. [Emphasis mine.]

What’s wrong with this? Well, the first bolded point is immediately contradicted by the ones that follow. Specifically, this cannot be a REST API, because it uses only one URL and one HTTP method. Two of the key features of HTTP-based REST are that

1. It’s object-oriented, where objects are identified by URLs. Each request’s URL identifies what object it operates on.
2. The methods to invoke on the objects are primarily indicated by the request’s method (GET, PUT, POST, DELETE). In fact this is why Tim Berners-Lee used the word “method” in the HTTP protocol in the first place.

So in a real REST API there would be a URL representing “my friends”, and a client could GET that URL to retrieve a list of friends, POST to it to add a friend (resulting in a new URL/resource representing that friend), PUT to a friend’s URL to update details, and DELETE to that URL to remove the friend relation. Instead, the actual Rdio API has a couple of dozen ad-hoc verbs including “addFriend” and “removeFriend”. I didn’t see one to get a list of friends or get info about a friend, but there is an Rdio-specific “get” verb that might work for those things. “Get” seems nicely general, but then for some reason there are also sixteen other specific getters ranging from “getActivityStream” to “getTracksInCollection”.

So it’s clear that Rdio’s “REST” API, like many other recent “REST” APIs such as DropBox’s*, isn’t REST at all. It’s more of an ad-hoc RPC scheme, which is ironic because there’s traditionally been a lot of enmity between REST proponents vs. those of RPC protocols like XML-RPC and SOAP. The SOAP boffins must be chortling behind their WSDLs at this.

Maybe we should just give up on the term REST, since it’s become so diluted as to mean nothing more than “HTTP API that’s not as hard to use as SOAP”?

• The hacky DropBox API especially annoys me because there’s already a perfectly good standardized and widely adopted REST API for most of what DropBox does.

I have backed up all the tweets from my Twitter account (@snej) to a local file, and am now mass-deleting all of them. This is a venerable form of protest that goes back to early BBSs like the WELL. Basically, I am no longer willing to donate my ‘valuable’ user-generated content to a centralized service that issues fuck-yous of this magnitude to its developers and users.

I could rant at length about the arrogance, stupidity and just plain creepiness of that message and the policies behind it, but I don’t know that it’s even worth it. Others have already done a pretty good job of deconstructing its marketroid Newspeak. I just can’t resist pointing out that two of the major components of Twitter’s content model—the @-mention and the #hashtag—were invented by early users and app developers, not by Twitter itself, then later integrated directly into the system to make them more useful. That’s a great example of collaborative development. Now, perversely, Twitter sees fit to tell app developers exactly how they can and can’t represent those same features in their UIs.

And yes, this is enforceable, because thanks to OAuth they can and will revoke an app’s access to Twitter at the flick of a switch. They brag about how they “revoke literally hundreds of API tokens / apps a week” [ibid]. I just now realized the implications of this, actually. OAuth may be more secure than traditional HTTP auth in that it doesn’t give apps access to your account password, but the centralization of control that it gives to service providers is really disturbing.

“But Jens”, you say, “you still have accounts on other centralized social networking sites such as Facebook, Tumblr, LiveJournal and flickr, many of which have also shown a similar disregard for users and developers. Why aren’t you deleting those accounts?”

Good question, anonymous readership. It comes down to three factors:

1. These other services feel more like real apps, with idiosyncratic features. Twitter has always seemed more like (and promoted itself as, at least to developers) a general purpose platform. It’s a series of tubes for publishing and subscribing to 140-character blobs. I could ignore its stupid star-shaped topology, centralized control, and frustrating payload limitations … as long as it stuck to being a generic service. Now they’re taking that back. It’s as though the phone company is telling me what color of telephone I’m allowed to plug into their lines and what size the touch-tone buttons have to be. (Think that’s a silly example? The old monopoly AT&T actually did enforce that ludicrous degree of control until court rulings in 1956 and 1968.)
2. Some of these services are, frankly, a lot more important to me than Twitter. I am basically on Twitter mostly to keep up with some friends/acquaintances who post about their daily lives there. (It makes me very sad that some of those friends once used to post far more meaningful content on a regular basis on LiveJournal. I miss those days.) But even though I keep my list of follows really small, my stream still has so many retweets and links to random URLs and unreadably-shorthanded opinions, that it’s often more frustrating than useful.
3. Finally some of those services just haven’t done anything particularly evil yet. Turns out I can put up with innocent everyday failure pretty well when I’m not paying anything for it.

The big question in my mind is what to replace Twitter with. Ironically (and perhaps pathetically) I think I will end up reading Facebook more, because some of my Twitter friends are also there. At least until the next time Facebook does something egregiously evil.

In a larger sense, it should not be rocket science to build some plumbing that does what Twitter does—publish and subscribe small blobs—with an actually-decentralized architecture. There are a lot of smart developers out there, but to some extent we’ve been seduced into suckling at the proprietary API teats of big providers, at the expense of developing the next generation of open protocols.

Yeah, in my current day job I’m as guilty of this as anyone else. But at home I’ve got a garage full of various pieces of half-built tech that attempt to solve that problem in one form or another, if I could ever finish any of them. A lot of the trouble is motivation. Anyone want to help out?

Twitter: Charming in brief doses, he tells you little one-liner jokes, then wanders off after two sentences to go talk at somebody else. He absolutely will not shut up for an instant, and namedrops shamelessly about his famous friends. When he’s outworn his welcome he passes out drunk on the floor and has to be dragged home.

MySpace: Who? Oh, right, this anorexic high-school girl who threw herself at you at a party once in 2005. She kept bragging about all the bands she knew (and which you could overhear on the tinny earbuds she wore.) After one too many Jägermeister Jell-O shots she barfed Day-Glo all over your shoes. Last you’ve heard, she’s found some 80-year-old media mogul to be her sugar daddy.

Facebook: You vaguely remember him from high school. He was a nonentity then and he’s equally uninteresting now, but he’s somehow infiltrated your circle of friends and shows up at every social event you go to, telling boring anecdotes about last night’s game and what he bought at Wal*Mart. Worse, it seems he’s joined some cult and wants you to join too so he can go up a level.

Tumblr: She’s got impeccable taste, a lovely apartment and fascinating stories, but after a while you realize she only talks about what other people have done; she doesn’t have an original thought in her head. She won’t carry on a conversation, either, so the only way to get her to pay attention to you is to repeat back something she’s already told you.

Soup.io: Similar to Tumblr, but with a cute Austrian accent. She’s more conversational, but on the downside she sometimes insists on talking to you in German.

LiveJournal: A mysterious Goth chick you were introduced to at a club. After you strike up a friendship with her, she starts telling you all her innermost secrets whenever you see her. This is terribly alluring at first, enough so that you can overlook her appallingly bad fanfic, but after a while you begin to realize how seriously disturbed she is. Around then she abruptly stops showing up, and you’re never sure whether she killed herself or just moved to a more elite social circle. You never learn her real name.

[Update: I’ve changed two of them to male. I wanted to be consistent in the personification, but in retrospect that leaves me open to charges of sexism, which absolutely wasn’t intended. They all have male counterparts, of course, whom I’d love to hear about if you want to write about them.]

I’ve never figured out whether I prefer Python or Ruby, so I’ve written things in both languages. Sometimes I start in one, then change my mind and decide I’d rather use the other. Unfortunately, changing over is painful, even though both have fairly similar syntax. For instance, converting to Ruby means inserting zillions of “end” statements!

Having a need to do this recently, I lazily looked around for a script that would do the grunt-work of Python-to-Ruby translation. I couldn’t find one, so I ended up writing one myself. And I’ve uploaded it for the benefit of others who might have the same need, and who might even improve it.

So here it is: py2.rb. Use it wisely. In particular, pay attention to the caveat found in the file’s header comments:

This script just does the obvious, easy transformations, giving you more time to work on the harder ones :) It is NOT a real parser, just a bunch of kludgy regex operations, so it can’t do anything fancy. It may get some things wrong, and won’t even attempt some other things that it’s very likely to get wrong. The output will definitely have to be hand edited by someone familiar with both languages, before it can be expected to compile as Ruby, much less run correctly. The goal is simply to require less hand editing, and less mechanical replacing, than you would have had to do without this script.

It’s in the public domain. “Do what thou wilt” shall be the whole of the law.

Note: Please don’t comment with Python advocacy or, worse, anti-Ruby flames. That’s not the point. I like Python fine; I just prefer to remain promiscuous. And I think it’s ultimately in everyone’s interest for ideas to be able to flow from one form of expression to another.