Problems seem to occur if the kernel panics or, worse, the disk abruptly loses power. Disk controllers have an internal cache, and they re-order sector writes to minimize the amount of head travel. If the controller resets/dies before it can flush its cache, some random subset of the most recent sector writes will be lost.

I’m reading through the “Atomic Commit In SQLite” document now — it’s very interesting, and I don’t believe it was around when I was working with sqlite in ‘05-‘07. It sounds like current versions of it are more robust to disk power failures … maybe as a result of the problems we had at Apple ;)

A related problem that came up much more often was denial-of-service to processes accessing the database, due to another process being hung while it holds a lock. On my Leopard system this seemed to happen every few months to PubSub. This is my fault, not sqlite’s, of course, but it does mean that a failure in one process propagates into others, which is bad. Worse, the way to recover from it is to move the database aside and re-create it, just as though it had been corrupted. This makes me appreciate the way that in an append-only file format, readers can never be locked out.

This was a total pain to implement, and is why I really want to write something that won’t have these problems, even if it isn’t otherwise as powerful as sqlite.

So uh, how did you manage to do that with sqlite? :)