Two of the downsides of the iPhone are the two-year indentured servitude to AT&T, and the cost of the service plans (well, they’re not that expensive, but it’s 50% more I was spending on my Sidekick.) But I inadvertently found a way around both those problems.
Read the rest of this entry »
There’s an interesting kerfluffle going on regarding the scaling woes that Twitter.com is going through, especially since it’s built on Ruby On Rails. Here’s the original interview with one of the Twitter coders, the somewhat evasive reply by the lead Rails architect, and Mark Pilgrim’s cruel-but-funny dissection of the latter.
Ruby is a lovely language and Rails is a lovely framework, but both of them trade performance for aesthetics and convenience. That is, they’re slow. No problem, though — in the magical world of web apps, you just solve performance problems by throwing hardware at the problem, usually in the form of more and more web servers. (If I sound sarcastic, I’m just envious because I write client-side software, where we don’t have that convenient fallback.)
Then, since 99.99% of web apps use a SQL database, the next bottleneck is the database server that all the web servers are talking to. The next step of course is to buy more database servers, but then you have to distribute/replicate the database across those servers, which is even more challenging. (Or if you’re a crazed wunderkind like LiveJournal founder Brad Fitzpatrick, you invent a memory-based distributed hashtable as a cache to put in front of the database.)
This whole thing has me wondering why it is that SQL databases are used as the all-purpose hammer for solving all data storage problems for web apps. To some degree this has to be a historical accident, because I don’t think it’s necessarily obvious. My guess is that (a) a lot of companies already used these databases for storing their, uh, data; (b) in the mid-’90s all these companies were desperate to Get On The Web; so (c ) countless web developers wrote “three-tier” frameworks for gluing those SQL databases onto web servers.
I’ve been using SQL a lot (via sqlite, as a data store for the Syndication framework) and it’s quite nice. But I don’t think SQL — or any sort of query-based database system — is the right tool for every job, in particular these kinds of social-messaging apps like Twitter.
The figure that stands out to me is “…up to 11,000 requests per second”. Jesus Christ, that’s a lot. Where does that come from? Even assuming a million members who each post ten times a day, that’s only about 100 posts per second. If each member runs a client app polling for updates every 15 minutes, 24 hours a day, that’s still only about 1,000 hits/sec. There’s still an order of magnitude unaccounted for. (Maybe the apps poll every 1.5 minutes?)
But think about the polling again. Each of these requests just boils down to the client asking “what messages did I get since the last time I asked?” The SQL way of answering that question is to run a complicated join that looks up all of the members I’m subscribed to, looks up all the messages of those members, and selects the ones whose timestamp is later than the date of the client’s last request. Sure, there are indexes, and databases are optimized for this stuff, but it’s still going on a thousand times a second.
Now, a more sane way to do this (if I may be so bold) is to keep a message queue for every member, and whenever someone posts a new message, copy a pointer to that message into the queue of every member who’s subscribed. Then when someone polls, you just have to get the messages out of their queue and send them along. If that sounds kind of familiar, it’s because that’s exactly how email works — the Twitter posting interface is like a listserv, and the polling interface is like a POP server.
The cool thing is that you don’t even need a database to do this. You can store all of this stuff using very simple file formats, with one file per message queue. Since this is so much simpler than triple-decker SQL joins, one server can handle more requests; and if that’s too much, you can use a networked filesystem (or even a SAN) to distribute it across servers.
And of course you could keep going with this novel “distributed” idea, and make this into an actually-distributed system, where individual users (or groups of them) can run their own Twitter servers that queue their incoming messages and relay their outgoing ones. Imagine!
The trouble is, the idea of centralizing is so entrenched, and there’s so much industry support for “scalability” by means of piling on more servers and moving to bigger data-centers with fatter pipes, that I can see that this is going to go on for some time. (I’ve heard that AIM handles over a billion IMs a day, every one of which streams in and out of an unfathomably fat pipe in a huge AOL data center in Virginia. Shudder.)
In some moods (the same sorts of moods where I fantasize about how $5-per-gallon gas would kick-start desperately needed changes in our transportation system) I think about what would happen if all the DNS servers mysteriously vanished, all the data centers blew up, and we had to re-implement everything in a distributed peer-to-peer fashion. It’s actually really interesting …
NYT: Computer Science Takes Steps to Bring Women to the Fold
I find this article baffling and the comments on it aggravating.
“The nerd factor is huge,” Dr. Cuny said. … This image discourages members of both sexes, but the problem seems to be more prevalent among women. ‘They think of it as programming,’ Dr. Cuny said. ‘They don’t think of it as revolutionizing the way we are going to do medicine or create synthetic molecules or study our impact on the climate of the earth.’
Well, of course it’s programming. Trying to sell CS as somehow not being about programming is false advertising — it’s like telling kids that chemistry isn’t about molecules or mathematics isn’t about numbers. Sure, there are scientists using computers to design medicines or study the climate. But they’re not computer scientists, they’re biologists or geologists. Computer science is about software.
How is it that people can be so excited by the Internet and digital media, but totally turned off by the prospect of designing the stuff that makes those things work? They seem to confuse computer science with data-entry, or boring MIS drudgery like writing payroll systems. Or do they just totally not care about where things like web search and MP3 codecs and 3D graphics and peer-to-peer protocols come from … are they just some magic that falls out of the sky and no one should give a second thought to?
Another issue here is the distinction between Computer Science as a field of academic study, versus actual software engineering / programming / hacking. This is a young field (just as chemistry and aeronautics used to be) and not very codified yet. That’s part of the fun — it’s all being made up as we go along, and there aren’t so many tomes of knowledge that have to be digested first. I had a great time in college … but I have to admit that it didn’t directly teach me that much about programming. I learned a great deal of useful science and mathematics, I had access to a lot of computers and time to do a lot of hacking, and I hung around with brilliant people whom I got some good ideas from. But I’ve worked with people who are great software engineers despite not having a college degree … or having a degree in a different field like physics or psychology. So the number of people (and their M/F ratio) getting degrees in CS may not be that relevant.
On the downside, I’ve looked at the source code to enough open source projects to realize that there are a lot of people out there writing code whose basic programming skills are very, very poor; they probably could have used a solid undergraduate CS introduction. (Not to start any flames, but PHP seems especially prone to this … I think it’s kind of the BASIC of the modern era. The barrier to entry is so low, and the language so ad hoc, that people can just flail around in it.)
And I’m not even addressing the main point, the low and diminishing proportion of women in the field. In fact, my experience is that the ratio is a lot worse than the figures like 25% that they quote in the article: I’ve almost never worked in an environment where more than about 10% of the technical people were female. Which is really sad, in many ways.
My friend Tanya is a gateway to the strange and exotic worlds of fan- and slash-fiction. Today she pointed out a whole LiveJournal community, mac_hearts_pc, devoted to mostly-smutty extrapolations of Apple’s anthropomorphized Mac-vs-PC ads. Wow.
In her post about this, she says
“I tend to think of Macs as so feminine as to be, well, female”
which is really making me think…
How do we map computer behaviors onto ‘male’ and ‘female’? In American culture the æsthetic sense, intuition and readiness to please that exemplify “Mac-ness” bring to mind images of women, but if I challenge myself I can also imagine them as aspects of a courtly gentleman, like a royal servant or even a knight. Or perhaps of the British “gentleman’s gentleman”, such as Bunter or Jeeves … though here the details of the æsthetics don’t work, as this calls to mind a much darker, subtler UI theme, all leather, oak, brass and polished silver. The colors in Aqua fit better with the color scheme of medieval Europe or Japan.
From my perspective as a programmer, I consider the underpinnings as well. The Unix core of Mac OS is undeniably male: strong, efficient, taciturn, more rugged than pretty, ready to do exactly what you told it with no backtalk and no second-guessing about what you might have “meant instead”. And this core is there for anyone who wants it, it hasn’t been suppressed or denied. Yet the OS has learned that getting along in the world, and taking a role subservient to its user, requires it to adopt feminine traits as well.
At this point a light bulb LED turns on and I say “Aha! Jungian psychology!” According to Carl Jung and his successors, every male has a feminine aspect to his personality (the “anima”) and every female a masculine aspect (the “animus”). They’re like the small dots in the yin/yang symbol. If they’re not acknowledged or consciously used, they’re still there, and we may unconsciously use them in our relations with others, to generally negative results — like acting a role we haven’t properly rehearsed, adopting only the stereotypes without the essence. [John A. Sanford’s The Invisible Partners is a fascinating book on this topic.]
Do you see where I’m going with this? Both Mac and PC (and PC’s hippie cousin Linux) are essentially male, because the innards of any machine embody traits we see as stereotypically masculine (rigid, powerful, unforgiving, rule-based…). But in most of our interactions with computers, we want a more even gender balance, or even one that’s weighted toward the feminine archetypes (intuitive, harmonious, calming…) In other words, we want the computer to have gone through some Jungian analysis, to have acknowledged its anima and integrated it into its personality, to have “gotten in touch with its female side”.
Of course I think the Mac is pretty well advanced psychologically these days, though it’s had a tumultuous past. It started out practically denying its masculine self, projecting a totally femininized image, in reaction to the crudity of its then-peers. This caused all kinds of interior strain and complexes — which I can vouch for, having been one of its many therapists during that era. The masculine traits were there, but deeply repressed inside and revealed only to experienced programmers, and the strain made any further growth nearly impossible. (This also got the Mac beaten up a lot by bullies.) The key to progress was a reunion with NeXT, which had similar goals but had attained a different balance. The early relationship was rocky, but by 2001 the partners had fused in an alchemical wedding and the stage was set for serious growth.
I don’t want to stoop to Windows-bashing, but I’ll just say briefly that in my opinion it still has more therapeutic work to do to get over its hyper-masculine DOS past (complicated by its promiscuous desires to run on any hardware it sees.) Its progress toward integrating its anima has been steady, though slow, but I believe it still relies too much on adopting stereotypes (“are my windows translucent enough?”, like a neophyte cross-dresser) and on following fashions set by the Mac, without yet fully integrating the feminine strengths into its core being. But at least all parties have similar goals nowadays, which is a good thing for all.
There’s an interesting online competition going on called My Dream App. The idea is that a bunch of people pitch their ideas for a Mac application, and the set of ideas gets winnowed down in several rounds of public voting, until one is left. A group of experienced developers have promised to implement the winning idea as a shareware app, whatever it turns out to be.
It’s a fun concept, but it highlights some of the problems of having end users design software. A number of the proposals give me a particular sinking feeling I associate with user-interface design meetings: lots of ideas that sound super-cool as one-sentence pitches, accompanied by irresistibly glitzy faked-up screen shots (all replete with translucency, rounded corners, and this year’s de rigeur reflections). But too often there’s no “there” there. It’s all so vague that I can sense that these people haven’t thought through the difficult bits or worked out a coherent idea of what the app will do. Read the rest of this entry »
It just occurred to me that my newish MacBook gives me no less than four sensory modalities for detecting high CPU usage:
Still missing is taste, and maybe balance and proprioception, but I think I don’t want my computer using those pathways to communicate with me.
Lua is an interesting scripting language. I can’t say I have much familiarity with it; I’ve only read the book, and a couple of papers, and downloaded and built the interpreter (which takes less than a minute). But what I’ve seen of it gives me a warm feeling, like reading a concise little poem, a haiku. It’s a small language, but what’s there is well-considered, and it appears that you can build bigger things (like object models, whether class- or prototype-based) out of its building blocks pretty easily.
The implementation of the Lua 5.0 runtime is also interesting, as described in an excellent paper. One of the smaller details that’s been fascinating me is that Lua, it turns out, uses unique string objects.
When you use any kind of garbage-collected (or ref-counted) framework, string objects accumulate like dust bunnies. I’ve profiled Java apps and seen tens of thousands of instances of java.lang.String. Cocoa apps also have large numbers of NSStrings lying around. Some of these are temporary strings that just haven’t been garbage-collected (or drained from an autorelease pool) yet; but I think a lot more of them are duplicates.
Lua’s approach, which I have to say I hadn’t thought of before, is to make all string instances unique: there are never two different string objects with the same contents. In practice this means that the runtime maintains a big hash table (a Set) of weak references to all the string objects, so before it creates a new string it can first check if one with the same contents already exists. This seems kind of expensive. But consider the advantages:
One of the secondary issues, which only occurred to me after a little further reflection, is that this really works only if strings are all immutable. Otherwise you can take one string and change it to be identical to another string, yet still a distinct object. This is a bit of a drawback; I have to say that I prefer the Cocoa API, where the immutable NSString has a subclass NSMutableString, to the Java API where the mutable StringBuffer class is not type-compatible with the regular String. (Speaking of which, a Lua tech note presents a typically elegant way to implement an efficient appendable string buffer.)
I wouldn’t be at all surprised if the unique-strings approach predates Lua, perhaps going back to some primordial string-processing language like SNOBOL. Still, it was new to me, and it’s been interesting to think about.
Much of what I’m consumed with (at work) boils down to a question of: what is the right shape for the small but plentiful bits of writing that we are all creating daily? Here shape means largely visual representation but also sequencing and topology.
It’s a problem of hypertext, primarily. The World Wide Web established one shape for hypertext: individual pages with one-way links in the text, replacing one another in a back-forwards chain. It’s proven to be a pretty good shape, but it’s not the only one, and earlier thinkers like Engelbart and Nelson had lots of other ideas. Read the rest of this entry »
By way of introducing my favorite hypertext, I have to digress a bit. Last March [2001] I had a particularly nasty flu for about three weeks, which ended up as a wretched dry cough. I couldn’t go one minute without coughing, and I had a horrible sharp pain in my ribs caused by a sprained chest muscle. One night I was lying on the couch (so I wouldn’t keep Diana up all night) trying in vain to sleep, and decided to look up exciting Drug Facts about various medications I was taking. Guaifenasin wasn’t very interesting (just an expectorant, toxic in large doses) but Dextromethorphan turned out to be fascinating stuff – chemically similar to opiates, it suppresses coughs but doesn’t bind to any opiate receptors. What the mainstream medical websites don’t tell you, however, is that it does have very potent psychedelic effects at large doses. I’d heard about people getting stoned on cough syrup and assumed it was just codeine, but no, DXM is quite odd stuff in its own right.
Disclaimer. I would never try high doses of Dextromethorphan myself and don’t recommend that you do. Not only does chugging cough syrup sound disgusting, but DXM can become addictive, and the long term effects are known to include brain damage. Of course, that doesn’t mean we can’t safely enjoy ourselves by reading about other people’s unwise use of this interesting drug!
Tussin space. The Erowid archive has quite a lot of information about the effects of DXM, including a mention of the term _tussin space_, which refers to the sort of self-consistent alternate reality or apparent physical construct that seems to be produced by the drug. “Many find it to be a vast, open space, full of exotic and alien constructs, buildings and shapes. Some see crystalline towers and cloud-like lifeforms; others see the Tussin Space as a forest, desert, or other natural setting. Your Mileage May Vary.” I love this idea – I’m a sucker for things involving alternate realities or overarching shapes behind the world – so I googled about for other references to “tussin space” and found … Big Fun.
Big Fun was a commune of sorts inhabited by a group of young slackers in the mid ’90s. It was immortalized by one of its members, The Gus, who compiled a glossary of the group’s slang, rituals, characters and situations and posted it on the web. Glossary is of course a natural form for hypertext, and you can happily traipse around inside this one for hours on end (as I did that night). It gives a vivid kaleidoscopic image of what that scene must have been like. I have to say I would never want to live in that kind of squalor, but Gus describes it with a romanticism that makes it great fun to read and to imagine oneself in:
Despite its nearly-unlivable conditions and constant social tension, Big Fun is a place unlike any in the world. Big Fun people are full of ambition and interesting ideas, but for now they are distracted by the fragrance of life. So they stroll along for now, enjoying the considerable fun that the greater Charlottesville area has to offer. They can always buckle down and begin life in earnest when they have worked out whatever it is that is joyfully blocking them. Perhaps, as is the case for this glossary, a great creative force can be tapped from the unconventional decadence that surrounds them.
Yay for youthful idealism!
Some thoughts on hypertext. The experience of the Big Fun Glossary hypertext is quite different than that of a normal narrative. There’s definitely a story, and one could imagine a novel or film set there. But when I think of Big Fun, it’s in terms of the individual events and characters, and associations between them, not in any sort of chronological structure (I don’t think I could describe the timeline in any detail if my life depended on it!) I think this qualifies The Gus’s glossary of it as an entirely different form of literature. One could of course get the same effect on paper with a written glossary (and it’s been done: Milorad Pavic’s Dictionary Of The Khazars) but looking up references in a paper book is sufficiently awkward that this is a form that, I think, really needs the computer to be practical.
The problem of leisure What to do for pleasure Ideal love a new purchase A market of the senses—Gang Of Four, “Natural’s Not In It”
“He realizes everything he does is second-hand, a waking dream, a dream someone else has dreamed for him—when he walks into a pub and greets friends with a line he’s half-consciously lifted from last night’s sit-com, he’s an advertisement. He hears himself and he feels tricked, humiliated, so he smashes into his friends … as if he’s never seen them before but knows they mean him no good. Screaming, he insists that he’s happy.”—Greil Marcus
Like most of us, I derive much of my sense of identity from mass-marketed consumer commodities, and present my identity to other people by prominently displaying these brand names and logos. But it’s okay, because I adopt a knowing, ironic Generation-X attitude towards it.
And really, apart from Apple t-shirts (they give them to me for free) and other shirts so deeply ironic they can’t be taken seriously (Spam, Mister Bubble) the major brand names I freely advertise are those of musical groups that haven’t quite taken over the world yet. Many of them aren’t even signed to deals with multinational corporations. Yet I have bought so deeply into the pop music dream that I believe the names and identities of my favorite bands are indicative of who I am and what I stand for.
