Having a need to do this recently, I lazily looked around for a script that would do the grunt-work of Python-to-Ruby translation. I couldn’t find one, so I ended up writing one myself. And I’ve uploaded it for the benefit of others who might have the same need, and who might even improve it.

So here it is: py2.rb. Use it wisely. In particular, pay attention to the caveat found in the file’s header comments:

This script just does the obvious, easy transformations, giving you more time to work on the harder ones :) It is NOT a real parser, just a bunch of kludgy regex operations, so it can’t do anything fancy. It may get some things wrong, and won’t even attempt some other things that it’s very likely to get wrong. The output will definitely have to be hand edited by someone familiar with both languages, before it can be expected to compile as Ruby, much less run correctly. The goal is simply to require less hand editing, and less mechanical replacing, than you would have had to do without this script.

It’s in the public domain. “Do what thou wilt” shall be the whole of the law.

Note: Please don’t comment with Python advocacy or, worse, anti-Ruby flames. That’s not the point. I like Python fine; I just prefer to remain promiscuous. And I think it’s ultimately in everyone’s interest for ideas to be able to flow from one form of expression to another.

My current gold standard for templaters is the Python library Genshi (which was inspired by Kid)—what I like about these is that your templates are valid XML: the parameters and control structures are expressed as special XML attributes and tags. This makes it easy to edit your templates in syntax-checking XML editors, and guarantees that your app serves valid XML or HTML.

But as I said, none of the Ruby template engines I’ve seen work this way. They’re all either generic macro systems that intermix markup and Ruby (like ERB), or they’re ruby APIs that output markup (like Builder, HAML and Markaby). I don’t really like either approach.

If you’ve got a good suggestion, I’ve got a gold star ready to stick on your forehead!

I learned things from it, as from many of the comments on my post. I even agree with most of what he says, but it hasn’t really changed my opinion, because it seems like we’re not talking about the same things.

From the standpoint of writing Python code, Python 3 sounds great. It’s so nice not to have to deal with crufty APIs that don’t make sense anymore, or half-baked mechanisms like the original string class.

But from the standpoint of using Python libraries, I still think this is terrible. Any time I want to use a library I’ll have to check whether it works with the version of Python I’m on. I may find that a library I use, and need updates to, will no longer support Python 2.x going forward. And the porting and branch-maintenance process will definitely take time out of library developers’ schedules, slowing down other work.

(Since James had several great quotes in his article, here’s one making my point: In Elbonia, cars have always driven on the left side of the road. Finally, the government decides that it’s time to harmonize this with the country’s neighbors, and announces that Elbonia will change to the more standard right-side driving. But they’re going to phase it in: next Monday, trucks will switch to driving on the right, and the Monday after that, the cars will switch. Rim-shot! The weakness of this analogy is that the sensible choice for Elbonia—making everyone switch at once—doesn’t exist for something as distributed and anarchic as a programming-language community.)

For me, the second issue predominates. I’m more of a library-user than a hardcore Python coder. In fact, on balance, I like the core Python language less than Ruby. What won me over were the libraries: there are a lot of really well-designed, well-documented, highly modular libraries out there that I’ve really enjoyed using. (Genshi, to name just one, is the best templating engine I’ve ever used.)

So my perspective of the Python 3 release is that it heralds a difficult period, where library development and maintenance may slow, and grazing the Python ecosystem will become more difficult. Hopefully this won’t last too long. But the worst-case scenario is that this forking process may discourage enough software developers to reduce the popularity of Python in the long term. That’s bad not because popularity is important in itself, but because a vibrant community is important for maintaining forward progress, and because popularity brings useful things like books and tools.

Looking at the official What’s New page, my takeaway is that the benefits are:

• A bunch of minor language features cleaned up
• Better (at least less confusing) Unicode support
• Some nice but minor new syntactic features

while the drawbacks are:

• Won’t run any pre-existing Python software. Just one minor example: the xrange() function no longer exists; instead, range() behaves the way xrange did. I’m pretty sure I’ve written something like “for i in xrange(a,b)” in every significant Python program I’ve ever written, causing all of them to break in 3.0 until I update them. Which is trivial; but there are dozens and dozens of changes like this. And many of them won’t cause easy-to-diagnose exceptions like the removal of xrange; some just cause subtle changes in behavior that will likely require debugging if they’re overlooked. (There is apparently an automated code converter, but of course it won’t fix everything.)

• As a side-effect of the above: All existing Python books are now out-of-date. (Sure, much of each book’s coverage is still accurate, but the reader who’s not a Python 3 expert already won’t know which parts are OK and which are now wrong.)

• Programs written for Python 3.0 probably won’t run in any earlier version of the interpreter, since most of the changes aren’t backward-compatible.

• Slower performance: “Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5.”

Something seems wrong here! the benefits are minor, the drawbacks are huge. Where’s the incentive to use it?

Compare this to the Ruby 1.9 story: A number of language improvements; they’re mostly backward-compatible, although with enough exceptions to break lots of large apps like Rails). But oh yeah: it’s three times as fast. That’s a very big deal, considering the poor performance of today’s Ruby (and Python). Once the 1.9 tech appears in a stable release, I am definitely upgrading.

Or compare to an operating system upgrade. How would you respond to a pitch like: “The ‘Meerkat’ release is going to be incredible! We’ve made a number of improvements to the system libraries. Of course, all existing apps will break, so you’ll have to get upgrades from the developers (which won’t run on earlier OS’s, by the way.) And afterwards you’ll notice everything running 10% slower! But we’re sure you’re going to love the enhancements to iterators and Unicode support…”

So what did I miss? I don’t mean this as an attack; I’m just genuinely confused…

Update: See also my further thoughts in response to James Bennett’s article.

Until there is more real documentation, this is a basic idea of Blocks: it is closures for C. It lets you pass around units of computation that can be executed later. For example:

void call_a_block(void (^blockptr)(int)) {
blockptr(4);
}

void test() {
int X = ...
call_a_block(^(int y){ print(X+y); });  // references stack var
snapshot

call_a_block(^(int y){ print(y*y); });
}

In this example, when the first block is formed, it snapshots the value of X into the block and builds a small structure on the stack. Passing the block pointer down to call_a_block passes a pointer to this stack object. Invoking a block (with function call syntax) loads the relevant info out of the struct and calls it. call_a_block can obviously be passed different blocks as long as they have the same type.

This is very exciting: it’s the kind of new abstraction the C family has needed for years. As you know if you’ve worked in Ruby or Python or Smalltalk or any functional language, the ability to declare an anonymous function inline, and pass it as a parameter to another function, opens the door to creating new and useful control structures. Blocks are to control structures as struct is to data structures.

For example, one of the things that blew my mind about Smalltalk-80 when I first learned it was how the basic conditional and loop operations were not hardwired into the language; instead, they were just methods of built-in classes. For example:

root := y>=0 ifTrue: [y sqrt] ifFalse: [-1]

(Apologies if I’ve messed up the syntax; it’s been a long time.) Yes, this is an if-then-else expression. But literally, it’s the message ifTrue:ifFalse: being sent to the (boolean) result of the expression y>0, with its parameters being two blocks. And what happens at runtime is that the class True implements ifTrue:ifFalse: by evaluating the first block parameter and returning its value, while the class False implements it by evaluating the second block.

This is, not surprisingly, a bit too expensive a way to implement the ubiquitous if-then-else, so in practice the compiler optimizes this expression into hardwired bytecodes instead. But it demonstrates that you can define comparable control structures yourself for your own purposes … such as for URL routing in a web application, which many Ruby and Python frameworks allow to be customized with blocks.

I think blocks will also make it possible to implement better concurrency patterns like Actors in a clean way. You may recall that I was investigating these, but after a few days I gave up because the necessary stack munging confused the Obj-C runtime too much. Once blocks are available (and hopefully integrated into Objective-C and Foundation) I’ll have to give it another try.

This turns JavaScript into a much more useable language, that I could see doing serious development in. I’m well aware that today’s JS isn’t a “toy”, that it’s got an interesting prototype-based object model under the hood; but I’ve found the language so loose as to be difficult to use. This new version adds a full class model, much better support for iteration, a form of generators/coroutines, and more.

Intriguingly, it also offers type-checking, which has been absent from most mainstream dynamic languages. I know it’s trendy to use “duck typing”, but I find that languages with no compile-time type-checking frustrate me because I make so damn many mistakes that aren’t caught until runtime; which means things I could have fixed in two minutes in C++ or Objective-C take half an hour to fix as I run the program over, and over, and over, discovering and fixing one problem at a time.

Given the concurrent developments in Apple’s JavaScript interpreter, I’m getting pretty excited for JavaScript. I hope that in addition to making it blindingly fast, Apple will also add ECMAScript 4 support, and make it easy to access the language outside of a WebView.

What I’d really like to use are Actors. In a nutshell, an Actor is an object that has its own [cooperative] thread and message queue. Actors interact by message-passing instead of shared state. The idea is to eliminate the need for standard synchronization primitives like semaphors and locks, and get rid of the race conditions and deadlocks that plague multi-threaded programs.

The page I linked to above is from Revactor, a new Actor implementation for Ruby 1.9; Actors are also built into languages like Erlang and Io.

The only hard part about implementing Actors in Obj-C appears to be the underlying dependency on coroutines. Steve Dekorte [author of Io] has a C coroutine implementation ; when I discovered that last year, I then found an Objective-C wrapper for it, but that relies on a HigherOrderMessaging library which is incompatible with OS X 10.5 and hasn’t yet been updated.

So I’ve gone DIY.

I started with Steve Dekorte’s coroutine library, but unfortunately it doesn’t build on 10.5 … and when I got it to build, I then ran into a bug in the system header … and by the time I’d worked around that, I knew the code well enough that I pretty much rewrote it. Doing this made it much shorter and clearer, since Steve’s code is cross-platform and supports three different implementations, while all I care about is OS X 10.5. I got it down to 150 lines of C.

Then I wrote a pretty simple Objective-C wrapper around it, under 200 lines. (Here’s the header.)

Come ‘n’ get it…

The work in progress is in my Mercurial repository if you want to check it out. It’s grandly titled “Actors”, but all that’s in it so far is the coroutine implementation.

Needless to say, this is currently for amusement purposes only; do not use it in real code. In particular, I strongly suspect that there are going to be problems getting Objective-C exceptions and autorelease pools to work correctly in coroutines. I’ll try to explore that next.

This is exciting news. Now, Mac OS X already has excellent Ruby support, with a complete Ruby 1.8 and Rails stack (including extras like Mongrel and Capistrano) included in the box. There’s also a glue library called RubyCocoa that bridges between the Ruby and Objective-C languages, making it easy to write all or part of a Cocoa app in Ruby (or to call useful Obj-C frameworks from a Ruby app.)

So why MacRuby? The big deal is performance. By re-implementing the Ruby object model on top of the Objective-C runtime — so all Ruby objects are bona fide Objective-C objects and vice versa — it will eliminate serious amounts of overhead:

• Parameter marshalling whenever a Ruby method has to call into Objective-C or vice versa
• Bookkeeping data structures that map between Ruby and Objective-C objects and classes
• Incompatible threading models
• Multiple garbage collectors

It also takes advantage of the major speed boost of the new bytecode interpreter in Ruby 1.9.

From my experiences working on Apple’s Java runtime, I know that bridging to and from native code is a major performance issue. The amount of overhead for a single inter-language call may not seem like much (a few microseconds), but it’s one of those “death by a thousand cuts” effects, because in an application that uses native libraries it happens so often.

(For example, one of the ways I was able to speed up AWT graphics performance by about 5x in MRJ 2.1 was to pipeline the calls, so instead of each java.awt.Graphics method calling directly into the native QuickDraw API, it just appended the parameters to a display-list array, which periodically got flushed through a single call to a native C++ method that walked the list and made all the QuickDraw calls. I didn’t initially believe this would be a big win, but Steve Zellers wrote a prototype version that proved it.)

    person.setFirstName_lastName(first,last)

    person.setFirstName first, lastName: last

    person.setFirstName first, :lastName => last

Finally, another good thing that I didn’t realize right away is that it’s an official Apple open-source project. So not only does that make it very likely that a finished version of MacRuby will be built into 10.6; it also means that, if it’s necessary to tweak the Objective-C runtime to make MacRuby work better, that’s likely to happen too. The people working on runtimes and scripting languages at Apple are really smart, and I’m glad to see they’re not resting on their laurels after all they achieved in 10.5.

My first exposure to network programming was in Java, whose approach to asynchrony is to use threads. Lots of ‘em. The API calls are [almost] all blocking, so you run them on background threads. This is good because it makes the way the API works more intuitive: you call a method, and it returns a value when it finishes. This makes your own code more intuitive, as it just performs the operations in order, like: open connection, send request, read response, parse response, close connection, return.

The downside is that making heavily threaded code work correctly can be very hard, and the problems are subtle, hard to understand and debug, and sometimes almost impossible to reproduce. Edward Lee’s paper The Trouble With Threads describes a complex Java server that was excruciatingly well-designed, code-reviewed and tested.

“No problems were observed until the code deadlocked in April 2004, four years later. Our relatively rigorous software engineering practice had identified and fixed many concurrency bugs. But that a problem as serious as a deadlock could go undetected for four years despite this practice is alarming.”

My teammates and I had similar experiences while implementing the Java AWT libraries on Mac OS 9 and 10.0. We kept running into weird concurrency bugs, fixing them (which often meant restructuring code and making it more complex), and then running into more. By the time we were done, threading didn’t seem like such a good idea anymore.

Cocoa (and the underlying CFNetwork framework) instead make the asynchrony explicit, by using an event- and callback-based model. When you invoke a network operation, the call returns immediately; but you set up a delegate object or callback function that will later be invoked when the operation completes. A fancy event loop called a “runloop” at the top level takes care of dispatching events in order. For the most part, you can write your code using only a single thread and still let your app perform tasks in parallel without blocking the UI.

The drawback is that your code gets more complex, since the high-level control flow no longer matches the semantics of the programming language. A multi-step operation now involves a separate function to implement each step, and any state has to be stored externally in instance variables. Explicit state machines often pop up. There’s a lot of boilerplate for setting delegates, adding and removing observers, parsing notification dictionaries, and so forth, which obscures the actual logic of the code.

As you can guess, I’m knee-deep in this sort of thing right now. Just like many times before, I’m wishing there were a third way. I’d like to be able to keep each operation’s flow of control simple, as with threads, but at the same time limit the interactions between operations to keep the overall flow of control from turning into race-condition spaghetti.

Oh, and I want to keep writing in Objective-C. There are intriguing multiprocessing techniques in some higher-level languages, like Lua coroutines and Io actors, but I don’t think I’m ready to start writing the core of my app in one of them. I wonder if there’s a way to implement some of those techniques cleanly in Objective-C.

I’m not offering any solutions, just describing the problem I see. Maybe I’ll get one of those great “What, you’re not using XYZ? It does exactly what you want” comments! Or maybe not. Anyway, here are some things I’ve run across that offer useful techniques:

• Higher-Order Messaging framework—adds some higher-level functional primitives for Cocoa programming, that makes delegation and callbacks simpler to use. ( Update 3PM: Appears to be incompatible with 10.5. It compiles, with warnings, but the self-tests quickly fail, with exceptions raised by NSInvocation. Dang!)

• libcoroutine, a cross-platform implementation of coroutines (similar to cooperative threads) in C. (There are several of these; this is a newish one by Steve Dekorte which, unlike others I’ve tried, works correctly on Mac OS X.)

• Actors, as found in the Io language and others. An actor is basically an object that runs a private event loop and thread: you communicate with it by sending it messages. This helps encapsulate state, since the object itself is only doing one thing at a time, and its internal state is hidden from other threads.

• Update: I forgot to list NSOperation and NSOperationQueue, new classes in OS X 10.5. These aren’t “magic” in the way the above are, they’re just a little framework for managing simultaneous tasks, but they’re useful. I’m just getting started with NSOperation: it’s what I’m using for now to manage the nasty asynchronous bits.

Got any more useful links or ideas? Post them, and I’ll add ‘em here.

“Arc only supports Ascii. MzScheme, which the current version of Arc compiles to, has some more advanced plan for dealing with characters. But it would probably have taken me a couple days to figure out how to interact with it, and I don’t want to spend even one day dealing with character sets. Character sets are a black hole. I realize that supporting only Ascii is uninternational to a point that’s almost offensive [...] But the kind of people who would be offended by that wouldn’t like Arc anyway.”

That last bit [emphasis mine] sort of flummoxed me. Is he saying that LISP only appeals to native English speakers?[1] Or that no one in their right mind would use LISP to write software for end-users?[2] Or maybe that internationalization is just some sort of abstract feel-good political-correctness issue, since none of those third-worlders even have computers anyway?[3]

He makes similarly eye-opening assertions about HTML, too. Arc has HTML-generating libraries, but they “just do everything with tables” instead of CSS. Why? Because apparently CSS-based Web designs are less agile than ones made out of tables. Somehow I don’t think most people who’ve done web design both ways would agree—those old-school layouts made with infinitely-nested tables were about as agile as a house of cards.

Anyway. Normally I’m a big slut for new programming languages, but it would probably take me a couple days to figure out Arc, and I don’t want to spend even one day on Yet Another LISP Variant that won’t even let me write code I could use in the real world.

Wait—let me not end on such a snarky note. Since I’m such a big smarty, what would I have done differently? I would simply have made the language’s “character” data type 16 bits wide instead of 8, and provided four trivial library routines to convert such strings to and from UTF-8 and CP-1252 encodings for I/O purposes. That’s about an hour’s work, and all you need for really basic Unicode support; once you have that, you can add further Unicode niceties (and there are admittedly a zillion of ‘em) a few at a time without completely breaking old code.

[1] And only those native English speakers who don’t care about foo-foo details like “curly quotes”—or emdashes—or other Arcane Symbols™. Remember, to the true old-school hacker, even lowercase letters are an inessential frill.

[2] Like, say, Reddit.com. From SecretGeek’s awesome overview of LISP:

“Reddit is proof that lisp is really powerful. Paul Graham originally wrote reddit, in lisp, on the back of a napkin while he was waiting for a coffee. it was so powerful that it had to be rewritten in python just so that ordinary computers could understand it. Because it was written in lisp it was almost no effort to rewrite the entire thing, and the rewrite was completed in-between two processor cycles.”

[3] I’m reminded of a meeting between Apple and Sun (JavaSoft) back in 1996. I was there to discuss OpenDoc and JavaBeans, but each company also had text and I18N engineers there, who were talking about Unicode and text-layout technology for the upcoming Java2D graphics engine. There was an exchange that went something like this:

Apple engineer: …and the layout needs to take into account ligatures and contextual forms, where adjacent letters change glyphs depending on neighboring characters, or even merge into a single glyph.

Sun engineer: C’mon, is this important? How many people need advanced typographic features like that, anyway?

Apple engineer: [after a pause] Well, there are over 900 million of them in India alone, and another 200 million or so in the Arabic world>