On the other hand, 8-bit encodings are troublesome. There actually is a reason why Python and Ruby are putting in all the effort to get off of 8-bit strings. UTF-8 isn’t generally too horrible, but there are too many not-uncommon text manipulations where you operate on characters and character positions, and then it becomes a pain in the ass. Worst of all, if you’re an English-speaking developer, you don’t hit most of those edge cases very soon, unless you’re very careful about I18N testing. Then your app barfs in weird ways when you get users or installations in the rest of the world.

(And using HTML-style escaping as an encoding for working with 8-bit strings in memory, as “Anonymous” suggested, is just nuts. HTML-escaped text is nasty to deal with — even simple stuff like string comparisons becomes difficult and expensive, because there are so many ways to represent any one string.)

Unicode normalization is a real pain, but that usually only matters when doing I/O and is a library issue.

I’d probably choose a canonical representation of 32 bits for a Unicode character, but perhaps that’s hopeless from a performance point of view.

(constant (global ‘☼) MAIN)

(context ‘☺)

(define (☻ ✄ ☁ ⍾)
(print ✄ ☁ ⍾))

(define (‽)
(println {‽}))

(context ☼)

(set ‘℥ “what ” ‘ᴥ “the ” ‘ᴒ “dickens”)

(☺:☻ ℥ ᴥ ᴒ)

(☺:‽)
(exit)

as for HTML… CSS sucks. period. If you think otherwise you have not had to write it and account for all the variations of browsers. CSS was meant to be written by tools not humans… that said CSS sadly is becoming the black magic of choice for some reason…

Also his arguments seems little backwards. It goes like this:

1. Python was not decided with right character representation in the beginning and it took a year to change that

2. I would not want to do that, so I will use character representation which is unlikely to satisfy most of the world population and is pretty much obsolete so I would not have the problem.

Vadim

Some are talking about string-representation, and string-access time. Others are talking about automatic io-coding. Both of these things have further questions: what to do for invalid codings, or how to handle un-normalized code points.

But even before you get into what you think the answers to these questions are, start with what exactly you need the language to help with. What part of unicode is least pleasant to deal with directly- so much so that language-support could help? Then how would you keep your new feature from pissing off the people who that doesn’t bother?

I don’t mean to suggest there aren’t answers: just that they aren’t obvious; this isn’t a “simple” thing by a long shot.