Unicode is a standard for coding multingual text. ISO has standardized a portion of Unicode as ISO646
Unicode is complex. Whereas ASCII defines 127 characters, Unicode defines 1,114,112 code points, and characters are composed of one or more code points. Unicode provides code charts, but doesn't stop there. The following things are also specified by Unicode:
RS: Until version 3.0, 16 bits (\u0000-\uFFFD: the "Basic Multilingual Plane", BMP) were sufficient for any Unicode. From 3.1, we must expect longer codes - up to 31 bits long, as specified in ISO 10646. Why 31 bits? Because that is the maximum that can be expressed in UTF-8: 6 bytes, omitting the taboo values \xFE and \xFF.
1111110a 10aaaaaa 10bbbbbb 10bbcccc 10ccccdd 10dddddd
, where small letters stand for "payload" bits of bytes a..d, highestmost has only 7 bits
comp.lang.tcl 2008-04:
Newsgroups: comp.lang.tcl From: [email protected] Date: Sat, 26 Apr 2008 11:55:45 -0700 (PDT) Local: Sat, Apr 26 2008 2:55 pm Subject: unicode - get character representation from \uxxx notation Hello, to show my problem see the following example: > set tcl_patchLevel 8.5.3b1 > set str "n\u00E4mlich" nämlich > set c 0xE4 > set str "n\\uformat %04.4X $chmlich" n\u00E4mlich How do I get the \u00E4 in the character representation let's say iso8859-1 ? > encoding convertto iso8859-1 $str Newsgroups: comp.lang.tcl From: [email protected] Date: Sat, 26 Apr 2008 14:21:27 -0700 (PDT) Local: Sat, Apr 26 2008 5:21 pm Subject: Re: unicode - get character representation from \uxxx notation To convert the hex number expressed as a string 0x00e4 to a Unicode character, use: format "%c" 0x00e4 You can then use encoding convertto to convert this to another encoding, e.g.: encoding convertto iso8859-1 format "%c" 0x00e4
LV 2008-07-08:
I've a request from a developer concerning whether Tcl is capable of handling characters larger than the Unicode BMP. His application was using tdom and it encountered the 𝒜 character, which is a script-A, unicode value 0x1D49C, which tdom reports it can't handle because it is limited to UTF-8 chars up to 3 bytes in length.
What do Tcl programmers do to properly process the longer characters?
Note this is in an enterprise setting. Finding a solution is critical in the publishing (web or print) arena.
RS 2008-07-09: Unicode out of BMP (> U+FFFF) requires a deeper rework of Tcl and Tk: we'd need 32 bit chars and/or surrogate pairs. UTF-8 at least can deal with 31-bit Unicodes by principle.
LV During July, 2008, there was some discussion in the TCT mailing list [L1 ] (let's see how long that URL lasts...) about ways that the Tcl code itself could evolve to handle things better. But for right now, users have to face either dealing with their wide unicode via a different programming language in some way (whether converting wide characters to some other similar character, using some sort of macro representation, etc.)