Subject: Re: UNICODE Series

From: Ross Paterson <[email-masked]>
Date: Wed, 21 Jul 1993 14:07:44 BST

Another development that may of interest is the UCS Transmission
Formats (UTFs) that have been proposed for Unicode/ISO-10646 (UCS =
Universal Character Set, another name for ISO-10646).  These aim to
encode these wide characters as byte sequences with some degree of
backward compatability.  Now some people are using them as external
representations, e.g. in files as well as for communication.

UTF-1 (formerly called UTF) is described in an informative (non-normative)
annex to ISO-10646.  ISO-10646 characters are encoded as follows:

        ISO-10646       UTF
        ---------------------
        0000 - 009F     00-9F
        00A0 - 00FF     A0 A0-FF
        0100 - 4015     A1-F5 XX
        4016 - 038E2D   F6-FB XX XX
        038E2E and up   FC-FF XX XX XX XX

where XX means G0 (21-7E) plus G1 (A0-FF) (190 positions).  (Currently
codes are assigned only for the 16-bit subset, which is identical to
Unicode 1.1).  Some properties:
- ASCII files are still correct.
- UTF files using only Latin-1 characters are legible when interpreted
  as Latin-1 files.
- C0 (00-20: control codes and SPACE) and C1 (7F-9F) characters appear
  only as themselves, and so can be used to re-synchronize.
Disadvantages include the amount of arithmetic involved in conversion,
and the difficulty in synchronization.  For an implementation, see
metis.com:pub/utf.c.

UTF-2 (a.k.a. FSS-UTF -- File System Safe UTF) was proposed by X/Open.
It provides synchronization, simpler conversion, and 2-way ASCII
compatability, at the cost of using bytes in the range 80-9F, some
longer encodings, and the loss of the partial Latin-1 compatability.
The following description is excerpted from the comments in
metis.com:pub/utf-fss.c, written by Ken Thompson:

 * The proposed UCS transformation format encodes UCS values in the range
 * [0,0x7fffffff] using multibyte characters of lengths 1, 2, 3, 4, 5,
 * and 6 bytes.  For all encodings of more than one byte, the initial
 * byte determines the number of bytes used and the high-order bit in
 * each byte is set.  Every byte that does not start 10xxxxxx is the
 * start of a UCS character sequence.
 *
 * An easy way to remember this transformation format is to note that the
 * number of high-order 1's in the first byte signifies the number of
 * bytes in the multibyte character:
 *
 * Bits  Hex Min  Hex Max  Byte Sequence in Binary
 *   7  00000000 0000007f 0vvvvvvv
 *  11  00000080 000007FF 110vvvvv 10vvvvvv
 *  16  00000800 0000FFFF 1110vvvv 10vvvvvv 10vvvvvv
 *  21  00010000 001FFFFF 11110vvv 10vvvvvv 10vvvvvv 10vvvvvv
 *  26  00200000 03FFFFFF 111110vv 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv
 *  31  04000000 7FFFFFFF 1111110v 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv 10vvvvvv
 *
 * The UCS value is just the concatenation of the v bits in the multibyte
 * encoding.  When there are multiple ways to encode a value, for example
 * UCS 0, only the shortest encoding is legal.

Synchronization is much easier, ASCII files are still correct, and only
ASCII characters use ASCII bytes.

These properties mean that many 8-bit clean programs and subroutines that
don't place any special meaning on non-ASCII bytes (e.g. a string search
routine) will also work on UTF text without change.  And, of course, any
existing ASCII file is also a UTF file.  For these reasons, it is used
as the external encoding in Plan 9 (a research Unix system).  For more
details, see research.att.com:dist/plan9man/09utf.ps.Z.  They didn't
address the rendering of non-spacing characters (diacritics), but those
are not used in Chinese computing.

As for code length, if you compare the above descriptions with
the layout of the code space (see the Unicode books, or tables in
unicode.org:pub/MappingTables and metis.com:pub/csmaps), you get the
following character lengths in bytes:

                        now     Unicode UTF1    UTF2
----------------------------------------------------
ASCII                   1       2       1       1
Latin-based alphabets   1       2       1+      1+
Europe & Middle East    1       2       2       2
Asian alphabets         1       2       2       3
CJK Auxilliary          2       2       2       3
Hangul syllable blocks  2       2       2/3     3
Hanzi/Kanji/Hanja       2       2       3       3

The big losers here are a belt of alphabets from India to Cambodia, plus
Mongolian and alphabetic Hangul, not that these need worry CCNET readers.

So if you can stand a 50% blow-up of your hanzi text, the gain is a fairly
smooth transition to a universal character set, with more hanzi, and both
simplified and traditional characters.  I've recently hacked the X windows
client 'xedit' to handle UTF text in a Plan 9-like way.  It's certainly
a lot simpler than the current scheme for internationalization of X.
--
Ross Paterson <[email-masked]>
Department of Computing, Imperial College, London SW7