Subject: [glenn: Re: What are the Character Sets?]

From: Yolin H u a n g <[email-masked]>
Date: Sat, 30 Jan 1993 22:23:25 EST

If you haven't seen this, here is a message about several codes for Han
script.  Among these, also more on the "EUC" code asked about earlier on
CCNet.  Quite a mind-boggling plethora.  Are they really in use?  What's
their respective relation to each National Standard?  What's the
advantage?  The variable length encoding is probably not that
straightforward to program.  But, thought software workers might find
this message useful.
  Glenn posted quite a few fine articles there on Usenet.  Hope he got
enough sleep.

----- Forwarded message # 1:

From: [email-masked] (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: What are the Character Sets?
Date: 16 Jan 93 20:51:05 GMT
References: <[email-masked]>
Organization: MIT Artificial Intelligence Laboratory
Lines: 317

In article <[email-masked]> [email-masked] (Lawrence
 Crowl) writes:

>EUC

 Extended Unix Code, an ISO2022 8-bit compliant portmanteau encoding which
 is divided into four codesets (0-3), each of which serve as containers for
 different character set repertoires.

 Two encoding forms are defined in EUC systems, a variable length multibyte
 encoding intended primarily for interchange, and a fixed length encoding
 primarily intended for processing.  The variable length encoding is called
 a "multibyte character"; the fixed length encoding is called a "wide
 character," here wide means >8bits (nominally speaking).

 The multibyte encoding works as follows:  if bit7 of the first byte is
 zero, then it is a single byte encoding in codeset 0; if bit7==1 &
 byte0 != 0x8e|0x8f, then it is a multibyte encoding in codeset 1; if
 byte0 == 0x8e, then a multibyte encoding in codeset 2 follows; if
 byte0 == 0x8f, then a multibyte encoding in codeset 3 follows.  The
 single shift bytes, 0x8e and 0x8f, are called SS2 and SS3 in ISO2022
 terminology.

 Given that codesets can be mapped to different character repertoires,
 EUC is not a single character reperoire, but encodes families of repertoires;
 that is, there is not just one EUC encoding, but as many as you like.
 There are at least five common EUC encodings that I'm aware of:

 EUC-C  Chinese (PRC) Euc               Multibyte Length        Widechar Length
  codeset 0     ASCII                           1                       2
  codeset 1     GB 2312 1980                    2                       2
  codeset 2     undefined                       -                       -
  codeset 3     undefined                       -                       -

 EUC-H  Han (Taiwanese) Euc
  codeset 0     ASCII                           1                       4
  codeset 1     CNS 11643 1986 (plane 1)        2                       4
  codeset 2     CNS 11643 1986 (planes 2-255)   4*                      4
  codeset 3     undefined                       -                       -

 EUC-JX Japanese Euc
  codeset 0     ASCII                           1                       2
  codeset 1     JIS X 0208 1983                 2                       2
  codeset 2     JIS X 0201 1976                 2*                      2
  codeset 3     undefined                       -                       -

 EUC-JX Japanese Extended Euc
  codeset 0     ASCII                           1                       2
  codeset 1     JIS X 0208 1983                 2                       2
  codeset 2     JIS X 0201 1976                 2*                      2
  codeset 3     JIS X 0212 1990**               3*                      2

 EUC-K  Korean Euc
  codeset 0     ASCII                           1                       2
  codeset 1     KS C 5601 1987                  2                       2
  codeset 2     undefined                       -                       -
  codeset 3     undefined                       -                       -

 *includes one byte for the single shift code (0x8e or 0x8f)
 **recently added to EUC-J form an extended EUC-JX (I'm not sure what
   this is being called officially)

 As is clear from the above, there is not just one EUC encoding; nor is
 their one uniform set of EUC multibyte widths or one uniform wide character
 width ala Euc.  This can lead to difficulties in environments which wish
 to interoperate between different Euc platforms, especially since nothing
 in an Euc character string indicates which version of Euc applies.

>ISO 646

 =ASCII

>ISO 2022

 "Information processing -- ISO 7-bit and 8-bit coded character sets --
  Code extension techniques"

 A set of mechanisms whereby 7-bit and 8-bit encodings may be extended to
 incorporate characters not normally representable in a single 7-bit or
 8-bit encoding.

 7-bit extensions are achieved through the use of Escape (ESC), Shift-Out (SO),
 Shift-In (SI); 8-bit extensions are achieved through the use of these 7-bit
 extension mechanisms along with Locking-Shift Two (LS2), Locking-Shift Three
 (LS3), Single-Shift Two (SS2), and Single-Shift Three (SS3).  The semantics
 of these mechanisms and 37 different escape sequences are defined by this
 standard in order to achieve a generalized extension mechanism.

 The extensions supported by this standard revolve around three types of
 mechanisms: announcements, designations, and invocations.  Announcers are
 used to make explicit default assumptions and to provide information about
 what follows; designations are used to associate character repertoires
 with particular control mechanisms (invocation sequences); and invocations
 are the use of particular control or escape sequences to enable a mapping
 between a designated character repertoire and a set of codepoints (character
 code element values).

 The designation and invocation mechanism of ISO2022 is general enough to
 support multibyte encodings and arbitrary "other coding systems".  This
 "other coding system" allows for the incorporation of any kind of encoding
 system,  e.g., ISO10646 UCS2 (Unicode) or UCS4, in an ISO2022 conformant
 encoding string.  Both returnable and non-returnable other coding systems
 are supported.  [Returnable means that one can escape out of the other
 coding system without having to understand the other encoding system itself.
 ISO10646 UCS2 and UCS4 require the non-returnable mode since the means
 by which a return is implemented requires the recognition of the escape
 sequence ESC 2/5 4/0 (the three octets 0x33 0x25 0x40), e.g., this sequence
 would occur if the two ISO10646 UCS2 characters DIGIT THREE (0x0033) and
 FORMS UP HEAVY AND DOWN HORIZONTAL LIGHT (0x2540) occurred on a big endian
 machine, or if the characters SQUARED APAATO (0x3300) and HANGUL SYLLABLE
 LIEUL-YE-DIGEUD (0x4025) occurred on a little endian machine.  In contrast,
 all of the proposed UTF encoding forms of ISO10646 can use a returnable
 mode, since they preserve escape sequences.  It might be fair to say that
 an ISO2022 parser *could* be taught to recognize the invocation of certain
 other encoding systems, like ISO10646 UCS2 and UCS4; this is possible since
 the other coding system invocation sequences are known a priori.  In this
 case, the parser could switch its escape sequence parser to use 2-byte or
 4-byte encodings, by means of which it could recognize the 2-byte or 4-byte
 form of the return escape sequence: for UCS2 -- 0x0033 0x0025 0x0040; for
 UCS4 0x00000033 0x00000025 0x00000040.  The registry for ISO2022, ECMA,
 has already assigned invocation and designation sequences for ISO10646
 UCS2, UCS4, and UTF-1 encodings.]

>ISO 6937

 ISO6937-1

 "Information processing -- Coded character sets for text communication --
  Part 1: General Introduction"

 ISO6937-2

 "Information processing -- Coded character sets for text communication --
  Part 2: Latin alphabetic and non-alphabetic graphic characters"

 ISO6937-2 AD 1

 "Information processing -- Coded character sets for text communication --
  Part 2: Latin alphabetic and non-alphabetic graphic characters Addendum 1"

 ISO6937-3

 "Information processing -- Coded character sets for text communication --
  Part 3: Control Functions for Page-image Format"

 ISO6937-7

 "Information processing -- Coded character sets for text communication --
  Part 7: Greek graphic Characters"

 ISO6937-8

 "Information processing -- Coded character sets for text communication --
  Part 8: Cyrillic graphic Characters"

 A series of standards developed by ISO/TC97 which, among others, represents
 the needs of Library and Bibliographic communities.  Both 7-bit and 8-bit
 encodings are provided according to ISO2022 encoding techniques and ISO6429
 control function definitions.

 One difference from ISO6937 and other character sets like ISO8859-* is its
 explicit use graphical symbol composition by means of combining characters,
 in particular, characters which represent diacritics, accents, and other types
 of non-spacing marks.

 ISO6937 differs from ISO10646 (& Unicode) in its use of combining characters
 in two different ways:

  (1) it uses a prefix rather than a postfix encoding of combining characters;
      i.e., a combining character precedes the base character to which it
      is applied; and

  (2) it explicitly limits (and specifies) all combinations which may occur
      (ISO10646 does not limit combinations).

>ISO 8859

 A series of 8-bit character sets which conform to ISO2022 8-bit
 encoding techniques.  There are currently six parts which cover various
 Latin script based alphabets (with more on the way); there is one
 for Greek, one for Cyrillic, one for Hebrew, and one for Arabic.  All
 of these incorporate ISO646 as the G0 encodings (2/0 through 7/14).

 ISO Latin1 is another name for ISO8859-1.

>ISO 10646

 "Information technology -- Universal Multiple-Octet Coded Character
  Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane"

 accepted in June 1992, to be published in 1993.

 Basic multilingual plane (BMP) is derived from Unicode 1.0 with some minor
 additions and changes.  Defines two encoding forms UCS2 and UCS4, for
 universal character set 2-byte form, and universal character set 4-byte
 form.  UCS2 is equivalent to UCS4 with the upper 16 bits equal to zero
 in the UCS4 form.  Only 31-bits of UCS4 are usable for encoding, the MSB
 is reserved and MBZ.

 The goal of the ISO10646 basic multilingual plane is to support the
 encoding of of written languages on a world-wide scale.

 No characters are currently defined in ISO10646 outside of the basic
 multilingual plane.

>JIS

 One of any number of Japanese Industrial Standards produced by the
 Japanese Standards Association, such as JIS X 0201, JIS X 0208, JIS X 0212.
 JIS X 0201 (1976) defines two 7-bit character sets and an 8-bit character
 set.  The first 7-bit character set is equivalent to ASCII except for code
 position 5/12, which has the YEN SIGN instead of BACKSLASH and code
 position 7/14, which has an OVERSCORE isntead of TILDE; the second 7-bit
 set contains Katakana symbols in what is called the "hankaku" or half-
 width form (as compared with the normal width of a Kanji glyph, which
 is called "zenkaku," or full-width form).  The 8-bit set is the union
 of the two 7-bit sets, with the hankaku katakana located in the G1
 space (10/0 through 15/15) of which only 10/1 through 13/15 are used.

 JIS X 0208 (1983) defines a set of 6353 Kanji characters, 169 Kana
 characters, 166 Latin, Greek, and Cyrillic characters, 10 numeral
 characters, 147 special characters, and 32 ruled line element characters,
 a total of 6877 characters.  The characters are organized according to
 "wards" which are blocks of 94 characters that are easily mapped into
 the ISO2022 G0 code set (2/1 through 7/14 = 94 codepoints).  Kanji
 characters in this standard are divided into two sets: 2965 level 1
 Kanji, and 3388 level 2 Kanji.  The two different levels are ordered
 according to different principles:  level 1 is ordered phonetically,
 level 2 is ordered by radical and stroke count, with ties in radical
 and stroke count broken by onyomi phonetic order.  The phonetic order
 applied to level 1 Kanji operates by choosing a representative "On-Kun"
 an onyomi (Japanized Chinese pronunciation) or kunyomi (indigenous
 Japanese pronunciation); ties in "On-Kun" are broken by ordering by
 onyomi alone followed by kunyomi alone; further ties are broken by
 radical order then stroke count.  All JIS X 0208 require two bytes
 for their encoding; JIS X 0208 is designated and invoked in an 7-bit
 ISO2022 conformant string by the escape sequence ESC 2/4 4/2; in an
 8-bit ISO2022 conformant string, it is designated by the same escape
 sequence, and invoked by Shift-In (SI).

 JIS X 0212 (1990) defines a set of nearly 6,000 additional kanji which
 were not included in JIS X 0208.  These kanji are ordered like those
 of level 2 in JIS X 0208, i.e., by radical, stroke count, then onyomi.

>Runic

 A cute name given by AT&T Plan 9 designers to ISO10646 UCS2 (Unicode)
 fixed width 16-bit encodings.  These 16-bit encodings are there called
 "runes".  [I really hate cute names that give absolutely no information
 and are misleading.  In my opinion, this term should not be used for
 this purpose, particularly since ISO10646 UCS2 (Unicode) will eventually
 have the actual Rune characters included.]

>UCS-2
>UCS-4

 See above under 10646.

>UTF

 UCS Transformation Format.  The first such format, defined in Annex F
 of DIS 10646-1.2:1992(E).  Nowadays called UTF-1 to distinguish it from
 other transformation formats.

>UTF-2

 File System Safe UTF (FSS-UTF), an alternative transformation format defined
 by Ken Thompson, and used in the Plan 9 operating system.

>Unicode

 A character set defined and promoted by Unicode, Inc., a non-profit
 consortium of computer hardware and software manufacturers, library
 and bibliographic organizations, information service providers, and
 individual members.  Unicode is derived from earlier work in the design
 of multilingual software support done by Xerox and Apple Computers.

 The Unicode Worldwide Character Set Standard, Version 1.0, Volume 1 was
 published by Addison Wesley in the fall of 1991 at a time when the first
 draft of DIS10646 was being voted on; the vote failed and discussions of
 a merger began, with the subsequent production of the second draft of
 10646 in December 1991 which used Unicode 1.0 as a basis for the basic
 multilingual plane.  Volume 2 of the Unicode standard, containing the
 Han character encodings of Unicode 1.0, was published by Addison Wesley
 in late summer 1992.  Subsequent to the successful vote for the second
 draft of DIS10646 in June 1992, Unicode 1.0.1 was published as an addendum
 to Unicode 1.0; Unicode 1.0.1 is a proper subset of the ISO10646 basic
 multilingual plane.  Unicode 1.1 is scheduled for release in the near
 future as a technical report which makes Unicode code for code identical
 to the basic multilingual plane of ISO10646.

 Unicode has the goal of encoding the characters needed to support the
 representation of all the written languages of the world, including those
 no longer written today.  It also had the goal of supporting interoperability
 with imporant existing character sets; consequently, Unicode is a superset
 of most character sets used in computer systems today worldwide.  As
 defined by the Unicode standard, a character code element constitutes a
 16-bit unsigned integral value in the range 0x0000 through 0xffff; somewhat
 more than half of these 65,536 code elements are now assigned to characters
 from nearly 25 different scripts.  A small number of scripts are not yet
 represented by Unicode, but have draft encodings which are now being
 reviewed by concerned parties:  Burmese, Ethiopic, Khmer, Sinhalese,
 and Tibetan (the Tibetan encodings in Unicode 1.0 will be removed by
 Unicode 1.1 because they were removed from ISO10646 pending further study).
 A number of other scripts used by relatively small communities or by
 historical writing systems, such as the vertical Mongolian script, Cree,
 Cherokee, Egyptian, etc., are being studied for further inclusion in a
 future version of Unicode.

 Two important features of Unicode are its unification of Chinese, Japanese,
 and Korean uses of Han characters (hanzi/kanji/hanja) into a single collection
 of unified CJK characters, and its support for the arbitrary combination of
 non-spacing marks in order to create novel or unusual (i.e., hard to
 ascertain) graphical symbol composites (an extremely imporant feature for
 bibliographic uses and for representing minority alphabets).

Glenn Adams
Cambridge, Massachusetts

----- End of forwarded messages