From Ricky.Yeung@Eng.Sun.COM Thu Jun 23 16:13:43 1994 Return-Path: Date: Thu, 23 Jun 1994 14:12:42 +0800 From: Ricky.Yeung@Eng.Sun.COM (Ricky Yeung) To: xiaofei@ifcss.org Cc: Ricky.Yeung@Eng.Sun.COM, lee@umunhum.stanford.edu Subject: Fung Lee's ehz-cns-big5.spec Content-Length: 8959 In the file ifcss.org:/software/soft-author-discussion/ehz-cns-big5.spec, for some reason, all the "~" characters in the document turned into the "c" characters. Could you please fix that by replacing it with a correct version? ======================================== EHZ: Extended HZ for GB, CNS and BIG5 (Preliminary Draft) Fung Fung Lee * June 18, 1994 1. Introduction Following the HZ extension framework recently proposed by Ricky Yeung, I hereby propose two specific extensions to HZ for 7-bit encoding of CNS11643-1986 and BIG5. First, let me explain the terminology. The original HZ encoding scheme I proposed in 1989 can now be referred as HZ-1989, or just HZ for short. The current extended HZ will be referred as EHZ-1994, or just EHZ for short. Possible future extensions will be designated with the "EHZ" prefix. EHZ is a superset of HZ (the set of strings that are "accepted" by the EHZ parser is a superset of that acceptable by the HZ parser). Any future EHZ should always be a superset of previous EHZ. This is possible due to the highly extensible extension framework proposed by Yeung (see his proposal for details). The subset of EHZ that deals only with ASCII and a non-ASCII code X can be referred as EHZ-X. EHZ-CNS and EHZ-BIG5 are the topics of this paper. By following the methodolgy implied in this paper, it should be relatively straightforward to define other extensions, such as EHZ-JIS, in the future, if such goal is desirable. 2. Extension Scheme To allow for direct switching from one non-ASCII mode to another non-ASCII mode, say, BIG5 Plane 1 to BIG5 Plane 2, it is most natural to choose a two-byte escape sequence with '~' as the leading byte. Motivated by the escape sequences of ISO-2022 chosen for GB and CNS, we choose the following escape sequences: ~A GB ~G CNS Plane 1 ~H CNS Plane 2 ~g BIG5 Plane 1 ~h BIG5 Plane 2 ~f BIG5 Plane 0 As BIG5 is not an ISO registered standard, no escape sequence has been pre-defined for BIG5. The sequences ~g and ~h are chosen to reflect the approximate equivalence of the CNS and BIG5 character sets. The sequence ~f is for potential user-defined BIG5 codes (as explained later). The reader is reminded that ISO-2022 is used as a reference to limit the arbitrariness of the choice of the second bytes of the two-byte escape sequences. EHZ is NOT ISO-2022. As proposed by Yeung, the default non-ASCII mode is GB, mandated by backward compatibility, i.e., HZ should be a subset of EHZ. For example, the following examples are syntactically correct EHZ strings: This is ASCII. This is GB: ~{cccccc~}. This is also GB: ~{~Acccccc~}. This is CNS: ~{~Gcccccc~}. This is also CNS: ~{~Gcccc~Hcccc~Gcccccc~}. This is BIG5: ~{~gcccccc~}. This is also BIG5: ~{~hcc~gcccccccccc~}. This is mixed: ~{gcccc~Gcccccc~gcccccc~Acccc~Hcc~hcc~Acccc~}. For CNS11643-1986 (a subset of CNS11643-1992), or CNS for short, the choice of mapping is trivial, because CNS was defined as a 7-bit code based on graphical ASCII character in the range 0x21-0x7E. There exists a CNS internal code that has a straightforward one-to-one correspodence to the CNS exchange code: setting the high bit of the first byte, and setting or not setting the high bit of the second byte depending on whether the code is in Plane 1 or Plane 2. As for BIG5, the situation is somewhat tricky, because there is no pre-defined 7-bit mapping as in the case of CNS and GB. However, a natural starting point is to find "anchor code points" using the approximate correspondence between CNS and BIG5. Three such anchor points are the first member of the common symbols (the space symbol), the first member of the frequently-used hanzi, and the first member of the less-frequently-used hanzi. This "anchoring" principle says that each of these three members should share the same 7-bit code point in both EHZ-CNS and EHZ-BIG5. Successive members of the three above ranges in BIG5 are then assigned successive code points in EHZ-BIG5, as shown in Table 1. These three regions cover all the vendor-independent common parts of BIG5. For the two vendor-defined ranges (0xA3C0-0xA3FE, 0xC6A1-0xC8FE), the BIG5 codes could not be mapped to successive EHZ-BIG5 codes, because images of codes in the former range would overlap some existing CNS symbol code range, which is undesirable, and images of codes in the later range would extend beyond the end of Plane 1 (217D7E, 0x7D7E of Plane 1). Hence, a mapping is defined so that they do not overlap any defined CNS code ranges, are packed in order and assigned integer codes smaller than those of existing CNS symbols and larger than those of hanzi. In addition, it seems nice to map 0xA3C0 to 212940, and map 0xC6A1 to 212A21, because the respective second bytes differ by 0x80. There is indeed some arbitrariness in this process. Note that this is NOT an attempt to unify CNS and BIG5. For example, the 30 CNS numerals (212621-21263E) are present in one or the other of the above two vendor-defined ranges, depending on which vendors, such as KuoChiao and ETen. Such a unification would imply some kind of vendor affinity. This kind of bias is not desirable because it is not necessary at all in the first place. As for the remaining code range (0xF9D6-0xFEFE) within 0xA140-0xFEFE, the BIG5 codes can take on successive EHZ-BIG5 codes following the mapping of 0xF9D5. The BIG5 user-defined code range 0x8140-0xA0FE is too large to fit into the remaining unused code space in Planes 1 and 2 of EHZ-BIG5, even if the range is fragmented. A new code plane, Plane 0, is needed for the completeness of the BIG5 to EHZ-BIG5 transformation, although the use of any user-defined code on Internet for open communication is discouraged in general. Table 1: EHZ mapping tables --------------------------------------------------------------------------- EHZ-GB (HZ) GB2312-1980 internal code ("high-bit set") 2121-7D7E <-> A1A1-FDFE (2nd byte range: A1-FE) in more details: 2121-297E <-> A1A1-A9FE 846 symbols (including "blank" ones) 3021-5779 <-> B0A1-D7F9 3755 frequently-used hanzi 5821-777E <-> D8A1-F7FE 3008 less-frequently-used hanzi other code points no offical glyphs assigned ----------------------------------------------------------------------------- EHZ-CNS CNS11643-1986 internal code (ETen) (CNS11643-1992 Planes 1 and 2) 212121-217D7E <-> A1A1-FDFE (2nd byte range: A1-FE) 222121-227D7E <-> A121-FD7E (2nd byte range: 21-7E) in more details: a. *212121-212570 <-> A1A1-A5F0 408 common symbols + 48 "blanks" b. 212621-21263E <-> A6A1-A6BE 30 numerals c. 212721-212939 <-> A7A1-A9B9 213 radicals f. 214221-214241 <-> C2A1-C2C1 33 other symbols g. *214421-217D4B <-> C4A1-FDCB 5401 frequently-used hanzi h. *222121-227244 <-> A121-F244 7650 less-frequently-used hanzi *227245-227246 <-> F245-F246 no official glyphs assigned other code points no official glyphs assigned ----------------------------------------------------------------------------- EHZ-BIG5 BIG5 internal code (ETen, Chinese Windows 3.1, MacOS) (2nd byte ranges: 40-7E, A1-FE) u. 202121-20564A <-> 8140-A0FE 5024 user defined symbols a. *212121-212540 <-> A140-A3BF 408 common symbols d. 212940-21297E <-> A3C0-A3FE 63 vendor defined hanzi/symbols e. 212A21-212E40 <-> C6A1-C8FE 408 vendor defined hanzi/symbols g. *214421-217D4B <-> A440-C67E 5401 frequently-used hanzi h. *222121-227246 <-> C940-F9D5 7652 less-frequently-used hanzi i. 227247-227B32 <-> F9D6-FEFE 826 vendor defined hanzi/symbols other code points unused ----------------------------------------------------------------------------- Note that the common symbols (a), the frequently-used hanzi (g), and the less-frequently-used hanzi (h) of CNS and BIG5 both map to the same code ranges by conscious design (with the minor exception of BIG5 having 2 more hanzi in region h). However, two identical code points, one from EHZ-CNS, another from EHZ-BIG5, do not necessarily correspond to the same glyph. This is because CNS was supposed to be a cleanup of BIG5, involving re-ordering of glyphs according to proper stroke counts, and removal of redundant and/or incorrect glyphs. Even ignoring the difference in the non-hanzi part and the vendor-specific part of the character sets, conversion between CNS and BIG5 is far from straightforward. ----------------------------------------------------------------------------- * Author's email address: lee@csl.stanford.edu Disclaimer: This work has no relation to my employer.