[This file was generated from the HTML document "http://ifcss.org:8001/www/pub/software/info/cjk-codes/".] NOTES ON CJK CHARACTER CODES AND ENCODINGS The character codes covered are: Big-5 A code for traditional Chinese, widely used in Hong Kong, Taiwan and elsewhere. 94x94 Character Sets These codes represent each character by a pair of numbers 1-94 (decimal), called row and column respectively ("kuten" in Japanese, "qu1wei4" in Chinese). The idea is to map onto the 94 graphic ASCII bytes, by adding 0x20 to each (0xA0 for shifted ASCII). Chinese - GB Traditional Chinese - CNS Japanese - JIS Korean - KSC Encodings CCCII and ANSI Z39.64-1989 (EACC) Two closely related 3-byte codes, intended to cover all CJK languages. Unicode/ISO-10646 A new code covering most of the world's languages. Encodings SEE ALSO International Standard--Information Processing-- ISO 7-bit and 8-bit coded character sets--Code extension techniques, ISO 2022:1986. (= ANSI X3.41, JIS X0202, BS 6856, FIPS 35, ECMA 35) Compound Text Encoding, Version 1.1 [ftp: ftp.cc.utexas.edu source/X11R6/xc/doc/hardcopy/CTEXT/ctext.PS.Z], Robert W. Scheiffer, 1989 (Also in TROFF/TBL format [ftp: ftp.cc.utexas.edu source/X11R6/xc/doc/specs/CTEXT/ctext.tbl.ms]). Electronic Handling of Japanese Text(japan.inf), Ken Lunde, 1992. Mostly English, with some Japanese, in the following formats: ISO-2022-JP [ftp: ftp.ora.com pub/examples/nutshell/ujip/doc/japan.inf-032092.jis]. Shift-JIS [ftp: ftp.ora.com pub/examples/nutshell/ujip/doc/japan.inf-032092.sjs]. EUC-J [ftp: ftp.ora.com pub/examples/nutshell/ujip/doc/japan.inf-032092.euc]. Understanding Japanese Information Processing, Ken Lunde, O'Reilly & Associates, Inc, 1993. International Standard--Information Technology-- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane, ISO/IEC 10646-1:1993. The Unicode standard : worldwide character encoding(vols 1 & 2) the Unicode Consortium, Reading MA : Addison-Wesley, 1991-1992. ISBN: 0-201-56788-1 (pbk. : v. 1), 0-201-60845-6 (pbk. : v. 2) LC: QA76.76.C53U6 Hello World [ftp: ftp.research.att.com dist/plan9man/09utf.ps.Z], Rob Pike and Ken Thompson, Usenix Conference, Jan 1993. Unicode: A Universal Character Code, Jürgen Bettels and E. Avery Bishop, Digital Technical Journal 5:3 (Sep 1993) 21-31 (also in plain text format [ftp: gatekeeper.dec.com pub/DEC/DECinfo/DTJ/I18N-unicode.txt]). T Comments on this document are most welcome. Thanks to:Hu Bo, Nelson Chin, Ken'ichi Handa, Ken Lunde, Man-Chi Pong and Bob Sandstrom. Last modified: 15th February 1995. Ross Paterson Big-5 (Traditional Chinese) This is a 2-byte code, designed to represent traditional Chinese characters. The first byte is A1-F9; the second byte 40-7E,A1-FE. The code ranges are: A140-A24E punctuation and other symbols A24F-A258 various units A259-A261 Chinese units A262-A2AE box-drawing pieces and shapes A2AF-A2B8 Arabic numerals A2B9-A2C2 Roman numerals A2C3-A2CE Hangzhou numerals A2CF-A2E8 Latin capital letters A2E9-A343 Latin small letters A344-A35B Greek capital letters A35C-A373 Greek small letters A374-A3BA zhuyin symbols A3BB-A3BF zhuyin diacritics A440-C67E frequently used hanzi (5401) C6A1-C6F7 hiragana C6F8-C7B0 katakana C7B1-C7E8 Cyrillic letters C7E9-C7F2 circled numbers C7F3-C7FC parenthesized numbers C940-F9D5 less frequently used hanzi (7652) Each of the hanzi blocks is ordered by total number of strokes, then Kangxi radical no. A variant is the Taiwan standard CNS-11643. The font HKU-Ch16 (a.k.a. chinese.16) uses a slight variant: the less frequent hanzi are moved down to occupy C6A1-F755. The usual encoding is a simple interleaving with plain US-ASCII text. Relationship between Big5 and CNS 11643-1986 The Big5 non-hanzi and level 1 hanzi correspond to CNS level 1: A140-A1F5 <-> 0101-0254 A1F6 <-> 0256 A1F7 <-> 0255 A1F8-A2AE <-> 0257-0346 A2AF-A3BF <-> 0401-0580 # digits, Latin, Greek, bopomofo **** <- 0701-0925 # Kangxi radicals **** <- 3401-3433 # symbols for control characters A440-ACFD <-> 3601-5102 # start of hanzi level 1 ACFE <-> 5551 AD40-AFCF <-> 5103-5550 AFD0-BBC7 <-> 5552-7547 BBC8-BE51 <-> 7549-7959 BE52 <-> 7548 BE53-C1AA <-> 7960-8520 C1AB-C2CA <-> 8522-8722 C2CB <-> 8521 C2CC-C360 <-> 8723-8812 C361-C3B8 <-> 8814-8867 C3B9 <-> 8869 C3BA <-> 8868 C3BB-C455 <-> 8870-8965 C456 <-> 8813 C457-C67E <-> 8966-9343 # end of level 1 C6A1-C7E8 -> ****** # kana, Cyrillic (different in ETEN) C7E9-C7FC <-> 0601-0620 # enclosed numerals (different in ETEN) Big5 level 2 hanzi correspond to CNS level 2: C940-C949 <-> 0101-0110 # start of level 2 C94A -> 3634 # duplicate of A461 C94B-C96B <-> 0111-0143 C96C-C9BD <-> 0145-0192 C9BE <-> 0144 C9BF-C9EC <-> 0193-0244 C9ED-CAF6 <-> 0246-0424 CAF7 <-> 0245 CAF8-D6CB <-> 0425-2378 D6CC <-> 3067 D6CD-D779 <-> 2380-2493 D77A <-> 3174 D77B-DADE <-> 2494-3066 DADF <-> 2379 DAE0-DBA6 <-> 3068-3173 DBA7-DDFB <-> 3175-3603 DDFC -> 3386 # duplicate of DCD1 DDFD-E8A2 <-> 3604-5342 E8A3-E975 <-> 5344-5501 E976-EB5A <-> 5503-5807 EB5B-EBF0 <-> 5809-5930 EBF1 <-> 5343 EBF2-ECDD <-> 5931-6073 ECDE <-> 5502 ECDF-EDA9 <-> 6074-6183 EDAA-EEEA <-> 6185-6424 EEEB <-> 6815 EEEC-F055 <-> 6425-6634 F056 <-> 6184 F057-F0CA <-> 6635-6722 F0CB <-> 5808 F0CC-F162 <-> 6723-6814 F163-F16A <-> 6816-6823 F16B <-> 7165 F16C-F267 <-> 6824-6982 F268 <-> 7320 F269-F2C2 <-> 6983-7044 F2C3-F374 <-> 7046-7164 F375-F465 <-> 7166-7319 F466-F4B4 <-> 7321-7365 F4B5 <-> 7045 F4B6-F4FC <-> 7366-7442 F4FD-F662 <-> 7444-7649 F663 <-> 7443 F664-F976 <-> 7650-8169 F977-F9C3 <-> 8171-8219 F9C4 <-> 8170 F9C5 <-> 8220 F9C6 <-> 8232 F9C7-F9D1 <-> 8221-8231 F9D2-F9D5 <-> 8233-8236 GuoBiao (Chinese) Guo2jia1 Biao2zhun3(Chinese for "national standard", abbreviated Guo2Biao1 or GB) refers to standards promulgated by the PRC. The following character sets are included: THE BASIC SET: GB 2312-80 GB 2312-80 (GB0, GB)is designed to represent simplified Chinese characters, and is a national standard in the PRC. It is also used in Singapore. The rows are: 01 special symbols (94) 02 paragraph numbers (72) 03 GB 1988-80 (94, = ISO 646-CN) 04 hiragana (83) 05 katakana (86) 06 Greek (48) 07 Cyrillic (66) 08 pinyin accented vowels (26) and zhuyin symbols (37) 09 box and table drawing pieces (76) 16-55 hanzi level 1 (3755, ordered by pinyin) 56-87 hanzi level 2 (3008, ordered by radical, then stroke) The GB 1988-80 set is the Chinese extension of ISO 646. It differs from US-ASCII only in having a yuan/yen symbol in place of the dollar sign (24) and an overline in place of the tilde (7E). VARIATIONS AND EXTENSIONS OF GB 2312-80 GB 6345.1-86 is a revision of GB 2312-80: The form of "g" (row 3 cell 71) is changed to the open form. Six pinyin letters are added (row 8, cells 27-32). Row 10 is added as a half-width version of row 3. Row 11, cells 1-32 is added as a half-width version of the corresponding positions of row 8. The form of one hanzi (row 79 cell 81) is altered. Further details are available in English [ftp: ftp.ifcss.org pub/software/fonts/gb/misc/gb6345.txt] and Chinese [ftp: ftp.ifcss.org pub/software/fonts/gb/misc/gb6345.gb] (encoded in GB 2312-80). GB 8565.2-89 (GB8) is intended as an extension of GB 2312-80, and contains additional simplified hanzi in the following rows: 12 hanzi (92, from GB 7590-87, ordered by radical) 13 hanzi (50, from GB 7589-87, ordered by radical) 15 date and time symbols (69) and miscellaneous hanzi (24, ordered by radical) 90-94 hanzi (470, from GB 7589-87, ordered by radical) The character set ISO-IR-165 comprises GB 2312-80, as modified by GB 6345.1-86 (except for the alteration to row 79, cell 81), and GB 8565.2-89, plus 44 hanzi added to row 13 row 14, containing 92 hanzi 3 more hanzi? 22 background characters added to row 6 (cells 60-81) GB 12345-90 (GB1) is a code for traditional hanzi, parallel to GB 2312-80 with the following additional rows: 88-89 hanzi level 3: an extra 103 characters so that the traditional counterparts of GB 2312-80 level 1 hanzi are uniquely represented (103, ordered by pinyin). Further information is available on the mapping between the two sets. OTHER SETS Some other sets have been defined, but (like GB1) are not widely supported: GB 7589-87 (GB3) 7237 simplified hanzi in radical order (rows 16-92) GB 7590-87 (GB5) 7039 simplified hanzi in radical order (rows 16-90) General Purpose Han Characters for Modern Chinese (GB7) 43 simplified hanzi (row 1) GB 12052-89 (Korean) 94 Korean Idu characters (row 71 cells 89-94 and row 72 cells 1-88) GB 13131-94 traditional version of GB 7589-87 GB 13132-94 traditional version of GB 7590-87 SPECIAL VERSIONS USED IN ISO/IEC 10646 ISO/IEC 10646 (and the Unicode cross-reference tables) refer to a number of codes in GB sets beyond GB 2312-80 that are not part of the official standards, but were added by China solely for this purpose. They include: GB 12345-90 additional rows: 92 Hong Kong hanzi (58) 93 Korean Idu characters (94, from GB 12052-89) GB 8565.2-89 The characters added for ISO-IR-165, plus 8 hanzi (?) in row 8. Further, the versions of GB 7589-87 and GB 7590-87 submitted by China for this purpose used the traditional forms of the hanzi. Mapping from GB 12345-90 to GB 2312-80 In almost all cases, for each codepoint in GB 2312-80, the corresponding codepoint in GB 12345-90 contains the most common traditional form of the GB 2312-80 character, so that the mapping of characters in levels 1 and 2 is trivial. The exceptions are three instances where GB2312-80 includes (in level 2) a traditional form corresponding to a simpler character in level 1. In each of these cases, the simpler character was a rare character that was substituted for a common character with more strokes. In GB 12345-90, these pairs are interchanged, so that the more common traditional form occurs in level 1. In mapping from GB 12345-90 to GB 2312-80, both forms should be mapped to the simplified form in level 1: 1-6565 -> 0-2683 # hou4 "empress" vs. "behind" 1-6637 -> 0-5164 # yu2 "I, me" vs. "remainder" 1-8765 -> 0-3520 # yao1 "one, small" vs. me5 (question particle) GB 12345-90 also adds a level 3, containing additional variants (103) of some level 1 characters used in the traditional orthography. Their level 1 counterparts are: 1-8801 -> 0-1658 1-8836 -> 0-2910 1-8871 -> 0-4165 1-8802 -> 0-1669 1-8837 -> 0-3001 1-8872 -> 0-4285 1-8803 -> 0-1777 1-8838 -> 0-3061 1-8873 -> 0-4341 1-8804 -> 0-1780 1-8839 -> 0-3077 1-8874 -> 0-4353 1-8805 -> 0-1823 1-8840 -> 0-3143 1-8875 -> 0-4408 1-8806 -> 0-1837 1-8841 -> 0-3168 1-8876 -> 0-4408 1-8807 -> 0-1907 1-8842 -> 0-3207 1-8877 -> 0-4408 1-8808 -> 0-1969 1-8843 -> 0-3215 1-8878 -> 0-4419 1-8809 -> 0-1983 1-8844 -> 0-3216 1-8879 -> 0-4531 1-8810 -> 0-1986 1-8845 -> 0-3259 1-8880 -> 0-4537 1-8811 -> 0-2117 1-8846 -> 0-3279 1-8881 -> 0-4582 1-8812 -> 0-2119 1-8847 -> 0-3290 1-8882 -> 0-4721 1-8813 -> 0-2177 1-8848 -> 0-3317 1-8883 -> 0-4721 1-8814 -> 0-2212 1-8849 -> 0-3343 1-8884 -> 0-4744 1-8815 -> 0-2223 1-8850 -> 0-3417 1-8885 -> 0-4782 1-8816 -> 0-2281 1-8851 -> 0-3462 1-8886 -> 0-4875 1-8817 -> 0-2302 1-8852 -> 0-3525 1-8887 -> 0-5009 1-8818 -> 0-2322 1-8853 -> 0-3541 1-8888 -> 0-5022 1-8819 -> 0-2365 1-8854 -> 0-3541 1-8889 -> 0-5184 1-8820 -> 0-2420 1-8855 -> 0-3541 1-8890 -> 0-5189 1-8821 -> 0-2441 1-8856 -> 0-3554 1-8891 -> 0-5185 1-8822 -> 0-2540 1-8857 -> 0-3570 1-8892 -> 0-5224 1-8823 -> 0-2546 1-8858 -> 0-3579 1-8893 -> 0-5238 1-8824 -> 0-2567 1-8859 -> 0-1757 1-8894 -> 0-5264 1-8825 -> 0-6756 1-8860 -> 0-3827 1-8901 -> 0-5402 1-8826 -> 0-2690 1-8861 -> 0-3830 1-8902 -> 0-5427 1-8827 -> 0-2714 1-8862 -> 0-3843 1-8903 -> 0-5427 1-8828 -> 0-2756 1-8863 -> 0-3845 1-8904 -> 0-5434 1-8829 -> 0-2767 1-8864 -> 0-3851 1-8905 -> 0-5438 1-8830 -> 0-2781 1-8865 -> 0-3907 1-8906 -> 0-5454 1-8831 -> 0-2802 1-8866 -> 0-3909 1-8907 -> 0-5476 1-8832 -> 0-2824 1-8867 -> 0-4743 1-8908 -> 0-5494 1-8833 -> 0-2850 1-8868 -> 0-3979 1-8909 -> 0-5528 1-8834 -> 0-2859 1-8869 -> 0-3990 1-8835 -> 0-2886 1-8870 -> 0-4023 Since the most common traditional counterpart occurs in the position of each GB 2312-80 character, in some cases a GB 2312-80 character has been moved to level 3, while the original position is occupied by a more complex character. CNS (Traditional Chinese) The Taiwan standard CNS 11643-1986 is a pair of 94x94 planes (or levels) derived from Big-5 and another 94x94 plane of user characters (rows 1-71), called plane 14. It may also be treated as a 3-byte 94-character code. Plane 1 comprises 1-3 punctuation and other symbols (Big-5 characters A140-A2AE one minor change) 4-5 various numerals and assorted alphabets (Big-5 characters A2AF-A3BF) 6 enclosed numerals (Big-5 characters C7E9-C7FC) plus small Roman numerals 7-9 Kangxi radicals 34 symbols for control characters 36-93 frequently used hanzi (Big-5 characters A440-C67E) Plane 2 is 1-82 less frequently used hanzi (BIG-5 characters C940-F9D5 except C94A and DDFC, eliminated as duplicates of A461 and DCD1 respectively) The hanzi planes are ordered by stroke count and then Kangxi radical as in Big-5, but a few of the counts or radicals have been corrected (and the characters relocated) so the mapping is not continuous. The revised code CNS 11643-1992 carries over planes 1 and 2 unchanged. Other planes are: 3 6148 hanzi from plane 14 of CNS 11643-1986 (rows 1-66) 4 7298 rarely used hanzi, including some used in names, 171 more hanzi from plane 14 of CNS 11643-1986 and hanzi in ISO/IEC 10646-1:1993 not in previous planes (rows 1-78) 5 8603 even rarer hanzi (rows 1-92) 6 6388 variant forms (yi4ti3zi4) with up to 14 strokes (rows 1-68) 7 6539 variant forms (yi4ti3zi4) with 15 or more strokes (rows 1-70) Hanzi in the last 3 planes were supplied by the Ministry of Education. An eighth plane of about 7000 characters is in preparation. JIS (Japanese) The first of the 94x94 codes was the Japanese standard JIS C 6226-1978 (Old-JIS). This was re-arranged to accommodate the 1981 Jouyou Kanji list (all of which are in level 1), resulting in JIS C 6226-1983 (New-JIS, renamed JIS X0208-1983 in 1987). Two kanji have since been added to level 2, giving the current JIS X0208-1990: 01-02 punctuation, symbols 03 ISO 646 (alphanumerics only) 04 hiragana 05 katakana 06 Greek 07 Cyrillic 08 line drawing 16-47 kanji level 1 (2965, ordered by on-yomi) 48-83 kanji level 2 (3384, ordered by Kangxi radical, then stroke) 84 miscellaneous kanji (6) [This seems an appropriate place to mention JIS X0201-1976(called JIS C 6220 until 1987), an 8-bit code. Its left half (JIS-Roman) is the Japanese national variant of ISO 646 and differs from US-ASCII only in having a yen symbol in place of the backslash (5C) and an overline in place of the tilde (7E). Its right half (A1-DF) comprises Japanese punctuation and katakana.] An extension is defined by JIS X0212-1990: 02 more punctuation, symbols 06 accented Greek 07 non-Russian Cyrillic 09 extended Latin 10 uppercase accented Latin 11 lowercase accented Latin 16-77 kanji level 3 (5801, ordered by Kangxi radical, then stroke) The non-kanji parts of JIS X0208-1990 and JIS X0212-1990 occupy disjoint code ranges. KSC (Korean) The Korean standard KSC 5601-1987(formerly KIPS, updated in 1989 and 1992) has the following rows: 01-02 punctuation, symbols 03 KSC 5636 (ISO 646-KR) 04 alphabetic Hangul 05 roman numbers, Greek 06 line drawing 07 various units 08-09 miscellaneous 10 hiragana 11 katakana 12 Cyrillic 16-40 Hangul syllable blocks (2350) 42-93 hanja (4888, ordered by Hangul, then radical) The KSC 5636 set is the Korean extension of ISO 646. It differs from US-ASCII only in having a Won sign in place of the backslash (5C). The few hanja that have more than one pronunciation have extra codes, so that the 4888 hanja codes cover 4620 unique hanja. The revisions did not alter the code assignments, only some annexes relating to alternative coding methods. Annex 3 of KSC 5601-1992 describes the 2-byte johab-hyeong code for modern Hangul syllables. This is a 16-bit code, with the top bit usually set, and then 5 bits each for initials (choseong) 14 simple consonants + 5 "double" consonants vowels (jungseong) 10 simple vowels + 10 complex vowels finals (jongseong) 14 simple consonants + 13 clusters + 1 filler giving a total of 11,172 possible syllable blocks. Annex 4 describes a one-byte code for internal use. KSC 5657-1991 has the following rows: 16-36 Hangul syllable blocks, level 1 (1930) 37-54 Hangul syllable blocks, level 2 (1676) 55-85 hanja (2856) Encodings of 94x94-character sets One popular way of mixing US-ASCII and a 2-byte set is to set the high bit on the first byte of 2-byte characters. Some schemes also set the high bit on the second byte. An example is the scheme commonly called Shift-GB. Note that this is quite different from Shift-JIS (MS Kanji). There are schemes that use ASCII sequences to switch between ASCII and GB modes to accommodate both sets in graphic ASCII text. For example, the HZ scheme used on the Usenet group alt.chinese.text wraps unshifted GB text in the strings "~{" and "~}". For more details, see the HZ package [ftp: ftp.ifcss.org pub/software/unix/convert/HZ-2.0.tar.gz]. ISO 2022 CODING ISO 2022:1986ISO 7-bit and 8-bit Coded Character Sets - Code Extension Techniquesprovides several ways of encoding n-byte 94- or 96-character sets, using escape sequences. It uses a two-stage mapping: Graphic character sets are assigned to four character set slots, called G0, G1, G2 and G3. G0 may hold a 94-character set, while the others may hold either a 94- or a 96-character set. Initially G0 contains ISO 646 IRV(1983), and G1-3 are undefined. (This set differs from US-ASCII only in that it has a currency symbol in the place of the dollar sign; this difference is often ignored.) These assignments may be changed with designator sequences. These slots are assigned to two byte ranges: GL (graphic left): byte values 20-7F for 96-character sets, 21-7E for 94-character sets (in which case SPACE and DEL are defined). GR (graphic right): byte values A0-FF for 96-character sets, A1-FE for 94-character sets. Initially G0 is assigned to the GR range and G1 to the GL range. These assignments may be changed with shift sequences: A single shift changes an assignment for a single character (one or more bytes). A locking shift makes a permanent assignment. Some subset implementations are widely used. 7-BIT ENCODINGS (USING GL ONLY) JUNET or ISO-2022-JP [ftp: ftp.internic.net rfc/rfc1468.txt] Designator sequences are used to switch G0 between 94-character sets. The guidelines explicitly require switching back to an ISO 646 variant (such as US-ASCII or JIS-Roman) before the end of each line. This scheme is widely used in Japan. ISO-2022-JP-2 [ftp: ftp.internic.net rfc/rfc1554.txt] An extension of the previous scheme, using designator sequences to switch G2 between 96-character sets, and single shifts to include characters from these sets. ISO-2022-KR [ftp: ftp.internic.net rfc/rfc1557.txt] A designator sequence is used to assign the KSC 5601-1987 set to G1, and locking shifts to switch GL between G0 and G1. ISO-2022-INT-1 [ftp: ftp.internic.net internet-drafts/draft-ohta-text-encoding-01.txt] A mixture of the previous two. G1 is used for KSC 5601-1987 and 96-character sets, G0 for all other sets. ISO-2022-CN A proposed scheme combining GB and CNS sets using designator sequences and locking shifts. See the related mailing list [ftp: apng.org apng/mail.archive/apng-cc]. 8-BIT ENCODINGS (USING BOTH GL AND GR) Extended Unix Code (EUC) A locale-dependent fixed codeset assignment is assumed, and single shifts are used to access G2 and G3. It is common practice to leave US-ASCII in G0. The X Windows Compound Text Encoding [ftp: ftp.cc.utexas.edu source/X11R6/xc/doc/hardcopy/CTEXT/ctext.PS.Z] Only G0 and G1 are used, locked into GL and GR respectively. Designator sequences are used to switch character sets. "Left half" sets should be used in G0 only, "Right half" sets in G1 only, others (including the 2-byte CJK sets) in either position. (There is also provision for non-ISO-2022 character sets.) For newer schemes, the convention appears to be: G0 US-ASCII. G1 Primary non-Latin character set (usually a 2-byte set in the CJK context). G2 Supplementary Latin-based character set (or, in the CJK context, an auxiliary 1-byte set). G3 Auxiliary non-Latin character set (usually a 2-byte set in the CJK context). For many mono-lingual (or mono-lingual+English) applications, it suffices to keep a fixed configuration. Shift-JIS (MS-Kanji) Shift-JIS (MS Kanji) is a widely-used encoding of JIS. It uses the following 1- or 2-byte encoding: left half of JIS X0201-1976 (JIS-Roman) 00-FF right half of JIS X0201-1976 (katakana) A1-DF JIS X0208-1990 rows 01-62 (odd) 81-9F 40-9E (except 7F) JIS X0208-1990 rows 01-62 (even) 81-9F 9F-FC JIS X0208-1990 rows 63-94 (odd) E0-EF 40-9E (except 7F) JIS X0208-1990 rows 63-94 (even) E0-EF 9F-FC This arrangement arose because JIS X0201-1976 was used first, and the 2-byte set was squeezed in later as a backward-compatible extension. Extended UNIX Code (EUC) This is a fixed ISO 2022 configuration with G1 (and possibly G2 and/or G3) pre-defined for a particular locale. No designator sequences or locking shifts are used, and the 8-bit single shifts are used to access G2 and G3. Moreover, the glyphs of the 2-byte sets in G1 and G3 are assumed to be twice as wide as those of the 1-byte set in G2 (and the US-ASCII characters in G0). Thus the sets and their byte representations are as follows: set 0: 00-7F US-ASCII (narrow) set 1: GR GR 2-byte characters (wide) set 2: 8E GR extra 1-byte set (narrow) set 3: 8F GR GR extra 2-byte set (wide) Some examples: SHIFT-GB G1 = GB 2312-80 Chinese EUC G1 = GB 2312-80, G2 = row 8 of GB 2312-80 (half width) Japanese EUC G1 = JIS X0208-1990, G2 = right half of JIS X0201-1976, G3 = JIS X0212-1990 Korean EUC G1 = KSC 5601-1987 Somewhat different is Taiwanese EUC G1 = CNS 11643-1986 level 1, G2 = 3-byte form of CNS 11643-1986 ISO 2022 designator sequences The following escape sequences may be used to alter the assignments of the graphic character sets. esc ( ID shift 94-character set into G0 esc ) ID shift 94-character set into G1 esc * ID shift 94-character set into G2 esc + ID shift 94-character set into G3 esc $ ( ID shift multi-byte 94-character set into G0 esc $ ) ID shift multi-byte 94-character set into G1 esc $ * ID shift multi-byte 94-character set into G2 esc $ + ID shift multi-byte 94-character set into G3 esc - ID shift 96-character set into G1 esc . ID shift 96-character set into G2 esc / ID shift 96-character set into G3 Some implementations (for example, Mule) also use esc , ID shift 96-character set into G0 but ISO 2022 does not permit a 96-character set in G0. The ID, which ISO-2022 calls a final character, may be in the range 30-7E. Values in the subrange 30-3F are reserved for private use, and 7E means an empty set. Others are to be registered with ECMA before use. A summary of the registry [ftp: ftp.cc.columbia.edu kermit/charsets/iso-registry.txt] is available online. Values of ID for 94-character sets include: B US-ASCII (= left half of ISO 8859 sets) I right half of JIS X0201-1976 (katakana) J left half of JIS X0201-1976 (JIS-Roman) Values of ID for multi-byte 94-character sets are: @ JIS C 6226-1978 A GB 2312-1980 B JIS X0208-1990 C KSC 5601-1987 D JIS X0212-1990 E GB 2312-1980 plus GB 8565-1989 G CNS 11643-1986 level 1 H CNS 11643-1986 level 2 I CNS 11643-1992 plane 3 J CNS 11643-1992 plane 4 K CNS 11643-1992 plane 5 L CNS 11643-1992 plane 6 M CNS 11643-1992 plane 7 For the first 3 of these an older G0 designator esc $ IDis also allowed. Values of ID for 96-character sets include: A right half of ISO 8859-1:1987 (ISO Latin-1) B right half of ISO 8859-2:1987 (ISO Latin-2) C right half of ISO 8859-3:1988 (ISO Latin-3) D right half of ISO 8859-4:1988 (ISO Latin-4) F right half of ISO 8859-7:1987 (ISO Latin-Greek) G right half of ISO 8859-6:1988 (ISO Latin-Arabic) H right half of ISO 8859-8:1988 (ISO Latin-Hebrew) L right half of ISO 8859-5:1989 (ISO Latin-Cyrillic) M right half of ISO 8859-9:1989 (ISO Latin-5) T Thai character set TIS 620-2533:1990 V right half of ISO 8859-10:1992 (ISO Latin-6) NOTE ON ISO NOTATION In ISO documents, bytes are called octets, and (except in ISO/IEC 10646) are represented by pairs of decimal numbers. For example, the hexadecimal code 2D (the character `-' in US-ASCII) is represented as 2/13. ISO 2022 shift sequences SHIFTS FOR USE WITH 7-BIT ENCODINGS The GL range may be switched between G0-3 with the locking shift sequences: si (called LS0 in 8-bit form) lock G0 into GL so (called LS1 in 8-bit form) lock G1 into GL esc n LS2: lock G2 into GL esc o LS3: lock G3 into GL A single character from G2 or G3 may be placed in GL by preceding it with esc N SS2: shift single character from G2 into GL esc O SS3: shift single character from G3 into GL SHIFTS FOR USE WITH 8-BIT ENCODINGS The GR range may be switched between G1-3 with the locking shift sequences: esc ~ LS1R: lock G1 into GR esc } LS2R: lock G2 into GR esc | LS3R: lock G3 into GR In an 8-bit environment, the following single shifts are used: 8E SS2: shift single character from G2 8F SS3: shift single character from G3 The high bit on the following character is ignored. See also the Note on ISO notation. CCCII and ANSI Z39.64-1989 (EACC) CCCII CCCII was developed by the Chinese Character Analysis Group, based in Taiwan. It is used in libraries there. The 3 numbers making up a code are called plane, section and position. A layer is a group of 6 consecutive planes. There are 15 layers, representing parallel versions of the same characters. The layers of CCCII are: 1 traditional characters 2 simplified characters 3-12 other variant forms 13 Japanese kanji from JIS C 6226 14 Korean hanja (?) 15 reserved Layout of first layer: 1:1 graphic ASCII 1:11 Chinese punctuation marks (35) 1:12-14 Kangxi radicals (214) 1:15 Chinese numerical marks (41), phonetic symbols (37) 1:16-67 most frequently used hanzi (4808) 1:68-3:64 next most frequently used hanzi (17,032) 3:65-6:5 other hanzi (20,583) Each group of hanzi is ordered by: Kangxi radical number count of residual strokes lexicographical stroke order on residue: dian3, heng2, shu4, pie3, na4. Sections 1-15 of other layers are said to be user space. Other planes: 91 shifted JIS C 6226 kanji 92 reserved for Korean KIPS 93 reserved for supplementary Chinese characters 94 reserved for non-chinese characters of CB2 ANSI Z39.64-1989 (EACC) ANSI Z39.64-1989 (RLIN East Asian Character Code Set, REACC or EACC) was created by the Research Libraries Group (RLG) in collaboration with the US Library of Congress and the Chinese Character Analysis Group. It has been used by the Library of Congress and other libraries (mainly in North America) since 1983. (Perhaps EACC and REACC denote slightly different sets??) The structure of the code is based on CCCII, but there are a few differences: there is no "other hanzi" set. many rare characters and variants are omitted. there are some substitutions, to further the "one character, one code" principle. Characters that are now variants but used to have a life of their own are coded as variants (e.g. simplified wan4 "10,000" or yun2 "cloud"). Also layer 2 is used mainly for official simplified forms, rather that just any non-orthodox variant with fewer strokes. layer 13, plane 1 contains 293 additional codes to cover the rest of JIS C 6226-1983 (mainly kana and kokuji). layer 14, plane 1 contains Korean hangul to cover KIPS. layer 14, plane 2 contains 44 additional codes to cover the rest of GB 2312-80. An ISO 2022 encoding of a 94x94x94-character set would use the multi-byte designators with an ID in the range 60-6F, but none seem to be registered. Unicode/ISO-10646 ISO 10646 was originally conceived as approximately a disjoint union of national standards, yielding a 32-bit internal code with compact stateful external encodings. Unicode was a competing standard, aimed at unifying identical characters from different variants of the same script, to give a 16-bit code. They have since been unified: ISO/IEC 10646-1:1993 has two forms, a 31-bit form (UCS-4) and a 16-bit form (UCS-2). UCS-2 is a subset of UCS-4 (plane 0 of group 0), and is code-for-code identical with Unicode 1.1 (though Unicode imposes extra semantics on some characters). The aim is to cover all the (reasonably widely-used) scripts in the world. ISO/IEC 10646 was adopted as Chinese National Standard GB 13000 in December 1993. A Japanese translation is in the process of being adopted as a Japanese standard, JIS X0221. Unicode 1.0 is described by the Unicode books (see References). The changes to make Unicode identical to ISO 10646 UCS-2, yielding first Unicode 1.0.1 and then Unicode 1.1, are relatively minor, and do not affect the hanzi set. They are available as Unicode Technical Report #4. Various tables and appendices describing Unicode 1.1, plus mapping tables to various CJK codes are available at the Unicode site [ftp: unicode.org pub/]. See also the Unicode page maintained by Glenn Adams. UCS-2/UNICODE The overall layout of this unified 16-bit code (also called the Base Multilingual Plane or BMP of ISO-10646) is: 0000 - 1FFF A-ZONE: Alphabets: The first 256 codes are identical to ISO 8859-1 (Latin-1), except that the 65 control codes are excluded. 2000 - 2FFF A-ZONE: Symbols and Punctuation 3000 - 4DFF A-ZONE: CJK Auxiliary 4E00 - 9FFF I-ZONE: CJK Unified "Ideographs" A000 - DFFF O-ZONE: Reserved for future assignment (but see UTF-16 for a proposed use) E000 - FFFD R-ZONE: Restricted use - the codes FFFE and FFFF are excluded. All characters are also assigned names, but for the "ideographs" these are merely "CJK UNIFIED IDEOGRAPH-4E00" and so on. THE REST OF UCS-4 In the first version of ISO/IEC 10646, groups 60-7F and planes E0-FF of group 0 are reserved for private use, while all other planes are reserved for future standardization. This is expected to change in the forthcoming revision: groups 01-0E of plane 0 will become available for definition, groups 0F and 10 of plane 0 will become available for private use, while the rest of the code space will be reserved (see UTF-16). COMBINING CHARACTERS These are characters for graphemes such as Semitic and Indic vowels, IPA diacritics, tone marks and Hangul jamos. These characters are placed after the base character; the rendering process is expected to generate the composite glyph, in some cases by a table lookup. ISO 10646 defines 3 levels of subset implementation: level 1: base characters only. level 2: base characters and certain combining characters, roughly those required by Semitic and Indic-derived scripts. level 3: all characters. Many redundant "presentation forms" are also provided (especially in the additions that make Unicode 1.1), including the tone-letter combinations used in Hanyu pinyin (but not those used in Zhuyin fuhao) and pre-composed Hangul syllable blocks. Unicode/ISO-10646 Alphabets Groups marked with * have not yet been finalized. 0020 - 007E Basic Latin (graphic part of US-ASCII) 00A0 - 00FF Latin-1 Supplement (right half of ISO 8859-1) 0100 - 017F Latin Extended-A 0180 - 024F Latin Extended-B (extended in 1.1) 0250 - 02AF IPA Extensions 02B0 - 02FF Spacing Modifier Letters 0300 - 036F Combining Diacritical marks 0370 - 03CF Basic Greek (based on ISO 8859-7) 03D0 - 03FF Greek Symbols and Coptic 0400 - 04FF Cyrillic (based on ISO 8859-5, extended in 1.1) 0500 - 052F unassigned 0530 - 058F Armenian 0590 - 05CF Hebrew Extended-A 05D0 - 05EA Basic Hebrew (based on ISO 8859-8) 05EB - 05FF Hebrew Extended-B 0600 - 0652 Basic Arabic (based on ISO 8859-6) 0653 - 06FF Arabic Extended (extended in 1.1) 0700 - 08FF * Ethiopic (?) 0900 - 097F Devanagari (based on ISCII 1988) 0980 - 09FF Bengali (based on ISCII 1988) 0A00 - 0A7F Gurmukhi (based on ISCII 1988) 0A80 - 0AFF Gujarati (based on ISCII 1988) 0B00 - 0B7F Oriya (based on ISCII 1988) 0B80 - 0BFF Tamil (based on ISCII 1988) 0C00 - 0C7F Telugu (based on ISCII 1988) 0C80 - 0CFF Kannada (based on ISCII 1988) 0D00 - 0D7F Malayalam (based on ISCII 1988) 0D80 - 0DFF * Sinhala 0E00 - 0E7F Thai (based on TIS 620-2533:1990) 0E80 - 0EFF Lao (based on TIS 620-2533:1990) 0F00 - 0F7F * Burmese 0F80 - 0FDF * Khmer 1000 - 105F * Tibetan 1060 - 109F * Mongolian 10A0 - 10CF Georgian Extended 10D0 - 10FF Basic Georgian 1100 - 11FF Hangul Jamo (added in 1.1) 1200 - 125F * Ethiopian (?) 1E00 - 1EFF Latin Extended Additional (added in 1.1) 1F00 - 1FFF Greek Extended (added in 1.1) Unicode/ISO-10646 CJK Auxiliary 3000 - 303F CJK Symbols and Punctuation 3040 - 309F Hiragana (based on JIS X0208-1990) 30A0 - 30FF Katakana (based on JIS X0208-1990) 3100 - 312F Bopomofo (based on GB 2312-80) 3130 - 318F Hangul Compatibility Jamo (based on KSC 5601-1987) 3190 - 319F CJK Miscellaneous (Kaeriten) 3200 - 32FF Enclosed CJK Letters and Months (months added in 1.1) 3300 - 33FF CJK Compatibility (extended in 1.1) 3400 - 3D2D Hangul (based on KSC 5601-1987) 3D2E - 44B7 Hangul Supplementary-A (rows 16-36 of KSC 5657-1991, added in 1.1) 44B8 - 4DFF Hangul Supplementary-B (rows 37-54 of KSC 5657-1991, added in 1.1) The punctuation set does not include separate codes for punctuation (e.g. brackets) in horizontal and vertical text. The rendition of these characters is considered outside the scope of the code. The advantage of this approach is that text may be reset in a different direction by merely changing the outside declarations. The Hangul elements here are redundant presentation forms that can be coded with Hangul Jamo 1100-11FF. Unicode/ISO-10646 Unified "Ideographs" The CJK character set is derived from a merger of a set proposed by Unicode (an extension of the EACC list) and a draft for a proposed Chinese standard GB 13000. Since February 1990, it has been developed by the CJK Joint Research Group. A set comprising 20,902 characters was issued in March 1992, and is variously called UniHan (by Unicode), Han Character Set or HCS (by the Chinese), Ideographic CJK Character Unified Repertoire and Ordering, or URO 2.0 (by ISO, because the term "Han" is inappropriate in a Japanese context, and only ISO WG2 is allowed to create a "Character Set"). It includes all hanzi/kanji/hanja from the following sets: GB 2312-80, GB 12345-90 and GB 8565-89 CNS 11643-1986 planes 1 and 2 JIS X0208-1990 and JIS X0212-1990 KSC 5601-1987 and KSC 5657-1991 The successor of the CJK-JRG is the Ideographic Rapporteur Group (IRG) of ISO SC2/WG2. The IRG is compiling lists of additional characters: HCS-B comprises about 8000 characters, and China has asked for another 1000 in addition to these. There is a lot of competition for the remaining space in UCS-2. HAN UNIFICATION This refers to the unification of Chinese-derived characters used in China, Taiwan, Korea, Japan and Vietnam, without which a 16-bit character set would be unattainable. The idea was that two characters having the same abstract shape would be unified, regardless of meaning or font differences. Users of different languages will require different visual representations of the codes, but this is regarded as a font/language issue to be handled by declarations extraneous to the codeset. Of course, the distinction between a variant form and a font difference is sometimes subjective. Considerations of character identity are overridden by the Source Set Separation Rule: distinct codepoints in any of the following source sets are mapped to distinct codepoints in the new set: GB 2312-80, GB 12345-90, GB 7589-87, GB 7590-87 and GB 8565-89 CNS 11643-1986 planes 1, 2 and 14 JIS X0208-1990 and JIS X0212-1990 KSC 5601-1987 (minus the repeated hanja mentioned there) and KSC 5657-1991 The importance of this is that if you translate text encoded in a national character set into Unicode and back again, you get the same text you started with. The result is a fairly conservative unification, e.g. simplified forms of most radicals make for different characters. ORDERING The ordering of the CJK characters is based on the following dictionaries, in order of priority: Kangxi Zidian (7th ed.) Zhonghua Bookstore, Beijing, 1989. Dai Kanwa Ziten (revised ed.) Taisyuukan Syoten, Tokyo, 1986. Hanyu Da Zidian (1st ed.) Sichuan Cishu Publishing, Chengdu, 1986. Dae Jaweon (1st ed.) Samseong Publishing, Seoul, 1988. From the Unicode book, volume 2, page 14: When a character is found in the KangXi Zidian, it follows the Kang Xi Zidian order. When it is not found in the KangXi Zidian and is fou nd in Dai Kanwa Ziten, it is given a position extrapolated from the K angXiposition of the preceding character in Dai Kanwa Ziten. When it is not found in either KangXi or Dai Kanwa, Hanyu Da Zidian and Dae J aweondictionaries are consulted in a similar manner. Hence the ordering is approximately Kangxi radical, then stroke count. However, a number of simplified are treated separately, and are placed immediately after the corresponding traditional radical. Unicode/ISO-10646 Restricted Zone E000 - F8FF Private use Area F900 - FAFF CJK Compatibility Ideographs (the duplicate hanja from KSC 5601-1987, the two duplicate Big-5 characters excluded from CNS 11643-1986, and some extra characters from corporate codesets) FB00 - FB4F Alphabetic Presentation Forms (added in 1.1) FB50 - FDFF Arabic Presentation Forms-A (added in 1.1) FE20 - FE2F Combining Half Marks (added in 1.1) FE30 - FE4F CJK Compatibility Forms (verticals and overlines) FE50 - FE6F Small Form Variants FE70 - FEFE Arabic Presentation Forms-B (re-arranged in 1.0.1) FF00 - FFEF Halfwidth and Fullwidth Forms (for full round-trip compatibility with existing ASCII+CJK coding schemes; extended in 1.1) FFF0 - FFFD Specials There is also a special character FEFF ZERO WIDTH NO-BREAK SPACE (formerly BYTE ORDER MARK) Note also FFFD REPLACEMENT CHARACTER which can be used for characters not representable in Unicode/ISO-10646. Unicode/ISO-10646 Symbols and Punctuation 2000 - 206F General Punctuation 2070 - 209F Superscripts and Subscripts 20A0 - 20CF Currency Symbols 20D0 - 20FF Combining Diacritical Marks for Symbols 2100 - 214F Letterlike Symbols 2150 - 218F Number Forms 2190 - 21FF Arrows 2200 - 22FF Mathematical Operators 2300 - 23FF Miscellaneous Technical (extended in 1.1) 2400 - 243F Control Pictures 2440 - 245F Optical Character Recognition 2460 - 24FF Enclosed Alphanumerics 2500 - 257F Box Drawing 2580 - 259F Block Elements 25A0 - 25FF Geometric Shapes 2600 - 26FF Miscellaneous Symbols 2700 - 27BF Dingbats Encodings of Unicode/ISO-10646 Unicode/UCS-2 text is intended to be represented internally (for processing) as 16-bit values. For transmission and external storage a number of byte encodings have been proposed. DIRECT ENCODINGS The simplest is a direct representation as two bytes, most significant first, as specified in ISO-10646, and in the UNICODE-1-1 [ftp: ftp.internic.net rfc/rfc1641.txt] scheme registered for use with MIME [ftp: ftp.internic.net rfc/rfc1521.txt] (Multipurpose Internet Mail Extensions). A variant (see informative Annex F of ISO-10646) allows the order of the two bytes to depend on the machine that produced the text, but to begin the stream with the character FEFF ZERO WIDTH NO-BREAK SPACE (formerly BYTE ORDER MARK) which will establish the byte order, since FFFE is not a legal code. UTF-1 The UCS Transformation Format (UTF, or UTF-1) for ISO-10646 (see informative Annex G of ISO/IEC 10646-1:1993) specifies a variable-length encoding avoiding C0, C1, DEL and SPACE octets, but appears to be dead already. UTF-8 Another encoding, proposed by X-Open, is variously called UTF-8, FSS-UTF, UTF or UTF-2. It has been proposed as normative Annex P to ISO/IEC 10646. See also Hello World [ftp: ftp.research.att.com dist/plan9man/09utf.ps.Z], by Rob Pike and Ken Thompson. Some properties of this encoding: ASCII files are also UTF-8 files. only ASCII characters use ASCII bytes. simple synchronization. many operations need not decode. chars 0800-4015 (Asian alphabets, symbols & CJK auxiliary) have longer encodings than in UTF-1. both UCS-2 and UCS-4 are covered. UTF-7 UNICODE-1-1-UTF-7 [ftp: ftp.internic.net rfc/rfc1642.txt] (UTF-7), an encoding of UCS-2 using only mail-safe bytes, has been registered for use with MIME [ftp: ftp.internic.net rfc/rfc1521.txt]. Some properties: The encoding of mail-safe characters is readable. Other characters are encoded in a base-64 code, with 3 characters occupying 8 bytes. UCS-4 text may be pre-coded with UTF-16. Conversion functions for UTF-8 and UTF-7 [ftp: unicode.org pub/Programs/ConvertUTF/] are available from the Unicode site [ftp: unicode.org pub/]. UTF-16 UTF-16 (formerly UCS-2E) is not a byte encoding, but a scheme for representing certain UCS-4 codes in a UCS-2 stream. (Recall that no UCS-4 codes outside UCS-2 have yet been defined.) It has been proposed as normative Annex O to ISO/IEC 10646. Characters in the range 10000 to 10FFFF are represented by a pair of O-zone codes, the first in the range D800-DBFF and the second in the range DC00-DFFF.