[This file was generated from the HTML document
"http://ifcss.org:8001/www/pub/software/info/cjk-codes/".]
NOTES ON CJK CHARACTER CODES AND ENCODINGS
The character codes covered are:
Big-5 A code for traditional Chinese, widely used in Hong
Kong, Taiwan and elsewhere.
94x94 Character Sets These codes represent each character by a pair of
numbers 1-94 (decimal), called row and column
respectively ("kuten" in Japanese, "qu1wei4" in
Chinese). The idea is to map onto the 94 graphic ASCII
bytes, by adding 0x20 to each (0xA0 for shifted
ASCII).
Chinese - GB
Traditional Chinese - CNS
Japanese - JIS
Korean - KSC
Encodings
CCCII and ANSI Z39.64-1989 (EACC)
Two closely related 3-byte codes, intended to cover
all CJK languages.
Unicode/ISO-10646 A new code covering most of the world's languages.
Encodings
SEE ALSO
International Standard--Information Processing-- ISO 7-bit and 8-bit
coded character sets--Code extension techniques, ISO 2022:1986. (= ANSI
X3.41, JIS X0202, BS 6856, FIPS 35, ECMA 35)
Compound Text Encoding, Version 1.1 [ftp: ftp.cc.utexas.edu
source/X11R6/xc/doc/hardcopy/CTEXT/ctext.PS.Z], Robert W. Scheiffer, 1989
(Also in TROFF/TBL format [ftp: ftp.cc.utexas.edu
source/X11R6/xc/doc/specs/CTEXT/ctext.tbl.ms]).
Electronic Handling of Japanese Text(japan.inf), Ken Lunde, 1992. Mostly
English, with some Japanese, in the following formats:
ISO-2022-JP [ftp: ftp.ora.com
pub/examples/nutshell/ujip/doc/japan.inf-032092.jis].
Shift-JIS [ftp: ftp.ora.com
pub/examples/nutshell/ujip/doc/japan.inf-032092.sjs].
EUC-J [ftp: ftp.ora.com
pub/examples/nutshell/ujip/doc/japan.inf-032092.euc].
Understanding Japanese Information Processing, Ken Lunde, O'Reilly &
Associates, Inc, 1993.
International Standard--Information Technology-- Universal Multiple-Octet
Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual
Plane, ISO/IEC 10646-1:1993.
The Unicode standard : worldwide character encoding(vols 1 & 2) the
Unicode Consortium, Reading MA : Addison-Wesley, 1991-1992. ISBN:
0-201-56788-1 (pbk. : v. 1), 0-201-60845-6 (pbk. : v. 2) LC:
QA76.76.C53U6
Hello World [ftp: ftp.research.att.com dist/plan9man/09utf.ps.Z], Rob
Pike and Ken Thompson, Usenix Conference, Jan 1993.
Unicode: A Universal Character Code, Jürgen Bettels and E. Avery Bishop,
Digital Technical Journal 5:3 (Sep 1993) 21-31 (also in plain text format
[ftp: gatekeeper.dec.com pub/DEC/DECinfo/DTJ/I18N-unicode.txt]).
T
Comments on this document are most welcome.
Thanks to:Hu Bo, Nelson Chin, Ken'ichi Handa, Ken Lunde, Man-Chi Pong and
Bob Sandstrom.
Last modified: 15th February 1995.
Ross Paterson
Big-5 (Traditional Chinese)
This is a 2-byte code, designed to represent traditional Chinese characters.
The first byte is A1-F9; the second byte 40-7E,A1-FE. The code ranges are:
A140-A24E punctuation and other symbols
A24F-A258 various units
A259-A261 Chinese units
A262-A2AE box-drawing pieces and shapes
A2AF-A2B8 Arabic numerals
A2B9-A2C2 Roman numerals
A2C3-A2CE Hangzhou numerals
A2CF-A2E8 Latin capital letters
A2E9-A343 Latin small letters
A344-A35B Greek capital letters
A35C-A373 Greek small letters
A374-A3BA zhuyin symbols
A3BB-A3BF zhuyin diacritics
A440-C67E frequently used hanzi (5401)
C6A1-C6F7 hiragana
C6F8-C7B0 katakana
C7B1-C7E8 Cyrillic letters
C7E9-C7F2 circled numbers
C7F3-C7FC parenthesized numbers
C940-F9D5 less frequently used hanzi (7652)
Each of the hanzi blocks is ordered by total number of strokes, then Kangxi
radical no.
A variant is the Taiwan standard CNS-11643.
The font HKU-Ch16 (a.k.a. chinese.16) uses a slight variant: the less
frequent hanzi are moved down to occupy C6A1-F755.
The usual encoding is a simple interleaving with plain US-ASCII text.
Relationship between Big5 and CNS 11643-1986
The Big5 non-hanzi and level 1 hanzi correspond to CNS level 1:
A140-A1F5 <-> 0101-0254
A1F6 <-> 0256
A1F7 <-> 0255
A1F8-A2AE <-> 0257-0346
A2AF-A3BF <-> 0401-0580 # digits, Latin, Greek, bopomofo
**** <- 0701-0925 # Kangxi radicals
**** <- 3401-3433 # symbols for control characters
A440-ACFD <-> 3601-5102 # start of hanzi level 1
ACFE <-> 5551
AD40-AFCF <-> 5103-5550
AFD0-BBC7 <-> 5552-7547
BBC8-BE51 <-> 7549-7959
BE52 <-> 7548
BE53-C1AA <-> 7960-8520
C1AB-C2CA <-> 8522-8722
C2CB <-> 8521
C2CC-C360 <-> 8723-8812
C361-C3B8 <-> 8814-8867
C3B9 <-> 8869
C3BA <-> 8868
C3BB-C455 <-> 8870-8965
C456 <-> 8813
C457-C67E <-> 8966-9343 # end of level 1
C6A1-C7E8 -> ****** # kana, Cyrillic (different in ETEN)
C7E9-C7FC <-> 0601-0620 # enclosed numerals (different in ETEN)
Big5 level 2 hanzi correspond to CNS level 2:
C940-C949 <-> 0101-0110 # start of level 2
C94A -> 3634 # duplicate of A461
C94B-C96B <-> 0111-0143
C96C-C9BD <-> 0145-0192
C9BE <-> 0144
C9BF-C9EC <-> 0193-0244
C9ED-CAF6 <-> 0246-0424
CAF7 <-> 0245
CAF8-D6CB <-> 0425-2378
D6CC <-> 3067
D6CD-D779 <-> 2380-2493
D77A <-> 3174
D77B-DADE <-> 2494-3066
DADF <-> 2379
DAE0-DBA6 <-> 3068-3173
DBA7-DDFB <-> 3175-3603
DDFC -> 3386 # duplicate of DCD1
DDFD-E8A2 <-> 3604-5342
E8A3-E975 <-> 5344-5501
E976-EB5A <-> 5503-5807
EB5B-EBF0 <-> 5809-5930
EBF1 <-> 5343
EBF2-ECDD <-> 5931-6073
ECDE <-> 5502
ECDF-EDA9 <-> 6074-6183
EDAA-EEEA <-> 6185-6424
EEEB <-> 6815
EEEC-F055 <-> 6425-6634
F056 <-> 6184
F057-F0CA <-> 6635-6722
F0CB <-> 5808
F0CC-F162 <-> 6723-6814
F163-F16A <-> 6816-6823
F16B <-> 7165
F16C-F267 <-> 6824-6982
F268 <-> 7320
F269-F2C2 <-> 6983-7044
F2C3-F374 <-> 7046-7164
F375-F465 <-> 7166-7319
F466-F4B4 <-> 7321-7365
F4B5 <-> 7045
F4B6-F4FC <-> 7366-7442
F4FD-F662 <-> 7444-7649
F663 <-> 7443
F664-F976 <-> 7650-8169
F977-F9C3 <-> 8171-8219
F9C4 <-> 8170
F9C5 <-> 8220
F9C6 <-> 8232
F9C7-F9D1 <-> 8221-8231
F9D2-F9D5 <-> 8233-8236
GuoBiao (Chinese)
Guo2jia1 Biao2zhun3(Chinese for "national standard", abbreviated Guo2Biao1
or GB) refers to standards promulgated by the PRC. The following character
sets are included:
THE BASIC SET: GB 2312-80
GB 2312-80 (GB0, GB)is designed to represent simplified Chinese characters,
and is a national standard in the PRC. It is also used in Singapore. The
rows are:
01 special symbols (94)
02 paragraph numbers (72)
03 GB 1988-80 (94, = ISO 646-CN)
04 hiragana (83)
05 katakana (86)
06 Greek (48)
07 Cyrillic (66)
08 pinyin accented vowels (26) and zhuyin symbols (37)
09 box and table drawing pieces (76)
16-55 hanzi level 1 (3755, ordered by pinyin)
56-87 hanzi level 2 (3008, ordered by radical, then stroke)
The GB 1988-80 set is the Chinese extension of ISO 646. It differs from
US-ASCII only in having a yuan/yen symbol in place of the dollar sign (24)
and an overline in place of the tilde (7E).
VARIATIONS AND EXTENSIONS OF GB 2312-80
GB 6345.1-86 is a revision of GB 2312-80:
The form of "g" (row 3 cell 71) is changed to the open form.
Six pinyin letters are added (row 8, cells 27-32).
Row 10 is added as a half-width version of row 3.
Row 11, cells 1-32 is added as a half-width version of the corresponding
positions of row 8.
The form of one hanzi (row 79 cell 81) is altered.
Further details are available in English [ftp: ftp.ifcss.org
pub/software/fonts/gb/misc/gb6345.txt] and Chinese [ftp: ftp.ifcss.org
pub/software/fonts/gb/misc/gb6345.gb] (encoded in GB 2312-80).
GB 8565.2-89 (GB8) is intended as an extension of GB 2312-80, and contains
additional simplified hanzi in the following rows:
12 hanzi (92, from GB 7590-87, ordered by radical)
13 hanzi (50, from GB 7589-87, ordered by radical)
15 date and time symbols (69) and miscellaneous hanzi (24,
ordered by radical)
90-94 hanzi (470, from GB 7589-87, ordered by radical)
The character set ISO-IR-165 comprises GB 2312-80, as modified by GB
6345.1-86 (except for the alteration to row 79, cell 81), and GB 8565.2-89,
plus
44 hanzi added to row 13
row 14, containing 92 hanzi
3 more hanzi?
22 background characters added to row 6 (cells 60-81)
GB 12345-90 (GB1) is a code for traditional hanzi, parallel to GB 2312-80
with the following additional rows:
88-89 hanzi level 3: an extra 103 characters so that the
traditional counterparts of GB 2312-80 level 1 hanzi
are uniquely represented (103, ordered by pinyin).
Further information is available on the mapping between the two sets.
OTHER SETS
Some other sets have been defined, but (like GB1) are not widely supported:
GB 7589-87 (GB3) 7237 simplified hanzi in radical order (rows 16-92)
GB 7590-87 (GB5) 7039 simplified hanzi in radical order (rows 16-90)
General Purpose Han Characters for Modern Chinese (GB7)
43 simplified hanzi (row 1)
GB 12052-89 (Korean) 94 Korean Idu characters (row 71 cells 89-94 and row
72 cells 1-88)
GB 13131-94 traditional version of GB 7589-87
GB 13132-94 traditional version of GB 7590-87
SPECIAL VERSIONS USED IN ISO/IEC 10646
ISO/IEC 10646 (and the Unicode cross-reference tables) refer to a number of
codes in GB sets beyond GB 2312-80 that are not part of the official
standards, but were added by China solely for this purpose. They include:
GB 12345-90 additional rows:
92 Hong Kong hanzi (58)
93 Korean Idu characters (94, from GB 12052-89)
GB 8565.2-89 The characters added for ISO-IR-165, plus 8 hanzi (?)
in row 8.
Further, the versions of GB 7589-87 and GB 7590-87 submitted by China for
this purpose used the traditional forms of the hanzi.
Mapping from GB 12345-90 to GB 2312-80
In almost all cases, for each codepoint in GB 2312-80, the corresponding
codepoint in GB 12345-90 contains the most common traditional form of the GB
2312-80 character, so that the mapping of characters in levels 1 and 2 is
trivial.
The exceptions are three instances where GB2312-80 includes (in level 2) a
traditional form corresponding to a simpler character in level 1. In each of
these cases, the simpler character was a rare character that was substituted
for a common character with more strokes. In GB 12345-90, these pairs are
interchanged, so that the more common traditional form occurs in level 1. In
mapping from GB 12345-90 to GB 2312-80, both forms should be mapped to the
simplified form in level 1:
1-6565 -> 0-2683 # hou4 "empress" vs. "behind"
1-6637 -> 0-5164 # yu2 "I, me" vs. "remainder"
1-8765 -> 0-3520 # yao1 "one, small" vs. me5 (question particle)
GB 12345-90 also adds a level 3, containing additional variants (103) of
some level 1 characters used in the traditional orthography. Their level 1
counterparts are:
1-8801 -> 0-1658 1-8836 -> 0-2910 1-8871 -> 0-4165
1-8802 -> 0-1669 1-8837 -> 0-3001 1-8872 -> 0-4285
1-8803 -> 0-1777 1-8838 -> 0-3061 1-8873 -> 0-4341
1-8804 -> 0-1780 1-8839 -> 0-3077 1-8874 -> 0-4353
1-8805 -> 0-1823 1-8840 -> 0-3143 1-8875 -> 0-4408
1-8806 -> 0-1837 1-8841 -> 0-3168 1-8876 -> 0-4408
1-8807 -> 0-1907 1-8842 -> 0-3207 1-8877 -> 0-4408
1-8808 -> 0-1969 1-8843 -> 0-3215 1-8878 -> 0-4419
1-8809 -> 0-1983 1-8844 -> 0-3216 1-8879 -> 0-4531
1-8810 -> 0-1986 1-8845 -> 0-3259 1-8880 -> 0-4537
1-8811 -> 0-2117 1-8846 -> 0-3279 1-8881 -> 0-4582
1-8812 -> 0-2119 1-8847 -> 0-3290 1-8882 -> 0-4721
1-8813 -> 0-2177 1-8848 -> 0-3317 1-8883 -> 0-4721
1-8814 -> 0-2212 1-8849 -> 0-3343 1-8884 -> 0-4744
1-8815 -> 0-2223 1-8850 -> 0-3417 1-8885 -> 0-4782
1-8816 -> 0-2281 1-8851 -> 0-3462 1-8886 -> 0-4875
1-8817 -> 0-2302 1-8852 -> 0-3525 1-8887 -> 0-5009
1-8818 -> 0-2322 1-8853 -> 0-3541 1-8888 -> 0-5022
1-8819 -> 0-2365 1-8854 -> 0-3541 1-8889 -> 0-5184
1-8820 -> 0-2420 1-8855 -> 0-3541 1-8890 -> 0-5189
1-8821 -> 0-2441 1-8856 -> 0-3554 1-8891 -> 0-5185
1-8822 -> 0-2540 1-8857 -> 0-3570 1-8892 -> 0-5224
1-8823 -> 0-2546 1-8858 -> 0-3579 1-8893 -> 0-5238
1-8824 -> 0-2567 1-8859 -> 0-1757 1-8894 -> 0-5264
1-8825 -> 0-6756 1-8860 -> 0-3827 1-8901 -> 0-5402
1-8826 -> 0-2690 1-8861 -> 0-3830 1-8902 -> 0-5427
1-8827 -> 0-2714 1-8862 -> 0-3843 1-8903 -> 0-5427
1-8828 -> 0-2756 1-8863 -> 0-3845 1-8904 -> 0-5434
1-8829 -> 0-2767 1-8864 -> 0-3851 1-8905 -> 0-5438
1-8830 -> 0-2781 1-8865 -> 0-3907 1-8906 -> 0-5454
1-8831 -> 0-2802 1-8866 -> 0-3909 1-8907 -> 0-5476
1-8832 -> 0-2824 1-8867 -> 0-4743 1-8908 -> 0-5494
1-8833 -> 0-2850 1-8868 -> 0-3979 1-8909 -> 0-5528
1-8834 -> 0-2859 1-8869 -> 0-3990
1-8835 -> 0-2886 1-8870 -> 0-4023
Since the most common traditional counterpart occurs in the position of each
GB 2312-80 character, in some cases a GB 2312-80 character has been moved to
level 3, while the original position is occupied by a more complex
character.
CNS (Traditional Chinese)
The Taiwan standard CNS 11643-1986 is a pair of 94x94 planes (or levels)
derived from Big-5 and another 94x94 plane of user characters (rows 1-71),
called plane 14. It may also be treated as a 3-byte 94-character code.
Plane 1 comprises
1-3 punctuation and other symbols (Big-5 characters
A140-A2AE one minor change)
4-5 various numerals and assorted alphabets (Big-5
characters A2AF-A3BF)
6 enclosed numerals (Big-5 characters C7E9-C7FC) plus
small Roman numerals
7-9 Kangxi radicals
34 symbols for control characters
36-93 frequently used hanzi (Big-5 characters A440-C67E)
Plane 2 is
1-82 less frequently used hanzi (BIG-5 characters C940-F9D5
except C94A and DDFC, eliminated as duplicates of A461
and DCD1 respectively)
The hanzi planes are ordered by stroke count and then Kangxi radical as in
Big-5, but a few of the counts or radicals have been corrected (and the
characters relocated) so the mapping is not continuous.
The revised code CNS 11643-1992 carries over planes 1 and 2 unchanged. Other
planes are:
3 6148 hanzi from plane 14 of CNS 11643-1986 (rows 1-66)
4 7298 rarely used hanzi, including some used in names,
171 more hanzi from plane 14 of CNS 11643-1986 and
hanzi in ISO/IEC 10646-1:1993 not in previous planes
(rows 1-78)
5 8603 even rarer hanzi (rows 1-92)
6 6388 variant forms (yi4ti3zi4) with up to 14 strokes
(rows 1-68)
7 6539 variant forms (yi4ti3zi4) with 15 or more strokes
(rows 1-70)
Hanzi in the last 3 planes were supplied by the Ministry of Education. An
eighth plane of about 7000 characters is in preparation.
JIS (Japanese)
The first of the 94x94 codes was the Japanese standard JIS C 6226-1978
(Old-JIS). This was re-arranged to accommodate the 1981 Jouyou Kanji list
(all of which are in level 1), resulting in JIS C 6226-1983 (New-JIS,
renamed JIS X0208-1983 in 1987). Two kanji have since been added to level 2,
giving the current JIS X0208-1990:
01-02 punctuation, symbols
03 ISO 646 (alphanumerics only)
04 hiragana
05 katakana
06 Greek
07 Cyrillic
08 line drawing
16-47 kanji level 1 (2965, ordered by on-yomi)
48-83 kanji level 2 (3384, ordered by Kangxi radical, then
stroke)
84 miscellaneous kanji (6)
[This seems an appropriate place to mention JIS X0201-1976(called JIS C 6220
until 1987), an 8-bit code. Its left half (JIS-Roman) is the Japanese
national variant of ISO 646 and differs from US-ASCII only in having a yen
symbol in place of the backslash (5C) and an overline in place of the tilde
(7E). Its right half (A1-DF) comprises Japanese punctuation and katakana.]
An extension is defined by JIS X0212-1990:
02 more punctuation, symbols
06 accented Greek
07 non-Russian Cyrillic
09 extended Latin
10 uppercase accented Latin
11 lowercase accented Latin
16-77 kanji level 3 (5801, ordered by Kangxi radical, then
stroke)
The non-kanji parts of JIS X0208-1990 and JIS X0212-1990 occupy disjoint
code ranges.
KSC (Korean)
The Korean standard KSC 5601-1987(formerly KIPS, updated in 1989 and 1992)
has the following rows:
01-02 punctuation, symbols
03 KSC 5636 (ISO 646-KR)
04 alphabetic Hangul
05 roman numbers, Greek
06 line drawing
07 various units
08-09 miscellaneous
10 hiragana
11 katakana
12 Cyrillic
16-40 Hangul syllable blocks (2350)
42-93 hanja (4888, ordered by Hangul, then radical)
The KSC 5636 set is the Korean extension of ISO 646. It differs from
US-ASCII only in having a Won sign in place of the backslash (5C).
The few hanja that have more than one pronunciation have extra codes, so
that the 4888 hanja codes cover 4620 unique hanja.
The revisions did not alter the code assignments, only some annexes relating
to alternative coding methods. Annex 3 of KSC 5601-1992 describes the 2-byte
johab-hyeong code for modern Hangul syllables. This is a 16-bit code, with
the top bit usually set, and then 5 bits each for
initials (choseong) 14 simple consonants + 5 "double" consonants
vowels (jungseong) 10 simple vowels + 10 complex vowels
finals (jongseong) 14 simple consonants + 13 clusters + 1 filler
giving a total of 11,172 possible syllable blocks. Annex 4 describes a
one-byte code for internal use.
KSC 5657-1991 has the following rows:
16-36 Hangul syllable blocks, level 1 (1930)
37-54 Hangul syllable blocks, level 2 (1676)
55-85 hanja (2856)
Encodings of 94x94-character sets
One popular way of mixing US-ASCII and a 2-byte set is to set the high bit
on the first byte of 2-byte characters. Some schemes also set the high bit
on the second byte. An example is the scheme commonly called Shift-GB. Note
that this is quite different from Shift-JIS (MS Kanji).
There are schemes that use ASCII sequences to switch between ASCII and GB
modes to accommodate both sets in graphic ASCII text. For example, the HZ
scheme used on the Usenet group alt.chinese.text wraps unshifted GB text in
the strings "~{" and "~}". For more details, see the HZ package [ftp:
ftp.ifcss.org pub/software/unix/convert/HZ-2.0.tar.gz].
ISO 2022 CODING
ISO 2022:1986ISO 7-bit and 8-bit Coded Character Sets - Code Extension
Techniquesprovides several ways of encoding n-byte 94- or 96-character sets,
using escape sequences. It uses a two-stage mapping:
Graphic character sets are assigned to four character set slots, called
G0, G1, G2 and G3. G0 may hold a 94-character set, while the others may
hold either a 94- or a 96-character set.
Initially G0 contains ISO 646 IRV(1983), and G1-3 are undefined. (This
set differs from US-ASCII only in that it has a currency symbol in the
place of the dollar sign; this difference is often ignored.) These
assignments may be changed with designator sequences.
These slots are assigned to two byte ranges:
GL (graphic left): byte values 20-7F for 96-character sets, 21-7E for
94-character sets (in which case SPACE and DEL are defined).
GR (graphic right): byte values A0-FF for 96-character sets, A1-FE for
94-character sets.
Initially G0 is assigned to the GR range and G1 to the GL range. These
assignments may be changed with shift sequences:
A single shift changes an assignment for a single character (one or more
bytes).
A locking shift makes a permanent assignment.
Some subset implementations are widely used.
7-BIT ENCODINGS (USING GL ONLY)
JUNET or ISO-2022-JP [ftp: ftp.internic.net rfc/rfc1468.txt]
Designator sequences are used to switch G0 between
94-character sets. The guidelines explicitly require
switching back to an ISO 646 variant (such as US-ASCII
or JIS-Roman) before the end of each line. This scheme
is widely used in Japan.
ISO-2022-JP-2 [ftp: ftp.internic.net rfc/rfc1554.txt]
An extension of the previous scheme, using designator
sequences to switch G2 between 96-character sets, and
single shifts to include characters from these sets.
ISO-2022-KR [ftp: ftp.internic.net rfc/rfc1557.txt]
A designator sequence is used to assign the KSC
5601-1987 set to G1, and locking shifts to switch GL
between G0 and G1.
ISO-2022-INT-1 [ftp: ftp.internic.net
internet-drafts/draft-ohta-text-encoding-01.txt]
A mixture of the previous two. G1 is used for KSC
5601-1987 and 96-character sets, G0 for all other
sets.
ISO-2022-CN A proposed scheme combining GB and CNS sets using
designator sequences and locking shifts. See the
related mailing list [ftp: apng.org
apng/mail.archive/apng-cc].
8-BIT ENCODINGS (USING BOTH GL AND GR)
Extended Unix Code (EUC)
A locale-dependent fixed codeset assignment is
assumed, and single shifts are used to access G2 and
G3. It is common practice to leave US-ASCII in G0.
The X Windows Compound Text Encoding [ftp: ftp.cc.utexas.edu
source/X11R6/xc/doc/hardcopy/CTEXT/ctext.PS.Z]
Only G0 and G1 are used, locked into GL and GR
respectively. Designator sequences are used to switch
character sets. "Left half" sets should be used in G0
only, "Right half" sets in G1 only, others (including
the 2-byte CJK sets) in either position. (There is
also provision for non-ISO-2022 character sets.)
For newer schemes, the convention appears to be:
G0 US-ASCII.
G1 Primary non-Latin character set (usually a 2-byte set
in the CJK context).
G2 Supplementary Latin-based character set (or, in the CJK
context, an auxiliary 1-byte set).
G3 Auxiliary non-Latin character set (usually a 2-byte set
in the CJK context).
For many mono-lingual (or mono-lingual+English) applications, it suffices to
keep a fixed configuration.
Shift-JIS (MS-Kanji)
Shift-JIS (MS Kanji) is a widely-used encoding of JIS. It uses the following
1- or 2-byte encoding:
left half of JIS X0201-1976 (JIS-Roman)
00-FF
right half of JIS X0201-1976 (katakana)
A1-DF
JIS X0208-1990 rows 01-62 (odd)
81-9F 40-9E (except 7F)
JIS X0208-1990 rows 01-62 (even)
81-9F 9F-FC
JIS X0208-1990 rows 63-94 (odd)
E0-EF 40-9E (except 7F)
JIS X0208-1990 rows 63-94 (even)
E0-EF 9F-FC
This arrangement arose because JIS X0201-1976 was used first, and the 2-byte
set was squeezed in later as a backward-compatible extension.
Extended UNIX Code (EUC)
This is a fixed ISO 2022 configuration with G1 (and possibly G2 and/or G3)
pre-defined for a particular locale. No designator sequences or locking
shifts are used, and the 8-bit single shifts are used to access G2 and G3.
Moreover, the glyphs of the 2-byte sets in G1 and G3 are assumed to be twice
as wide as those of the 1-byte set in G2 (and the US-ASCII characters in
G0). Thus the sets and their byte representations are as follows:
set 0: 00-7F US-ASCII (narrow)
set 1: GR GR 2-byte characters (wide)
set 2: 8E GR extra 1-byte set (narrow)
set 3: 8F GR GR extra 2-byte set (wide)
Some examples:
SHIFT-GB G1 = GB 2312-80
Chinese EUC G1 = GB 2312-80, G2 = row 8 of GB 2312-80 (half width)
Japanese EUC G1 = JIS X0208-1990, G2 = right half of JIS
X0201-1976, G3 = JIS X0212-1990
Korean EUC G1 = KSC 5601-1987
Somewhat different is
Taiwanese EUC G1 = CNS 11643-1986 level 1, G2 = 3-byte form of CNS
11643-1986
ISO 2022 designator sequences
The following escape sequences may be used to alter the assignments of the
graphic character sets.
esc ( ID shift 94-character set into G0
esc ) ID shift 94-character set into G1
esc * ID shift 94-character set into G2
esc + ID shift 94-character set into G3
esc $ ( ID shift multi-byte 94-character set into G0
esc $ ) ID shift multi-byte 94-character set into G1
esc $ * ID shift multi-byte 94-character set into G2
esc $ + ID shift multi-byte 94-character set into G3
esc - ID shift 96-character set into G1
esc . ID shift 96-character set into G2
esc / ID shift 96-character set into G3
Some implementations (for example, Mule) also use
esc , ID shift 96-character set into G0
but ISO 2022 does not permit a 96-character set in G0.
The ID, which ISO-2022 calls a final character, may be in the range 30-7E.
Values in the subrange 30-3F are reserved for private use, and 7E means an
empty set. Others are to be registered with ECMA before use. A summary of
the registry [ftp: ftp.cc.columbia.edu kermit/charsets/iso-registry.txt] is
available online.
Values of ID for 94-character sets include:
B US-ASCII (= left half of ISO 8859 sets)
I right half of JIS X0201-1976 (katakana)
J left half of JIS X0201-1976 (JIS-Roman)
Values of ID for multi-byte 94-character sets are:
@ JIS C 6226-1978
A GB 2312-1980
B JIS X0208-1990
C KSC 5601-1987
D JIS X0212-1990
E GB 2312-1980 plus GB 8565-1989
G CNS 11643-1986 level 1
H CNS 11643-1986 level 2
I CNS 11643-1992 plane 3
J CNS 11643-1992 plane 4
K CNS 11643-1992 plane 5
L CNS 11643-1992 plane 6
M CNS 11643-1992 plane 7
For the first 3 of these an older G0 designator esc $ IDis also allowed.
Values of ID for 96-character sets include:
A right half of ISO 8859-1:1987 (ISO Latin-1)
B right half of ISO 8859-2:1987 (ISO Latin-2)
C right half of ISO 8859-3:1988 (ISO Latin-3)
D right half of ISO 8859-4:1988 (ISO Latin-4)
F right half of ISO 8859-7:1987 (ISO Latin-Greek)
G right half of ISO 8859-6:1988 (ISO Latin-Arabic)
H right half of ISO 8859-8:1988 (ISO Latin-Hebrew)
L right half of ISO 8859-5:1989 (ISO Latin-Cyrillic)
M right half of ISO 8859-9:1989 (ISO Latin-5)
T Thai character set TIS 620-2533:1990
V right half of ISO 8859-10:1992 (ISO Latin-6)
NOTE ON ISO NOTATION
In ISO documents, bytes are called octets, and (except in ISO/IEC 10646) are
represented by pairs of decimal numbers. For example, the hexadecimal code
2D (the character `-' in US-ASCII) is represented as 2/13.
ISO 2022 shift sequences
SHIFTS FOR USE WITH 7-BIT ENCODINGS
The GL range may be switched between G0-3 with the locking shift sequences:
si (called LS0 in 8-bit form) lock G0 into GL
so (called LS1 in 8-bit form) lock G1 into GL
esc n LS2: lock G2 into GL
esc o LS3: lock G3 into GL
A single character from G2 or G3 may be placed in GL by preceding it with
esc N SS2: shift single character from G2 into GL
esc O SS3: shift single character from G3 into GL
SHIFTS FOR USE WITH 8-BIT ENCODINGS
The GR range may be switched between G1-3 with the locking shift sequences:
esc ~ LS1R: lock G1 into GR
esc } LS2R: lock G2 into GR
esc | LS3R: lock G3 into GR
In an 8-bit environment, the following single shifts are used:
8E SS2: shift single character from G2
8F SS3: shift single character from G3
The high bit on the following character is ignored.
See also the Note on ISO notation.
CCCII and ANSI Z39.64-1989 (EACC)
CCCII
CCCII was developed by the Chinese Character Analysis Group, based in
Taiwan. It is used in libraries there.
The 3 numbers making up a code are called plane, section and position. A
layer is a group of 6 consecutive planes. There are 15 layers, representing
parallel versions of the same characters.
The layers of CCCII are:
1 traditional characters
2 simplified characters
3-12 other variant forms
13 Japanese kanji from JIS C 6226
14 Korean hanja (?)
15 reserved
Layout of first layer:
1:1 graphic ASCII
1:11 Chinese punctuation marks (35)
1:12-14 Kangxi radicals (214)
1:15 Chinese numerical marks (41), phonetic symbols (37)
1:16-67 most frequently used hanzi (4808)
1:68-3:64 next most frequently used hanzi (17,032)
3:65-6:5 other hanzi (20,583)
Each group of hanzi is ordered by:
Kangxi radical number
count of residual strokes
lexicographical stroke order on residue: dian3, heng2, shu4, pie3, na4.
Sections 1-15 of other layers are said to be user space.
Other planes:
91 shifted JIS C 6226 kanji
92 reserved for Korean KIPS
93 reserved for supplementary Chinese characters
94 reserved for non-chinese characters of CB2
ANSI Z39.64-1989 (EACC)
ANSI Z39.64-1989 (RLIN East Asian Character Code Set, REACC or EACC) was
created by the Research Libraries Group (RLG) in collaboration with the US
Library of Congress and the Chinese Character Analysis Group. It has been
used by the Library of Congress and other libraries (mainly in North
America) since 1983. (Perhaps EACC and REACC denote slightly different
sets??)
The structure of the code is based on CCCII, but there are a few
differences:
there is no "other hanzi" set.
many rare characters and variants are omitted.
there are some substitutions, to further the "one character, one code"
principle. Characters that are now variants but used to have a life of
their own are coded as variants (e.g. simplified wan4 "10,000" or yun2
"cloud"). Also layer 2 is used mainly for official simplified forms,
rather that just any non-orthodox variant with fewer strokes.
layer 13, plane 1 contains 293 additional codes to cover the rest of JIS
C 6226-1983 (mainly kana and kokuji).
layer 14, plane 1 contains Korean hangul to cover KIPS.
layer 14, plane 2 contains 44 additional codes to cover the rest of GB
2312-80.
An ISO 2022 encoding of a 94x94x94-character set would use the multi-byte
designators with an ID in the range 60-6F, but none seem to be registered.
Unicode/ISO-10646
ISO 10646 was originally conceived as approximately a disjoint union of
national standards, yielding a 32-bit internal code with compact stateful
external encodings. Unicode was a competing standard, aimed at unifying
identical characters from different variants of the same script, to give a
16-bit code. They have since been unified: ISO/IEC 10646-1:1993 has two
forms, a 31-bit form (UCS-4) and a 16-bit form (UCS-2). UCS-2 is a subset of
UCS-4 (plane 0 of group 0), and is code-for-code identical with Unicode 1.1
(though Unicode imposes extra semantics on some characters). The aim is to
cover all the (reasonably widely-used) scripts in the world.
ISO/IEC 10646 was adopted as Chinese National Standard GB 13000 in December
1993. A Japanese translation is in the process of being adopted as a
Japanese standard, JIS X0221.
Unicode 1.0 is described by the Unicode books (see References). The changes
to make Unicode identical to ISO 10646 UCS-2, yielding first Unicode 1.0.1
and then Unicode 1.1, are relatively minor, and do not affect the hanzi set.
They are available as Unicode Technical Report #4. Various tables and
appendices describing Unicode 1.1, plus mapping tables to various CJK codes
are available at the Unicode site [ftp: unicode.org pub/]. See also the
Unicode page maintained by Glenn Adams.
UCS-2/UNICODE
The overall layout of this unified 16-bit code (also called the Base
Multilingual Plane or BMP of ISO-10646) is:
0000 - 1FFF A-ZONE: Alphabets: The first 256 codes are identical to
ISO 8859-1 (Latin-1), except that the 65 control codes
are excluded.
2000 - 2FFF A-ZONE: Symbols and Punctuation
3000 - 4DFF A-ZONE: CJK Auxiliary
4E00 - 9FFF I-ZONE: CJK Unified "Ideographs"
A000 - DFFF O-ZONE: Reserved for future assignment (but see UTF-16
for a proposed use)
E000 - FFFD R-ZONE: Restricted use - the codes FFFE and FFFF are
excluded.
All characters are also assigned names, but for the "ideographs" these are
merely "CJK UNIFIED IDEOGRAPH-4E00" and so on.
THE REST OF UCS-4
In the first version of ISO/IEC 10646, groups 60-7F and planes E0-FF of
group 0 are reserved for private use, while all other planes are reserved
for future standardization. This is expected to change in the forthcoming
revision: groups 01-0E of plane 0 will become available for definition,
groups 0F and 10 of plane 0 will become available for private use, while the
rest of the code space will be reserved (see UTF-16).
COMBINING CHARACTERS
These are characters for graphemes such as Semitic and Indic vowels, IPA
diacritics, tone marks and Hangul jamos. These characters are placed after
the base character; the rendering process is expected to generate the
composite glyph, in some cases by a table lookup. ISO 10646 defines 3 levels
of subset implementation:
level 1: base characters only.
level 2: base characters and certain combining characters,
roughly those required by Semitic and Indic-derived
scripts.
level 3: all characters.
Many redundant "presentation forms" are also provided (especially in the
additions that make Unicode 1.1), including the tone-letter combinations
used in Hanyu pinyin (but not those used in Zhuyin fuhao) and pre-composed
Hangul syllable blocks.
Unicode/ISO-10646 Alphabets
Groups marked with * have not yet been finalized.
0020 - 007E Basic Latin (graphic part of US-ASCII)
00A0 - 00FF Latin-1 Supplement (right half of ISO 8859-1)
0100 - 017F Latin Extended-A
0180 - 024F Latin Extended-B (extended in 1.1)
0250 - 02AF IPA Extensions
02B0 - 02FF Spacing Modifier Letters
0300 - 036F Combining Diacritical marks
0370 - 03CF Basic Greek (based on ISO 8859-7)
03D0 - 03FF Greek Symbols and Coptic
0400 - 04FF Cyrillic (based on ISO 8859-5, extended in 1.1)
0500 - 052F unassigned
0530 - 058F Armenian
0590 - 05CF Hebrew Extended-A
05D0 - 05EA Basic Hebrew (based on ISO 8859-8)
05EB - 05FF Hebrew Extended-B
0600 - 0652 Basic Arabic (based on ISO 8859-6)
0653 - 06FF Arabic Extended (extended in 1.1)
0700 - 08FF * Ethiopic (?)
0900 - 097F Devanagari (based on ISCII 1988)
0980 - 09FF Bengali (based on ISCII 1988)
0A00 - 0A7F Gurmukhi (based on ISCII 1988)
0A80 - 0AFF Gujarati (based on ISCII 1988)
0B00 - 0B7F Oriya (based on ISCII 1988)
0B80 - 0BFF Tamil (based on ISCII 1988)
0C00 - 0C7F Telugu (based on ISCII 1988)
0C80 - 0CFF Kannada (based on ISCII 1988)
0D00 - 0D7F Malayalam (based on ISCII 1988)
0D80 - 0DFF * Sinhala
0E00 - 0E7F Thai (based on TIS 620-2533:1990)
0E80 - 0EFF Lao (based on TIS 620-2533:1990)
0F00 - 0F7F * Burmese
0F80 - 0FDF * Khmer
1000 - 105F * Tibetan
1060 - 109F * Mongolian
10A0 - 10CF Georgian Extended
10D0 - 10FF Basic Georgian
1100 - 11FF Hangul Jamo (added in 1.1)
1200 - 125F * Ethiopian (?)
1E00 - 1EFF Latin Extended Additional (added in 1.1)
1F00 - 1FFF Greek Extended (added in 1.1)
Unicode/ISO-10646 CJK Auxiliary
3000 - 303F CJK Symbols and Punctuation
3040 - 309F Hiragana (based on JIS X0208-1990)
30A0 - 30FF Katakana (based on JIS X0208-1990)
3100 - 312F Bopomofo (based on GB 2312-80)
3130 - 318F Hangul Compatibility Jamo (based on KSC 5601-1987)
3190 - 319F CJK Miscellaneous (Kaeriten)
3200 - 32FF Enclosed CJK Letters and Months (months added in 1.1)
3300 - 33FF CJK Compatibility (extended in 1.1)
3400 - 3D2D Hangul (based on KSC 5601-1987)
3D2E - 44B7 Hangul Supplementary-A (rows 16-36 of KSC 5657-1991,
added in 1.1)
44B8 - 4DFF Hangul Supplementary-B (rows 37-54 of KSC 5657-1991,
added in 1.1)
The punctuation set does not include separate codes for punctuation (e.g.
brackets) in horizontal and vertical text. The rendition of these characters
is considered outside the scope of the code. The advantage of this approach
is that text may be reset in a different direction by merely changing the
outside declarations.
The Hangul elements here are redundant presentation forms that can be coded
with Hangul Jamo 1100-11FF.
Unicode/ISO-10646 Unified "Ideographs"
The CJK character set is derived from a merger of a set proposed by Unicode
(an extension of the EACC list) and a draft for a proposed Chinese standard
GB 13000. Since February 1990, it has been developed by the CJK Joint
Research Group. A set comprising 20,902 characters was issued in March 1992,
and is variously called
UniHan (by Unicode),
Han Character Set or HCS (by the Chinese),
Ideographic CJK Character Unified Repertoire and Ordering, or URO 2.0 (by
ISO, because the term "Han" is inappropriate in a Japanese context, and
only ISO WG2 is allowed to create a "Character Set").
It includes all hanzi/kanji/hanja from the following sets:
GB 2312-80, GB 12345-90 and GB 8565-89
CNS 11643-1986 planes 1 and 2
JIS X0208-1990 and JIS X0212-1990
KSC 5601-1987 and KSC 5657-1991
The successor of the CJK-JRG is the Ideographic Rapporteur Group (IRG) of
ISO SC2/WG2. The IRG is compiling lists of additional characters: HCS-B
comprises about 8000 characters, and China has asked for another 1000 in
addition to these. There is a lot of competition for the remaining space in
UCS-2.
HAN UNIFICATION
This refers to the unification of Chinese-derived characters used in China,
Taiwan, Korea, Japan and Vietnam, without which a 16-bit character set would
be unattainable. The idea was that two characters having the same abstract
shape would be unified, regardless of meaning or font differences. Users of
different languages will require different visual representations of the
codes, but this is regarded as a font/language issue to be handled by
declarations extraneous to the codeset. Of course, the distinction between a
variant form and a font difference is sometimes subjective.
Considerations of character identity are overridden by the Source Set
Separation Rule: distinct codepoints in any of the following source sets are
mapped to distinct codepoints in the new set:
GB 2312-80, GB 12345-90, GB 7589-87, GB 7590-87 and GB 8565-89
CNS 11643-1986 planes 1, 2 and 14
JIS X0208-1990 and JIS X0212-1990
KSC 5601-1987 (minus the repeated hanja mentioned there) and KSC
5657-1991
The importance of this is that if you translate text encoded in a national
character set into Unicode and back again, you get the same text you started
with. The result is a fairly conservative unification, e.g. simplified forms
of most radicals make for different characters.
ORDERING
The ordering of the CJK characters is based on the following dictionaries,
in order of priority:
Kangxi Zidian (7th ed.) Zhonghua Bookstore, Beijing, 1989.
Dai Kanwa Ziten (revised ed.) Taisyuukan Syoten, Tokyo, 1986.
Hanyu Da Zidian (1st ed.) Sichuan Cishu Publishing, Chengdu, 1986.
Dae Jaweon (1st ed.) Samseong Publishing, Seoul, 1988.
From the Unicode book, volume 2, page 14:
When a character is found in the KangXi Zidian, it follows the Kang
Xi Zidian order. When it is not found in the KangXi Zidian and is fou
nd in Dai Kanwa Ziten, it is given a position extrapolated from the K
angXiposition of the preceding character in Dai Kanwa Ziten. When it
is not found in either KangXi or Dai Kanwa, Hanyu Da Zidian and Dae J
aweondictionaries are consulted in a similar manner.
Hence the ordering is approximately Kangxi radical, then stroke count.
However, a number of simplified are treated separately, and are placed
immediately after the corresponding traditional radical.
Unicode/ISO-10646 Restricted Zone
E000 - F8FF Private use Area
F900 - FAFF CJK Compatibility Ideographs (the duplicate hanja from
KSC 5601-1987, the two duplicate Big-5 characters
excluded from CNS 11643-1986, and some extra
characters from corporate codesets)
FB00 - FB4F Alphabetic Presentation Forms (added in 1.1)
FB50 - FDFF Arabic Presentation Forms-A (added in 1.1)
FE20 - FE2F Combining Half Marks (added in 1.1)
FE30 - FE4F CJK Compatibility Forms (verticals and overlines)
FE50 - FE6F Small Form Variants
FE70 - FEFE Arabic Presentation Forms-B (re-arranged in 1.0.1)
FF00 - FFEF Halfwidth and Fullwidth Forms (for full round-trip
compatibility with existing ASCII+CJK coding schemes;
extended in 1.1)
FFF0 - FFFD Specials
There is also a special character
FEFF ZERO WIDTH NO-BREAK SPACE (formerly BYTE ORDER MARK)
Note also
FFFD REPLACEMENT CHARACTER
which can be used for characters not representable in Unicode/ISO-10646.
Unicode/ISO-10646 Symbols and Punctuation
2000 - 206F General Punctuation
2070 - 209F Superscripts and Subscripts
20A0 - 20CF Currency Symbols
20D0 - 20FF Combining Diacritical Marks for Symbols
2100 - 214F Letterlike Symbols
2150 - 218F Number Forms
2190 - 21FF Arrows
2200 - 22FF Mathematical Operators
2300 - 23FF Miscellaneous Technical (extended in 1.1)
2400 - 243F Control Pictures
2440 - 245F Optical Character Recognition
2460 - 24FF Enclosed Alphanumerics
2500 - 257F Box Drawing
2580 - 259F Block Elements
25A0 - 25FF Geometric Shapes
2600 - 26FF Miscellaneous Symbols
2700 - 27BF Dingbats
Encodings of Unicode/ISO-10646
Unicode/UCS-2 text is intended to be represented internally (for processing)
as 16-bit values. For transmission and external storage a number of byte
encodings have been proposed.
DIRECT ENCODINGS
The simplest is a direct representation as two bytes, most significant
first, as specified in ISO-10646, and in the UNICODE-1-1 [ftp:
ftp.internic.net rfc/rfc1641.txt] scheme registered for use with MIME [ftp:
ftp.internic.net rfc/rfc1521.txt] (Multipurpose Internet Mail Extensions).
A variant (see informative Annex F of ISO-10646) allows the order of the two
bytes to depend on the machine that produced the text, but to begin the
stream with the character
FEFF ZERO WIDTH NO-BREAK SPACE (formerly BYTE ORDER MARK)
which will establish the byte order, since FFFE is not a legal code.
UTF-1
The UCS Transformation Format (UTF, or UTF-1) for ISO-10646 (see informative
Annex G of ISO/IEC 10646-1:1993) specifies a variable-length encoding
avoiding C0, C1, DEL and SPACE octets, but appears to be dead already.
UTF-8
Another encoding, proposed by X-Open, is variously called UTF-8, FSS-UTF,
UTF or UTF-2. It has been proposed as normative Annex P to ISO/IEC 10646.
See also Hello World [ftp: ftp.research.att.com dist/plan9man/09utf.ps.Z],
by Rob Pike and Ken Thompson. Some properties of this encoding:
ASCII files are also UTF-8 files.
only ASCII characters use ASCII bytes.
simple synchronization.
many operations need not decode.
chars 0800-4015 (Asian alphabets, symbols & CJK auxiliary) have longer
encodings than in UTF-1.
both UCS-2 and UCS-4 are covered.
UTF-7
UNICODE-1-1-UTF-7 [ftp: ftp.internic.net rfc/rfc1642.txt] (UTF-7), an
encoding of UCS-2 using only mail-safe bytes, has been registered for use
with MIME [ftp: ftp.internic.net rfc/rfc1521.txt]. Some properties:
The encoding of mail-safe characters is readable.
Other characters are encoded in a base-64 code, with 3 characters
occupying 8 bytes.
UCS-4 text may be pre-coded with UTF-16.
Conversion functions for UTF-8 and UTF-7 [ftp: unicode.org
pub/Programs/ConvertUTF/] are available from the Unicode site [ftp:
unicode.org pub/].
UTF-16
UTF-16 (formerly UCS-2E) is not a byte encoding, but a scheme for
representing certain UCS-4 codes in a UCS-2 stream. (Recall that no UCS-4
codes outside UCS-2 have yet been defined.) It has been proposed as
normative Annex O to ISO/IEC 10646. Characters in the range 10000 to 10FFFF
are represented by a pair of O-zone codes, the first in the range D800-DBFF
and the second in the range DC00-DFFF.