Subject: GB vs BIG-5: technical point of view

From: [email-masked]
Date: Wed, 27 Nov 1991 19:06:47 EST

The fundamental problem faced by the any Chinese encoding scheme designer
is that there are too many Chinese characters.  Many consider Big5 better
than GB2312-80 because Big5 has more characters.  However, from technical
point of view, the Big5 has several problems.

The biggest problem is that Big5 contains too many characters in one
character set, so that them can not be put into one 2-byte coding plane.
In ISO 2022 (international standard for encoding and combining character
sets), a 2-byte code set is encoded in two 7-bit octets, each octet in
0x21-0x7e.  So a 2-byte code set can at most hold 94x94 = 8836 characters.
Therefore, Big5 shouldn't be defined as a 2-byte encoding at all.  It
can be a 3-byte encoding as in CNS 11643.

Not conforming to ISO 2022 standard, Big5 has many technical problems.

First, Big5 is not an acceptable standard in many internationalized
software system, although it is widely used in Taiwan and elsewhere.
For example, Big5 does not work well with X window system.  You cannot
use Big5 in inter-client communication such as cut and paste, window
properties, resources, etc.  You have to either sacrifice Big5, as in
kterm, where Big5 code is converted to GB first before cut & paste; or
sacrifice the the generality, as in cxterm, where X11 Compound Text
Encoding standard is not followed.  Either way, Big5 is frustrated.

Second, Big5 cannot be efficiently converted to 7-bit internal code
representation.  Although the most practical systems exclusively use the
double-byte 8-bit internal code representation, i.e. use "1XXXXXXX ?XXXXXXX"
for a Chinese character, GB2312-80 defines only 7 bits in each byte,
(i.e. "_XXXXXXX _XXXXXXX").  GB can be represented in 7-bit environment
(i.e. "0XXXXXXX 0XXXXXXX") by using standard ESC sequence, as defined in
ISO 2022.  Efficient representation of a code set in 7-bit environment
is essential because such code set can extend and co-exist with other
code sets.  Once needed, all the GB text we currently used, such as XHWZ,
poems, etc, can be easily converted into 7-bit and fit into ISO platform.
Moreover, if many uncommon Chinese characters are required, a new code set
of rarely used characters can be designed and join in.  Big5 simply
doesn't have such expanding capability.

Third, the mixture of two coding planes in one code set in Big5 adds extra
difficulty to the software development.  For example, in GB encoding, it
is straightforward to tell whether any given byte is a part of a Chinese
character or not.  But in Big5, it is much more complicated, sometimes you
need to trace back to the beginning of the text.  As another example,
elvis/celvis already has multi-font feature.  Make it a 7-bit Chinese editor
for GB encoding is quite possible, but for Big5 encoding, I don't have
any solution yet.  In my development of cxterm and celvis, I found Big5
had led to many problems which would not have existed otherwise.  Therefore,
Big5-oriented system is intrinsically more complex.

It is quite interesting that many software systems I know of, which are
developed outside China, HK, and Taiwan (means that you have absolute
freedom to choose GB or Big5), are either designed for GB exclusively,
or designed for GB first and added Big5 later.  For instance, ChTeX, HZ,
and many PC Chinese editors support GB only.  Cxterm, celvis, xhzview,
pmTeX, etc., were initially designed to support GB only, Big5 supporting
feature were added later.

Forth, Big5 cannot be easily split into 2 ISO conforming character sets.
Big5 is divided into 2 levels, level 1 (5401 characters) and level 2
(7650 characters).  These two levels should have been put into separated
planes to ensure efficiency.  However, Big5 isn't arranged this way.
Level 1 occupies the lower space of both GL plane ("1XXXXXXX 0XXXXXXX")
and GR plane ("1XXXXXXX 1XXXXXXX"); level 2 occupies the higher space
of both.  Since GL and GR set must use different escape sequences, one
has to switch back and fore all the time, adding unnecessary overhead.
If the two levels were put into separated planes, say, level 1 in GR
and level 2 in GL, it would be more efficient.  We could simply stick
with GR and occasionally switch to GL (because the level 2 is not so
commonly used), which was acceptable.

I think GB may be a technically better way to start.  It conforms ISO
standard and makes itself really adaptable.  It is extendible to work
together with other character sets.

If we stay with double-byte code, we can also absorb Big5 into a "twin
character sets double-byte" scheme: put the common used character set,
GB2312-80 in GR plane ("1XXXXXXX 1XXXXXXX"), and put another less
frequently used character set (including traditional characters) in GL
plane ("1XXXXXXX 0XXXXXXX").  This scheme has many advantages over Big5:
it can be made acceptable to most internationalized systems; it can be
easily expanded and efficiently converted into 7-bit; and it can have
more characters than Big5 (max: 17672 = 2*94*94).  The disadvantage is
that there is no immediately distinction between a Chinese byte and a
ASCII byte, hence it is hard to process, but no harder than Big5.

CNS 11643 is another choice.  It's a 3-byte 7-bit encoding, level 1 and
level 2 of Big5 are put into different planes in CNS 11643.  However,
if CNS 11643 is put in double-byte internal code representation, there
will be no place for many frequently used simplified characters.

Here I am not intended to judge which one is overall better, Big5 or
GB, nor to say which one you should use.  Neither do I comment the
future trends of encoding systems, including ISO 10646 and Unicode,
which require rewriting of almost everything.  I simply stated the
technical problem I saw of Big5, under the current framework of ISO
2022 encoding system.  Comments are welcome.

Yongguang Zhang
Purdue Dept of Computer Sciences

Appendix 1:  GB encoding standard

By GB we usually refer GB2312-1980, "Code of Chinese Graphic Character
for Information Interchange, Primary Set".   It was announced in 1980,
and registered with ISO.  The "final character" of ESC sequence assigned
by ISO is `A'.  (`B' is assigned to Japanese JIS X0208-1983, and `C'
to Korean KS C5601-1987.)

Around 1983, a CCIPS (China Computer & Information Processing Standard-
ization) technical committee is formed to be parallel with ISO/TC97;
CCIPS/SC2 is formed to be parallel with ISO/TC97/SC2. In 1984 CCIPS/SC2
proposed 6 sets of "Code of Chinese Graphic Character for Information
Interchange".  All sets were designed to be in double-byte 7-bit encoding.
Here is the list.

Basic Set:
  GB2312-1980, national standard, announced in 1980.
  7445 graphic characters: 202 symbols, 60 ordinals, 22 numbers,
	52 Latin, 169 Kana, 48 Greek, 66 Russian, 26 pinyin symbols,
	37 zhuyin (bopomofo), 6763 Hanzi (simplified).
    hanzi:
	level 1: 3755 (in pinyin pronunciation order),
	level 2: 3008 (in radical order).

1st Aux Set:
  GB12345-90, recommended national standard, announced in 1990.
  Traditional counterpart of the Basic Set.
  7583 graphic characters: 682 non-Hanzi (same as in the Basic Set),
	35 punctuations and pinyin, 6886 Hanzi (traditional).
    in 6886 Hanzi:
	level 1: 3755,
	level 2: 3008,
	level 3: 103 (many-to-one traditional characters in level 1, 2).

2nd Aux Set:
  GB7589-87, announced in 1987.
  Less frequently used characters than the Basic Set (GB2312).
  7237 graphic characters: all simplified Hanzi (in radical order).

3rd Aux Set:
  yet to be announced, under final preparation.
  Traditional counterpart of the 2nd Aux Set.
  7237 corresponding traditional Hanzi.

4th Aux Set:
  GB7590-87, announced in 1987.
  Really less frequently used characters.
  7039 graphic characters: all simplified Hanzi (in radical order).

5th Aux Set:
  yet to be announced, under final preparation.
  Traditional counterpart of the 4th Aux Set.
  7039 corresponding traditional Hanzi.

No software system or font for any GB Aux set is available.

Reference:
The Status and Progress of Chinese Language Processing Technology
ACCC, International, Beijing, China, 1991