Subject: More =\= Better -- GB or BIG5

From: Benjamin Ao <[email-masked]>
Date: Sun, 24 Nov 1991 10:13:21 -0500

The evaluation of BIG5 versus GB is not a simple one. I think
this should be described in terms of relative, not absolute, merits.

	GB contains fewer characters, and is therefore more likely
to encounter the problem of not including an existing and useful
character. This is defintely its disadvantage. Yet, being a smaller
character set, both the coding table and the font files are smaller
in size. In cases where the memory and the storage are not unlimited,
this is clearly an advantage.

	On the other hand, BIG5 is undoubtedly less likely to have
a character missing problem, being a larger character set. Yet, just
because it's larger, it occupies more memory and storage space, and
therefore does pose a problem for smaller computers.

	A simple arithmetic shows that, ignoring the punctuation
marks and other symbols, the GB set contains 94 x 72 = 6768 (6763
used) characters, and the BIG5 set contains 191 x 84 = 16044 (15875
used) characters. Therefore, a 16x16 GB font is 216.576 KB, but a
16x16 BIG5 font is 513.408 KB; and a 24x24 GB font is 487.296 KB,
but a 24x24 BIG5 font is 1155.168 KB. There is a 296.832 KB difference
in the first case, and a 667.872 KB difference in the second. Consider
cases where more fonts with higher resolution are needed, the
difference in storage requirement between GB and BIG5 will become
greater. These figures may not sound very impressive for people who
are used to computers with almost unlimited memory and storage space,
but for those who have to confine themselves to the limits of small
computers used by private owner and small businesses, these are
serious problems.

	Generally speaking, more is not always better. It all depends
on what most people need most of the time. If we concede that most
people use Chinese word processors to process modern Chinese language
data, then an appropriate character set should contain about the same
number of characters as those used in modern Chinese. According to
a 1985 Chinese character frequency assessment, whose scale is the
largest in history, a total of 7,754 characters appear at least once
in a literature composed of 11,873,453 characters, which was published
in mainland China from 1977 to 1982 covering subjects in natural and
social sciences. Allowing for some variance, 8000 is probably a close
estimate of the number of characters used in modern Chinese. If this
is true, then the 6768 total of GB is definitely closer to this
estimate than the 16044 total of BIG5. In other words, about half of
all the characters defined in BIG5 will probably never be used.

	On the other hand, even BIG5 proves to be too small a character
set if we are going to process pre-modern Chinese texts. Kangxi Zidian
contains 47035 characters, and the largest collection of Chinese
characters appears to be a Chinese-Japanese dictionary called Kanwa
Dai Jiten, which reportedly contains well over 90,000 characters. I
haven't checked out that dictionary and counted the characters in it
myself. But if this report is true, then it's bad bad news for
advocates of UNICODE who believes 65 K is more than enough. The worst
thing about this that one never can tell which of the tens of thousand
pre-modern character may appear in any given pre-modern text, and there
has been no statistics of the frequency of usage of these characters.

	Given the statistical properties of Chinese characters, and to
achieve the maximum cost-effectiveness, I don't think it is wise to
advocate for arbitrarily large character sets. Rather, we should be
content with a basic set which contains somewhere in the neighborhood
of 8000 characters, or maybe simply with a slightly modified GB set
that no longer has the problem of the one-to-many relationship between
simpified and traditional characters, and that includes some regional
characters used in Japan, Hong Kong etc. Additional and pre-modern
characters could be supplies as additional font packages. In other
words, modern and pre-modern characters may share the same code, but
are differentiated with their font names.

	This solution to the Chinese character coding problem may not
be very elegant, but is much more efficient than coding systems that
enclude tens of thousand almost useless archaic characters to engulf
our otherwise valuable and useful memory and storage space.

	Benjamin Ao
	Department of Linguistics
	Ohio State University

---

From: Ya-Gui Wei <[email-masked]>
Date: Sun, 24 Nov 1991 12:24:56 -0500

Benjamin,

I agree with you whole heartedly. Increasing the size of the
code table is not a solution for solving the rare character
problem. But rather, the solution should be to define a set
of character construction rules.

Neither the ASCII table nor the English keyboard layout pretends
to include all the symbols that might appear in a piece of
English text. The solution is left to the individual text
processing softwares. This is where software packages like Latex
comes in.

I posted a proposal to CCNET not long ago about the character
construction rules. Perhaps it is somewhat crude and far from
complete - I was only meaning to illustrate a principle.  Note
that one does not neccessary need a set of infinitely scalable
fonts to serve certain construction rules - bitmap fonts can
be shrunk with respectable results.

  Ya-Gui Wei