Subject: Re: GB versus BIG-5

From: Benjamin Ao <[email-masked]>
Date: Tue, 26 Nov 1991 10:12:31 -0500

I agree with Stephen that BIG-5 is more useful, because it's a superset
of GB. I think it's rather narrow-minded to say that we don't need any
of the edge of BIG-5 over GB as long as we stick to the simplified
characters. Traditional characters are a reality to be reckoned with by
any serious language scientist, as long as there are people using them
and there are materials written or printed in them. The difference
between traditional and simplified characters is not the issue here.
The real issue is how many characters should be included in a character
set to be used on a daily basis, and what can be done about characters
that are not included. I believe that there is an optimal solution to
this problem. And such a solution should be based on the statistical
properties of the characters. True, there has never been any research
of the statistical properties of all the characters used in all walks
of life in the past hundred or so years. Yet, the probability of
occurrence of any linguistic unit, including the character, is dictated
to a large degree by the structure of language. Whatever material we
use, the more frequently used characters will be the same, although each
character's ranking in relative frequency of occurrence may vary.
According to the study I referred to in my last posting, 3000 characters
already cover 99.15% of all the characters in that corpus of data. Any
other corpus of data will yield the same result, although the composition
of the 3000 character may not always be the same. In other word, the over-
whelming majority of characters have a very low probability of occurrence.
If 7754 characters already covered 100% of the over 11 million characters
used in a huge corpus of data that included studied in literature,
philosophy, linguistics, etc, then it's not likely that any other character
will have a more than 1 millionth probability of occurrence. This is not
to say that none of those infrequent characters will ever be used in a
text, in fact some of them do pop up occasionally. The question is whether
this justifies the inclusion of most or all of the infrequent characters
in a basic charater set, and if not, how big should the basic set be.

---

From: Ned Walsh <[email-masked]>
Date: Wed, 27 Nov 1991 00:29:10 EST

Just my 2 cents...

Benjamin Ao wrote:

>of GB.

Not exactly.  Big-5 is not a superset of GB.  It does not include
simplified characters, which should be considered variants of their
traditional forms.  Very often I need traditional characters,
simplified characters, and Japanese touyou kanji and kana all in the
same file.  This is impossible with a GB-based system.  It is
possible, though awkward, in ETen and other Big-5 systems by putting
simplified characters in a separate userfont file allowed for by the
coding system; of course, though, there is no standardized simplified
character userfont file, so if you want to share a Chinese file the
userfont file must go with it.  ChinaStar's character set allows
mixing simplified and complex characters if you're willing to pay
something like $900 for their ROM-card system; but then you're stuck
with a file that can only be read by ChinaStar (there is no utility to
convert ChinaStar files to GB or Big-5 code).  Tianma and Guoqiao
support both traditional and simplified characters--but only one set
at a time; you can't mix the two in the same file.

There are plenty of serious Chinese computer users who would be
willing to sacrifice RAM and disk space to accomodate a broader
character set than that offered by either Big5 or GB.  Not that every
version of every application must support all the characters in a set,
but there needs to be a standardized code and public domain bitmap
fonts for far more than 8,000 or 13,000 characters.

It would be nice if the Taiwanese Ministry of Education would design a
series of standardized Big5 userfonts--one for simplified characters
(one country, two systems:) ) and others for, say, Chinese
Buddhist texts, or classical poetry, or dialect-specific characters,
or Touyou kanji.

As I understand it, the PRC's GuoBiao code was supposed to include a
primary set of codes (GB 2312) for commonly used, simplified
characters, and auxiliary sets to cover traditional forms and
dialect- or region-specific characters.  I assume the auxiliary sets
idea never got very far.  Anyone know?

Where's Unicode when you need it?

And while I'm in a whining sort of mood, I might as well mention
there's still no solution to the sorting problem I mentioned last
month.

/Ned

p.s.

As for simplified vs traditional: Is there any hard data indicating
that simplified characters are any easier to learn than traditional
characters?  or indicating that they are easier to read (i.e. fewer
"misreadings")?  I suspect it makes no difference, that the only
significant effect of simplifcation is to increase writing speed and
make printing easier.  It also is my impression that people who learn
simplified characters first have a very difficult time adjusting to
traditional characters, whereas people who have learned traditionals
first adjust to simplified rather quickly after the initial shock
wheres off.  I suppose it's the old problem of non-commutivity of
simplification rules.  In any case, in most situations I suppose it
doesn't really matter whether you use simplified or traditional forms.
But in some situations it's essential that you reproduce as closely as
possible the forms of the characters used in the original text--for
example, in poetry, when you might be trying to spot visual puns, or
clues for dating and placing a text. It's somewhat analogous to
preserving old spellings for pre-modern English literature.  But I
digress from CCNET-ish interests...forgive me.

---

From: Benjamin Ao <[email-masked]>
Date: Wed, 27 Nov 1991 09:28:22 EDT

> simplified characters, which should be considered variants of their

	When I said BIG-5 was a superset, I was assuming that the simplified
characters share the same coding with their traditional counterparts, but
are put in a different font.

> same file.  This is impossible with a GB-based system.  It is

	This is not true. As long as the text editor can handle multiple
fonts in a text, it is possible, provided that the missing characters
involved in the one-to-many mapping are also supported.

> userfont file must go with it.  ChinaStar's character set allows

	Standardization is indeed very important.

> at a time; you can't mix the two in the same file.

	You may tell the developers about this. It shouldn't be too hard
to fix this problem.
> fonts for far more than 8,000 or 13,000 characters.

	In theory, we need as many formally coded characters as there
have been on the earth, i.e. well over 90,000. Imagine, for examply,
what would be called for if someone decided to eletronicize Dai Kanwa
Jiten. In practice, however, this does not mean every font file must
contain over 90,000 characters, although they probably should all be
formally coded. The coding scheme need not be one dimensional. Rather,
all these characters can be divided into groups of say 8000 or even even
fewer characters. Each group is treated as a different font, and given
a standard font name. Thus, character A and character B may be totally
unrelated, but nevertheless share the same code in two different fonts.
This idea is similar to Ned Walsh's suggestion below. This way, one does
not need to sacrifice too much for a single giant font, but has the
flexibility of choosing from a number of small font according to his/her
needs. Of course, in order to do so, the text editor must have a good
font management capability, which is not at all difficult to implement.