Subject: Big 5 and scholarly usage

From: Dorothy Day <[email-masked]>
Date: Sun, 24 Nov 1991 14:35:00 EST

Recent postings have pointed out that suitability of the size and
selection of the character database of a textprocessing system depends
heavily on the uses to which it is put. Thus, daily business needs
(such as letter-writing) can often be met even by the numerous
electronic typewriters with extremely limited character availability
produced in Taiwan, Japan, and mainland China.

However, traditional literature, history, and other scholarly areas
require a much larger character base. This need not be "every
character in the Dai Kanwa Jiten (ca. 90,000)"; a well-selected
character set could be far smaller and still meet 99.44% of one's
needs. China Star's character generator ROM card contains something
over 40,000 traditional and 20,000 simplified characters; in several
years working with traditional Chinese fiction (probably the richest,
or at least largest, vocabulary in all the literature), I have had to
customize maybe five obscure characters that could not be pulled off
the ROM card.

For a long time I felt that systems that did not provide at least this
large a character base were unsuitable for working is traditional
areas of scholarship. This meant that TianMa (whose character base is
taken from the set in Xinhua Zidian) would be unsuitable, even though
its character input is superior to any other except the Xerox Star
system. Both use contextual analysis to select the correct characters
for phrases typed in.

I'm beginning to suspect that the selection in the Xinhua Zidian has
been done with sufficient care that 98-99% of the characters in any
given document will still be found in the character base. In a word
processor, having to customize the creation of one character per
document is a nuisance but not a hardship, if the character creation
software is sufficiently friendly. This, TianMa may abe more adequate
than I had supposed.

In a character encoding system like Unicode, much closer to universal
coverage is essential for electronic transmission. However, that still
does not have to mean "every character in the Dai Kanwa Jiten." A
substantial number of the occurances in that dictionary have been
determined to have occurred in perhaps only one or two source
documents, where a clerk made a copying error. An error that is picked
up and repeated becomes part of the written language, but isolated
errors do not make for separate characters. The committees working on
Unicode have been able to eliminate a substantial number of these
isolated erroneous forms. If need be, they can be put in separate
subsets defined by user groups; this is how Unicode will handle many
EXTREMELY rare characters from the world's languages, and space for
"special subsets" will be reserved in the code, to be enabled by the
appropriate set tag.

BTW, has anyone on this list tried Xin TianMa, which is now entirely
hard-disk based (but with a hardware "key" on the parallel port)? I
know it's been released, but haven't spoken with anyone who has used
it.

Dorothy Day, Indiana University
DAY@IUBACS or [email-masked]