From: [email-masked]
Date: Mon, 4 Nov 1991 17:40:33 GMT

Dear Xiaofei and all others who might be interested in the problem,

Xiaofei indeed points out a very important question. I try to give some
hints by by looking at the way other people have dealt with the problem.

- Shenzhen University: Hong Lou Meng Retrieval System and the Quan Tang Shi
  Retrieval System:
  At Shenzhen University both projects used the GB 2312 code with only
  some 7000 simplified characters. I will not discuss the problem of
  encoding classical Chinese text with simplified characters. As long as
  you use a reference edition in simplified characters the problem will
  not be a problem for the computing people, but the problem must have
  been solved already by the editors of the paper edition. Mainland China
  scholars have produced quit a lot of final editions of important
  classical texts in simplified characters (the best edition for example
  of `Ru Lin Wai Shi' is the Shanghai Guji Chubanshe edition of 1984 in
  simplified characters).
  For Hong Lou Meng there have been some 300 extra characters not found in
  the GB 2312 code. These characters have been created by the character
  generater program (which should be part of every Chinese system), and
  which have been place on empty positions in GB-code. Empty positions
  are for example QU 10 to 15 and QU 88 to 94, altogether 13 QU. Since
  each QU ca accomodate 94 characters, there can be accomodated additional
  1222 characters. Of course, these additional characters do not corespond
  to any standard coding scheem. So if somebody else than the original
  author wants to work with the Hong Lou Meng System, it will not be enough
  to give the electronically available text, but there must go with the
  text the enlarged character fonts.

  With the Tang-Dynasty Poetry the Shenzhen people had much greater problem,
  because there have been more than 2000 characters not found in the
  GB-code. Since there have not been enough empty positions they also put
  Tang-Poetry characters on GB-code positions with characters which
  do not appear in Tang-Poetry. Thus many of the special characters,
  including Japanese hiragana and katakana are overwritten by Chinese
  characters. Personally I do not consider this a very elgant solution,
  but sticking to GB-2312 there is no other way!

- Academy of Social Science, Beijing: Quan Tang Shi System (also!).
  They chose a different approach. First they voted for traditional
  characters. They designed their own coding system together with their
  own Laser Printer system. I think this approach is not advisable,
  because it's not based on any accepted standard. Moreover one has
  to do every thing by oneself. Why did they do so? Because nobody should
  be able to illegaly copy the texts. Or, if one gets a illegal copy
  nobody can use it, because of the non-standard coding system. I can
  understand this approach, since this is a very excellent electronic
  version of the Quan Tang Shi with many times of proofreading and
  quite a few man-years (or should I say woman-years, when I think of
  the many young woman doing the hard work of inputing) invested into
  the project. Nevertheless the responsible people contemplate to
  produce a BIG-5 version of the Quan Tang Shi system

- Academia Sinica, Taibei: 25 History project: Through this project
  the 25 Official Histories are available in electonic form. The datefiles
  are available in two coding scheems: 1) CCCII-code, which is developed
  and maintained by people closely related to the Academia Sinica.
  The CCCII-code is a multy-byte code with up to 50.000 Chinese
  characters, which needs special software to handle the code.
  Readily availble is a library system and a text system. The retrieval
  system for the 25 Histories has been specially programmed for the
  project. Thus it might be difficult to use the CCCII code.

  The Academia Sinica can convert the text of the 25 Histories also
  to the BIG-5 code. In the BIG-5 code - at least in the version of
  ETEN - the user defined characters are collected in a special file,
  which allows up to 5000 characters. Thus: should one ever get some
  date files form the 25 Histories project, they will be acompanied
  by some USERFONTS containing the non-BIG-5 characters.

Having reviewd some famous projects with classical texts it should
be possible to come to some conclusion or suggestion of how to handle
such texts with the available codings scheems.

- Each project concerned with one larger text or a group of related
  text should be coded with a standard coding scheem. It should be
  roughly known whow many non standard characters are to be expected.
  If it's more than some 1000 characters one should use BIG-5 sytems
  such as ETEN, because the GB-code does not allow comfortably for more
  than some 1000 new characters. If there are not too many new characters,
  one can consider the source: if in simplified characters, then GB-2312,
  if in tradtional characters, then BIG-5.

- A definit advantage of a sytem like ETEN (BIG-5 coding) is, that the
  user defined characters are in a separate font, which is much easier
  shipped together with the text than if standard and user font are
  in one big file.

- I think the special fonts for non-standard characters should go together
  with one special text (or may by a groupe of realted texts). I think in
  this way the fonts are easier to handle, than to accumulate all the
  non-standard characters which might appear - say on the level of a
  sinological institute - in one single file. Such large font files might
  become very confusing expecially when date are exchanged between different
  institutions.

- Special fonts which go together with some texts or text groupes should
  be documented. Some programs (as ETEN) provides such an option within
  the character generator program. Such a printed documentation will also
  be useful or even necessary for the input of these characters.
  As soon as larger coding scheems (as UNICODE oder 10646) and user
  software based on it become available, it will not be very hard to
  convert textfiles from a standard coding scheem (GB-2312 or BIG-5)
  supplemented by some user characters to the new system.

What we considered until now is how to handel missing characters when
doing electronic editing. But what to do when transmitting some text
via email. Normaly it is not possible or at least not very handy to
transmit also the necessary font files.

Wei Ya-gui suggested a standard language to describe characters that
cannot be found in a standard character set. If there exists such a
language it could indeed be a useful tool for occasional missing
characters.

Here I would like to suggest another solution, which might be more
appealing within sinological scholarship. When we don't know a character
we look it up in a dictionary. When a coding scheem doesn't know a
character we could also turn to a dictionary and indicate a *unique*
identification of the character in that dictionary. To be useful for
communication such a dictionary should fullfil two creteria: 1) the
characters should be numbered consecutively (no reference by page etc.) and
this numbering should be rather constant during conscutive editions.
2) Such a dictionary should be available in all the major Sinological
institutions.

As reference dictionaries could be used:

- MOROHASHI, within western Sinology this dictionary should be quite
  popular, but I don't know whether it is also accepted in Sinological
  institutions in Mainland China and Taiwan. Contains characters numbered
  from 1 to 48.902.
- Zhon Wen Da Ci Dian, Taibei 1968. Contains characters numbered from
  1 to 49.888.
- The Han Yu Da Zi Dian, Wuhan 1990, which seems to be one of the most
  complete character dictionaries, unfortunately contains no consecutive
  numbering.

In order to distinguish such a number from ordinary numbers, that might
appear in a Chinese text, they should be inclosed by braces (e.g. XYXY
XY<23456><01666>XYXY where XY stands for a Chinese character). When one
gets some files containing such numbers one can look up the corresponding
Chinese character. If necessary I might then incorporate this character
with the fontgenerating program into a USERFONT of my system. With a
simple replace machanism of any editing system these numbers will then
be replace by the characters in my userfont.

Since by now I am no more a active classical sinologist, I would like
very much to hear from experts, what dictionary - if at all - could be
useful. I would like very much to summarize different opinions on
the subject and thus help to find a workable solution for it.

urs

----------------------------------------------------------------------------
Urs Widmer            Tel.:  (0049)421-218-3188      CHINESISCH UND COMPUTER
PC-Labor, MZH         email: <[email-masked]>    Journal for Computing
Universitaet Bremen   FAX:   (0049)421-218-2720       in Oriental Languages
2800 Bremen 33                                             (in German)
Germany
----------------------------------------------------------------------------