Subject: Chinese coding, input, etc.

From: Wang Wei <[email-masked]>
Date: Fri, 12 Mar 1993 17:38:47 -0500

Hi, fellow netters,

This is Wang Wei again. Here I'd like to post some of what  I
think about  Chinese  computing. A  couple  of weeks  ago,  I
commented on Mr  Xu's article on  Chinese software on  MS-DOS
using some  offending  words. I  got  flames from  Mr  Xu  of
course. I'd apologise here. But as a matter of fact, I  still
have some disagreement with what Mr Xu has stated in  several
of his articles, which has nothing to do with my personality.

As Mr Xu said, Chinese computing  started back from 40  years
ago. If the argument  could stand, we  could then extend  the
history much further back to Fu2Xi1Ba1Gua4 thousands of years
ago. That was close to binary coding and related to  Chinese,
too. Right? But I see no point in there. I suppose there must
be at least a  computer involved when  we talk about  Chinese
computing.

Actually I'm not thrilled to write these "old craps" and post
them to the net. And actually  I do not have any  "influence"
as a poor  student. I  was and  still am  stating the  truth,
which may not be so cheerful to us.

OK. I apologise to  whom ever I offended.  Let's get down  to
business. It's a pity that Mr Xu doesn't reply my request for
his first article. I  wanted to base my  writing on his  one,
since I don't  have much material  handy. Everything  written
down here are based on my personal experiences before I  came
out of China  in mid 1991  and the postings  appear on  CCNET
recently. Comments, flames, whatever are always welcome.
_____________________________________________________________

To my personal knowledge,  Chinese computing started back  in
mid or  late 1970's,  or maybe  a  little earlier,  on  mini-
computers. But  since it  was  difficult to  display  Chinese
characters lacking graphical display, practical or commercial
Chinese hardware and  software add-on  modules appear  around
1980 when micro-computers based on 8080/Z80, and then IBM  PC
and clones gradually becoming more and more available. Before
CCDOS got  so popular  on the  market,  there was  already  a
LianXiang Chinese video  card and some  sort of  modification
package for micros based on S-100 bus and 8080/Z80 CPUs  with
dumb-terminals. It was nothing comparing with today's Chinese
"systems" but pretty cool back then.

I would  say CCDOS  is quite  a landmark  in the  history  of
Chinese  computing  in   the  main  land   China.  Its   main
contribution is  that it  made  Chinese computing  so  easily
available (Although  maybe the  lack of  software  protection
contributed to its popularity,  too). The evolution of  CCDOS
and the  emergence of  many its  clones  show that  CCDOS  is
probably going to survive the market competition.

1. Chinese character coding

Besides the modified version of GB  coding scheme (or we  can
call it GB11 comparing to GB10 below) used in CCDOS and other
similar systems,  there are  other coding  schemes like  GB10
(Hi-bit of  hi-byte  set, of  lo-bayte  reset) on  the  Stone
typewriters,  EBCDIC   "spot-filling"  on   IBM   mainframes,
leading-escape-code,  escape-code-quoted,   and  A9I   coding
scheme for  some not-remembered  systems :-).  Many  systems,
such as AT&T  Unix, IBM  5550, and  WANG MVS  (Thanks to  Guo
Jin), use their property coding scheme to represent  Chinese.
I believe  there are  some more.  And now  there's the  zW/Hz
coding scheme, which is "pure 2-byte  GB" code as claimed  by
Mr Wei Yagui in his article in a recent issue of HXWZ.  Among
these GB11 is definitely the most popularly used one in  main
land China, maybe just like Big5 in Taiwan.

As I mentioned  in my comments  to Mr Xu's  article, using  a
specific  coding  scheme  does  not  automatically  make  the
software "compatible" to the  others. And as  in the case  of
Beijing Library system, and  other situations, where odd  (or
maybe even dead)  characters are often  used, GB  is way  not
sufficient. As I know, the Beijing Municipal Public  Security
Bureau had problems coping with the strange characters people
used in their  names. That  also happens  a lot  in the  news
reports of Xinhua News Agency. So many people kind of  modify
the GB  table to  fit their  own needs.  Many have  used  the
unoccupied rows to put the extra characters when they have to
deal with both simplified and traditional characters and  the
conversion between  them.  Many  others have  used  in  these
undefined rows to store user-defined characters (I guess this
is also what CCDOS does).

I believe  the  end users  like  the secretaries  in  China's
offices would never care  how the characters are  represented
inside the computers. What they see  is how they input  their
texts, how they arrange things on a page, and how they  print
them out nicely.

Sometimes the using of a coding  scheme is restricted by  the
hardware and software environment. For example, when  writing
a software  to support  Chinese in  Email,  zW/Hz is  a  good
choice for most of us. But  for AT&T, they'd rather stick  to
their EUC (?: Extended Unix Code or whatever).

I personally  see UNICODE/ISO  10646 as  very promising.  But
there's a  danger  that the  GB2312  is going  to  appear  in
UNICODE untouched (I  read an article  about UNICODE on  this
net months ago).  Although adopting  a new  coding system  is
definitely costly, but  GB2312 is  definitely not  sufficient
either.

2. CCDOS and other MSDOS add-ons

CCDOS and most  of its clones  are add-on  modules to  MSDOS.
Most likely such an add-on module is divided into three  sub-
modules: input,  display, and  print. Most  of these  add-ons
achieve what Mr Xu referred as "BIOS-level compatibility"  to
MSDOS, which means if a software running on top of the add-on
module use only DOS  and BIOS function  calls to do  keyboard
input, screen  display, and  printing, it  will have  Chinese
processing capability.

Some software  add-on packages  have  the ability  to  handle
direct video memory mapping, which Mr Xu referred as "DMA  or
register level"  compatibility.  But DMA  and  registers  are
terms used  to talk  about hardware.  I don't  consider  them
suitable here. And  the technique such  packages use are  not
any more "secrets".  The simplest  way to  do it  is using  a
dual-buffer video  card, like  Hercules mono  or VGA,  and  a
timer  interrupt  (or  better  memory  fault  exception  with
286/386/486 in protected mode? I don't know). You can "cheat"
on BIOS and the user software to use one of the buffers while
setup the hardware to use the other one. The timer  interrupt
can be used to update the screen. (I haven't done this, but I
guess that's the way it works.)

Pure software packages normally  work as TSR  (Terminate-and-
Stay-Resident) programs in  DOS. They are  started after  DOS
booted. Then they change the  interrupt vectors, INT 10H  for
video display, INT 16H (or even INT 09H) for keyboard  input,
and INT 17H for printer, and then stay in memory. So  there's
another point here, when  you need to  write a program  which
must work with CCDOS and handle  INT 16H itself, you need  to
run your program  AFTER the  CCDOS stuff  rather than  before
them. You may take over INT 16H in your program, and route an
unwanted call  back  to the  old  INT  16H in  your  INT  16H
handler. It may not work totally flawlessly. But you may  not
get any control  of INT 16H  at all the  other way  as Mr  Xu
stated in his first article.

There are many hardware  supported MSDOS add-on packages  now
in the main land China market. But the most remarkable one is
the LianXiang Chinese video card  designed by Mr Ni  GuangNan
and his colleagues in the Research Institute of Computational
Technology, Academia Sinica, manufactured and distributed  by
the  Legend  Group  headquatered  in  ZhongGuanCun,  Beijing.
Another good  product back  in about  1984  was the  New  Era
Chinese video card from XiaoFeng.

The LianXiang card are remarkable  not only for its  hardware
design given the IC supply condition  in China, but also  for
its supporting software system.  The LianXiang card  realized
real register-level  compatibility  for both  text  mode  and
graphic mode  display at  the first  place.  So the  INT  10H
interrupt is  not changed  as with  CCDOS. The  card has  1MB
built-on RAM, which is  used to store  the 16x16 bit  display
font, input method data, and part of the 24x24 print font. So
it can  support both  simplified and  traditional  characters
easily. The user can switch between a number of input methods
in the fly, and  install any number of  input methods with  a
couple of DOS command lines. Since  the CPU is released  from
updating the  screen  and  the buffer  memory  is  relatively
large, its response to keyboard input and screen paging is as
fast as  non-Chinese systems.  Printing is  also much  faster
than other Chinese systems. All  these also made it  possible
to have disk-less Chinese workstations for PC networks.

3. Input methods

I don't agree with Mr Xu's judgement on PinYin input  method,
in which I mean that most of the Chinese key-in work is  done
by the professionals using  things like WuBiZiXing,  CangJie,
DaZhong, Eten,  etc.. PinYin  for them  is just  a  fallback.
PinYin may be  the most popular  one among  us the  so-called
"intellectuals"  who  can't  even  spell  the  right   PinYin
ourselves (Sorry for this comment ;-).  But how much work  do
we do? Several pages a week? That's just one or two hours for
a professional.

I'd     rather      agree      with     Mr      Jack      Sun
([email-masked]),   WBZX/CangJie    and
PinYin target to different groups of users.

I have a  question here:  How many of  us on  this net  could
input the  last  character in  GB  table easily  with  PinYin
(QuWei: 8794)? Huh? Just try it yourself.

That's one  point of  using the  writing-based input  methods
like  WBZX.  Most  of  the  professionals  would  "spell"   a
character out using WBZX or  whatever without knowing how  to
pronounce it.  And  besides, there's  no  point for  them  to
"think" about the text they type  while typing. And even  for
the people who  "think" while typing  (I remember reading  an
article in People's Daily about a Chinese writer who has a PC
and writes his novels on it), they remember things  gradually
while using a method and become spontaneous when typing.

Comparing to so many writing-based input methods available on
the market,  PinYin  has many  weak  points: there  are  many
homophones, even using word/phrase based input packages. This
makes PinYin extremely powerless when input  single-character
words. Many  have  praised the  sentence-based  PinYin  input
methods on this net. I don't know how many of them have  ever
tried to use  them to  input a Tang  Dynasty poem.  I bet  it
would make you  sweat out.  I know  that this  is an  extreme
example. But  the fact  is, there  are  many words  in  daily
spoken Chinese that are  single-character ones, such as:  Wo3
Ai4 Ni3 (I  love you),  Wo3 Xiang3  Qing3 Ni3  Bang1 Wo3  Ge4
Mang2 (I  want  to  ask  you  for  a  favour).  Every  single
character is an independent word in these two sentences. So I
may conclude that even the best "sentence-based" PinYin input
method  is  only  good  for  written  Chinese,  things   like
"Hong2tou2wen2jian4", not  good  enough for  the  writers  to
write novels with.

Another  point  is,  when  you  don't  know  how  to  read  a
character, you  have no  way to  input  it with  PinYin.  And
further more, there  are so many  people in  China who  can't
spell PinYin correctly (as some one on this net pointed out a
few days ago, there's  no 'n consonant,  there's only 'ng  in
Shanghai dialect).

But I still view PinYin as  the most promising input  method.
Not only because the younger  generation in China are  taught
with PinYin in  primary schools, not  only because there  are
just about 430  syllables (or roughly  1300 counting all  the
tones), but  also  because  it's  the  only  way  to  achieve
something  with  speech  recognition  (There  has  been  some
significant things  done  by Tsinghua  University  and  China
National Electronic Devices  Corp.). Besides  I also  believe
there're much  space  to  improve the  various  PinYin  input
methods we  are having  now (given  the example  provided  by
Haizi Xu  <[email-masked]> on  how  a
new word is added to XiaLiBaRen's dictionary: there are  more
than 10 steps  involved). The simplicity  of a  writing-based
input method may have  some score here:  it's so simple,  you
got nothing to complain about. :-)

In this specific case, I guess  the new word adding  approach
in PJS (a PinYin and phrase based input package developed  by
a Chinese  governmental research  institute and  somebody  in
Singapore) is a somewhat better one. Upon receiving an  input
string for some none-existent phrase, it'll try to break  the
string into single syllables and start to display the choices
for each of them. As  you finishing selecting characters  for
the phrase, it's added  into the dictionary  and sent to  the
application program which  is waiting for  input at the  same
time. I guess that's also what Mr Ed Lai's CiHui on Macintosh
does, but I haven't tried it, so no further words.

As to me personally, a perfect input package better have  two
input methods at  the same time,  by which I  mean I can  use
them both without switching back and  forth. I, as the  user,
can choose one  as my primary  input method,  and can  easily
fall  back  to  the  secondary  one  when  I  encounter   any
difficulty with the  primary one.  Things like  adding a  new
phrase into  the dictionary  should be  easy to  do.  There's
better some intelligence to  accept my not-so-correct  PinYin
input (as for 'n and 'ng,  c' and ch', etc.). Actually I  see
the problem with Chinese  input not any  more with the  input
methods themselves (given the fact  we have over 500  hundred
of them in main land China  alone), rather with the  software
packages that implement them (as Bo HU  <[email-masked]>
pointed out  in his  article dated  8 Mar.).  The  difference
between using a better package and a not-as-good one is  like
that between using MSDOS and MS Windows on the same PC.

To me, LianXiang is not a very good thing in a input  package
if it takes much effort to  integrate it. I agree with  Haizi
Xu again  on  this.  I  add  only  one  con  about  it:  it's
physically too tiring  for the eyes  when you  really use  it
cause you must scan the prompt line(s) to find what you want.

I also agree  with Haizi Xu  on separating  phrases on  their
scopes of usage. It's a great idea. But to implement it there
must be a  lot of  computational linguistics  involved. Am  I
right about  that? I  guess that  might also  be a  trend  of
future  Chinese  input  packages.  There  might  be   Chinese
"spelling-checker" or "grammar-checker" sort of things out on
the market in the near future.

4. Fonts

Just a few words. Mr Xu stated in his article that there  was
a law  prohibiting  the  existence  of  non-standard  Chinese
character coding schemes. I guess my friend Guo Jin is  right
about  this.  There  might  be  such  an  action  about  font
patterns. There might be  a reason for them  to do so. But  I
see it as merely  a commercial issue:  it's just because  the
government (or some specific  government agencies) has  spent
so much time and money to develop these font patterns (By the
way, I agree that they are really well done), they don't like
them to be wasted. They want some money back as compensation.
But I think such a law as unnecessary: (1) They can't ban all
the non-standard fonts, the others  may simple label them  as
"artistic"; (2) The  sense of sophistication  of fonts is  an
artistic term rather than a scientific one, which may develop
in time; (3)  A standard  should help  the manufacturers  and
vendors to develop better systems, rather than kick them  out
of the market.
_____________________________________________________________

Phew! I finally finished this  lengthy "old crap". I  promise
you I won't  do it again.  I tremblingly hope  this helps  my
"personality development".

Again, thanks for your patience  with this. I prefer  emailed
comments if it not interesting to anyone else.

Wang Wei