Subject: Some consideration for design of Chinese encoding standard

From: [email-masked]
Date: Fri, 8 Feb 91 20:13:00 EST

This is follow-up to Ed's message about consideration for designs.
I'll just tell what I think based on my experience writting ZWWATCH.

A little background info about CCDOS: the PC video can work two different
ways: text mode and graphic mode. Most PC applications run in text mode,
in which ASCII codes of data are stored in the frame buffer. CCDOS turns
the video into graphic mode - in which data are stored in the frame
buffer as bit-maps. (I am using the term frame buffer loosely, should
have used "video memory".)  Most versions of CCDOS intercepts system
calls to write characters to the screen. If they see Chinese text data
coming in, they draw the characters on screen, otherwise they draw the
ASCII characters in graphic mode.

ZWWATCH acts as an extra layer between cCDOS and the application programs,
interrupting the keyboard, video and printer system calls just like
CCDOS does. It does all the neccessary encoding and decodings as the
data is passed between CCDOS and the application, so that CCDOS only
sees its own format and the application only sees 'zW'. The conversion
can be tricky at times.

The printer decoder part of ZWWATCH is simplest. Printer output is treated
as a sequential stream, and the 'zW' string is checked for at the beginning
of each line. This works reasonably well for pure text (GB or ASCII), but
could conceivably have problem with escape codes intended as printer commands.
A scheme which allows for intraline mixing of GB and ASCII may be better
in solving this problem.

The keyboard/screen part - if we simply treats every thing we type as
a sequential stream, this part is very simple too. You simply does
'zW-encode' at the keyboard end and 'zW-decode' at the video end. You
don't even normally have escape commands to worry about in most cases.

Complication comes when I was trying to make it work with full screen
editors. As ED pointed out, this requires that Chinese text data be
indicated locally - which 'zW' does quite well. Everytime the cursor
position changes, ZWWATCH checks to determine whether the current line
is Chinese or ASCII. It does by reading the first two characters from
the beginning of the line to see if they are 'zW'.

Reading characters from a graphic screen is a very slow process. It
involves comparing the bit patterns on screen with the 256 font patterns.
The delay is not noticeable when only the first two characters need
to be read. But for schemes that allows intraline mixing of text, the
entire line would need to be read, which could be an expensive process.

In the end, I sort of got it working with full screen editors like
VMS EDT or vi on unix. It's not completely glitch-free, however.

JD's suggestion of writing something similar but not depending on
CCDOS is a reasonable suggestion. But if we have a very good implementation
of CCDOS, perhaps fewer would ask for that. The idea of using CCDOS
is that things can be more modular; different input schemes of different
versions of CCDOS can be used - otherwise our program has to take care
of converting input codes to internal codes and converting characters
etc. But it sure would be nice if a version of CCDOS can be written
with the our need in mind, so as to make some of our works -- like
determining if the current context is Chinese or ASCII -- easier.

If someone already has a set of good routines for inputing and outputing
Chinese characters, it definitely is a good idea to write a standalone
Chinese terminal. The problem with me was that I didn't have any input
routines, I have a set of output routines but no fonts.

yawei

---

From: [email-masked]
Date: Sat, 9 Feb 91 07:50:57 MST

I've condensed Ed Lai's considerations and added a few things.  Its a
little fuzzy, so please tell me if I missed anything or have
something wrong.

1. Encoding must be 7-bit ASCII printable characters.

   This is for those nodes/backbones that still can't transmit 8-bit
   characters.  Also avoids control characters that might be
   interpreted as commands by modem or other software.

2. Encoding must be transparent to any system or software it passes
   through with the exception of software designed to interpret it.

   No matter what system the text passes through, there must be no
   special interpretation of the contents of the text except by
   programs specifically designed to deal with the encoding method.
   There must also be no loss of text due to any special
   interpretations.

3. Encoding must accommodate on-line/batch processing transparently.

   With respect to on-line display, the user should not have to know
   that any conversion is occuring while displaying, editing, entering
   text.

   With respect to batch processing, the user should never need to
   explicitly interact with any conversion occuring in a batch
   process (i.e. batch printing, batch conversion and archiving,
   etc.).

4. Encoding must be interpreted in the same way in any environment.

   The hardware and software being used should in no way require
   special handling of the text other than what is needed to
   differentiate the encoded text from other text.  This must be
   consistent for any machine/program being used.

Other thoughts on encodings, editing and input method interactions
with editors:

   A. Editing text

      Use of HZ and zW with terminals seems to work quite well.  But they
      seem to bog down in an editing environment.  Here's another
      view (of course I may be misunderstanding why the encodings are
      a problem in editing).

      An example:

      Assume we have a file that has been encoded using the HZ system
      (no special preference for HZ, just for example).

      An editor would convert the text to 8-bit GB when reading it in.
      This is easy because HZ encoding is easily recognized in a text
      file.  Even mixed 8-bit GB and HZ is easy to detect in a text file.

      While the user is in an editing session, the text is 8-bit GB
      internally.  When the time comes to write the text out, the user
      can specify whether 8-bit GB or HZ output is desired or use a
      default output encoding.  This collides somewhat with
      consideration number 3 above.  But if the user works in one
      encoding most of the time, its not much of a problem.

      Actual input methods for Chinese could produce either 8-bit GB or
      HZ format text, and any conversions needed could easily be hidden
      from the user.

      The output encoding can be determined automatically, with the
      editor assuming HZ output because the input file was in HZ format
      or 8-bit GB because the input file was 8-bit GB.  If a mixed 8-bit
      GB and HZ file was read in, the last encoding detected would
      determine the output encoding.  If the user desires a different
      output format, they can choose it from a menu, set it with a
      keystroke, or in the case of programmable editors, set it with a
      function.

      The costly process here is the conversion.  If the conversion is
      limited to the beginning and ending of an editing session, it
      should work reasonably well with PC and Mac systems.  Input of
      Chinese is another matter.

   B. Input method interactions with programs

      One idea that seems to be taking hold in the X11 environment, is
      the idea of an input method process that is separate from the
      application.  The idea is that no matter which application you are
      using, a special key press or menu choice pops up the input
      method process for use, grabbing the keyboard away from the
      application.  All of the input is done with this special
      process, and certain keystrokes or menu selections send the
      results to the application.  If the application is designed
      right, it almost feels like you are doing the input in the
      application directly.  Of course, some keystrokes are special to
      the application, so an easy method of switching keyboard control
      is needed, or if both the input method and application are
      designed right, the application specific keystrokes pass through
      the input method to the application.

      Once again, any conversion of encodings during the input stage
      can be hidden in either or both the input method and the
      application.  If the application is designed right, it can tell
      the input method what encoding to use and no conversion needs to
      take place.

Well, enough of my long-winded, early morning thoughts.  I hope people
will comment on or correct these ideas as they wish.
-----------------------------------------------------------------------------
[email-masked]                      "I laughed.
Mark Leisher                                I cried.
Computing Research Lab                          I fell down.
New Mexico State University                        It changed my life."
Las Cruces, NM                     - Rich [Cowboy Feng's Space Bar and Grille]

---

From: Rupert Zhu <[email-masked]>
Date: Sat, 9 Feb 91 11:50:34 EST

Mark's summary provides a framework for future Chinese software
design and network design.  It is general enough to accommodate
various approaches.  After some revision, it can serve as a Guideline
for future development, I guess.

I TOTALLY agree on most of the points which Mark summarized.
Some very slightly different idea is presented in the following.
My comments are preceded with  "****".

---------------------------------------------------------------

> 3. Encoding must accommodate on-line/batch processing transparently.

****  Mostly agree.  (But Mark's point A makes equally good sense
      to me.)

>    consistent for any machine/program being used.

****  I did not quite understand this.  Maybe Mark is emphasizing
      consistency of the application softwares, or maybe this point
      already resides in some other points in this "guideline" of Mark.

>       or 8-bit GB because the input file was 8-bit GB.

**** This part is quite nice.

>       If a mixed 8-bit

**** It seems to me that this part is unnecessarily too complicated.
     How can such a messy mixture occur at first place?  As an
     alternative, we may change our policy to prohibit such mixtures.

>    B. Input method interactions with programs

**** This is the idea of modular design, (or object-oriented programming),
     Am I right?

------------------------------------------------------

I am learning a lot these days from pioneer experts like Mark, Yagui, Ed,
Stephen, Dorothy and others.  I haven't made my mind straight yet.
But I found the major considerations in my mind are compatible with
this Guideline summarized by Mark.

Thank you very much.

-- Rupert Zhu
   University of Waterloo

---

From: [email-masked]
Date: Sat, 9 Feb 91 10:18:28 MST

>       already resides in some other points in this "guideline" of Mark.

You're right.  The point is made in other places.  Sometimes I get a
little out of hand :-)

 >      alternative, we may change our policy to prohibit such mixtures.

I would vote to assume/require that the text should be in one encoding
only.

 > **** This is the idea of modular design, (or object-oriented programming),

Yes.  Depending on how it is implemented it could be looked at in
either way.  In the X11 environment, this idea tends to look like an
object-oriented approach when the input method is made part of an
application and modular design if the input method is made into a
separate program that "talks" with the application.

I would like to say thanks to Rupert for not giving up on the encoding
idea easily.  I think that in the past, jobs, school work and other
considerations :-) have stopped a serious effort towards working on an
acceptable 7-bit standard.  Thanks to Rupert, I am becoming really
interested in this effort again!  The trick now is to keep it going.
-----------------------------------------------------------------------------
[email-masked]                      "I laughed.
Mark Leisher                                I cried.
Computing Research Lab                          I fell down.
New Mexico State University                        It changed my life."
Las Cruces, NM                     - Rich [Cowboy Feng's Space Bar and Grille]

---

From: [email-masked]
Date: Sat, 9 Feb 91 13:43:00 EST

I do not quite understand the part of Mark's summary about
editors.

If the editor is running on my mainframe account, and I am
connected to it using a terminal which does the conversion,
then anything that ever reaches the mainframe account is
in our 7-bit codes (zW or HZ or whatever). The editor only
sees this 7-bit code, so of course it will save its files
in this format as well.

If the editor is running on my local CCDOS environment, then
it will be using whatever internal code your local environment
uses. It can be CCDOS's internal code, or it can be something
more compatible with our 7-bit code. (It's perfectly ok if
a version of CCDOS is written that only uses our 7-bit codes.)

If the local editor uses internal codes that are different
from our 7-bit codes, then the conversion occurs when we
upload it to the mainframe account. The conversion process
can be transparently incorporated into the upload/download
protocols.

yawei
aka Ya-Gui Wei

---

From: Rupert Zhu <[email-masked]>
Date: Sat, 9 Feb 91 16:35:59 EST

Thank you too, Mark.

I am thinking to start from a small scale.  We have less than 30
universities in Canada.  I am at a position to recommend to
Federation of Chinese Students in Canada for a practical plan of
establishing Chinese text network connecting every university.
The text flow in this network will be very low. (One Chinese
text file in every two weeks.)

Beside the pure scientific aspect, there are also practical concerns
in my mind.  At the user end, the number one concern is a good editor
(and screen and printer drivers built around the editor).  And also
what kind of machine is most popular, (if I am unable to provide
support to all kinds of machines).  And finally, how easy and how
trouble-free the method is to non-expert users.

It seems what in my mind may not be the best from pure scientific
point of view, but may be a fairly practical one at current stage.
On the other hand, I wish the network I am trying to build can be
easily upgraded at later time, when better softwares are ready.
That is why I try to learn all these things before I finally launch
the project in Canada.

Thank you very much.

-- Rupert