Subject: Some consideration for design of Chinese encoding standard

From: Ed Lai <[email-masked]>
Date: Fri, 8 Feb 91 15:48:14 -0800

I like to point out some of the thing we have to consider when we are
designing an encoding standard.

1. It should be able to accomdate both batch processing and on-line display
. In on-line display, the Chinese character will be display automatically
as it comes without the user doing anything. This is not a dream, we now
have several program for different machines that can do this. For those
without Chinese capability in the terminal program, they are not locked out,
any protocol that allows on-line display can also allowed batch processing.
Although it can be agrued that if we design a protocol that is for batch
processing only, then it would be more compact than one that allows on-line
display, but the saving will be so minimal that this will be a very short
sighted design. (Interestingly, what is proposed as a batch protocol such as
uuencode is in fact less compact than most of the on-line protocol using
pure GB).

2. It should function well in the enviroment we have now where the bulk of
mail/news is in 7-bit ASCII roman text. This means that the default mode
should be English, with escape character staring the Chinese encoding.
Otherwise we would be mis-reading most of our mail/news, making non-sensical
Chinese out of what is really English mail/news.

3. Because we are far more likely to be modifying the system in the terminal
end rather than the mainframe end. We must have a protocol that works without
the cooperation of the mainframe software being aware of we have Chinese text
over here. As a consequence, we need to have our English-to-Chinese escape
sequence at the beginning of every line so that we would not lose the mode
information when we skip ahead and backward n lines. As far as I understand
that is what the Japanese are doing in their system too.

4. Assuming we accept point number 3, there are still two way to do it. In the
first way you always uses a start Chinese escape and then you use a end
Chinese escape. If all your text are in Chinese it would end up having each
line starting with a start Chinese escape sequence and each line ending with
a end Chinese escape. However this also means you are sending mail and you
are absolutely sure that the other party is going to do batch processing, then
you may violate what we have said so far and use a single begin Chinese
escape in the whole file and a end Chinese at the end of the file, resulting
in a more compact file. The other alternative is to assume that there is a
implied end Chinese escape at the end of every line so it is not necessary to
put one at the end, this will be more compact in general, but it will not let
you use the trick of a single begin Chinese escape for the whole file. BTW
this is the approach of 'zW'. Each approach has its own merit and drawback,
but I have no problem with using either approach.

5. Now to choose the escape character/sequence. I really have no strong
perference, but obviously we want one that works. I hear there is problem
with $1B, so we should try to avoid it. We need to avoid something that is
likely to occur in normal English text, and the ending sequence should avoid
using a valid GB Chinese character. The latter does not sound like a problem
since there are still many undefined GB character code position. However, we
should research more into the problem, I heard that there is a GB standard
defined in Mainland China that is very close to the one we normally see, but
have some extra code defined to alleviate the n-to-1 mapping problem between
traditional and simplified character. If possible we should try to be able to
accomdate this broader GB.

6. We are talking mostly about protocol for displaying Chinese character
gracefully in a terminal, and zW/HZ is probably a good approximation already.
But they would not fare so well if we are doing input. Pai Chou has a terminal
emulator for Big-5 that uses a escape for every Chinese character. Most of us
may not want to go that far, but it does a reasonable job of inputting when
we are using vi or mail, so this is something we may want to think about.

/* Disclaimer: All statments and opinions expressed are my own */
/* Edmund K. Lai                                               */
/* Apple Computer, MS37-UP                                     */
/* 20525 Mariani Ave,                                          */
/* Cupertino, CA 95014                                         */
/* (408)974-6272                                               */
zW@h9cOi