Subject: Re: Extending HZ

From: [email-masked]
Date: Tue, 6 Apr 93 20:23:49 MDT

Almost forgot.  For info on Japanese coding, I have a file called
'japan.inf' by Ken Lunde that describes most of the major Japanese
encodings.

crl.nmsu.edu:pub/japanese/japan.inf  [128.123.1.18]

-----------------------------------------------------------------------------
[email-masked]
Mark Leisher
Computing Research Lab                    "Elegance is not optional!"
New Mexico State University                  -- Attributed to Richard O'Keefe
Box 3CRL, Las Cruces, NM  88003

---

From: Ed Lai <[email-masked]>
Date: Wed, 7 Apr 93 10:02:45 -0700

If you consider the number of printable character, then HZ scheme can
be used to cover 96*96=9216 characters for every two byte pair. That is
not enough to cover all of big-5 code. So for big-5 code three bytes are
needed at least in some cases.

Pai Chou in TaipeiTerm uses 3 bytes for every big-5 char and it actually
works quite well.

However, a lot of the big-5 characters are almost never used. And if it
is an extension of HZ it is better to be upward compatiable. I think the
people who wants big-5 code is more interested in reading traditional
character. They can do it to some extent now with a gb traditional font.
The problem are those n:1 mapping of simplified to tradtional characters.
So really the idea is to extend the gb, using the unused character space
in gb to include these muliple mapping tradtional characters.

Fung Fung Lee had discussed this issue before and called it extended GB,
so I would leave it to him to discuss it again. I just want to add a few
suggestions. I think there is a PRC standard that came after gb80 that
cover this problem, so that can be used. If not, we may pick some
commerical system like TianMa that let you switch between simplied
and traditional font. It is likely that they are already using such an
extended gb characters set and we can adopt it.

If it is felt that it is still necessary to cover the rest of the Big5 code,
then we can have an escape character so that the Big5 code with no
GB/extended GB equivalent will have a 3 (or maybe 4) bytes
representation.

To summarize again, we can have an extended gb character set that we are encourage to use.

1. For people who want to keep using simplied character, if they use
existing software, they may miss a few characters, but often it can be corrected just by using a slightly larger font file. The various HZ
software can be upgraded to remap these characters either because
it is more space efficient than a bigger font file, or in the case where
it is not practical to have a larger font file (e.g. the software is
depending on the Chinese system to display the characters).

2. For people who want to read it as traditional character, they can
use an extended gb font file in traditional character.

3. For people who want to use extened HZ from a Big5 system, when
the file is displayed, it will be converted from extended GB to Big5.
When they want to post from a Big5 system, the code will be
translated from Big5 to extended GB.

/* Disclaimer: All statments and opinions expressed are my own */
/* Edmund K. Lai                                               */
/* Apple Computer, MS37-AS                                     */
/* 20525 Mariani Ave,                                          */
/* Cupertino, CA 95014                                         */
/* (408)974-6272                                               */
zW@h9cOi

---

From: ~{QG9p~} <[email-masked]>
Date: Wed, 7 Apr 1993 12:50:38 -0500

An extended GB scheme would certainly add more characters. I've
heard that China already has extended the GB table to include
more characters (GB level 2?), just never seen any detail about
it.

One problem with an extended GB scheme is that it probably 
won't support simultaneous simplified and traditional character
displays.

An extended HZ scheme can have a lot of room to play if you
are willing to consider other tilde- sequences.  We may, for
example, use ~{ for GB2312 (as we do now), use ~( for Big-5,
etc.  Since there are a max possible 92 tilde- sequence to
be exploited, an extended HZ scheme can support a max of
   92x94x94 = ~800,000
characters!  This is about 12 times more than Unicode can
accomodate.:-)

 Ya-Gui

---

From: [email-masked] (Pai Chou)
Date: Wed, 7 Apr 93 19:11:27 -0700

>>Date: Wed, 7 Apr 93 10:02:45 -0700
>>From: Ed Lai <[email-masked]>
>
>Since Pai Chou ([email-masked]) is on this list
>([email-masked]), perhaps he can give us a brief introduction
>to his "T3" code.

Hi everybody,

Here is the documentation file for Taipei-100 and the T3 code.
Sorry I have a paper deadline on Monday, so I can't promise
to reply mails promptly.

Pai

(September 3, 1990)

Encoding Format: T3
-----------------------

Design Considerations
---------------------

   The 3-byte "T3" encoding is designed for 2-byte BIG-5
encoding.  This encoding method satisfies several criteria:

1 Only printable ASCII characters are used (i.e. [$20-$7E]).
   In effect, line feeds and escape characters are preserved.

2 Given any line without the knowledge of surrounding lines,
   the terminal emulator can always correctly display Hanzi.
   This is important for text editors, where you can be jumping
   back and forth between lines.

3 Maintain character integrity.  That is, one ASCII contributes
   to exactly one Hanzi's encoding, but never to more than one.

4 To comfortably cover the entire BIG-5 range (13051 chars)
   without using a lookup table or very complicated encoding
   method (such as variable-length Huffman encoding, etc).
   No complex mathematical operations are required: the most
   complex operation is multiply by 5, which can be synthesized
   using an add and a shift.

   To satisfy the first criterion, we can use at most 95 different
values per character.  Two ASCII's can therefore represent
95**2 = 9025 different Hanzi, which is not enough for the entire
set (13051) -- which violates criterion #4.  This does not even
include any escape sequences to distinguish Hanzi from Romazi.

   One attempt to satisfy criterion #2 is to have an escape
sequence to signify either Hanzi or Roman mode that follows
until another escape sequence or upon reaching the end of
the line.  However, to satisfy criteria #3, and #4, each Hanzi
requires three ASCII's to encode anyway, so the extra escape
sequences will waste space.

   My solution is to use two characters as escape characters:
` and ^.  I avoided ~ because the UNIX mail program uses it
as an escape character for invoking commands from the mail
message.  ` and ^ are not used frequently in regular mail or
news messages.  To make encoding easier, since `_^ are
three consecutive ASCII characters, _ is not used either. 
This results in 95-3 = 92 different values for the remaining
two bytes following the `^ escape.  Therefore, a total of
92**2 = 8464 different values can be represented.  But since
there are two escape characters, they can also contribute to
the encoding as well.  As a result, the total number of
representable characters using the three-character sequence
= 8464 * 2 = 16928.  In reality, the T3 format uses only 89
different values for the first byte, so a total of 89 * 92 * 2
= 16376 can be represented, which is still comfortable for BIG-5.

Encoding format
----------------

   A Big-5 encoding consists of two bytes. 
The first byte has the range of $A1-$F9, or 89 different
values.  It can easily be assigned to its own byte.
The algorithm is

=============
var
(* import *)
    bs: char;  (* higher byte's source *)
(* export *)
    bt: char;  (* higher byte's target *)

   bt := bs - $A1 + $21;
   if bt >= '^' then bt := bt + 3;
=============

   The second byte, however, is trickier.  The range of the
second byte is $40-$7E, $A0-$FE, with a few last-digit F's
missing.  But let's just treat them as if the range were
$40-$7F, $A0-$FF.

   The lower-order nibble can stay where it is.

   The upper nibble has the range [4,5,6,7,A,B,C,D,E,F], or 10
different values.  Since each ASCII can hold at most 92 values,
92 / 16 = 5.75.  It can represent at most 5 different values in
the upper nibble.

   The solution comes from the use of two different escape
characters.  Nibbles in the set [4,5,6,7,A] are assigned the
` escape, while those in [B,C,D,E,F] get ^.

The set [4,5,6,7,A,B,C,D,E,F] get mapped to [0,1,2,3,4,5,6,7,8,9],
modulo 5, and multiplied to the lower nibble.

The algorithm is
=============
var
  (* Import *)
      BS: char;  (* Lower byte's source *)
  (* Export *)
      BT: char;  (* Lower byte's target *)
      BE: char;  (* escape byte *)
  (* Utility *)
      LN, HN: integer; (* lower and upper nibble *)
 
      LN := AND( BS, $F );
      HN := AND( SHIFT_RIGHT(BS, 4), $F );
      if HN > $7 then HN := HN - 6 else HN := HN - 4;
      BT := OR( LN, Shift_Left( (HN mod 5), 4) ) + $21;
      if BT >= '^' then BT := BT + 3;
      if HN >= 5 then BE := '^' else BE := '`';
=============

To Map from ASCII to BIG-5 is not difficult.

var
  (* import *)
   T0, T1, T2: char; (* 3-byte encoding *)
  (* export *)
   B0, B1: char;  (* 2-byte BIG-5 encoding *)
  (* utility *)
   HN: integer; (* higher nibble *)

(* first, you should make sure it's a legal sequence *)
   if (T0 in ['`', '^'])
      and (T1 in [$21..'^'-1, '`'+1..$7E])
      and (T2 in [$21..'^'-1, '`'+1..$7E]) then begin

     if T1 > '`' then B0 := T1 - $3 - $21 + $A1 else B0 := T1 - $21 + $A1;
     if T2 > '`' then B1 := T2 - $3 - $21 else B1 := T2 - $21;
     HN := AND( Shift_Right( B1, 4 ), $F );
     if T0 = '^' then HN := HN + 5;
     if HN <= 3 then HN := HN + 4 else HN := HN + 6;
     B1 := OR( Shift_Left( HN, 4 ),  AND( B1, $F) );
  end;

Using T3 code in Taipei-100
--------------------------

Taipei-100 does the conversion between Big5 and T3 on the fly,
for every character typed or displayed.
When you type normally, you probably are not aware of this
translation until you do backspace, or when you move the
cursor to the middle of a Hanzi in a text editor such as vi or
emacs.  This is because to delete a Hanzi, you must kill three
bytes on the UNIX side; similarly, to move the cursor over a
Hanzi, you must move it three times on the UNIX side.

   Taipei-100 makes it explicit in these situations.  It displays
the Hanzi as the three bytes that UNIX sees when the cursor
is in the middle of the three-byte sequence.

   For example, suppose you typed

   [ H1 ]  [ H2 ]  [ H3 ]  [ H4 ]  [ H5 ]  [ H6 ] |
    gong1   xi3     fa1     cai2    hong2   se4
   which says Congratulations and Be Prosperous (for the new year)
         so (Give Me the)  Red  Color ...;  | denotes the cursor.

  Oops - you meant to type "red envalope" (hong2bao1).  The last Hanzi
is represented on the UNIX side by ^&S, so when you backspace
once, you see

    [ H1 ]  [ H2 ]  [ H3 ]  [ H4 ]  [ H5 ] ^ &|
    gong1   xi3     fa1     cai2    hong2

where the last S was deleted.  This way, you see that you need
to delete two more characters to erase the entire Hanzi.
Finally, after deleting two more ASCII's, you can type the
correct character over, and you get the correct result.

   There is one case where you might need to hit backspace
four times.  When the last character is a backslash, which can
be used as the "quoting" character in UNIX, the first backspace
gets quoted (replacing the backslash), and the next three
backspaces actually erase the characters.

   When you run a text editor (vi, emacs), if the cursor is not at
Hanzi boundaries, then the character gets displayed as the
three byte sequence, but it is restored as soon as the cursor
moves to Hanzi boundaries.  For example,

     [ H1 ] [ H2 ] ` 5|P  [ H4 ]

shows the cursor (|) in the middle of the third Hanzi.

vi-mouse and emacs-mouse
---------------------------

  Taipei-100 makes the 3-byte sequence explicit to the user
when the cursor is placed in the middle of a Hanzi.  This lets
the user know how many more characters to delete or move.
However, this could become a dread to the user, because the
computer must recompute and update the screen every time
the cursor is moved.  It slows down the response time, and
switching between the 3-byte sequence and Hanzi can be
confusing if not annoying.

   To alleviate this problem, I implemented vi-mouse and
emacs-mouse.  By simply clicking the mouse at the desired
target location, Taipei-100 generates the cursor-movement
commands to place the cursor there.

   Vi-mouse is invoked by holding down the command-key
(clover) while clicking the mouse button.  YOU MUST BE IN THE
_INSERT_MODE_ in vi.  This is particularly useful when editing
Hanzi, because it places the cursor at Hanzi boundary.

   Emacs-mouse is invoked by holding down the Option key
while clicking the mouse button.  If you hold down both
the Command-key and the Option-key, then Taipei-100
generates vi-mouse commands.

   Unfortunately, Taipei-100's cursor-mouse commands do
not always place the cursor accurately when the line is
indented (i.e. begins with a blank or a tab), or if you use
line numbers in vi.  There is no solution to this problem.

---

From: [email-masked] (Ricky Yeung)
Date: Thu, 8 Apr 93 12:00:16 PDT

>   Actually I thought about this issue again last year after the establishment
  >   of alt.chinese.text, and had a private discussion with Ricky Yeung
  >   about two possible schemes.  As we didn't perceive enough interests back
  >   then, we stopped working on it.
  >   I would leave it to Ricky to discuss the schemes.

Here are some notes that I recall from our discussion last year:

- As in HZ, ~{ ~} is the only esc sequence to switch between ASCII and
  non-single byte codes.

- For codes other than GB, the HZ coding would be ~{~x ... ~} where x
  indicates what coding scheme the following codes are in.
  For examples,
  	BIG5   		~{~T.......~}
  	Unicode		~{~U.......~}
  	Japanese	~{~J.......~}
  	Korean		~{~K.......~}

Issues:

+ This scheme simplies the implementation of HZ filters that don't want
  to or can't handle other coding schemes.

+ For BIG5 codes, we can either "Qui Wei"lize (segment :-) it into
  three-byte codes (like Pai Chou's T3 code?) or have two portions of
  BIG5, and use ~t for the less frequently used codes.

+ One question rasied was that should we allow switching code schemes
  inside HZ?  My inclination was that its added complexity may not be
  justified (to save 8 bytes per switch), since mixing codes are not
  very common. 

  Examples:

  Mixing GB and BIG5, if not supporting switching inside:  

    ~{xxxxxx~}~{~Txxxxx~}~{xxxxx~}

  If supporting internal switching:

    ~{xxxxxx~Txxxxx~Gxxxxx~}	assuming ~G for GB

+ Internal switching can be added as a future expansion, though.

+ For Unicode, we can either "Qui Wei"lize it into three-byte ASCII
  codes, or just use the four-byte numeric code.

+ We discussed the use of other escape sequences for BIG5, like ~[ ~]
  and ~( ~).  I thought it's not general enough and suggested the above.
  Also Fung Lee pointed out the fact that the characters like (, ), [,
  ,] can occur as the 1st byte in GB, which could be visually confusing
  if used for mode switching.

-Ricky