Subject: Extended HZ coding scheme...

From: Duh? <[email-masked]>
Date: Tue, 9 Nov 1993 04:07:52 EST

Hi all,
Through private communication with J.Zhang ([email-masked]), my mind
became clearer in the possibility for an extended coding scheme based on HZ.
The idea is embarrasingly simple -- chances are that it's been out there or
discussed years ago. If that were the case, my apologies. I'll give an outline
first so that you wouldn't have to waste too much time if it turns out to be
worthless. In any case, I intend this post as an oppotunity to learn more
about Chinese programming.

The scheme:
1. Use two-byte coding with full 256 spaces for each byte. A 7-bit ASCII
subset can be used as a compromise for e-mail.
2. Divide the 64k coding space into 8 ranges, each corresponding to a commonly
used Chinese font (Song4, Fang3Song4, Kai3, simplified, traditional, etc) with
8192 characters/symbols.
3. Expand the escape sequence for switching between ASCII and Chinese to 4 byte
long.

Advantages:
1. Optimal usage of coding space.
2. Enhance performance in font lib access, and even less memory usage.
3. Allow effortless support for mixing fonts/styles in one document.
4. Help developing a standard set of font libs.
5. Virtually eliminate mis-intepretation of ASCII <-> Chinese.

Now some detailed explanations:

1. I understand that the 94-member group of HZ coding was intended to
accomodate some brain-dead mailers that can only handle typable (printable?)
ASCIIs. A smallest-common-denominator scenario that we've all come too familiar
with. But I suspect that the portion of those brain-dead mailers in use today
is small and decreasing. Mailers capable of full 7-bit (or even 8-bit) ASCII
are becoming more and more popular.
   With full 256/byte coding, we have 64k slots to play with. For 7-bit ASCII
communication, we can easily take out that 16k subset and still get almost
twice the space as in HZ.
   Of course, you could go further. And there're some Grand Coding that intend
to include all human languages on earth. For the time being let's focus on
good'ol ASCII and Chinese.

2. You could use the full 64k coding space to include lots of weird Chinese
chars that no one but a few archeologists could recognize. Well, we shouldn't
let those archeologists drag us all down. I think 8k chars should be sufficient
for most "normal" usage.
   So let's break the coding space into 8 chunks, each 8k big. The first chunk
corresponds to Fang3Song4 font. Kai3 font goes from 2000h to 3fffh. And so on.
We have 8 different fonts included in one coding scheme. Alternatively, we
could have four fonts each for simplified and traditional styles.

3. Say the font defaults to Fang3Song4. To change to Kai3 font, the programm
simply adds 2000h to each char's default code.
   Mapping the code to the proper font lib, in this scheme, can and should be
done by a highly optimized and standard printf routine. The printf takes one
explicit argument -- the code. But it should implicitly (via header file or
env variables) take the names of font lib files. Conceivably, the printf does
font-lib mapping according to the higher 3 bits of the code, in assembly
language. Then it simply stripps the 3 bits and goes into the particular lib.
   Once such a printf routine is done, support for mixing fonts/styles becomes
a non-issue. As I understand, currently such support is far from trivial, and
everyone re-invents the wheel as to how to support font mixing. Furthermore,
there is no need for additional font tags. It's just the higher 3 bits of the
code, and interpreted by printf. The programmer using the printf can totally
forget about it.
   If the 7-bit ASCII subset is to be used, then the sub-coding-space would
be non-contageous: 0000h-007fh, 0100h-017fh, ...7f00h-7f7fh. We have two 8k
chunks. But switching among the two is essentially the same mechanism, just
with an offset value of 4000h.

   As to choosing between 8 fonts with single style or 4 fonts each for
simplified and traditional styles, I'd say the need for mixing styles doesn't
come up so often.

4. Currently HZ uses a 2-byte combo to signify transition from ASCII to
Chinese and then another combo for the reverse. In practice, I find that this
sometimes results in mis-interpretation.
   As a heuristic estimate, let's assume equal probability distribution for
the 8-bit combinations. The probability for a random hit of the two-byte
escape sequence is 1/64k -- one screw-up for every 64k of a file. Not small
enough in my judgment. Why not use 4 bytes? The chance for random hit would
be 1 in 4 billion. Or more, for that matter. The transition doesn't occur
often for typical files. Therefore, the increase in file size is trivial.
Only those bytes causing confusion would need extra processing in the new
scheme. So the performance degradation shoudl also be trivial.

All flames/comments/dsicussions welcome.

Bo Peng