Subject: More B7 and G7 encoding info

From: [email-masked]
Date: Thu, 25 Oct 90 03:57:24 MDT

going away now.  I fed the cat. - Zippy''

More information on my seven bit encoding scheme and background.

Disclaimer: I do not claim to be a representative of  any standards
            organizations, any official agency, or the MIT X
            Consortium.  This document is intended solely for purpose
            of gathering comments and possibly to introduce a seven
            bit encoding scheme for Chinese text into common usage.

[If you have comments or want a copy of the Compound Text Encoding
 standard document and/or the code for my seven bit encoding scheme,
 please send email to:
 Inernet: [email-masked]
 Bitnet : [email-masked]]

Subjects are in this order:

1. The Compound Text Encoding Standard.
2. Is a seven bit encoding standard for Chinese needed?
3. Why should we use the Compound Text Encoding standard?
4. The basics of Big5 and GuoBiao encoding.
5. The B7 and G7 seven bit encodings.

1. The Compound Text Encoding Standard.

   The MIT X Consortium wisely knew that there would be a time when
   people would want to mix languages in a single document.  This can
   lead to some very ambiguous situations in the text.

   The Compound Text Encoding standard was introduced with the X
   windowing system because the X windowing system contains many of the
   tools needed to write programs to work with multi-lingual text.

   The Compound Text Encoding standard introduces escape sequences that
   disambiguate mixed languages in one document.  This standard is based
   off of two existing International Standards Organization documents,
   ISO 2022 and ISO 6429(used for Control Sequence standards).

   It is important to note that the implied ISO 2022 starting state is
   assumed to be an 8-bit environment(see the Compound Text Encoding
   standard document for more details).

   The Compound Text Encoding standard has escape sequences defined for:

   ASCII
   7-bit ASCII graphics(ANSI X3.4-1968)
   JIS X0201-1976(reaffirmed 1984) Japanese 8-bit Alphanumeric-Katakana
   ISO 8859-{1,2,3,4} Latin alphabets 1 through 4
   ISO 8859-5 Latin/Cyrillic alphabet
   ISO 8859-6 Latin/Arabic alphabet
   ISO 8859-7 Latin/Greek alphabet
   ISO 8859-8 Latin/Hebrew alphabet
   ISO/IEC DIS 8859-9 Latin Alphabet No. 5
   GB2312-1980 (PRC) Chinese Hanzi Character Set
   JIS X0208-1983 Japanese Kanji Character Set, Level 2
   KS C5601-1987 Korean Graphic Character Set

   There are other escape sequences defined for:

   directionality
   (right-to-left or left-to-right text)

   non-standard encodings
   (non-ISO 2022 encodings, see "other coding systems" in ISO2022 doc,
    ISO registered "other coding systems" not used in Compound Text Encoding)

   extensions
   (for future extensions to the Compound Text Encoding standard by the X
    Consortium)

   other general control sequences
   (see the Compound Text Encoding standard document for more details)

2. Is a seven bit encoding standard for Chinese needed?

   Unfortunately, there are still many mail programs and terminal
   emulator programs in use that do not handle eight bit characters.
   This is rapidly changing, but until the world is reasonably sure that
   most systems can send and receive eight bit text, we should use a
   seven bit encoding system.

   There are many seven bit encoding schemes for Chinese that have
   appeared over the years, but none of them have gained popular usage.
   This is due to many reasons, but (I think) the two most important
   reasons are (1) the encoding scheme had a problem not forseen by the
   author and (2) people would prefer to use a scheme that has been
   designated as a standard by an official institution; insuring a stable
   and consistent standard.

   The GuoBiao 8 bit standard already exists, and the Big5 encoding is
   used so often, that it can be relied on not to make any sudden or
   significant changes.
   (I'm told that there is an attempt to make Big5 a ROC standard)

   These two encodings can be reliably used to form seven bit
   encoding standards.

3. Why should we use the Compound Text Encoding standard?

   The primary reason the Compound Text Encoding standard should be used,
   is stability.  It is based off of two internationally agreed upon
   standards documents(ISO 2022 and ISO 6429).

   Another good reason is that the X windowing system is now being used
   in many countries, and on many different kinds of machines.  It is
   becoming a de facto standard in many areas. For instance, the X
   windowing system recently became a US Government standard for user
   interface development.  All programs written under the X window
   system that use or work with multi-lingual text will probably use the
   Compound Text Encoding standard.

4. The basics of Big5 and GuoBiao encoding.

   At some time in the development of Chinese text encoding and
   processing methods, it was determined, that to represent the minimum
   set of Chinese characters needed for basic electronic text
   communication, two bytes would have to be used for each Chinese
   character.

   Both Big5 and GuoBiao use two bytes to represent one Chinese
   character.  However, they are a little different internally.

   A Big5 encoded Chinese character is defined as:

     Byte1 - has a value in the range 0xa1(161) through 0xfe(254)
     Byte2 - has a value in the range 0x40(64)  through 0x7e(126) or
                                      0xa0(160) through 0xfe(254)

     This encoding allows 14664 characters to be represented.

     In Big5 encoding, characters 0xa140 through 0xa3fe are used for
     symbols, Chinese numbers, an ASCII set, Greek, Roman numerals,
     and ZhuYinFuHao(not necessarily in that order).  The Chinese
     characters actually start at 0xa440.

   A GuoBiao encoded Chinese character is defined as:

     Byte1 - has a value in the range 0xa1(161) through 0xfe(254)
     Byte2 - has a value in the range 0xa1(161) through 0xfe(254)

     This encoding allows 8836 characters to be represented.

     In GuoBiao encoding, characters 0xa1a1 through 0xa3fe are used for
     symbols, Chinese numbers, an ASCII set, Greek, Roman numerals,
     Japanese Katakana and Hiragana, Russian, and BoPoMoFo(not
     necessarily in that order).  The Chinese characters actually
     start at 0xb0a1, and sometimes there are characters located in
     the range 0xaffb through 0xaffe.

5. The B7 and G7 encoding schemes.

   What is desirable in a seven bit text encoding scheme is to avoid
   ambiguity, avoid the use of control characters(with the exception of
   the escape character), avoid the possiblity of 0x00 occuring in the
   text, and represent the text as a sequence of printable characters,
   0x20 through 0x7e.

   Control characters, 0x01 through 0x1f, except 0x1b, should be avoided
   because of possible interpretation conflict with terminal emulation
   and mail programs.  The exception of 0x1b(the escape character) is
   because an unambiguous character is needed to introduce special
   sequences, and because the escape character has a long history of
   successfull use as a special sequence indicator.

   The 0x00 value is used to terminate strings in the C programming
   language.  It is possible that an occurrence of this value in the text
   could affect the interpretation of the text by functions and programs
   written in the C language.  There are other programming languages that
   interpret 0x00 in the same way.

   It is desirable to have the Chinese text represented as printable ASCII
   characters to avoid any possible misinterpretations on the part of any
   terminal emulator or mail program.

   The encoding scheme presented below meets these conditions.
   The seven bit representation of Big5 encoded Chinese text will
   henceforth be refered to as B7 and the seven bit representation of
   GuoBiao encoded Chinese text will henceforth be refered to as G7.

   B7 <-> Big5 conversion is problematic.  The second byte of the Big5
   encoded Chinese character can be 0x40 thru 0x7e or 0xa0 thru 0xfe.
   Three seven bit bytes are needed to represent a Big5 encoded Chinese
   character in B7, with one of the bytes being used to distinguish
   whether the second byte of the Chinese character was seven or eight
   bits.

   G7 <-> GuoBiao conversion only requires two bytes for the seven or
   eight bit representation.

B7 AND G7 ENCODING SPECIFICATION

---------------------------------------------------------------------

* A Compound Text extended segment is defined as:
*
* SS: ESC % / N M L NAME STX
*
* Where N is the number of bytes needed to represent one displayable
* character, M and L are combined to form a 14 bit number to determine
* the length of the text, including NAME and STX, NAME is the name of
* the encoding system, and STX(STart of teXt) is the value 0x02.  The
* valid range of values for N are 0x00 thru 0x04, with 0x00 indicating a
* varible number of bytes. The most significant bits of M and L are
* always set.  The function used for determining the length of the
* text from M and L is: ((M - 128) * 128) + (L - 128).

B7 is defined as a slightly modified Compound Text extended segment
using 3 bytes per character because there is no registered standard.

B7 encoding will be defined as:

SS: ESC % / 3 * * BIG5 0x02
ES: ESC ( B

B7 encoding does not use M or L because there are no 0x00 valued bytes
allowed in the encoding, and the most significant bit of M and L are
NOT set, but they should be replaced with printable ASCII
characters(the asterisks in the SS mean any printable ASCII character).
To implement a strict Compound Text encoded extended segment for Big5
text, all that has to be done is calculate the M and L values when
encoding the string into B7 format.

Each B7 encoded Chinese character lying between the SS and ES will be
defined as:

Byte1: 0x21 thru 0x7e
       This value is calculated by subtracting 0x80(128) from the
       first byte of the Chinese character.

Byte2: 0x37 or 0x38
       This value is determined by the value of the second byte of
       the Chinese character.  If the second byte of the Chinese
       character is in the range 0x40 through 0x7e, this value should
       be 0x37(the character '7') and if the second byte of the
       Chinese character is in the range 0xa0 through 0xfe, this
       value should be 0x38(the character '8').

Byte3: 0x40 thru 0x7e
       If the second byte of the Chinese character is in the range
       0x40 through 0x7e, it should not change.
 or
       0x20 thru 0x7e
       If the second byte of the Chinese character is in the range
       0xa0 through 0xfe, this value is calculated by subtracting
       0x80(128) from the second byte of the Chinese character.

---------------------------------------------------------------------

G7 encoding will be defined as:

SS: ESC $ ( A
ES: ESC ( B

Each G7 encoded Chinese character lying between the SS and ES will be
defined as:

Byte1: 0x21 thru 0x7e
       This value is calculated by subtracting 0x80(128) from the
       first byte of the Chinese character.
Byte2: 0x21 thru 0x7e
       This value is calculated by subtracting 0x80(128) from the
       second byte of the Chinese character.

---------------------------------------------------------------------

---

From: Ed Lai <[email-masked]>
Date: Thu, 25 Oct 90 21:16:14 -0700

>seven bit encoding system.

However there is a question how well these programs handle ESC, in that
case it would defeat the whole purpose.

>The GuoBiao 8 bit standard already exists

This is a minor point, GuoBiao is a 7 bit standard, not 8 bit standard.
What is commonly consider to be GuoBiao is in fact the internal code of
CCDOS, which is a variant of GuoBiao, not GuoBiao itself.

In other words, the core of what is called G7 encoding is the actual GuoBiao,
so there is little disagreement on this.

>whether the second byte of the Chinese character was seven or eight

However the 3 byte Big-5 encoding is more of a problem. Three bytes per
character is fairly wasteful. Both GB and Big-5 characters are divided into
two sets. The first set is in the front (e.g. in Big-5 it is 0xC6?? or less)
and is the more frequently used characters. The idea is that some system
may only want to put up level 1 and not level 2. Of course this is rather
silly because no system is known to do that. But we can take advantage of
this fact for more compact coding, use two bytes for level 1 coding (actually
it can be more than level 1, it can be as many can be fit into two bytes
encoding), one character of the first byte is reserved as an escape character
so that the rest can be coded as three byte. This way the weighted average
byte per character would be very close to 2 instead of 3 in the original
proposal.

/* Disclaimer: All statments and opinions expressed are my own */
/* Edmund K. Lai                                               */
/* Apple Computer, MS37-UP                                     */
/* 20525 Mariani Ave,                                          */
/* Cupertino, CA 95014                                         */
/* (408)974-6272                                               */
zW@h9cOi