Subject: A proposal for a GuoBiao and Big5 seven bit encoding scheme

From: [email-masked]
Date: Mon, 22 Oct 90 17:44:58 MDT

Below, I present what is hopefully, a reasonable, stable seven bit
encoding scheme for GuoBiao and Big5 encoded text.

Please alert me if I have left anything out or have something wrong.

[I have a program that implements the conversions between these
 encodings available.]

Send comments, bug reports, and code requests to:

Inernet: [email-masked]
Bitnet : [email-masked]

----------------------------- Article Start ----------------------

INTRO

In this article is presented an unambiguous and stable seven bit
encoding standard for Big5 or GuoBiao encoded Chinese electronic text.
It is based on the MIT X Consortium's "Compound Text Encoding 1.1"
standard specification.  The seven bit encodings are henceforth
refered to as B7 for seven bit encoded Big5 text and G7 for seven bit
encoded GuoBiao text.

WHY BIG5 AND GUOBIAO?

Big5, while not being an official standard, is probably used more
often than any other encoding scheme, especially in the Western world.

GuoBiao was officially registered by the PRC in 1980.

PROGRAMMING ISSUES

What is desirable in a seven bit encoding is, obviously, to represent
all of the text as seven bit values.  Although 0x00 is a valid value
in this range, it is desirable to avoid having this in the seven bit
text (for purposes of the C language and other 0x00 sensitive
systems).

B7<->Big5 conversion is problematic.  The second byte of the Big5
encoded Chinese character can be 0x40 thru 0xfe, or 190 possible
values, 127 of which are eight bit.  So three bytes are needed to
represent a Big5 encoded Chinese character in seven bits.  The first
byte of the Chinese character is treated the same way as it is in G7.
The second byte of the Chinese character is treated one of two ways:
if it is seven bits, don't change it, else if it is eight bits,
subtract 0x7e(127) from it.  This avoids having 0x00 occur in the text
and keeps the text seven bit.  Whether the second byte was seven or
eight bits, needs to be known for encoding/decoding purposes.

G7<->GuoBiao conversion is a simple process; inserting or
removing the start and end sequences, and setting and unsetting the
most significant bit.

B7 AND G7 ENCODING SPECIFICATION

---------------------------------------------------------------------

* A Compound Text extended segment is defined as:
*
* SS: ESC % / N M L NAME STX
*
* Where N is the number of bytes needed to represent one displayable
* character, M and L are combined to form a 14 bit number to determine
* the length of the text, including NAME and STX, NAME is the name of
* the encoding system, and STX(STart of teXt) is the value 0x02.  The
* valid range of values for N are 0x00 thru 0x04, with 0x00 indicating a
* varible number of bytes. The most significant bits of M and L are
* always set.

B7 is defined as a slightly modified Compound Text extended segment
using 3 bytes per character because there is no registered standard.

B7 encoding will be defined as:

SS: ESC % / 3 * * BIG5 0x02
ES: ESC ( B

B7 encoding does not use M or L because there are no 0x00 valued bytes
allowed in the encoding, and the most significant bit of M and L are
NOT set.  To implement a strict Compound Text encoded extended segment
for Big5 text, all that has to be done is add the M and L values.

Each B7 encoded Chinese character lying between the SS and ES will be
defined as:

Byte 1: 0x21 thru 0x7e
Byte 2: 0x07 or 0x08   (the number of bits in byte 2 of the Chinese character)
Byte 3: 0x40 thru 0x7f (if byte 2 of Chinese character was 7 bits)
        0x01 thru 0x7e (if byte 2 of Chinese character was 8 bits
                        calculated by subtracting 0x7e from byte 2 of
                        the Chinese character)

---------------------------------------------------------------------

G7 encoding will be defined as:

SS: ESC $ ( A
ES: ESC ( B

All bytes lying between the SS and ES will use 7 bits and have a value
0x21 thru 0x7e.

---------------------------------------------------------------------