Subject: B5encode/B5decode and HZ+S Block 1/2 definitions

From: Jing-Shin Chang <[email-masked]>

The recent postings from Prof. Simpson for HZ+S Version 0.4 can be modified
in a way that is conformant to ISO 2022 and uses only LOGICAL operations
(ANDs or ORs) instead of using multiplications and divisions in making
the mapping from Big5 codes to printable (or 7-bit ASCII) characters.

Look at the following BIG5 map:

/*
 * @(#) Jing-Shin Chang, NTHU, Hsinchu, Taiwan, ROC.
 *
ASCII:  0x00 - 0x1f     - control characters
        0x20 - 0x2f     - space and symbols
        0x30 - 0x39     - digits (0 - 9)
        0x3a - 0x40     - symbols
        (0x20- 0x40     - symbols and digits)
        0x41 - 0x5a     - upper case
        0x5b - 0x60     - symbols
        0x61 - 0x7a     - lower case
        0x7b - 0x7e     - symbols
        0x7f (127)      - DEL
Big5:   High Bytes:     0x81-0x8d,0x8e-0xa0,0xa1-0xfe
                        (0x01-0x0d,0x0e-0x20,0x21-0x7e,
                        when high bit stripped)
        Low  Bytes:     0x40-0x7e,0xa1-0xfe
        a0: ' ' | 0x80, a1: '!' | 0x80, fe: '~' | 0x80

    High Bytes
    00            7e 80 81  8d 8e     a0 a1 a3 a4  c6 c9  f9 fa fe ff
00 +----------------------------------------------------------------+
   |                                                                |
   |                                                                |
3f +----------------------------------------------------------------+
40 +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+
 : | 40-7e: 63 chars    * U  * . U     . * S * # S  # ! A  ! x U x  |
 : |                    * 3  * . 2     . * P * # T  # ! L  ! x 1 x  |
 : |                    *    * .       . * C * # D  # ! T  ! x   x  |
7e +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+
80 (NULL|0x80)
81 +----------------------------------------------------------------+

a1 +--------------------*----*-.-------.-*---*-#---#--!----!-x---x--+
 : | a1-fe: 94 chars    * U  * . U     . * bf* # S #  ! A  ! x U x  |
 : |                    * 3  * . 2     . * c0$ # T #  ! L  ! x 1 x  |
 : |                    *    * .       . * e0$ # D #  ! Td5! x   x  |
fe +--------------------*----*-.-------.-*-----#---#--!------x---x--+
ff (DEL)

*/

In Prof. Simpson's proposal, the Big5 codes are divided into two Blocks
with:
                                         +++++++++++ ************
                                         +         + *          *
                                         + Block 1 + *  Block 2 *
                                         + a140 -  + *  c940 -  *
                                         +  c87f   + *    fefe  *
                                         +++++++++++ ************

and the codes are moved to the printable area by adding shift offsets
to the codes. Such operations required multiplications and divisions
for each code.

We can actually aviod using multiplications and divisions if we think
Block 1 and Block 2 are divided as follows:

                                         a1                     fe
                                     40  ************************
                                         *                      *
                                         * Block 2              *
                                         *                      *
                                     7e  ************************
                                     a1  ++++++++++++++++++++++++
                                         +                      +
                                         + Block 1              +
                                         +                      +
                                     fe  ++++++++++++++++++++++++

The codes in Block 1 have the 8-th bit set in both their high bytes and
low bytes. On the other hand, in Block 2, the 8-th bit of the high bytes
are set but the 8-th bit of the low bytes are cleared. Therefore, we can
map the big5 codes into the printable area simply as follows:

        1. clear all the 8-th bits of the high bytes when we encounter
                a big5 code.

        2. clear the 8-th bits of the low bytes for codes in Block 1
                but keep them untouched if the codes belong to Block2

        3. a string of Big5 codes is quoted with a SO (shift out) code
                (e.g., 0x0e or "~1") and a SI (shift in) code (e.g., 0x0f
                or "~}") to distinguish it from ASCII codes.

        4. within a quoted Big5 code string, the codes are assumes to be
                in Block 1 by defaults; however, if a code in Block 2 is
                encountered during the mapping process, it is preceeded
                by an SS2 escape sequence (e.g., ESC 'N' or ~2), which
                represent a temporary shift to the 2nd code plane (i.e.,
                Block 2). (* one SS2 for each Block 2 code *)

(For simplicity of illustration, I only consider a mixed ASCII and BIG5 text
in the above steps. Extension to incorporate ASCII/GB/BIG5 is trivial.)
To implement the encoding and decoding of the above steps, we only need two
simple finite state machines (4 states for encoding and 2 states for decoding).
The main operations will be logical ANDs and ORs without using multiplications
and divisions. The computation load can thus be ignored.

Two programs, called "b5encode" and "b5decode" are attached in this posting
to show the simple implementation. The SO (0x0e), SI (0x0f), SS2 (ESC 'N')
control sequences are the same as the ones used in the SEEDNET news postings
as mentioned in my previous postings. It is conformat to ISO 2022 since all
control codes started with characters outside the 0x21 - 0x7e range.

In fact, the news postings in the SeedNet of Taiwan, which is based on CNS
codes also use the same 7-bit mapping process. The SunMicro implementation of
the Chinese/Japanese/Korean email transmission in its Asia Language
Environment also used the same mapping process to change the 8-bit codes
(Big5 or EUC) to 7-bit ISO 2022 compatible streams before the email is
deliveried. (EUC has a very trivial mapping with CNS-11643/86).The languages
(corresponding to the ~{, ~1, ~2, and ~} escapes in our current discussion)
are specified by "ESC $ ) '0'|'A'|'B'|'C'" for Chinese (Taiwan), Chinese
(Mainland), Japanese or Korean. Therefore, we can use the same b5encode and
b5decode procedures for many current implementations in news and email
postings.

The original HZ and HZ+S proposal have an obvious problem for using '~' (0x7e)
'{' (0x7b), '}' (0x7d) '1' and '2' as escape codes. All these codes belong
to the printable ASCII area; therefore, they are not compatible with ISO 2022.

Hope the above idea and information are useful for the current Big5 -> 7-bit
Big5 (or -> CNS) discussions. BTW. The CNS <-> Big5 converter had been
uploaded to ifcss.org. Personally, I prefer to use the 7-bit CNS for news
exchange. The cost for using a BIG5 <-> HZ+S algorithm is essentially the
same as using a BIGs <-> CNS because the two programs are already there.

        Shin -^^-
===
The following two programs are tested on a 15K big5 files. It seems OK.
However, since it was written within only about 2 hours, your testing
reports and comments are highly appreciated.
=== B5encode.c ===
/*
 * @(#)$Id: b5encode.c,v 1.1 1994/05/01 19:01:15 shin Exp shin $
 * @(#) encode a BIG5 text into 7-bit stream, ISO 2022 compatible
 */

#include        <stdio.h>

#define         ESC     033             /* 0x2b */
#define         SO      016             /* 0x0e */
#define         SI      017             /* 0x0f */
#define         ALE_LANG        HLE     /* default character set for
                                           Chinese (T) */
#define         HLE     '0'             /* Chinese/Taiwan */
#define         CLE     'A'             /* Chinese/Mainland */
#define         JLE     'B'             /* Japanese */
#define         KLE     'C'             /* Korean */

#define         CNS_SS2 0x4e            /* ESC 'N' -- to 2nd plane */
#define         CNS_SS3 0x4f            /* ESC 'O' */

#define         EUC_SS2 0x8e            /* EUC_SS2 0xa2-b0 for plane 2-16  */
#define         EUC_SS3 0x8f

#define         STAT_ASC        0
#define         STAT_B5_C1L     1
#define         STAT_B5_C1H     2
#define         STAT_B5_C0      3

#define is_high(x)      (x>=0xa1&&x<=0xfe)
#define is_low(x)       (!is_high(x))

main()
{
        int state = STAT_ASC;
        int so = 0;
        int     c, c0;

        while ( (c = getchar() ) != EOF ) {
                switch (state) {
                case STAT_ASC:
                        if ( is_high(c) ) {
                                if ( !so ) {
                                        putchar(SO);
                                        so = 1;
                                }
                                c0 = (c&0x7f);  /* clear high-bit */
                                state = STAT_B5_C0;
                        }
                        else {
                                if ( so ) {
                                        putchar(SI);
                                        so = 0;
                                }
                                putchar(c);
                        }
                        break;
                case STAT_B5_C0:
                        if ( is_high(c) ) {
                                putchar(c0);
                                putchar(c&0x7f);
                                state = STAT_B5_C1H;
                        }
                        else {
                                putchar(ESC);
                                putchar(CNS_SS2);
                                putchar(c0);
                                putchar(c);
                                state = STAT_B5_C1L;
                        }
                        break;
                case STAT_B5_C1H:
                case STAT_B5_C1L:
                        if ( is_high(c) ) {
                                c0 = (c&0x7f);  /* clear high-bit */
                                state = STAT_B5_C0;
                        }
                        else {
                                putchar(SI);
                                putchar(c);
                                so = 0;
                                state = STAT_ASC;
                        }
                        break;
                default:
                        break;
                } /* switch */
        } /* while */
}

/*
 * History:
 *
 * $Log: b5encode.c,v $
 * Revision 1.1  1994/05/01  19:01:15  shin
 * Initial revision
 *
 */

=== B5decode.c ===
/*
 * @(#)$Id: b5decode.c,v 1.1 1994/05/01 19:02:46 shin Exp shin $
 * @(#) decode a BIG5 file from a 7-bit stream encoded by b5encode
 */

#include        <stdio.h>

#define         ESC     033             /* 0x2b */
#define         SO      016             /* 0x0e */
#define         SI      017             /* 0x0f */
#define         ALE_LANG        HLE     /* default character set for
                                           Chinese (T) */
#define         HLE     '0'             /* Chinese/Taiwan */
#define         CLE     'A'             /* Chinese/Mainland */
#define         JLE     'B'             /* Japanese */
#define         KLE     'C'             /* Korean */

#define         CNS_SS2 0x4e            /* ESC 'N' -- to 2nd plane */
#define         CNS_SS3 0x4f            /* ESC 'O' */

#define         EUC_SS2 0x8e            /* EUC_SS2 0xa2-b0 for plane 2-16  */
#define         EUC_SS3 0x8f

#define         STAT_ASC        0
#define         STAT_B5_C1L     1
#define         STAT_B5_C1H     2
#define         STAT_B5_C0      3

#define         STAT_SO         4
#define         STAT_SO_ESC     5
#define         STAT_SO_ESC_SS2 6

#define is_high(x)      (x>=0xa1&&x<=0xfe)
#define is_low(x)       (!is_high(x))

main()
{
        int state = STAT_ASC;
        int so = 0;
        int     c, c0;

        while ( (c = getchar() ) != EOF ) {
                switch (state) {
                case STAT_ASC:
                        if ( c == SO ) {
                                so = 1;
                                state = STAT_SO;
                        }
                        else {
                                putchar(c);
                        }
                        break;
                case STAT_SO:
                        if ( c == SI ) {
                                so = 0;
                                state = STAT_ASC;
                        }
                        else if (c == ESC) {
                                if ( getchar() != CNS_SS2 )
                                        fprintf("warning: unexpected char\n");
                                putchar(0x80|getchar());
                                putchar(getchar());
                        }
                        else {
                                putchar(0x80|c);
                                putchar(0x80|getchar());
                        }
                        break;
                default:
                        break;
                } /* switch */
        } /* while */
}

/*
 * History:
 *
 * $Log: b5decode.c,v $
 * Revision 1.1  1994/05/01  19:02:46  shin
 * Initial revision
 *
 */

================== Related Discussion ======
> [[email-masked]] said:
>
> Nelson said:
>    In GB it's as simple as putting a ~{ in front of the GB code to
>    make it HZ compliant but can the same be said with BIG5, can i just
>    put ~1 or ~2 in front of BIG5 code to get it HZ+S compliant, or is
>    there a conversion required.
>
> A very simple conversion is required, namely the 8-bit Big5 blocks
> must be packed into the 7-bit code space.  See HZ+S 0.4 below.
>
>    If a conversion is required, i would say just go straight to using
>    CNS since a conversion is required anyways.
>
> The trouble is that CNS is not at all the same thing as Big5.  (I
> among others were guilty of spreading misinformation on this point.)
> Many people have Big5 software, but not many people have CNS software.
> However, I see your point.  Let's see if Jing-Shin Chang or somebody
> else can supply CNS<->Big5 conversion routines that are simple and
> that really work.
>
>   HZ+S Version 0.4
>
>       ... stuffs deleted ...
>
>   In order to give Big5 a standard 7-bit representation, we partition
>   the Big5 code range into two blocks.  Blocks 1 and 2 consist of the
>                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   ranges A140 to C87F and C940 to FEFE respectively, where the second
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   byte is restricted to the ranges 40 to 7F and A1 to FE.  Block 1
>                                                          ^^^^^^^
>   contains 5401 more frequently used Chinese characters plus numerals,
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   Roman and Greek and Cyrillic and Bopomofo alphabets, symbols,
>   numerals, Japanese Kana, etc.  Block 2 contains less frequently used
>                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   Chinese characters plus some additional symbols.  We pack each of
>   ^^^^^^^^^^^^^^^^^^                                ***************
>   the Block 1 and 2 ranges in order into an initial segment of the
>   ****************************************************************
>   range 2121 to 7D7E where both bytes are restricted to the range 21
>   ******************
>   to 7E.  This packing is implemented by a simple macro.  Thus each
>                         ******************************
>   character in each Big5 block is mapped to a code consisting of two
>   printable ASCII bytes.
>
>   In addition to the the above, we need a way to distinguish between
>   Big5 Block 1, Big5 Block 2, GB, and ASCII.  We can do so by using
>   four tilde escape sequences:
>
>     ~{ to switch into GB (just as in the current HZ standard)
>
>     ~} to switch into ASCII (just as in the current HZ standard)
>
>     ~1 to switch into Big5 Block 1 (this is new)
>
>     ~2 to switch into Big5 Block 2 (this is new)
>
> -- Steve
>
===
> [[email-masked]] further said:

>  > BTW, could you post your Big5<->HZS conversion algorithm?
>
> The programs are at ftp.math.psu.edu, /pub/simpson/chinese/hzps.  They
> are lightly hacked versions of Fung's GB <-> HZ converters.  At this
> point in time they are essentially untested.  The Big5 -> HZ+S routine
> is
>
>      big2hzp(b1, b2, a1, a2)
>          int b1, b2, *a1, *a2;
>      {
>          unsigned long n;
>
>          n = 157 * ((b1 < 0xC9) ? (b1 - 0xA1) : (b1 - 0xC9))
>                 + ((b2 < 0xA1) ? (b2 - 0x40) : (b2 - 0xA1 + 63));
>
>          *a1 = 0x21 + (int) (n / 94);
>
>          *a2 = 0x21 + (int) (n % 94);
>      }
>
> and the HZ+S -> Big5 routine is
>
>      hzp2big(p, a1, a2, b1, b2)
>          int p, a1, a2, *b1, *b2;
>      {
>          unsigned long n;
>          int r;
>
>          n = 94 * (a1 - 0x21) + (a2 - 0x21);
>
>          *b1 = ((p == 1) ? 0xA1 : 0xC9) + (int) (n / 157);
>
>          r = (int) n % 157;
>
>          *b2 = ((r < 63) ? (0x40 + r) : (0xA1 + r - 63));
>      }
>
> I hope I've got these algorithms right -- I haven't had time to test them.
>
> -- Steve
>
===
> [[email-masked]] further said:
>
> Your file here is not an ASCII/Big5 file.  The character after Schr is
> decimal 148 or hexadecimal 0x94, which is not a legal byte in either
> ASCII or Big5.  ASCII consists only of the range 0 to 127, that is
> 0x00 to 0xFF.  Big5 bytes are in the range 0x40 to 0x7E and 0xA1 to
> 0xFE.
>
> Nevertheless, thanks for pointing this out.  I'm glad you made such a
> thorough test!  What's interesting here is that big2hzps translates
> the two characters after Schr as ~1 ,~}, where   is ASCII 17 or
> 0x11, i.e. Control Q, which is not in the HZ+S preferred range 0x21 to
> 0x7D.  We probably don't want this kind of thing to happen (it makes
> HZ+S less robust), so I'll fix big2hzps to avoid it.  By the way, Fung
> F. Lee's gb2hz has a similar problem.
>
>  > Maybe this is what we have overlooked: the IBM PC extended codes.
>
> I had not overlooked this.  I was well aware that HZ+S as I formulated
> it would not be able cope with IBM PC extended ASCII, or ISO-8859-1,
> or any similar character set, that is any character set which uses
> bytes in the range 0xA1 to 0xFE in a way that is incompatible with GB
> and Big5.  This is a limitation of HZ that carries over to HZ+S.
>
> To solve this, it would be possible (but awkward) to extend HZ+S by
> adding additional tilde escape sequences for ISO-8859-1, IBM PC
> extended ASCII, etc etc.  HZ+S would start looking more and more like
> ISO-2022-JP-2.
>
> -- Steve
>