An Extension to HZ - A Framework for 7-bit Encoding Scheme for Arbitrarily Mixed ASCII and Non-ASCII characters (Preliminary Draft) Ricky Yeung* June 12, 1994 1. Introduction HZ has been the popular 7-bit data encoding scheme for GuoBiao (GB) for five years. There has been recent interest on extending HZ to handle popular Chinese character codes like BIG-5. Although this need is transient in nature, as systems are getting more 8-bit clean and universal 16-bit character coding schemes like Unicode are emerging, it is anyway desirable to have a 7-bit data exchange format because it seems that this transient period would not end soon in the foreseeable future. HZ was originally designed as an exchange code for files. However, because of its choice of graphical escape sequence, it makes it possible to write real-time encode/decode programs, such as hztty, that would allow existing applications to run on top without the awareness of HZ. This extension is an attempt to try to leverage on the success of HZ and extending it to handle other non-ASCII coding schemes. The purpose of this specification is to provide a framework and it does not target any SPECIFIC coding scheme. 2. Specification of the Extensions To support a non-ASCII code scheme, the extension requires the following. There exists a mapping to map the non-ASCII character codes to a fixed number of ASCII byte codes, such that each byte is within the range of $21-$7E (hexadecimals), except the first byte, which can only be in the range of $21-$7D. A graphical ASCII character is a byte in the range $21-$7E. A non-graphical ASCII character is a byte in the range $0-$20 or of the value $7F. The code $7E is ~. There are two modes, namely ASCII mode and non-ASCII mode. Inside non-ASCII mode, there could be sub-modes, such as GB and BIG-5 sub-modes. By convention, a non-graphical ASCII character should only appear in ASCII mode. The default mode is ASCII mode. In ASCII mode, a byte is interpreted as an ASCII character, unless a '~' is encountered. The character '~' is an escape character. Depends on the character immediately followed, it has the following special meaning. o The escape sequence "~~" is interpreted as a "~". o The escape-to-non-ASCII sequence "~{" switches the mode from ASCII to non-ASCII. o The escape sequence "~\n" is a line-continuation marker to be consumed with no output produced. o Any character c, other than the three characters mentioned above, that follows "~" is interpreted as ~c (i.e., "~" followed by itself). In non-ASCII mode, characters are interpreted as non-ASCII codes until the escape-from-non-ASCII code "~}" is read. This code switches the mode from non-ASCII back to ASCII. (Note that the escape-from-non-ASCII code "~}" ($7E7D) is outside the defined non-ASCII range.) Inside non-ASCII mode, a tilde escape sequence is used to switch into specific non-ASCII code scheme. To maintain backward compatibility with HZ, if the first byte that follows "~{" is not "~", by default it switches to GB sub-mode (as defined in the HZ specification). When picking code scheme specific escape sequences in the non-ASCII mode, it is recommended that graphical ASCII sequences similar to the ISO standards are used. For examples, use ~A for GB, and ~G and ~H for CNS. (Note that there is no such obvious escape sequences for BIG-5, as it is not a registered ISO Standard. Perhaps ~g and ~h could be chosen.) Also it is recommended that specific ISO code mappings from 16-bit non-ASCII codes to 7-bit ASCII to be used if possible. 3. Rationales a. The ISO 7-bit exchange standards. One of the biggest advantages of HZ is that it is possible to use many standard ASCII applications such as editors, mailers, and news readers for GB without any modification to the applications. For example, running a real-time HZ encoder/decoder program such as hztty on top of cxterm, which by itself has no knowledge of HZ, is sufficient to allow mixed ASCII and GB code processing. It would be extremely difficult, if not impossible, to re-use ASCII applications in this "HZ-unaware" manner, had ISO-type of escape sequences been used. It is because ISO chooses the non-graphical ASCII character ESC as the beginning of the escape sequences. The problem with ESC is that it does not have a unique graphical representation. Notice that the HZ Extension defines the ASCII and non-ASCII modes while the ISO has no explicit non-ASCII mode. b. The issue of maximal ASCII transparency. Informally, ASCII transparency is defined as how much ASCII information is lost when a pure ASCII file goes through a decoder for exchange format like HZ. Obviously, the more tilde escape sequences are defined, the more ASCII transparency will be lost. Therefore the approach of defining individual tilde escape sequences for various coding schemes was not taken. Instead, ASCII and non-ASCII modes are explicitly distinguished so that only one tilde escape sequence is needed to switch from ASCII to non-ASCII mode. Non-ASCII code schemes are then localized in the non-ASCII mode. Also to prevent ASCII transparency from further erosion, the HZ Extension defines the other two-byte tilde escape sequences that are originally undefined in HZ. Please refer to the HZ specification for the other rationales and recommendations regarding HZ. They are generally applicable to this HZ Extension. Acknowledgement The idea of creating a general framework for non-ASCII codes and the issue of ASCII transparency were inspired by the discussion on extending HZ to handle BIG-5 among the author, Fung Fung Lee, and Yagui Wei about a year ago, and the recent attempt on the net to come up with a unified 7-bit exchange format for Big-5. Fung Fung Lee has been very helpful in shaping the ideas in this article. I also thank him for his careful review of this draft and numerous interesting suggestions. *e-mail address: Ricky.Yeung@Sun.Com Disclaimer: This article has no connection with my employer.