From simpson@math.psu.edu Sun May 1 08:55:45 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12824; Sun, 1 May 94 08:55:45 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id JAA05764; Sun, 1 May 1994 09:53:45 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 1 May 1994 09:53:44 -0400 Date: Sun, 1 May 1994 09:53:44 -0400 Message-Id: <199405011353.JAA22970@boole.math.psu.edu> To: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: HZ+S tests In-Reply-To: <9405011202.AA29987@wangwei.math.ncu.edu.tw> References: <9405011202.AA29987@wangwei.math.ncu.edu.tw> Reply-To: simpson@math.psu.edu Status: O shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) writes: > The hzps<->big5 programs version 0.51 passed my original set of tests, > and the whole set of TangShi 300 Shou. OK, great! Thanks! But I'm not finished with you! In the meantime I made some further improvements. The new version of hzps2big is better because it preserves things like ~{....~}, that is, GB stuff. I also wrote hzps2gb which preserves things like ~1....~} and ~2....~}, that is, Big5 stuff. These are version 0.60. They are at math.psu.edu, /pub/simpson/chinese/hzps. Please test them. > When I test it with a file > with more ASCII and Big5 mixed together, I found one failure: > Original: > > > 5o+H$H: Schroedinger@totoro (Schrdinger), +H0O: NCUMA (The above quote from you is probably corrupted, due to our mailer not being 8-bit clean.) Your file here is not an ASCII/Big5 file. The character after Schr is decimal 148 or hexadecimal 0x94, which is not a legal byte in either ASCII or Big5. ASCII consists only of the range 0 to 127, that is 0x00 to 0xFF. Big5 bytes are in the range 0x40 to 0x7E and 0xA1 to 0xFE. Nevertheless, thanks for pointing this out. I'm glad you made such a thorough test! What's interesting here is that big2hzps translates the two characters after Schr as ~1,~}, where  is ASCII 17 or 0x11, i.e. Control Q, which is not in the HZ+S preferred range 0x21 to 0x7D. We probably don't want this kind of thing to happen (it makes HZ+S less robust), so I'll fix big2hzps to avoid it. By the way, Fung F. Lee's gb2hz has a similar problem. > Maybe this is what we have overlooked: the IBM PC extended codes. I had not overlooked this. I was well aware that HZ+S as I formulated it would not be able cope with IBM PC extended ASCII, or ISO-8859-1, or any similar character set, that is any character set which uses bytes in the range 0xA1 to 0xFE in a way that is incompatible with GB and Big5. This is a limitation of HZ that carries over to HZ+S. To solve this, it would be possible (but awkward) to extend HZ+S by adding additional tilde escape sequences for ISO-8859-1, IBM PC extended ASCII, etc etc. HZ+S would start looking more and more like ISO-2022-JP-2. -- Steve From simpson@math.psu.edu Sun May 1 17:53:45 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA29423; Sun, 1 May 94 17:53:45 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA07949; Sun, 1 May 1994 18:51:45 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 1 May 1994 18:51:44 -0400 Date: Sun, 1 May 1994 18:51:44 -0400 Message-Id: <199405012251.SAA23962@boole.math.psu.edu> To: Jing-Shin Chang Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: B5encode/B5decode and HZ+S Block 1/2 definitions In-Reply-To: <199405012150.RAA07584@leibniz.math.psu.edu> References: <199405012150.RAA07584@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > The recent postings from Prof. Simpson for HZ+S Version 0.4 can be modified > in a way that is conformant to ISO 2022 and uses only LOGICAL operations > (ANDs or ORs) instead of using multiplications and divisions in making > the mapping from Big5 codes to printable (or 7-bit ASCII) characters. > ...... Many thanks for your thoughtful contribution. I am making the latest information on HZ+S available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. Some technical advantages of HZ+S are listed in the documents there. My only motivation for proposing HZ+S was that the international Chinese community seems ready for a standard 7-bit representation of mixed Big5 / GB / ASCII for applications such as Internet e-mail and news. HZ+S was the simplest and most straightforward proposal I could think of that meets these requirements. I have no strong preference for HZ+S as against other proposals that would accomplish the same thing. ISO-2022 compatibility is an additional desideratum. We now have several proposals on the table: my HZ+S, Ya-Gui Wei's HZ-2, a proposal by Yidao Cai using ~] and ~[, and Jing-Shin Chang's b5encode. There is also Jing-Shin's idea of using 7-bit CNS. May I suggest that Jing-Shin and other interested parties join the software authors group at ifcss.org. I think that is a good forum in which to compare the purely technical aspects. Broader Chinese computing aspects can also be discussed on CCNET. > The original HZ and HZ+S proposal have an obvious problem for using > '~' (0x7e) '{' (0x7b), '}' (0x7d) '1' and '2' as escape codes. All > these codes belong to the printable ASCII area; therefore, they are > not compatible with ISO 2022. > > Hope the above idea and information are useful for the current Big5 > -> 7-bit Big5 (or -> CNS) discussions. BTW. The CNS <-> Big5 > converter had been uploaded to ifcss.org. Personally, I prefer to > use the 7-bit CNS for news exchange. The cost for using a BIG5 <-> > HZ+S algorithm is essentially the same as using a BIGs <-> CNS > because the two programs are already there. > > Shin -^^- -- Steve From simpson@math.psu.edu Mon May 2 12:31:23 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA03141; Mon, 2 May 94 12:31:23 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA13596; Mon, 2 May 1994 13:29:11 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 2 May 1994 13:29:10 -0400 Date: Mon, 2 May 1994 13:29:10 -0400 Message-Id: <199405021729.NAA29469@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu In-Reply-To: shin@hermes.ee.nthu.edu.tw's message of Mon, 2 May 1994 10:34:58 GMT Subject: 7-bit Big5 proposals. Part 1, ISO-2022 compatibility Reply-To: simpson@math.psu.edu Status: O There are several 7-bit Big5 proposals under consideration: my HZ+S, Ya-Gui's HZ-2, Jing-Shin's B5encode, and an unnamed proposal from Yidao Cai using ~[ and ~]. There is also a proposal by Jing-Shin to use 7-bit CNS. Information and software on HZ+S are available at math.psu.edu, /pub/simpson/chinese/hzps. Big5 <-> CNS conversion software has been uploaded to ifcss.org. Could the proponents of the other proposals please make their material similarly available? shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) writes: > In my proposal, however, easy encoding and decoding without > knowing the exact code space is the first consideration. The > filter writters of the various applications simply do not have > to hack the real arrangements of the GB/HZ/BIG5/XXX codes. This > criterion also results in a very negilible computation load as a > side effect. (This is probably important for PC users.) And, as > an additional advantage, the program can be modified very easily > to interface with other multi-language applications. This > additional advantage, in some sense, is also important to > interface between commercially available applications (say, from > Sun's ALE/HLE/CLE/JLE/KLE implementation) and the contributions > from the hacker groups. Your b5encode proposal has several advantages, notably the *extremely* simple interconvertibility with 8-bit Big5 (even simpler than in my HZ+S and Ya-Gui's HZ-2). However, I have a question about ISO-2022 compatibility. Although Big5 is not a registered ISO-2022 standard, CNS is one, with ISO-2022 final characters G and H respectively. Here G designates the block containing the more frequently used characters, and H designates the block containing the less frequently used characters. By contrast, in B5encode, you use a completely different way of dividing up the Big5 character set into two blocks, so that both the more frequently used characters and the less frequently used characters straddle both blocks. Therefore, B5encode looks very different from what is done ISO-2022. Am I missing something? In the above I ignored the distinction between Big5 and CNS. They are by no means equivalent, but there is in fact a rough equivalence so far as the more and less frequently used blocks are concerned. If we wanted serious ISO-2022 compatibility, we would not use Big5 at all, rather we would use 7-bit CNS with ~G and ~H as the escapes (as in a now abandoned early version of HZ+S). We would then have to use fairly complicated CNS <-> Big5 conversion algorithms. My preference is for using Big5 directly. My reason is that CNS is not widely used, while Big5 is extremely popular. For the immediate needs of the international Chinese Internet community, popularity overrides ISO-2022 compatibility. -- Steve From simpson@math.psu.edu Mon May 2 12:40:58 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA03543; Mon, 2 May 94 12:40:58 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA13691; Mon, 2 May 1994 13:38:31 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 2 May 1994 13:38:29 -0400 Date: Mon, 2 May 1994 13:38:29 -0400 Message-Id: <199405021738.NAA29486@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw Subject: 7-bit Big5 proposals, Part 2: Big5 conversion, Fonts Reply-To: simpson@math.psu.edu Status: O shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) writes: > The encoding/decoding efficiency is also one of my concern. However, > I guess Prof. Simpson's algorithm could be modified so that, even > with the current packed Big5 order, only shift and +/- operations > are used. (We knew that * is simply a sequencs of bit shifts in > the bytes from standard computer science text books. Moving code > space simply with shift operations should not be so hard as one > might think.) Yes, that is true. In my HZ+S <-> Big5 algorithms, I multiply and divide by 94 and 157, but that could be implemented by bit-twiddling for speed. I guess this would then be almost as fast as Ya-Gui's HZ-2 <-> Big5 algorithm or Jing-Shin's b5encode. Another technical issue occurs to me: FONTS. In my HZ+S proposal, I am packing Big5 ranges 1 and 2 in order into the 7-bit code space. This means that an HBF font for the Big5 part of HZ+S will take about 10 seconds to construct. All that is needed is to change two lines in the HBF header file for a Big5 font. I think the other proposals are not so convenient from this point of view. Would someone care to undertake a comparison? -- Steve From simpson@math.psu.edu Mon May 2 12:51:30 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA03901; Mon, 2 May 94 12:51:30 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA04557; Mon, 2 May 94 10:46:12 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA13722; Mon, 2 May 1994 13:41:54 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 2 May 1994 13:41:53 -0400 Date: Mon, 2 May 1994 13:41:53 -0400 Message-Id: <199405021741.NAA29496@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang), ccnet-l@uga.cc.uga.edu Subject: 7-bit Big5 proposals, Part 3: User needs. In-Reply-To: <199405021729.NAA29469@boole.math.psu.edu> References: <199405021729.NAA29469@boole.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) writes: > Therefore, I think we should address more on users' points of view > (e.g., lack of usable characters, easiness of software installation > etc.), availability of softwares (what softwares are needed and how > long it takes an end user to work under the current architecture > without too much changes), stability of the newsgroup (can the proposed > solution works and lasts for a long time? chould the users changes their > current installation nearly once-and-for-all?) Of course, the judgement Yes, user needs should be considered. Could someone please undertake to compare the various 7-bit Big5 proposals from this point of view? Are there any substantial differences that would affect users? > I think we could spend a little more time to investigate such problems > before we make the final decision to configure the new chinese.* > hierarchy. Personally, I think two or three months is not too long to > make appropriate decisions on the final configurations. We can wait. I think the chinese.* organizers do not want to wait so long. They have worked very hard, and they want to go ahead now. The decisions that they make now need not be final. -- Steve From simpson@math.psu.edu Mon May 2 16:17:40 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12809; Mon, 2 May 94 16:17:40 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA15743; Mon, 2 May 1994 17:15:04 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 2 May 1994 17:15:03 -0400 Date: Mon, 2 May 1994 17:15:03 -0400 Message-Id: <199405022115.RAA29867@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re: CNS converter. Re: 7-bit clean big5 In-Reply-To: <9405021819.AA08350@hermes.ee.nthu.edu.tw> References: <9405021819.AA08350@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > The (&0x8080, &0x8000) implementation is commonly referred to as > the internal code of the CNS char. set here in Taiwan. You can use > your ETen for processing such CNS codes by entering the CNS mode: > etctl/c:si1,1. Interesting. Carlos McEvilly and I were recently having a discussion about this. Carlos started by saying that etctl/c:si1,1 referred to CNS, but then he somehow convinced himself otherwise. Carlos also pointed out that ETen also has options for other coding schemes in the chcode package. It is all a little confusing, but I'm glad that CNS is not completely obscure. For the short-term 7-bit needs of the international Chinese community on the Internet, I am still favoring HZ combined with some form of 7-bit Big5, rather than 7-bit CNS. The main reason is that HZ and GB and Big5 are extremely popular, while CNS is not. -- Steve From CAI@neurophys.wisc.edu Mon May 2 17:17:59 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA15640; Mon, 2 May 94 17:17:59 CDT Received: from VMSmail by neurophys.wisc.edu; Mon, 2 May 94 17:13 CST Message-Id: <24050217134211@neurophys.wisc.edu> Date: Mon, 2 May 94 17:13 CST From: CAI@neurophys.wisc.edu Subject: 7-Bit Big5 and chinese.* groups To: soft-authors@ifcss.org X-Vms-To: IN%"soft-authors@ifcss.org" Status: O >From: IN%"simpson@math.psu.EDU" 2-MAY-1994 12:39:44.63 There are several 7-bit Big5 proposals under consideration: my HZ+S, Ya-Gui's HZ-2, Jing-Shin's B5encode, and an unnamed proposal from Yidao Cai using ~[ and ~]. There is also a proposal by Jing-Shin to ------- Mine is not a proposal at all. I just suggest you and Ya-Gui to use ~[ and ~] as escape codes, which I think look better than your ~1 ~2 under ASCII text mode (Ya-Gui abandoned his ~|). For the immediate needs of the international Chinese Internet community, popularity overrides ISO-2022 compatibility. ------ agree. Some of the following are related to chinese.* news froup. >From: IN%"simpson@math.psu.EDU" 2-MAY-1994 10:24:14.58 I'm very much in favor of going ahead full steam with the chinese.* hierarchy. Let's make this thing a reality as soon as possible. ----- I have the same feeling. I have a concern about coding schemes, namely that an HZ-only policy may tend to exclude Taiwanese participation. Therefore, I decided to ----- I did not perceive "HZ-only" from the proposal, although I think it could be better phrased to make people from taiwan feel better if they do not agree with the present phrasing, see my post on acc friday. Notice how Wei-Chang is somewhat upset about a perceived exclusion of Taiwanese, because of the HZ-only policy. I tried to calm him down and explain that chinese.* is for all Chinese speakers everywhere, and that the HZ-only policy is not *intended* to exclude anyone (even though it may have that unintentional effect). ------ again it is not HZ-only, so calm down. I am also in favor of bring Big5 or something else that most widely used in taiwan into to this land, but we don't need to wait untill everything is ready, right? I have to repeat that 8-bit Big5 is not practical at least at present, simply because it would not work for people who access usenet through a PC or a terminal, not directly from a workstation. It simply crashes the machine or freeze the screen at some point, I've experienced both. I heard that in Japan the terminals and PC are able to handle 8 bit directly, I think they must be configed different. (Can somebody provide more info on this aspect?) On the other hand, even if you allow 8-bit Big5, I am sure there are some "trouble maker" going to ask 8-bit GB, which I think is reasonable. Are we going to allow 8-bit GB too? If we do, then how do you distinguish 8-bit GB and 8-bit Big5, especially they are in the same article (it can happen in those replys) ? Then we may end up with two groups, one for GB or one for Big5, and cross post into two, like act <-> act.big5. Jing-Shin has a somewhat different perspective. He is a workstation and Chinese computing "power user" who sees coding problems as purely technical. His idea is that software can hide all coding problems ------ I also sees it purely technical and this is why I asked the discussion to be moved to software authors' list. not popular anywhere else, but according to Jing-Shin that's OK because perfect CNS <-> Big5 translation is embedded in the e-mail and news interface. I'm not quite sure how to answer Jing-Shin. He makes some very good points, but I fear he may be underestimating the technical difficulties faced by the average user with a PC or a Mac. ------- I am not against anything that makes users for taiwan "feel better" (if it helps) as long as it is 7-bit. But I favor the 7-bit big5 approach over CNS for reasons below. 1. Big5 is more popular. 2. Although CNS is 7-bit, it can not be directly used on network. Thus we have big5 -> CNS -> whatever-code instead of 8-bit big5 -> 7-bit big5. now. But any decisions made now can be changed later, as the Internet Chinese community grows and changes. ----- I fully agree. Yes. We should try to resolve these coding issues in a way that is comfortable for everybody. HZ+S is one idea, Big5 cross-posting is another, but let's resolve it somehow. I feel that HZ-only is not a good solution. ------ How about this: before 7-bit Big5 or CNS softwares come into existance, let's create a big5 mirror group under alt.* (since it is simpler) and cross-post, then gradually phase out the cross-posting. Another related issue: I consider the printable escape codes in Hz as a merit since you can see it and know it is there. And you can hand insert it under ASCII text mode. This is especially helpful to repair damaged files from my experience with Hz and JIS files. cai From simpson@math.psu.edu Mon May 2 19:16:57 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA19758; Mon, 2 May 94 19:16:57 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id UAA16563; Mon, 2 May 1994 20:14:18 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 2 May 1994 20:14:17 -0400 Date: Mon, 2 May 1994 20:14:17 -0400 Message-Id: <199405030014.UAA00284@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: handa@etlken.etl.go.jp (Kenichi Handa) Subject: 7-Bit Big5 and chinese.* In-Reply-To: <24050217134211@neurophys.wisc.edu> References: <24050217134211@neurophys.wisc.edu> Reply-To: simpson@math.psu.edu Status: O CAI@neurophys.wisc.edu writes: > From: IN%"simpson@math.psu.EDU" 2-MAY-1994 12:39:44.63 > There are several 7-bit Big5 proposals under consideration: my HZ+S, > Ya-Gui's HZ-2, Jing-Shin's B5encode, and an unnamed proposal from > Yidao Cai using ~[ and ~]. There is also a proposal by Jing-Shin to > > ------- Mine is not a proposal at all. I just suggest you and Ya-Gui > to use ~[ and ~] as escape codes, which I think look better > than your ~1 ~2 under ASCII text mode (Ya-Gui abandoned his ~|). I have to agree that ~[ and ~] look better, and I have nothing against them (except maybe that they suggest a matched pair like ~{ and ~}). Also, a couple of people have pointed out to me that ~ is an operator in C, so ~1 and ~2 can occur in C code. Kenichi Handa even suggested ~[ and ~] to avoid this conflict! I guess great minds think alike (Yidao's and Kenichi's I mean). I don't want to make too many changes too fast, but let's just say I'm now thinking I'll use ~[ and ~] in the next (final?) revision of the HZ+S spec. Apart from this, the spec has been stable for at least 48 hours! About the names HZ+S and HZ-2. If we on soft-authors can reach a consensus and present this proposal to the wider community, I'd like to call it HZ+, implying an enlargement of HZ. HZ+S is too ugly, and HZ-2 is too "negative" (suggests minus 2). > ------ again it is not HZ-only, so calm down. I am also in favor of > bring Big5 or something else that most widely used in taiwan > into to this land, but we don't need to wait untill everything > is ready, right? I agree. The chinese.* hierarchy should not wait for HZ+. But keep in mind that Big5 is a way of life in Taiwan. Given the high level of Internet activity in Taiwan, some form of Big5 will inevitably play a role in the chinese.* hierarchy. > I have to repeat that 8-bit Big5 is not practical at least at > present, simply because it would not work for people who access I have slowly come around to this view too. I used to think that the success of a.c.t.big5 means that 8-bit news is no problem, but several people have argued that this is not so. According to Yongguang Zhang, there are a lot of older Unix sites which use UUCP news feeds and have 7-bit-only tty drivers. There is no way for those folks to read 8-bit news. The upshot is that chinese.* needs some form of 7-bit Big5. I feel we can have a practical HZ+ or 7-bit Big5 in place very soon now, and the chinese.* organizers seem to recognize the need for such a standard. If chinese.* were to standardize on HZ+, this could be mutually beneficial for both HZ+ and the chinese.* hierarchy. I should mention that Ya-Gui, Yidao and I all belong to the volunteer committee to guide the chinese.* hierarchy. > ------ How about this: before 7-bit Big5 or CNS softwares come into > existance, let's create a big5 mirror group under alt.* (since > it is simpler) and cross-post, then gradually phase out the > cross-posting. Sounds good. This gives HZ+ software time to catch up, but nothing is set in stone. The chinese.* committee should get some input from Taiwan on this. > I consider the printable escape codes in Hz as a merit since you > can see it and know it is there. And you can hand insert it under > ASCII text mode. This is especially helpful to repair damaged > files from my experience with Hz and JIS files. Good point. ISO-2022 uses the Escape character 0x1E, which not everybody's editor understands. Of course there are tradoffs, because ~ makes it awkward to extend HZ in some desirable ways, e.g. European alphabet support via ISO-8859-1. In my view, the decisive factor is that ~ is part of the HZ standard, which is very well established and has proved its practicality. I see HZ+ as an attempt to build on the great success of HZ. -- Steve From shann@wangwei.math.ncu.edu.tw Mon May 2 22:09:21 1994 Return-Path: Received: from wangwei.math.ncu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA26073; Mon, 2 May 94 22:09:21 CDT Received: from qingzhao.math.ncu.edu.tw by wangwei.math.ncu.edu.tw (4.1/SMI-4.1) id AA03494; Tue, 3 May 94 11:06:09 CST Date: Tue, 3 May 94 11:06:09 CST From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Message-Id: <9405030306.AA03494@wangwei.math.ncu.edu.tw> To: CAI@neurophys.wisc.edu Subject: Re: 7-Bit Big5 and chinese.* groups Cc: soft-authors@ifcss.org Status: O CAI@neurophys.wisc.edu wrote: I have to repeat that 8-bit Big5 is not practical at least at present, simply because it would not work for people who access usenet through a PC or a terminal, not directly from a workstation. It simply crashes the machine or freeze the screen at some point, I've experienced both. I heard that in Japan the terminals and PC are able to handle 8 bit directly, I think they must be configed different. (Can somebody provide more info on this aspect?) I have totally reversed experiences. I thought 8-bit is no problem with PC (I don't know terminals), but sometimes troublesome with workstations. For instance, the only machine which cannot handle 8-bit e-mails, in my environment, is a DECstation 5125 Ultrix 4.2. And the gopher client on SPARCstations cannot read the Chinese filenames if I don't compile it with /usr/5bin/cc and define other parameters. But for PCs, all I have to do is to get an EXE file from an ftp site, and run it. I have never had any problems with PC news, gopher or www clients. Of course it crashes now and then, but I think it's just because "it is a PC". Don't you think so? PCs just easy to crash. I know Jananese PCs are different. But I don't know any differences in Taiwanese PCs. I believe they are identical as those we export to America or Europe. -Shann From mcevilly@netcom.com Tue May 3 01:01:25 1994 Return-Path: Received: from netcom.com (netcom9.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA02155; Tue, 3 May 94 01:01:25 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id XAA10196; Mon, 2 May 1994 23:00:40 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405030600.XAA10196@netcom.com> Subject: CNS and ETen To: ccnet-l@uga.bitnet Date: Mon, 2 May 1994 23:00:40 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <199405022121.OAA13750@netcom.com> from "Stephen G Simpson" at May 2, 94 05:15:03 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2036 Status: RO Stephen Simpson writes: > Interesting. Carlos McEvilly and I were > recently having a discussion about this. Carlos started by saying > that etctl/c:si1,1 referred to CNS, but then he somehow convinced > himself otherwise. Carlos also pointed out that ETen also has options I'm in trouble now. I just did a more thorough test, and indeed the above switch does enable CNS code. Here is a sample table of codepoints; readers who have ETen can try the above switch, and assuming all 8 bits make it through the mail to your computer, it should work fine. CNS plane one codepoints for Chinese hanzi numbers 1-10 ======================================================= Hanzi Hex ch Ch cH CH ===== ==== == == == == 1 4421 == D! Ä! D¡ Ä¡ 2 4428 == D( Ä( D¨ Ĩ 3 4435 == D5 Ä5 Dµ ĵ 4 463E == F> Æ> F¾ ƾ 5 446C == Dl Äl Dì Äì 6 447A == Dz Äz Dú Äú 7 4424 == D$ Ä$ D¤ Ĥ 8 442C == D, Ä, D¬ Ĭ 9 4426 == D& Ä& D¦ Ħ 10 4432 == D2 Ä2 D² IJ In the above cH means the first byte has the high bit off and the second byte has the high bit on. CH means the high bits are on in both bytes. ETen will correctly interpret the codes in the last column. I did try this bit twiddling last time, but apparently the codepoints I chose earlier were from another plane. I regret the error and hope it hasn't negatively influenced consideration of CNS. However, there is still an important unanswered question with ETen and CNS: how does one switch from one plane to another? CNS 11643 says a little about this, but I haven't been able to figure out the odd notation they use. There are escape codes to shift from one plane to another. The CNS 11643 intro says use of these escape codes, or control codes, is covered in CNS 7654--does anyone on the list have a copy of this? Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From mcevilly@netcom.com Tue May 3 01:15:33 1994 Return-Path: Received: from netcom.com (netcom9.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA02392; Tue, 3 May 94 01:15:33 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id XAA11829; Mon, 2 May 1994 23:14:09 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405030614.XAA11829@netcom.com> Subject: Re: CNS and ETen To: ccnet-l@uga.bitnet Date: Mon, 2 May 1994 23:14:04 -0700 (PDT) Cc: soft-authors@ifcss.org X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 928 Status: RO Just in case the high bits did get stripped in the previous message, here are the codepoints, uuencoded: ========== cut here ========= begin 600 test.cns M0TY3('!L86YE(&]N92!C;V1E<&]I;G1S(&9O"`@("`@("!C:"`@0V@@(&-(("!#2`H] M/3T]/2`]/3T]("`@("`@/3T@(#T]("`]/2`@/3T*,2`@("`@-#0R,2`@/3T@ M($0A("#$(2`@1*$@(,2A"C(@("`@(#0T,C@@(#T]("!$*"`@Q"@@($2H("#$ MJ`HS("`@("`T-#,U("`]/2`@1#4@(,0U("!$M2`@Q+4*-"`@("`@-#8S12`@ M/3T@($8^("#&/B`@1KX@(,:^"C4@("`@(#0T-D,@(#T]("!$;"`@Q&P@($3L M("#$[`HV("`@("`T-#=!("`]/2`@1'H@(,1Z("!$^B`@Q/H*-R`@("`@-#0R M-"`@/3T@($0D("#$)"`@1*0@(,2D"C@@("`@(#0T,D,@(#T]("!$+"`@Q"P@ M($2L("#$K`HY("`@("`T-#(V("`]/2`@1"8@(,0F("!$IB`@Q*8*,3`@("`@ :-#0S,B`@/3T@($0R("#$,B`@1+(@(,2R"@H@ ` end ========= cut here ========= Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From simpson@math.psu.edu Tue May 3 02:04:37 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA03910; Tue, 3 May 94 02:04:37 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id DAA18427; Tue, 3 May 1994 03:02:37 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 03:02:35 -0400 Date: Tue, 3 May 1994 03:02:35 -0400 Message-Id: <199405030702.DAA00932@boole.math.psu.edu> To: mcevilly@netcom.com (Carlos McEvilly) Cc: soft-authors@ifcss.org Subject: problem with twnmoe10 VM/CMS server In-Reply-To: <199405030410.VAA25730@netcom.com> References: <199405030014.UAA00284@boole.math.psu.edu> <199405030410.VAA25730@netcom.com> Reply-To: simpson@math.psu.edu Status: O Carlos McEvilly writes: > (I'm sending this to you, but feel free to cc: any reply to > the lists if you want). Thanks, Carlos. I'm copying to soft-authors, where the technical discussion on 7-bit Big5 is taking place. > Yet another small data point: the twnmoe10 machine in Taiwan > (twnmoe10.edu.tw or twnmoe10.bitnet) is a VM/CMS system run by > the Ministry of Education. When I was in Taiwan, this machine > was not 8-bit clean. It had another problem as well which may > be a consideration against ~[ and ~]: the '[' and ']' characters > would get deleted in any ASCII file transfer. They were visible > on the screen however, and present on the VM/CMS disk. A binary > file transfer would give the user an EBCDIC file, and I never did > find a good EBCDIC->ASCII conversion program (should have written > one, but that's behind me now :). This problem with the square > brackets may just be a matter of improper setup of the conversion > table on the twnmoe10 machine, and it may be solved by now. If > on the other hand it is a VM/CMS problem, there are probably > a fair number of users at twnmoe10 who cannot read Big5 news and > wouldn't be able to download HZ+S news (they would be able to > read it onscreen, of course, with the right software). > > I don't know how many people use twnmoe10, but it seemed to be > a significant number. This is the site used by anybody in Taiwan > whose organization does not have their own Internet-connected > computing center. This includes research entities, libraries, > small (officially recognized or government run) colleges, > universities, and institutes, and other government units. > Thanks very much for that information. Because of imperfect ASCII <-> EBCDIC translation, VM/CMS machines cannot handle ASCII properly. I'm talking about ordinary, 7-bit, printable ASCII here! This is a well-known, extremely irritating problem to which, last time I checked, there was no really good solution. It affects a lot of things. For instance, uuencoded files usually get mangled when they pass through VM/CMS. Very annoying. This is I think the principle reason why the MIME multimedia mail standard uses something called mmencode rather than uuencode. I don't know whether there is any way to cope with this problem. Maybe ~[ and ~] is not the best choice of escapes, but what choice if any can save us from VM/CMS weirdness? I am inclined to throw up my hands and say, any machine that cannot handle ASCII should be disconnected from the Internet. Does anybody have any ideas on this? -- Steve From simpson@math.psu.edu Tue May 3 03:23:16 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA05806; Tue, 3 May 94 03:23:16 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id EAA18688; Tue, 3 May 1994 04:20:42 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 04:20:40 -0400 Date: Tue, 3 May 1994 04:20:40 -0400 Message-Id: <199405030820.EAA01089@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re: Taiwanese participation in the chinese.* hierarchy In-Reply-To: <9405030240.AA24996@ifcss.org> References: <9405030240.AA24996@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes: > The alt.chinese.text FAQ contains answers about the HZ "mirror" of > Taiwan newsgroups and other pertinent facts. Thanks for pointing this out. I had of course read the a.c.t FAQ some time ago, but I somehow never noticed the information about the HZ mirror of Big5 Taiwan newsgroups. Thank you also, Ya-Gui, for your work in compiling this excellent FAQ. Equally pertinent is the lengthy FAQ for alt.chinese.text.big5. For those who don't know, let me mention that a.c.t.big5 is the Big5 cross-mirror of a.c.t., set up by people in Taiwan. What's especially interesting to me right now is that a great many of the answers in the a.c.t.big5 FAQ mention the words "8-bit clean". Reading both of these FAQs together, one gets the strong impression that the world needs some kind of 7-bit version of Big5. The Chinese software authors group, soft-authors@ifcss.org, is currently discussing various proposals for 7-bit Big5. (The specific proposals now being discussed are Ya-Gui's HZ-2, my HZ+S, Jing-Shin Chang's b5encode, and various ideas about 7-bit CNS.) I am hoping that soft-authors will quickly reach a consensus and present a practical 7-bit Big5 standard. I imagine that such a standard may be useful for the proposed chinese.* newsgroup hierarchy. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Tue May 3 03:43:06 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA06381; Tue, 3 May 94 03:43:06 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id EAA18778; Tue, 3 May 1994 04:41:08 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 04:41:07 -0400 Date: Tue, 3 May 1994 04:41:07 -0400 Message-Id: <199405030841.EAA01124@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Re: problem with twnmoe10 VM/CMS server In-Reply-To: <9405030810.AA04956@iitmax.acc.iit.edu> References: <199405030702.DAA00932@boole.math.psu.edu> <9405030810.AA04956@iitmax.acc.iit.edu> Reply-To: simpson@math.psu.edu Status: O Jianqing Hu writes: > Another point. [ and ] will be easily stripped out by a incorrect > Kermit transfer setup, say one end is in binary mode and the other > is in ASCII/Text mode.(Machine involved are PCs and UNIX hosts) Thanks for pointing out that problem. It's good also to know that the problem can be corrected by putting Kermit in ASCII mode at both ends. By contrast, the VM/CMS problem has no good solution. There is no way to make VM/CMS handle printable ASCII 100 percent correctly. A machine that can't handle printable ASCII 100 percent correctly can't deal with HZ, because HZ uses all of the printable ASCII bytes in an essential way. What do you think would be a good choice for the tilde escapes in HZ+S? As I recall, you were one of those who objected to ~1 and ~2 because ~ is an operator in C. -- Steve From simpson@math.psu.edu Tue May 3 06:00:17 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA09436; Tue, 3 May 94 06:00:17 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id GAA19163; Tue, 3 May 1994 06:58:19 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 06:58:18 -0400 Date: Tue, 3 May 1994 06:58:18 -0400 Message-Id: <199405031058.GAA01344@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: tilde escapes In-Reply-To: <9405030935.AA30037@iitmax.acc.iit.edu> References: <199405030841.EAA01124@boole.math.psu.edu> <9405030935.AA30037@iitmax.acc.iit.edu> Reply-To: simpson@math.psu.edu Status: O Jianqing Hu writes: > I have friends using IBM VM/CMS system enjoy ACT with no problem. In my opinion, your friends are lucky. The situation may be similar to uuencoded files. They can pass through certain VM/CMS machines unharmed, but a VM/CMS network connection can mangle them. For example, Carlos said that ASCII file transfer from one particular machine strips out [ and ]. This means that on that particular machine, all HZ codes containing [ and ] would be corrupted, no matter what tilde escapes are used. That is more than 4 percent of Chinese characters. > 4 Avoid troublesome characters, such as > > < > | - + / \ (Redirector, switch token) > [ ] (Said to be sensitive to CM/VMS conversion error) > ? ^ There are usually used to substitute missing characters. Are < and > really so troublesome? They serve as redirection symbols in shell scripts, but I don't think a shell script would contain ~< or ~>. It might be nice to use ~< and ~> as tilde escapes for Big5 in HZ+, with ~> switching to the more frequently used block and and ~< switching to the less frequently used block. Easy to remember. :-) > Could them be ~_ , ~', ~` , or ~%. Does anybody else have anything to say about this? Is there anything wrong with ~y and ~z? Maybe I should just stick to ~1 and ~2. I feel that all choices of tilde escapes will fail under certain circumstances, but most of the time everything will be fine, provided your network understands ASCII. -- Steve From simpson@math.psu.edu Tue May 3 07:03:53 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA10901; Tue, 3 May 94 07:03:53 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA19386; Tue, 3 May 1994 08:01:23 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 08:01:22 -0400 Date: Tue, 3 May 1994 08:01:22 -0400 Message-Id: <199405031201.IAA01463@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: HZ mirror of some *.tw Big5 newsgroups In-Reply-To: <9405031140.AA09520@hermes.ee.nthu.edu.tw> References: <9405031140.AA09520@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) writes: > Sorry, I didn't check the gopher services, although I knew that many > gopher servers do provide both BIG5 and HZ subject lines. I guess such > services are particularly for the HZ users, not the local users. You read the original tw.* Big5 newsgroups, so it's understandable that you would have overlooked the existence of the HZ mirrors. I see that you have not yet joined the soft-authors group at ifcss.org. We are trying to finalize a practical standard for 7-bit Big5. Your input would be welcome. We need to hear more Taiwanese viewpoints. -- Steve From ta_k@maths.su.oz.au Tue May 3 07:19:30 1994 Return-Path: Received: from adder.maths.su.oz.au by ifcss.org (4.1/IFCSS-Mailer) id AA11205; Tue, 3 May 94 07:19:30 CDT Date: Tue, 3 May 1994 22:17:04 +1000 From: ta_k@maths.su.oz.au (The Alan Tam Fan (~{L7S=wkCT~})) Message-Id: <199405031217.WAA25838@adder.maths.su.oz.au> Received: from ta_k@localhost by adder.maths.su.oz.au (8.6.7/14.7 to SMTP) id WAA25838; Tue, 3 May 1994 22:17:04 +1000 To: soft-authors@ifcss.org Subject: how to unsubscribe? Status: O Hi, Sorry for this here. But how do I unsubscribe? I don't log on often enough and my mail box is getting flooded. thanks very much bye Katya ============================================================================== Katya Ta ~{P;@v7<~} ta_k@maths.su.oz.au ~{ Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11415; Tue, 3 May 94 07:25:55 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA19539; Tue, 3 May 1994 08:23:17 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 08:23:15 -0400 Date: Tue, 3 May 1994 08:23:15 -0400 Message-Id: <199405031223.IAA01517@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Cc: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Subject: Re: 7-bit Big5 proposals, Part 2: Big5 conversion, Fonts In-Reply-To: <9405031207.AA09588@hermes.ee.nthu.edu.tw> References: <9405031207.AA09588@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O I wrote: > In my HZ+S proposal, I am packing Big5 ranges 1 and 2 in order into > the 7-bit code space. This means that an HBF font for the Big5 part > of HZ+S will take about 10 seconds to construct. All that is needed > is to change two lines in the HBF header file for a Big5 font. shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) replied: > In my vision, since the existence of a 7-bit Big5 encoding is to avoid > communication problems, not for processing problems, the data stream will > eventually be recoverd to its 8-bit form for the appropriate applications > at the user terminal. Then why should we care about or even create another > font that had been there already? You make a good point. However, things sometimes move in unexpected ways. HZ started out as a 7-bit mixed GB and ASCII for e-mail and news. But later Ya-Gui created ZWDOS, an HZ operating environment with its own screen display and printing capabilities. I was trying to look ahead to this sort of development. But your point is certainly correct. The immediate need is for 7-bit e-mail and news, not printing. -- Steve From CAI@neurophys.wisc.edu Tue May 3 10:00:37 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA15879; Tue, 3 May 94 10:00:37 CDT Received: from VMSmail by neurophys.wisc.edu; Tue, 3 May 94 09:56 CST Message-Id: <24050309562692@neurophys.wisc.edu> Date: Tue, 3 May 94 09:56 CST From: CAI@neurophys.wisc.edu Subject: problem with 8-bits To: soft-authors@ifcss.org X-Vms-To: WIRCS1::IN%"shann@wangwei.math.ncu.edu.tw", IN%"soft-authors@ifcss.org" Status: O >From: IN%"shann@wangwei.math.ncu.edu.tw" 2-MAY-1994 22:04:42.03 But for PCs, all I have to do is to get an EXE file from an ftp site, and run it. I have never had any problems with PC news, gopher or www clients. Of course it crashes now and then, but I think it's just because "it is a PC". Don't you think so? PCs just easy to crash. -Shann ----- No, it is not simply because "it is a PC", because it happens at least 10 times more often than usual. I don't know why but it just happens. cai From xiaofei Tue May 3 12:28:51 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA22335; Tue, 3 May 94 12:28:51 CDT Date: Tue, 3 May 94 12:28:51 CDT From: xiaofei (XiaoFei Wang) Message-Id: <9405031728.AA22335@ifcss.org> To: soft-authors, ta_k@maths.su.oz.au Subject: Re: how to unsubscribe? Status: O Hello, anyone who wish to unsubscribe the list `soft-authors', please mail a message to me at xiaofei@ifcss.org, thanks. From jiang Tue May 3 16:40:57 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA02406; Tue, 3 May 94 16:40:57 CDT Date: Tue, 3 May 94 16:40:57 CDT From: jiang (Yuan Jiang) Message-Id: <9405032140.AA02406@ifcss.org> To: soft-author, soft-authors Subject: Incorporating 7-bit Big5 Status: O I believe the major gpoints are already expressed on this matter. Our objective is clear -- putting HZ and a 7-bit Big5 scheme under one standard. The different schemes, such as HZ-S, Hz-2, etc, don't have much technical significance or make one stands out above the rest. I believe it's time to draw votes on the different schemes. This mailing list is a good place the draw the votes, but I'm not sure how well people from Taiwan is represented here. Most people here are in North America. Steve Simpson listed all the schemes. A detailed description can start the vote drawing. I would propose the ~[ variation should be considered as a different scheme. Yuan From simpson@math.psu.edu Tue May 3 16:50:45 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA02864; Tue, 3 May 94 16:50:45 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA23468; Tue, 3 May 1994 17:47:52 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 17:47:51 -0400 Date: Tue, 3 May 1994 17:47:51 -0400 Message-Id: <199405032147.RAA02351@boole.math.psu.edu> To: pei@mordor.com Cc: chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Subject: postings by you and Jeffrey Chen Reply-To: simpson@math.psu.edu Status: O Dear Professor Pei, Thanks for your thoughtful posting a few days ago, about the use of HZ in the chinese.* newsgroup hierarchy. I agree with your view of things, and I appreciate your positive comments about the effort to extend HZ to incorporate some kind of 7-bit Big5. A group of us on soft-authors@ifcss.org are working on this and I hope we will reach consensus soon. There was a somewhat ill-tempered posting today by Jeffrey Chen in alt.chinese.computing. I feel that, although Jeffrey Chen has some valid points, he is hostile because he is misapprehending what you and I and Ya-Gui and company are trying to accomplish. I think we should try to convert Jeffrey Chen from a foe to an ally. He thinks we are against Big5, and we need to show him otherwise. I'm planning to send him mail privately. If this is successful, we might even try to get him involved in the 7-bit Big5 effort. Do you think this is a good strategy? If you like, I could show you a draft of my remarks before I send them to Jeffrey Chen. I'm copying this note to the chinese.* volunteer committee and the soft-authors group. Sincerely, -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Tue May 3 17:44:27 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA05065; Tue, 3 May 94 17:44:27 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA23837; Tue, 3 May 1994 18:42:28 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 18:42:26 -0400 Date: Tue, 3 May 1994 18:42:26 -0400 Message-Id: <199405032242.SAA02423@boole.math.psu.edu> To: John Delacour Cc: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Subject: the HZ+ effort In-Reply-To: <199405032128.RAA23308@leibniz.math.psu.edu> References: <199405032128.RAA23308@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O To: John Delacour Cc: soft-authors@ifcss.org John, thanks very much for your CCNET-L postings in support of the effort to extend HZ to incorporate some kind of 7-bit Big5. I don't think the effort should be identified with me. If a consensus on HZ+ is reached, it will be the fruit of efforts by a number of people, especially Ya-Gui and Fung, going back as much as a year. Discussion began at least that long ago among some soft-authors folks. I am only trying to act as a cheerleader or sparkplug, to get this engine rolling. > I am not saying that Steve's HZ+ is the ne plus ultra of electronic > communication in Chinese, but ... Brother, you can say that again! But it is a simple, practical solution to an immediate problem. And, it is compatible enough with other things so that, when something better comes along, it will be easy to switch. > I would suggest for the moment that Steve Simpson elaborate his code > a little to deal with long lines or whatever and then... What do you have in mind? Fung's HZ --> GB program already had an option for dealing with long lines, and I carried that over to my HZ+S --> GB and HZ+S --> Big5 programs. In any case, long lines are an implementation issue. The HZ+ standard will consist not of software but of a coding system specification. My programs are only a way of testing the concept. > I don't read things very carefully or slowly, but I guess that talk of > CNS and ISO 2022 is not really relevant at the moment - I'm not saying > that in the long term they are irrelevant but for the moment we want > a little step of progress and HZ+ gives us that, and it's a step that > we can TAKE instead of just dream of. I agree. Soft-authors, what do you say? Shall we do it?!? -- Steve From simpson@math.psu.edu Tue May 3 19:39:46 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08835; Tue, 3 May 94 19:39:46 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id UAA26211; Tue, 3 May 1994 20:35:57 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 20:35:55 -0400 Date: Tue, 3 May 1994 20:35:55 -0400 Message-Id: <199405040035.UAA02620@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Incorporating 7-bit Big5 Cc: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang), shann@wangwei.math.ncu.edu.tw Reply-To: soft-authors@ifcss.org Status: O Yuan Jiang writes: > I believe the major gpoints are already expressed on this > matter. Our objective is clear -- putting HZ and a 7-bit > Big5 scheme under one standard. The different schemes, > such as HZ-S, Hz-2, etc, don't have much technical significance > or make one stands out above the rest. I believe it's > time to draw votes on the different schemes. I support that. There is a need for some kind of 7-bit Big5, and I think the technical information is enough to reach a consensus. > This mailing list is a good place the draw the votes, but I'm not > sure how well people from Taiwan is represented here. Most people > here are in North America. Right now soft-authors has only two people with a .tw address, and , and these two people have been silent. I keep trying to copy everything to shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) but I sometimes forget. Wei-Chang Shann is another Taiwanese whom I would like to involve. Are there any overseas Taiwanese here? Since Big5 is so very popular in Taiwan, it makes sense for us to get Taiwanese input. On the other hand, a lot of us in North America have experience with Big5 too. In addition, let's keep in mind that the standard we are trying to create is international, not Taiwanese. Our goal is to accommodate Big5 within a framework for international 7-bit Chinese news and e-mail. This is a step toward creating an Internet environment where all Chinese speakers everywhere can communicate and feel comfortable. In summary, I would like to have more Taiwanese input, but let's not forget our international perspective. -- Steve From mcpong@cs.ust.hk Tue May 3 20:05:13 1994 Return-Path: Received: from cssu2.cs.ust.hk by ifcss.org (4.1/IFCSS-Mailer) id AA09521; Tue, 3 May 94 20:05:13 CDT Received: by cssu2.cs.ust.hk (4.1/SMI-4.1) id AA04343; Wed, 4 May 94 09:03:09 HKT Date: Wed, 4 May 94 09:03:09 HKT From: mcpong@cs.ust.hk (Dr. Man-Chi PONG) Message-Id: <9405040103.AA04343@cs.ust.hk> X-Mailer: Mail User's Shell (7.2.5 10/14/92) To: soft-authors@ifcss.org Subject: Re: the HZ+ effort Status: O -- "Stephen G Simpson " wrote on 1994 May 4 (Wed): > > To: John Delacour > Cc: soft-authors@ifcss.org > > > I don't read things very carefully or slowly, but I guess that talk of > > CNS and ISO 2022 is not really relevant at the moment - I'm not saying > > that in the long term they are irrelevant but for the moment we want > > a little step of progress and HZ+ gives us that, and it's a step that > > we can TAKE instead of just dream of. > > I agree. Soft-authors, what do you say? Shall we do it?!? Based on what I read, it seems the discussion is more or less converged to the point of first supporting Big5 in HZ+. (CNS can be supported later, probably using escape ~G and ~H.) A sample implementation will help to spread the HZ+ "standard" (or scheme). I prefer the escape sequences ~< and ~> to ~1 and ~2, but it's a personal taste. Maybe it's time to make a vote on the various proposal. Would Steve post the vote? (to soft-authors?, CCNET-L?, alt.chinese.computing?) (It could be a draft vote, depending on whether there is overwhelming support for a particular pair of escape sequences.) -- mcpong -- in GB2312-80:~{ÅÓÃñÖÎ~}, in Big5: Ãe¥Áªv From simpson@math.psu.edu Tue May 3 20:47:21 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA11060; Tue, 3 May 94 20:47:21 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA18636; Tue, 3 May 94 18:45:57 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA26455; Tue, 3 May 1994 21:27:36 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 21:27:35 -0400 Date: Tue, 3 May 1994 21:27:35 -0400 Message-Id: <199405040127.VAA02694@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang), shann@wangwei.math.ncu.edu.tw Subject: information for a vote on 7-bit Big5 Reply-To: soft-authors@ifcss.org Status: O In-Reply-To: <9405032140.AA02406@ifcss.org> References: <9405032140.AA02406@ifcss.org> Reply-To: simpson@math.psu.edu Bcc: simpson --text follows this line-- Yuan Jiang writes: > Steve Simpson listed all the schemes. A detailed description > can start the vote drawing. The detailed description of HZ+S is in math.psu.edu, /pub/simpson/chinese/hzps/hzps.txt. Some sample HZ+S conversion filters are in the same directory. I have now put Ya-Gui's HZ-2 and Jing-Shin's B5encode proposals in the same directory, as hz-2.txt and b5encode.txt. However, I believe Ya-Wei mentioned that HZ-2 has seen a recent change involving two rows. So maybe we'd better wait to hear from Ya-Gui directly. Yuan Jiang writes: > I would propose the ~[ variation should be considered as a > different scheme. I would prefer not to split the votes for what is essentially the same proposal. I feel that, for instance, HZ+S with ~[ and ~] is not sufficiently different from the same proposal with ~1 and ~2 or with ~> and ~< or whatever. Could we instead do the following? Let's vote on each of HZ+S, HZ-2, and B5encode as one proposal. After that, let's have another vote on which pair of ~ escapes to use. There is a valid reason for not using ~1 and ~2, because ~ is an operator in C, but let's decide that later. The more significant differences between the three proposals involve how to divide Big5 into 2 blocks, and how to map those blocks into the graphical ASCII code range 0x2121 to 0x7D7E. HZ+S says: block 1 is the range 0xA140 to 0xC8FE (including the 5401 more frequently used characters, in order), block 2 is the range 0xC940 to 0xFEFE (starting with the 7652 less frequently used characters, in order), and each block is packed in order onto an initial segment of the ASCII code range, with no gaps. The advantages of this method are explained in my HZ+S spec. Another advantage that I forgot to mention there is that Big5 fonts can be used essentially as is. HZ-2 defines blocks 1 and 2 somewhat differently, so that block 1 contains not only the 5401 more frequently used characters but also an initial segment of the 7652 less frequently used characters. The mappings are given by a1 = ((b1 - 0xA1) << 1) + (b2 >> 7) + 0x21; a2 = b2 & 0x7F; for block 1 and a1 = ((b1 - 0xD0) << 1) + (b2 >> 7) + 0x21; a2 = b2 & 0x7F; for block 2. Thus the ranges of both mappings contain a lot of gaps. B5encode defines blocks 1 and 2 according to whether the high bit in the second byte of the Big5 character is 1 or 0. Thus blocks 1 and 2 both cut across the more and less frequently used characters. The mappings are given by setting the high bits of both bytes to 0. The range of the mapping for block 2 contains a lot of gaps. It has been said that the HZ-2 and B5encode mappings use fewer CPU cycles. However, I believe that nearly the same efficiency could be achieved in the HZ+S mappings, by using shifts and logical operations to implement multiplication and division by 94 and 157. This is a programming or implementation issue. Is the above information a sufficient basis for a vote? If we go ahead with a vote, I urge everyone to read the three specs and vote thoughtfully. -- Steve From simpson@math.psu.edu Tue May 3 20:58:59 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11434; Tue, 3 May 94 20:58:59 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA26567; Tue, 3 May 1994 21:57:00 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 3 May 1994 21:56:59 -0400 Date: Tue, 3 May 1994 21:56:59 -0400 Message-Id: <199405040156.VAA02729@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: the HZ+ effort In-Reply-To: <9405040103.AA04343@cs.ust.hk> References: <9405040103.AA04343@cs.ust.hk> Reply-To: simpson@math.psu.edu Status: O Dr. Man-Chi PONG writes: > A sample implementation will help to spread the HZ+ "standard" Do the HZ+S conversion filters in math.psu.edu /pub/simpson/chinese/hzps qualify as a sample implementation? What we'd really like is for Ya-Gui to write HZ+ support into ZWDOS, but I don't know whether he has time. :-) > I prefer the escape sequences ~< and ~> to ~1 and ~2, but > it's a personal taste. I'm leaning toward ~> and ~<, designating the "more" and "less" frequently used character blocks. Let's vote on that as a separate issue, after we have decided which 7-bit packing scheme to use. The packing scheme is the more serious issue. > Maybe it's time to make a vote on the various proposal. Would > Steve post the vote? (to soft-authors?, CCNET-L?, > alt.chinese.computing?) Well, I'm open to suggestions, but my first thought was to confine the initial vote to soft-authors. If we call for a vote now in ccnet-l or a.c.c, we will need to rehash a lot of the same technical issues that have already been resolved in soft-authors. Let's get a consensus in soft-authors, then present it to the wider public. That process worked pretty well in the case of HBF, didn't it? For those who don't know, Man-Chi Pong was the author of the original cxterm, and he was also a leader of the effort last year which produced the HBF file format. -- Steve From A7621GAC@AWIUNI11.EDVZ.UNIVIE.AC.AT Wed May 4 01:01:39 1994 Return-Path: Received: from AWIUNI11.EDVZ.UniVie.AC.AT (helios.edvz.univie.ac.at) by ifcss.org (4.1/IFCSS-Mailer) id AA18854; Wed, 4 May 94 01:01:39 CDT Message-Id: <9405040601.AA18854@ifcss.org> Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT by AWIUNI11.EDVZ.UniVie.AC.AT (IBM VM SMTP V2R2) with BSMTP id 4223; Wed, 04 May 94 07:59:17 MEZ Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT (NJE origin A7621GAC@AWIUNI11) by AWIUNI11.EDVZ.UNIVIE.AC.AT (LMail V1.2a/1.8a) with BSMTP id 2127; Wed, 4 May 1994 07:59:16 +0100 Date: Wed, 04 May 94 07:48:06 MEZ From: Werner Lemberg Subject: How to use Unicode in C soure code To: unicode@org, soft-authors@ifcss.org Status: O How can I input Unicode characters directly in C code ? ANSI C says that C code consists of a stream of 8bit values ... Inputting e.g. Chinese can be done with multibyte encoding schemes like GB or Big5, but how can I use the wide character feature ? Or is it better using UTF-2 to input Unicode ? How can the compiler know which mbtowc() - encoding he should use for which source code ? The UCS standard of IBM handles all things concerning input and output of Unicode but how to have Unicode in C source files directly (that an editor could read) ... Werner From shann@wangwei.math.ncu.edu.tw Wed May 4 02:19:44 1994 Return-Path: Received: from wangwei.math.ncu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA21068; Wed, 4 May 94 02:19:44 CDT Received: from qingzhao.math.ncu.edu.tw by wangwei.math.ncu.edu.tw (4.1/SMI-4.1) id AA03546; Wed, 4 May 94 15:17:45 CST Date: Wed, 4 May 94 15:17:45 CST From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Message-Id: <9405040717.AA03546@wangwei.math.ncu.edu.tw> To: soft-authors@ifcss.org Subject: consensus Cc: shin@hermes.ee.nthu.edu.tw, skhuang@csie.nctu.edu.tw, wchwang@csie.nctu.edu.tw Status: O > Right now soft-authors has only two people with a .tw address, > and , and these > two people have been silent. I keep trying to copy everything to > shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) but I sometimes forget. > Wei-Chang Shann is another Taiwanese whom I would like to involve. I don't know why Mr Wang and Huang are silent, maybe they are busy for their final exams or defenses. They may tell us later. I don't think I can make a fair vote now. I had some experiences with HZ-S and Prof. Simpson's conversion program. I like it and it passed the tests that I thought of. But I didn't compare it with HZ-2. > In summary, I would like to have more Taiwanese input, but let's not > forget our international perspective. I think this event is mostly an international business. Since within Taiwan we don't have much problem with 8-bit stuff. The major concern is the international communications. -Shann From simpson@math.psu.edu Wed May 4 07:42:01 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA28298; Wed, 4 May 94 07:42:01 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA28726; Wed, 4 May 1994 08:38:23 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 08:38:22 -0400 Date: Wed, 4 May 1994 08:38:22 -0400 Message-Id: <199405041238.IAA03567@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Cc: handa@etlken.etl.go.jp (Kenichi Handa), ccnet-l@uga.cc.uga.edu Subject: 7-bit Big5 proposals, ISO-2022 compatibility Reply-To: simpson@math.psu.edu Status: O We here on soft-authors have been looking at three proposals for adding 7-bit Big5 to the HZ standard. None of the three is truly ISO-2022 compatible, and we have pretty much agreed that ISO-2022 compatibility is less important than compatibility with the very popular Big5 coding scheme. Nevertheless, I would like to point out that we now have a good opportunity to open the way for POSSIBLE FUTURE compatibility with ISO-2022. I think we ought to take advantage of this opportunity. On the one hand, our 7-bit Big5 cannot be truly ISO-2022 compatible now, because Big5 is not now a registered ISO-2022 codeset. On the other hand, the closely related CNS-11643 is indeed a registered ISO-2022 codeset. Under ISO-2022, the symbols G and H designate CNS blocks or levels containing the more and less frequently used Chinese characters, respectively. CNS-11643 is not the same thing as Big5, but it is based on Big5 and contains almost exactly the same characters as Big5. The intention of CNS-11643 was apparently to standardize Big5 and correct some errors. Unfortunately, the error corrections made CNS-11643 sufficiently different from Big5 so that nobody likes to use it. That is why we are basing our work on Big5 rather than CNS. Big5 is popular while CNS is not. If in the future Big5 gets registered with ISO-2022, then I think it is very likely that Big5 will be handled similarly to the way CNS-11643 is now handled. This means that ISO-2022 would assign two symbols to designate the Big5 blocks of more frequently used characters (in the range 0xA140 to 0xC8FE) and less frequently used characters (in the range 0xC940 to 0xFEFE). I can't predict what these symbols will be, but that doesn't matter. The important point is that ISO-2022 compatibility will require partitioning Big5 into two blocks as just mentioned. [ The above is similar to what is already done in Mule, the multi-lingual extension of Emacs. Mule uses an unofficial extension of ISO-2022 in which Big5 is partitioned almost exactly as above. The only difference is that in Mule block 2 starts at 0xC6A1 rather than 0xC940, but I believe this is due to confusion over the spurious "HKU-Big5 standard". Clearly the structure of the Big5 character set (as implemented for instance in ETen, the most popular Big5 software) calls for 0xC940 rather than 0xC6A1. I am copying this note to Kenichi Handa, author of Mule and chairman of one of the ISO-2022 committees. Perhaps Kenichi can enlighten us. I hope that Mule and HZ+ can come to an agreement. ] In light of the above, let's now consider our three 7-bit Big5 proposals: HZ+S, HZ-2, and B5encode. The three proposals are all quite similar to each other and all accomplish the goal of 7-bit Big5. However, only one of them partitions Big5 into two blocks in the specific manner described above. To quote from my earlier posting: HZ+S says: block 1 is the range 0xA140 to 0xC8FE (including the 5401 more frequently used characters, in order), block 2 is the range 0xC940 to 0xFEFE (starting with the 7652 less frequently used characters, in order), and each block is packed in order onto an initial segment of the ASCII code range, with no gaps. HZ-2 defines blocks 1 and 2 somewhat differently, so that block 1 contains not only the 5401 more frequently used characters but also an initial segment of the 7652 less frequently used characters. B5encode defines blocks 1 and 2 according to whether the high bit in the second byte of the Big5 character is 1 or 0. Thus blocks 1 and 2 both cut across the more and less frequently used characters. >From this it is clear that HZ+S holds out much more hope of future ISO-2022 compatibility than the other two proposals. I view this as a significant advantage of HZ+S over HZ-2 and B5encode. Yes, it is true that our first goal is to incorporate 7-bit Big5 into HZ. But at the same time, we also have an opportunity to incorporate possible future ISO-2022 compatibility, with no extra work or sacrifice. Other things being equal, I think we should take advantage of this opportunity. Note that the existing HZ standard is already ISO-2022 compatible, in the sense that the HZ/GB-2312-1980 character set is already registered and designated by the ISO-2022 symbol A. Thus HZ+S tries to extend this desirable feature of HZ. This is part of the idea that HZ+S seeks to build on the success of HZ. -- Steve PS. Can our Taiwanese friends perhaps explain why Big5 has never been registered with ISO-2022? Perhaps this is a political matter. ISO is an international standards body, and CNS-11643 is an official ROC Taiwan standard. From shin@hermes.ee.nthu.edu.tw Wed May 4 13:04:29 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA08694; Wed, 4 May 94 13:04:29 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182112(1)>; Thu, 5 May 1994 02:07:45 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA11206; Thu, 5 May 94 02:00:56 CST Date: Thu, 5 May 1994 16:00:56 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405041800.AA11206@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: B5encode/B5decode, non-ISO compatible -^^- Cc: soft-authors@ifcss.org Status: O > Reply-To: simpson@math.psu.edu > > Jing-Shin Chang writes: > > For people who do not currently consider ISO-compatibility as an emergency > > task, here comes my non-compatible version of B5encode/Decode. It implement > > the switching of characters to/from ASCII/HZ/B5(7bit) with: > > > > ~{ : to HZ > > ~1 : to Big5 Block1 > > ~2 : to Big5 Block2 > > ~} : to ASCII > > Jing-Shin, is this new proposal intended as a replacement for your > earlier B5encode proposal? If you want your new proposal to be > considered side-byt-side with your earlier proposal, could you please > give it a different name? This is not a new proposal. The sprit of my proposal is still in easy mapping and easy coding for the prospective soft contributors. The way to divide the code space is still the same. This posting is just to show that one can easily change the ESC chars from one to another (I spend only a few hours to change it and test it). If people choose to retain the HZ-style encoding scheme, I am not stronly objective to that. It can easily be implemented. Anyway, many HZ users does not want to make radical changes, so I think their demands should be considered when we try to combine HZ and Big5. In fact, by replacing the two statements to set data bit in the program by your mapping routine, it should be identical to your proposal. > > In addition, for purposes of comparing your new proposal with other > proposals, could you please give us a verbal description of the > structure of blocks 1 and 2 and how you map them to the graphical > ASCII range? > > What do you consider to be the advantages of your new proposal? > In addition to non-ISO-compatibility, I mean. :-) > > -- Steve > From simpson@math.psu.edu Wed May 4 14:51:55 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA12110; Wed, 4 May 94 14:51:55 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA06899; Wed, 4 May 94 11:18:23 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA01540; Wed, 4 May 1994 14:15:15 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 14:15:14 -0400 Date: Wed, 4 May 1994 14:15:14 -0400 Message-Id: <199405041815.OAA03981@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Are we ready for a vote on HZ+ ? Reply-To: simpson@math.psu.edu Status: O I have slightly enlarged my verbal description of HZ+S in math.psu.edu, /pub/simpson/chinese/hzps/hzps.txt. I now list six advantages of HZ+S, instead of only four. :-) To summarize the situation, there are now three HZ+ proposals on the table: HZ+S, HZ-2, and B5encode. Are we ready to vote? Before I call for a formal vote, are they any objections? -- Steve From rap@doc.ic.ac.uk Wed May 4 14:52:22 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA12184; Wed, 4 May 94 14:52:22 CDT Received: from santos.doc.ic.ac.uk by cnd.org (4.1/4.7) id AA07709; Wed, 4 May 94 11:48:14 PDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0pylto-00070eC; Wed, 4 May 94 19:42 BST Message-Id: Date: Wed, 4 May 94 19:42 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Re: B5encode/B5decode, non-ISO compatible -~~- Cc: shin@HERMES.EE.NTHU.EDU.TW Status: O Jing-Shin Chang writes: > For people who do not currently consider ISO-compatibility as an emergency > task, here comes my non-compatible version of B5encode/Decode. It implement > the switching of characters to/from ASCII/HZ/B5(7bit) with: > > %{ : to HZ > %1 : to Big5 Block1 > %2 : to Big5 Block2 > %} : to ASCII This scheme of separating the blocks based on the high bit of the second byte saves a little processing, but will lengthen the text considerably. The probability of needing a switch between any two Big5 characters is 2*(94/157)*(1 - 94/157) or roughly 48%. If (instead of switching) you use a prefix before each character from the smaller block, this reduces to 63/157 or about 40%. On the other hand, fewer that 1 in a 1000 characters seem to come from the second hanzi level, so dividing the characters that way will not increase the size much. But if set switching is used, most level 2 characters (rare as they are) will require 2 of them. The cheapest of all would be to use a single character prefix for each level 2 character. -- Ross Paterson Department of Computing, Imperial College, London SW7 From simpson@math.psu.edu Wed May 4 14:52:31 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA12209; Wed, 4 May 94 14:52:31 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA04240; Wed, 4 May 94 09:55:16 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA00833; Wed, 4 May 1994 12:53:14 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 12:53:12 -0400 Date: Wed, 4 May 1994 12:53:12 -0400 Message-Id: <199405041653.MAA03873@boole.math.psu.edu> To: Jing-Shin Chang Cc: soft-authors@ifcss.org Subject: B5encode/B5decode, non-ISO compatible -^^- In-Reply-To: <199405041629.MAA00614@leibniz.math.psu.edu> References: <199405041629.MAA00614@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > For people who do not currently consider ISO-compatibility as an emergency > task, here comes my non-compatible version of B5encode/Decode. It implement > the switching of characters to/from ASCII/HZ/B5(7bit) with: > > ~{ : to HZ > ~1 : to Big5 Block1 > ~2 : to Big5 Block2 > ~} : to ASCII Jing-Shin, is this new proposal intended as a replacement for your earlier B5encode proposal? If you want your new proposal to be considered side-byt-side with your earlier proposal, could you please give it a different name? In addition, for purposes of comparing your new proposal with other proposals, could you please give us a verbal description of the structure of blocks 1 and 2 and how you map them to the graphical ASCII range? What do you consider to be the advantages of your new proposal? In addition to non-ISO-compatibility, I mean. :-) -- Steve From simpson@math.psu.edu Wed May 4 14:54:18 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA12503; Wed, 4 May 94 14:54:18 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA05643; Wed, 4 May 94 10:32:14 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA01074; Wed, 4 May 1994 13:27:00 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 13:26:58 -0400 Date: Wed, 4 May 1994 13:26:58 -0400 Message-Id: <199405041726.NAA03914@boole.math.psu.edu> To: kazax@u.washington.edu Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, chinesenet@uclink.berkeley.edu Reply-To: simpson@math.psu.edu Status: O To: kazax@u.washington.edu (Jeffrey Chen) Dear Jeffrey, I saw your interesting article in alt.chinese.computing. You commented on Ming Pei's earlier article, and you made some valid and often-overlooked points about Big5 users and Big5 software. I participate in (1) the advisory committee for the proposed chinese.* newsgroup hierarchy, and (2) an informal group at soft-authors@ifcss.org which is devising a 7-bit Big5 standard for use in Internet news and e-mail. Ming Pei is also participating in the advisory committee. Frankly, I would like to try to gain your sympathy for what we are trying to do. I feel that you will like us if you get to know us. My main point is that we want very much to accommodate Big5 users in an existing international framework where HZ already plays a significant role. This task is not completely easy. It will require patience and understanding from all parties. Your constructive input is welcome. Let me comment on some specific points in your article. > >(1) There are two (not 3) Chinese language codes: GuoBiao and Big5. > > Other experts will disagree with that. I think what Ming Pei was trying to emphasize here is that GB and Big5 are by far the two most popular Chinese character coding schemes. Some people have the false impression that HZ is another coding scheme. They don't realize that the currently popular HZ is only a 7-bit variant of GB. Professor Pei was trying to correct that possible misunderstanding. > I read that there's only one gateway in the US that will strip the > 8th bit, and that's it. For some time I thought that 8-bit news is unproblematic. However, recent discussions with very knowledgeable programmers have convinced me otherwise. It is not so much a gateway problem as a driver problem. In the US and possibly elsewhere, there are many older Unix machines where the kernel tty driver does not provide for an 8-bit mode. If such a machine gets its news by UUCP, then the 8th bits get stripped, because UUCP uses the tty driver. A solution to this problem is to switch from a UUCP news feed to a NNTP news feed, because NNTP does not run the news through the tty driver. Unfortunately, the NNTP option is not available at all sites. In addition to news, 7-bit e-mail presents even more of a problem. Many sites (including mine) give users no way to mail 8-bit text. There are many well-known workarounds such as uuencode and MIME. This e-mail problem affects news too, because many people use e-mail to send postings to newsgroups. Eventually, the whole world will be 8-bit clean. But this is not now the case, and I don't think it will be the case any time soon. That is why overseas Chinese developed HZ, namely to cope with the 7-bit nature of the international network. > >(4) The sensible thing to do is to extend HZ scheme to include Big5. > > That would leave 7-bit GB (HZ) and 7-bit big5(whatever name). There > still is a need for mirroring. I partially agree with your point here. I am in favor of extending HZ to include 7-bit Big5, but I recognize that this will not solve the GB versus Big5 problem. I posted a long article about this. I'd be glad to show it to you if you are interested. Our problem is, how to set up newsgroups so that both Big5 users and GB users can be reasonably comfortable. Mirroring is one possibility, but it is not the only possibility. Another possibility is for interested users to upgrade their software so that it can handle both GB and Big5. This presents difficulties of course, but in the long run it may be a better solution than mirroring, which involves a very imperfect translation between simplified and traditional characters. However, I certainly agree with you that the chinese.* newsgroup hierarchy needs to find SOME way to accommodate Big5 (be it mirroring, or 7-bit Big5, or whatever). This is the lesson to be learned from alt.chinese.text.big5 and from the HZ mirrors of Big5 tw.* newsgroups. > >(5) Naming the hierarchy as HZ is proper. This should not imply an > > altitude favoring Guobiao. It merely specifies the encoding > > scheme of HZ (or improved model of HZ as Simpson is proposing). > > Why IS it proper? If it's not favortism, then please be CLEAR about the > reason why it must be HZ. It's proper because it "merely specified" HZ? Here I must disagree slightly with Professor Pei. If we expand HZ to include 7-bit Big5, then for the sake of clarity I don't think we should keep the name HZ for the enlarged standard. Using the same name would create too much confusion. I have proposed the name HZ+. I think the name HZ+ makes a lot of sense. A frequently asked question would be, what is the difference between HZ and HZ+? The simple answer would be that HZ+ is a new and improved HZ, expanded to include 7-bit Big5. > > It should be made clear to everyone that a source document written > > in Big5 is not excluded from the proposed hierarchy, as long as > > it is (by whatever means) encoded into HZ before sending. > > What? What's the diffence? "You're not excluded as long as you > change yourself to be one of us" ??? I don't think Ming Pei was trying to suggest that Big5 is to be translated into the existing HZ/GB. When he says "encoded into HZ by whatever means", he is referring to HZ+. The encoding would be a very simple and 100 percent reversible conversion from 8-bit Big5 to 7-bit Big5. Professor Pei, could you please confirm that I have correctly interpreted your intention? > 100% no-loss translation is possible with HZ to Big5. Under this > criterion, the newsgroups should all be based in big5 instead. This is an excellent technical point. However, I don't think that HZ/GB users would like to be forced to translate their postings into Big5, any more than Big5 users would like to be forced to translate their postings into HZ/GB. That's why I think the best solution is a mixed standard which incorporates both GB and Big5. Actually, your 100% in the above should probably be corrected to 99.99 percent. As the linguistic scholar Martin Heijdra pointed out to me, there are one or two cases of n-to-1 mappings in the GB -> Big5 direction. Case: gan (as in dry) and qian (as in Qianlong/heaven) are separated in GB, but not in Big-5. But your point is essentially correct. The great majority of n-to-1 mappings are in the Big5 -> GB direction, as with Mian4 ("face") versus Mian4 as in Mian4 Bao1 ("bread"). > So you are saying that for you HZ users you can just go to your favorite > newsgroups and post, but a big5 or GB user must first read the news, find > a favorite thread, leave it unread, exit from newsreader, launch and > Chinese editor or compatible, writes a reply, use a converter to ... Again, you have a valid point. A 7-bit Big5 standard asks Big5 users to translate their news postings into 7-bit Big5 before posting. This may be somewhat inconvenient at first. However, there is a compensating advantage. The advantage is that the postings will reach more readers, namely those readers whose cannot receive 8-bit news. I don't have any statistics, but I am now convinced that the number of such readers is substantial. > You are assuming that big5 users are similar to HZ users. Big5 > users get most of their software commercially (like TB, ETen, > Han,Cwin31) to use their big5 applications (thousands). There are > big5 shareware like KC, but their numbers are limited. My point > is, big5 users don't depend on shareware for their Chinese > computing needs, especially those users in Taiwan. It's not that > easy to "switch" enviroments because it takes time to get use to > one and it costs money. This is an excellent point about Big5 users! And in other ways too, the Big5 user community is very different from the HZ user community. I have to admit that points such as this are often overlooked in discussions among the international Chinese computing community. All I can say is, I wish Big5 users would participate more in the international setting. Post more to groups like ccnet-l@uga.uga.edu and alt.chinese.computing! If you are a software developer, join soft-authors@ifcss.org! We would welcome your insight. I'm sure you can teach us a lot! For years on CCNET, I have strongly advocated Big5. But I always wished for more support from users of commercial Big5 products. Maybe this will start to happen more, now that Taiwan is participating more in the Internet. > Since most of these people are either 1) Taiwanese or 2) professionals > who uses commerical software all over the world, you are going to exclude > a lot of them from chinese newsgroup participation. > I would like to say that a unified code that everyone agreed on is > critical for true universal access for all Chinese to these newsgroups. > Like I've said above, if all of you avoid the issue and just use HZ as > you please, you ARE going to exclude a large number of people. I agree with you. A HZ/GB-only policy would effectively exclude or at least severely discourage Big5 users. I have tried hard to move the chinese.* newsgroup organizers away from a HZ/GB-only policy. I am convinced that they sincerely welcome Big5 users. I believe that they are amenable to a practical 7-bit mixed GB/Big5 standard. They are also somewhat amenable to cross-morroring, to some degree. They may not fully appreciate all of the linguistic and technical issues, but their hearts are in a good place. Rather than fight with them, let's try to encourage them and help them. I don't think I'm being naive. > IMHO, HZ doesn't even need to exist if the net is totally 8-bit clean. > It's in the process of duing just that has older software and hardware > get replaced. As soon as the net is clean, the need for HZ and "~{"'s > will cease to exist. I think it will take at least 10 to 15 years for the net to become totally 8-bit clean. In the meantime, HZ will become even more widely accepted than it is already. My prognosis is that HZ and similar standards will be with us for a long time. Instead of fighting against this success, let's build on it by expanding HZ to include 7-bit Big5. > I would also like to remind everyone that there those of us on the Net > are NOT really doing some avant-garde work with regards to Chinese > cumputing; there are millions of users today who uses Chinese everyday > with a large number of Chinese appilcations. I grant this point. HZ+ will not be anything esoteric or leading-edge. It is intended as a practical solution to an immediate problem. I am convinced that the world needs a 7-bit standard for international Chinese e-mail and news on the Internet. HZ+ can meet that need in a simple, practical way. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From shin@hermes.ee.nthu.edu.tw Wed May 4 15:08:41 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA13030; Wed, 4 May 94 15:08:41 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182122(1)>; Thu, 5 May 1994 04:10:38 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA11620; Thu, 5 May 94 04:03:43 CST Date: Thu, 5 May 1994 18:03:43 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405042003.AA11620@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: Are we ready for a vote on HZ+ ? Status: O > Reply-To: simpson@math.psu.edu > > I have slightly enlarged my verbal description of HZ+S in > math.psu.edu, /pub/simpson/chinese/hzps/hzps.txt. I now list six > advantages of HZ+S, instead of only four. :-) > > To summarize the situation, there are now three HZ+ proposals on the > table: HZ+S, HZ-2, and B5encode. Are we ready to vote? Before I call > for a formal vote, are they any objections? > There seem to be some questions. First of all, the mixed encoding schemes requires mltilingual readers to cope with reading/posting, and in particular, follow-up. I can't image how a reader follow up one's posting, which is written in another char set. If his/her viewer is only monolingual, strange things could happen. From simpson@math.psu.edu Wed May 4 17:09:01 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA17486; Wed, 4 May 94 17:09:01 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA03652; Wed, 4 May 1994 18:07:02 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 18:07:00 -0400 Date: Wed, 4 May 1994 18:07:00 -0400 Message-Id: <199405042207.SAA04286@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: soft-authors@ifcss.org Subject: Re: Are we ready for a vote on HZ+ ? In-Reply-To: <9405042003.AA11620@hermes.ee.nthu.edu.tw> References: <9405042003.AA11620@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O I wrote: > > To summarize the situation, there are now three HZ+ proposals on the > > table: HZ+S, HZ-2, and B5encode. Are we ready to vote? Before I call > > for a formal vote, are they any objections? Jing-Shin Chang writes: > There seem to be some questions. > > First of all, the mixed encoding schemes requires mltilingual readers > to cope with reading/posting, and in particular, follow-up. I can't > image how a reader follow up one's posting, which is written in > another char set. If his/her viewer is only monolingual, strange things > could happen. It seems to me that your question is about future implementation of HZ+. Right now we are only voting on the HZ+ specification, not on how to implement it in e-mail and news software. Full-fledged implementation must come later. -- Steve From simpson@math.psu.edu Wed May 4 17:55:20 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA19115; Wed, 4 May 94 17:55:20 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA05041; Wed, 4 May 1994 18:53:09 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 18:53:07 -0400 Date: Wed, 4 May 1994 18:53:07 -0400 Message-Id: <199405042253.SAA04363@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: information on 7-bit Big5 is available by anonymous FTP Reply-To: simpson@math.psu.edu Status: O Here is a README file describing some information and software related to a proposed standard for 7-bit Big5. This material is available by anonymous FTP. If you have any questions or comments, please send e-mail to simpson@math.psu.edu or to soft-authors@ifcss.org. -- Steve --------- README for math.psu.edu, /pub/simpson/chinese/hzps This file created 30 April 1994 by Stephen G. Simpson. This file last revised 4 May 1994 by Stephen G. Simpson. Recently there has been a lot of discussion on Chinese computing newsgroups (alt.chinese.computing, ccnet-l@uga.uga.edu, soft-authors@ifcss.org) about possible 7-bit representations of Big5 for use in e-mail, news postings, etc. I proposed a scheme called HZ+S. HZ+S is a 7-bit method of representing text that is a mixture of Big5, GB, and ASCII. HZ+S is a simple extension of the popular 7-bit HZ representation of mixed GB and ASCII. This directory contains documents and experimental software related to HZ+S. I place this material here as a reference for the Chinese computing newsgroup discussions. Contents of this directory: README -- this information file hzps.txt -- a document describing HZ+S hz.txt -- an old document describing HZ etfonts.txt -- an old document describing Big5 gb2hz.c -- a filter to convert GB files to HZ files hz2gb.c -- a filter to convert HZ files to GB files big2hzps.c -- a filter to convert Big5 files to HZ+S files hzps2big.c -- a Big5 filter for HZ+S files hzps2gb.c -- a GB filter for HZ+S files hz-2.txt -- an alternative proposal from Ya-Gui Wei b5encode.txt -- an alternative proposal from Jing-Shin Chang soft-authors -- the soft-authors@ifcss.org mail alias From simpson@math.psu.edu Wed May 4 20:32:34 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA24264; Wed, 4 May 94 20:32:34 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA06324; Wed, 4 May 1994 21:30:29 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 4 May 1994 21:30:27 -0400 Date: Wed, 4 May 1994 21:30:27 -0400 Message-Id: <199405050130.VAA04573@boole.math.psu.edu> To: Edmund Lai Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: 7-bit Big5 proposals, ISO-2022 compatibility Newsgroups: alt.chinese.computing In-Reply-To: <199405050025.TAA14319@cs.utexas.edu> References: <199405050025.TAA14319@cs.utexas.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > Recently I have been very busy and have not been following the > discussion closely. Ed, there has been a *lot* of discussion. If you'd like to join in, welcome! I'd be glad to help you catch up. How many megabytes do you want? :-) If you'd just like a snapshot of the current state of the discussion, use anonymous FTP to look at math.psu.edu, /pub/simpson/chinese/hzps. For those who don't know, Ed Lai was one of creators of the HZ standard that is now used in alt.chinese.text. > Let me ask a stupid question that may already been answered before. > How difficult is it to convert between Big5 and CNS? I don't consider this a stupid question, especially since I myself was confused about it for a long time. :-) The answer is that people in Taiwan have written programs to convert from Big5 to CNS and back, but these programs are far from one-liners. The relationship between Big5 and CNS is not nearly so simple as the relationship between GB and HZ (or HGB and GB in the terminology that you prefer). Historically, the development of CNS was quite different from the development of GB. What happened was that Big5 started out as a code set for a commercial integrated software package. In time Big5 became extremely popular, so a government standards committee set out to standardize it. Unfortunately, the committee wasn't able to resist the temptation to correct errors. They rearranged some symbols and alphabets, deleted two Chinese characters as duplicates, and relocated about 14 other Chinese characters, presumably on the basis of corrected stroke counts. This scrambles lots of codes. (I'm plagiarizing from Ross Paterson here.) The result is that CNS is sufficiently different from Big5 so that nobody likes to use it. Big5 remains extremely popular, while the government standard CNS is pretty much ignored. > In its spirit CNS is much closer to pure GB I can see why you wish that CNS were analogous to GB. Life would be so much simpler! Unfortunately, such is not the case. > so it make sense for any extension of HZ to use CNS That would make sense if CNS were closely related to some popular standard. Unfortunately, it is not. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From yawei Wed May 4 22:14:53 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA27655; Wed, 4 May 94 22:14:53 CDT Date: Wed, 4 May 94 22:14:53 CDT From: yawei (Ya-Gui Wei) Message-Id: <9405050314.AA27655@ifcss.org> To: lai@APPLE.COM, simpson@math.psu.edu Subject: Re: 7-bit Big5 proposals, ISO-2022 compatibility Cc: soft-authors, yawei Status: O Steve and others, I am currently working on the next version of ZWDOS (which probably isn't due until a few months from now.) Because of the uncertainty on Big5 encoding at this point, and because I don't want to change the program every time some new encoding comes up, I decided to use the following simple script to implement all future encoding methods. You are welcome to comment on it. One thing I am not sure is whether a CNS<->Big5 script can be constructed from this language: perhaps with a very long script? :-) -- Description of Script language used in code conversion: Script is stack-based language (with Postscript-like syntax), with the following operators: Input and output: input is 16-bit number as the only element on the stack (the high byte represents the first byte, and low byte representing the second byte of a double byte character). Output is 16-bit number left on top of stack. Numbers may be supplied as operands, and will be pushed onto the top of the stack as soon as they are encountered. Numbers must be specified in their decimal format, and acceptable numbers are ranged from 0 to 32767. Other 16-bit signed or unsigned numbers may be generated by combination of operators and numbers. All intermediate results are 16-bit values, and may be treated as signed or unsigned. An operator always appear after its operands. When an operator is encoutered, it usually pops its required operands off the stack, completes its operations, and push the result(s) back on the stack. Wherever "first", "second", "third" operands are used in the following text, we refer to the order they were pushed onto the stack, which is opposite from the order they were popped. The following operators uses the top-most two elements from the stack as operands. (Note that the / operator generates two results.) + - addition. - - subtract the second operand from the first operand. (Example: "2,1-" results in "1".) * - multiplication. / - division. the first operand is the dividend, the second is the divisor. After the division, the quotient is first pushed onto the stack, followed by the remainder. (The remainder is on top of the stack.) (Example: "5,2/" results in "2,1".) & - bitwise AND. | - bitwise OR. ^ - bitwise XOR. > - greater than test. if first operand is greater, result is 1. else 0. Operands are treated as signed numbers. < - less than test. = - equal test. [ - shift left. second operand is the shift count. ] - shift right. { - rotate left. } - rotate right. m - merge. The low byte is first operand is taken as the high byte of the result, and the low byte of the second operand is taken as the low byte of the result. This is identical in effect to "255&w255&8[|". (Example: "1,2m" results in 258.) Other operators: w - switch the position of the top two elements on the stack. Generates no output. l - split. opposite of "m". (Similar to "256/", but takes less CPU time.)) p - the top element of the stack is popped (discarded). d - the top element of the stack is duplicated. ~ - bitwise NOT. Takes one operand. ? - this is a three-operand operator. if the first operand is non-zero, the second operand is the result, otherwise the third operand (top-most) is the result. , - comma may be used to seperate operands whenever ambiquities may occur. e.g. "127,127m". "" - and the entire string may be enclosed in quotes. Examples: 1. Convert GB to HZ: "32639&" 2. Convert HZ to GB: "32639~|" 3. Convert GB or HZ to dictionary offset: "32639&l33-w33-94*+" 4. Convert dictionary offset to HZ: "94/33+w33+wm" 5. Convert Big5 to HZ-2 (Block 1): "dlw161-wm7]33+w127&m" 6. Convert HZ-2 (Block 1) to Big5: "d33-1]161+wlw33-1&7[|m" From simpson@math.psu.edu Wed May 4 23:55:13 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA00933; Wed, 4 May 94 23:55:13 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id AAA07109; Thu, 5 May 1994 00:53:14 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 00:53:12 -0400 Date: Thu, 5 May 1994 00:53:12 -0400 Message-Id: <199405050453.AAA04833@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Ya-Gui's script language for ZWDOS In-Reply-To: <9405050314.AA27655@ifcss.org> References: <9405050314.AA27655@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes: > I am currently working on the next version of ZWDOS (which > probably isn't due until a few months from now.) Because > of the uncertainty on Big5 encoding at this point, and because > I don't want to change the program every time some new encoding > comes up, I decided to use the following simple script to > implement all future encoding methods. You are welcome to > comment on it. Interesting. Most impressive. You are a real programmer. I guess this means we can assume that the next version of ZWDOS will include 7-bit Big5 support. Fantastic! -- Steve From xiaokun@stats.gla.ac.uk Thu May 5 03:40:56 1994 Return-Path: Received: from hillhead.cent.gla.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA07010; Thu, 5 May 94 03:40:56 CDT Received: from bb-gateway.stats.gla.ac.uk by jess.glasgow.ac.uk with SMTP (PP) id <20074-0@jess.glasgow.ac.uk>; Thu, 5 May 1994 09:38:43 +0100 From: Xiaokun Zhu Date: Thu, 5 May 94 09:42:07 BST Message-Id: <18756.9405050842@jupiter.stats> To: yawei@ifcss.org Subject: Re: 7-bit Big5 proposals, ISO-2022 compatibility Cc: soft-authors@ifcss.org Status: O yawei@ifcss.org wrote: > Numbers may be supplied as operands, and will be pushed ^^^^^^^ > onto the top of the stack as soon as they are encountered. > Numbers must be specified in their decimal format, ^^^^^^^ > and acceptable numbers are ranged from 0 to 32767. Other ^^^^^^^ > 16-bit signed or unsigned numbers may be generated by > combination of operators and numbers. ^^^^^^^ There is confusion between numbers and operand. ^^^^^^^ Can you define "number" as CONSTANT ? > One thing I am not sure is whether a CNS<->Big5 script can > be constructed from this language: perhaps with a very long > script? :-) Your script don't support LookupTable operation directly. "?" operater can simulate it but I think the script'll be very long! ---xiaokun From rap@doc.ic.ac.uk Thu May 5 05:33:47 1994 Return-Path: Received: from santos.doc.ic.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA09853; Thu, 5 May 94 05:33:47 CDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0pz0i3-00070fC; Thu, 5 May 94 11:31 BST Message-Id: Date: Thu, 5 May 94 11:31 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Re: 7-bit Big5 proposals, ISO-2022 compatibility Cc: lai@APPLE.COM Status: O Edmund Lai writes: > Let me ask a stupid question that may already been answered before. > How difficult is it to convert between Big5 and CNS? Here is what I think the mapping is, mostly generated from the Unicode mapping tables: # Big5 non-hanzi and hanzi level 1 <-> CNS level 1 # A140-A154 <-> 212121-212135 A155 <-> 21213A # a guess A156 <-> 212139 A157-A159 <-> 212136-212138 A15A-A1AA <-> 21213B-212169 # unsure about first 3 A1AB <-> 21216B A1AC <-> 21216A A1AD-A1E0 <-> 21216C-212241 A1E1 <-> 212243 A1E2 <-> 212242 A1E3-A1F5 <-> 212244-212256 A1F6 <-> 212258 A1F7 <-> 212257 A1F8-A2AE <-> 212259-21234E A2AF-A3BF <-> 212421-212570 # digits, Latin, Greek, bopomofo **** <- 212721-212939 # Kangxi radicals **** <- 214221-214241 # symbols for control characters A440-ACFD <-> 214421-215322 # start of hanzi level 1 ACFE <-> 215753 AD40-AFCF <-> 215323-215752 AFD0-BBC7 <-> 215754-216B4F BBC8-BE51 <-> 216B51-216F5B BE52 <-> 216B50 BE53-C1AA <-> 216F5C-217534 C1AB-C2CA <-> 217536-217736 C2CB <-> 217535 C2CC-C360 <-> 217737-21782C C361-C3B8 <-> 21782E-217863 C3B9 <-> 217865 C3BA <-> 217864 C3BB-C455 <-> 217866-217961 C456 <-> 21782D C457-C67E <-> 217962-217D4B # end of hanzi level 1 C6A1-C7E8 -> ****** # kana, Cyrillic (different in ETEN) C7E9-C7FC <-> 212621-212634 # enclosed numerals (different in ETEN) # # Big5 hanzi level 2 <-> CNS level 2 # C940-C949 <-> 222121-22212A C94A -> 214442 # duplicate of A461 C94B-C96B <-> 22212B-22214B C96C-C9BD <-> 22214D-22217C C9BE <-> 22214C C9BF-C9EC <-> 22217D-22224C C9ED-CAF6 <-> 22224E-222438 CAF7 <-> 22224D CAF8-D6CB <-> 222439-22376E D6CC <-> 223E63 D6CD-D779 <-> 223770-22387D D77A <-> 223F6A D77B-DADE <-> 22387E-223E62 DADF <-> 22376F DAE0-DBA6 <-> 223E64-223F69 DBA7-DDFB <-> 223F6B-224423 DDFC -> 224176 # duplicate of DCD1 DDFD-E8A2 <-> 224424-22554A E8A3-E975 <-> 22554C-225721 E976-EB5A <-> 225723-225A27 EB5B-EBF0 <-> 225A29-225B3E EBF1 <-> 22554B EBF2-ECDD <-> 225B3F-225C69 ECDE <-> 225722 ECDF-EDA9 <-> 225C6A-225D73 EDAA-EEEA <-> 225D75-226038 EEEB <-> 22642F EEEC-F055 <-> 226039-226242 F056 <-> 225D74 F057-F0CA <-> 226243-226336 F0CB <-> 225A28 F0CC-F162 <-> 226337-22642E F163-F16A <-> 226430-226437 F16B <-> 226761 F16C-F267 <-> 226438-226572 F268 <-> 226934 F269-F2C2 <-> 226573-22664C F2C3-F374 <-> 22664E-226760 F375-F465 <-> 226762-226933 F466-F4B4 <-> 226935-226961 F4B5 <-> 22664D F4B6-F4FC <-> 226962-226A4A F4FD-F662 <-> 226A4C-226C51 F663 <-> 226A4B F664-F976 <-> 226C52-227165 F977-F9C3 <-> 227167-227233 F9C4 <-> 227166 F9C5 <-> 227234 F9C6 <-> 227240 F9C7-F9D1 <-> 227235-22723F F9D2-F9D5 <-> 227241-227244 From simpson@math.psu.edu Thu May 5 08:18:23 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA13693; Thu, 5 May 94 08:18:23 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id JAA08749; Thu, 5 May 1994 09:16:06 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 09:16:05 -0400 Date: Thu, 5 May 1994 09:16:05 -0400 Message-Id: <199405051316.JAA09130@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: shin@hermes.ee.nthu.edu.tw Subject: 7-bit Big5, call for expression of viewpoints In-Reply-To: <9405050314.AA27655@ifcss.org> References: <9405050314.AA27655@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes: > I am currently working on the next version of ZWDOS (which > probably isn't due until a few months from now.) Because > of the uncertainty on Big5 encoding at this point, and because Your script language is flexible enough to handle any of the 7-bit Big5 schemes that are under consideration: HZ+S, HZ-2, B5encode, maybe even something based on CNS. However, other software developers may not want to implement so much flexibility. It seems better to agree now on one 7-bit Big5 standard, so that software developers can go ahead with confidence and fine-tune their algorithms. I have been trying to get a consensus here on soft-authors. There seems to be a fair amount of indifference. Most people have not said anything. Maybe they are in favor of 7-bit Big5 but don't care which scheme is used. Or, maybe they don't care at all about 7-bit Big5. Then there is the relatively small group that has commented on one aspect or another. Let's say Ya-Gui, Fung, Jianqing Hu, MC Pong, Ross, Carlos, Yuan Jiang, Nelson, Xiaokun Zhu, Jing-Shin Chang, Yidao, and me. These people have commented but mostly have not expressed a preference for one proposal over another. I have given my reasons for thinking HZ+S is best (see math.psu.edu, /pub/simpson/chinese/hzps/hzps.txt), but any of the proposals is acceptable to me, so long as we reach a consensus on a 7-bit Big5 extension of HZ. CALL FOR EXPRESSION OF VIEWPOINTS Could I ask everybody here on soft-authors to please send mail to soft-authors indicating your stance or viewpoint? Indicate your degree of interest and concern. You can say that you don't care at all about 7-bit Big5. You can say that you want a 7-bit Big5 standard but don't care to choose between the various proposals. You can indicate which of the proposals you would find acceptable. If you have a strong preference for one proposal or another, indicate that. If you have strong ideas about CNS, mention them. If possible, indicate reasons for your viewpoint. This is not a formal vote, but I am hoping a consensus will emerge, or at least we will see how things stand. -- Steve From simpson@math.psu.edu Thu May 5 10:21:23 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA17489; Thu, 5 May 94 10:21:23 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA09613; Thu, 5 May 1994 11:19:23 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 11:19:22 -0400 Date: Thu, 5 May 1994 11:19:22 -0400 Message-Id: <199405051519.LAA09291@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: advantages of HZ+S Reply-To: simpson@math.psu.edu Status: O I have again reworked my HZ+S proposal. Only the wording of the proposal has changed, not the specification itself. Here is the current version. Related material is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. Your comments are welcome. Note that the escape sequences for Big5 blocks 1 and 2 are specified as ~1 and ~2, but in accordance with recent discussions this could change to ~[ and ~] or ~> and ~<. Apart from this minor detail, the proposal is completely stable. -- Steve --------- HZ+S Proposal, Version 0.6 1. Introduction. By Big5 we mean the ETen implementation of Big5, chosen because ETen is the most popular Big5 software. The ETen implementation of Big5 is described precisely in my document which is available for anonymous FTP at ifcss.org, /software/fonts/d/etfonts.txt, or at math.psu.edu, /pub/simpson/chinese/hzps/etfonts.txt. HZ+S is my proposal for a standardized 7-bit representation of mixed Big5, GB, and ASCII for convenient e-mail transmission, etc. I intend HZ+S to be downward compatible with HZ, the popular 7-bit representation of mixed GB and Big5. If HZ+S becomes widely accepted, I would like to rename the resulting standard as HZ+. This name is intended to suggest an expanded or enlarged version of HZ. 2. Details of the HZ+S specification. In order to give Big5 a convenient 7-bit representation, I partition the Big5 code range into two blocks. Blocks 1 and 2 consist of the ranges A140 to C8FE and C940 to FEFE respectively, where the second byte is restricted to the ranges 40 to 7E and A1 to FE. (Note that block 1 contains the more frequently used Chinese characters plus numerals, Roman and Greek and Russian and Bopomofo alphabets, symbols, numerals, Japanese Kana, etc. Block 2 contains the less frequently used Chinese characters plus some additional symbols. The less frequently used Chinese characters start at C940, which is also the beginning of the block 2 range.) I pack each of the block 1 and 2 ranges in order onto an initial segment of the range 2121 to 7D7E, where the first byte is restricted to the range 21 to 7D and the second byte is restricted to the range 21 to 7E. This packing is implemented by a simple macro. Thus each character in each block is mapped to a code consisting of two graphical ASCII bytes. In addition to the the above 7-bit representation of Big5, I need a way to distinguish between Big5 block 1, Big5 block 2, GB, and ASCII. I do so by using four tilde escape sequences: ~{ to switch into GB (just as in the existing HZ standard) ~} to switch into ASCII (just as in the existing HZ standard) ~1 to switch into Big5 block 1 (this is new in HZ+S) ~2 to switch into Big5 block 2 (this is new in HZ+S) There are also a few additional tilde escape sequences, as defined in the HZ specification. 3. Advantages of HZ+S. (1). The proposed HZ+S or HZ+ standard is based firmly on two existing standards which are already very popular: HZ and Big5. This should enable HZ+ to win a degree of immediate acceptance. (2). HZ+S will be very easy to implement. I have already hacked up some Big5 <-> HZ+S conversion filters. This experimental software is based on Fung F. Lee's GB <-> HZ conversion filters and is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. (3). HZ+S fully respects the natural partition of the Big5 code space into blocks of more and less frequently used characters. This leads to several additional advantages, listed below. (4). HZ+S uses escape sequences sparingly. The ~2 escape sequence is needed only to switch into Big5 block 2, the block containing the less frequently used Chinese characters. Most of the time, we would use only ~1 to switch into Big5 block 1, plus ~{ and ~} as in HZ. (5). Existing Big5 fonts can be used essentially unchanged for HZ+S. (6). HZ+S is fully compatible not only with ETen Big5, but also with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less frequently used Chinese characters begin somewhere prior to C940. Thus HZ+S may help to overcome the difficulties associated with these corrupt variants. (7). The way is open for HZ+S to be compatible with a possible future incorporation of Big5 into ISO-2022-JP-2, similar to the present incorporation of CNS-11643. Such an unofficial extension of ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. Please send your comments to me. Stephen G. Simpson / simpson@math.psu.edu / May 5, 1994 From mcevilly@netcom.com Thu May 5 11:02:31 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA20073; Thu, 5 May 94 11:02:31 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id JAA00752; Thu, 5 May 1994 09:00:23 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405051600.JAA00752@netcom.com> Subject: Re: 7-bit Big5, call for expression of viewpoints To: soft-authors@ifcss.org Date: Thu, 5 May 1994 09:00:23 -0700 (PDT) Cc: shin@hermes.ee.nthu.edu.tw In-Reply-To: <199405051316.JAA09130@boole.math.psu.edu> from "Stephen G Simpson" at May 5, 94 09:16:05 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1735 Status: O > Could I ask everybody here on soft-authors to please send mail to > soft-authors indicating your stance or viewpoint? Indicate your > degree of interest and concern. You can say that you don't care at > all about 7-bit Big5. You can say that you want a 7-bit Big5 standard > but don't care to choose between the various proposals. You can > indicate which of the proposals you would find acceptable. If you My support is behind HZ+S for several reasons. 1) It has been through more scrutiny here; 2) it holds out the possibility of future ISO-2022 compatibility; 3) it maps more naturally onto the Big5 character set (this might be important in winning over Taiwanese programmers). HZ+2 has some good points, but these are also shared by HZ+S, and HZ+S has additional advantages. The third scheme is unacceptable because its mixing of codes from the low and high ranges in Big5 would mean a great deal of text bloat as well as overhead with the increased amount of shift in/shift out. CNS seemed interesting at first, but it adds its own additional layer of complexity and since it's hardly used at all even in Taiwan, it also might serve to further alienate HZ from Taiwanese users. Ya-Gui's planned script language is very encouraging, because it was unclear to me whether he would be willing to support a method other than HZ+2, and this development seems to answer that question in a positive way. It was a beautiful solution to the problem he faces developing a new version in the face of an uncertain coding environment. It buys us some breathing room, and suggests an innovative approach for any software developer with the same kind of need for flexibility. Nicely done. Carlos McEvilly mcevilly@netcom.com From simpson@math.psu.edu Thu May 5 11:56:49 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA22494; Thu, 5 May 94 11:56:49 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA09599; Thu, 5 May 94 09:54:41 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA10192; Thu, 5 May 1994 12:48:56 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 12:48:54 -0400 Date: Thu, 5 May 1994 12:48:54 -0400 Message-Id: <199405051648.MAA09472@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: acknowledgements Reply-To: simpson@math.psu.edu Status: O I just added the following to my HZ+S document. 4. Acknowledgements. The originator of the HZ standard was Fung Fung Lee, who also mentions significant contributions from Edmund Lai, Tin-Fook Ngai, Ya-Gui Wei, and Ricky Yeung. The present HZ+S proposal owes much to an earlier HZ-2 proposal by Ya-Gui Wei. In addition, HZ+S has benefitted greatly from my discussions with Yidao Cai, Jing-Shin Chang, Jianqing Hu, Carlos McEvilly, Ross Paterson, Ya-Gui Wei, and other members of soft-authors@ifcss.org. I regard my own contribution to HZ+S as being tiny compared to that of Ya-Gui Wei. -- Steve From lai@apple.com Thu May 5 13:14:13 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA25699; Thu, 5 May 94 13:14:13 CDT Received: from mac39.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA24129; Thu, 5 May 94 11:11:34 -0700 for soft-authors@ifcss.org Date: Thu, 5 May 94 11:11:34 -0700 Message-Id: <9405051811.AA24129@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: CCNET-L@uga.cc.uga.edu From: lai@apple.com (Edmund Lai) Subject: Re: 7-bit Big5 proposals, ISO-2022 compatibility Cc: soft-authors@ifcss.org Status: O >Here is what I think the mapping is, mostly generated from the Unicode >mapping tables: > ># Big5 non-hanzi and hanzi level 1 <-> CNS level 1 ># >A140-A154 <-> 212121-212135 many lines deleted >F9D2-F9D5 <-> 227241-227244 This does not look too bad to me and the conversion can be done with a table a page long in each direction. >CNS seemed interesting at first, but it adds >its own additional layer of complexity and since it's hardly used at >all even in Taiwan, it also might serve to further alienate HZ from >Taiwanese users. As I said the complexity is not that bad, and we are not talking about using CNS as the internal code, the conversion between Big5 and CNS probably exist already, and if not can certainly be easily written and distributed. Whatever encoding scheme we come up with other than CNS has certainly not used any where before at all. Rather than alienating user in Taiwan, we can tell them that we support the offical standard, we support the default internal code standard by providing CNS<->Big5 conversion, and ISO-2022 is not a problem. Any universities in Taiwan that wish to support HZ* in their server can have a good story to tell the goverment authority. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From khsc@SynOptics.COM Thu May 5 15:36:45 1994 Return-Path: Received: from SynOptics.COM ([134.177.1.95]) by ifcss.org (4.1/IFCSS-Mailer) id AA02014; Thu, 5 May 94 15:36:45 CDT Received: from taichi (taichi.synoptics.com) by SynOptics.COM (4.1/SMI-4.1) id AA11705; Thu, 5 May 94 13:34:23 PDT Received: by taichi (4.1/2.0N) id AA06491; Thu, 5 May 94 13:34:22 PDT Message-Id: <9405052034.AA06491@taichi> Date: Thu, 5 May 94 13:34:22 PDT From: Simon Chow ~{V\9zP[~} To: soft-authors@ifcss.org Subject: Re: 7-bit Big5, call for expression of viewpoints Status: O > CALL FOR EXPRESSION OF VIEWPOINTS > > Could I ask everybody here on soft-authors to please send mail to > soft-authors indicating your stance or viewpoint? Indicate your > degree of interest and concern. You can say that you don't care at > all about 7-bit Big5. You can say that you want a 7-bit Big5 standard > but don't care to choose between the various proposals. You can > indicate which of the proposals you would find acceptable. If you > have a strong preference for one proposal or another, indicate that. > If you have strong ideas about CNS, mention them. If possible, > indicate reasons for your viewpoint. > > This is not a formal vote, but I am hoping a consensus will emerge, > or at least we will see how things stand. > > -- Steve > I am very much in favour of the HZ* Big5 extension, I have basic understanding of all three proposal, but did not have time to fullying think out all the implications (read descripbing the proposals in terms of computer language such as C), But since Steve has done a good job on pointing out the advantages of HZ+S, I am comfortable with HZ+S, My votes are: Yes, we need a 7bit Big5 HZ standard I favour HZ+S, HZ-2 is my second choice |\/\/\/| =============================== | | ==================================== | | (o)(o) | | Simon Chow C _) UUCP: att!synoptics.com!schow | | SynOptics Communication Inc. | ,___| INTERNET: schow@synoptics.com | | 4401 Great America Pkwy, | / Voice: +1 (408) 764-1327 | | Santa Clara, CA 95052-8185 /____\ Fax: +1 (408) 988-5525 | ============================================================================= #include From simpson@math.psu.edu Thu May 5 16:21:32 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA04441; Thu, 5 May 94 16:21:32 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA17767; Thu, 5 May 94 14:19:04 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA12361; Thu, 5 May 1994 17:16:55 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 17:16:54 -0400 Date: Thu, 5 May 1994 17:16:54 -0400 Message-Id: <199405052116.RAA09813@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Cc: ccnet-l@uga.cc.uga.edu Subject: Big5 or CNS ? In-Reply-To: <9405051811.AA24129@apple.com> References: <9405051811.AA24129@apple.com> Reply-To: simpson@math.psu.edu Status: RO Edmund Lai writes: > certainly not used any where before at all. Rather than alienating > user in Taiwan, we can tell them that we support the offical > standard I have the feeling that supporting CNS *would* alienate many users, both in Taiwan and elsewhere. Remember that Big5 is the defacto standard and is extremely popular. Most people in Taiwan and many people elsewhere use it for their daily work. CNS on the other hand is an official ROC government standard which nobody uses, not even in Taiwan. I don't think it is for us to tell users in Taiwan or elsewhere which standard is best for them. It makes more sense for us to follow as closely as possible the standard that they actually use. The standard they actually use is Big5. Perhaps it's worth recalling that the purpose of HZ+ is to serve as a practical tool for international 7-bit Chinese news and e-mail. Our primary goal should be practicality and wide popular acceptance, not compatibility with ROC government standards. -- Steve From lai@apple.com Thu May 5 18:06:00 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA08664; Thu, 5 May 94 18:06:00 CDT Received: from mac39.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA02761; Thu, 5 May 94 16:03:47 -0700 for soft-authors@ifcss.org Date: Thu, 5 May 94 16:03:47 -0700 Message-Id: <9405052303.AA02761@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw From: lai@apple.com (Edmund Lai) Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu Status: O >Edmund Lai writes: > > certainly not used any where before at all. Rather than alienating > > user in Taiwan, we can tell them that we support the offical > > standard > >I have the feeling that supporting CNS *would* alienate many users, >both in Taiwan and elsewhere. Remember that Big5 is the defacto >standard and is extremely popular. Most people in Taiwan and many >people elsewhere use it for their daily work. CNS on the other hand >is an official ROC government standard which nobody uses, not even in >Taiwan. I don't think it is for us to tell users in Taiwan or >elsewhere which standard is best for them. It makes more sense for us >to follow as closely as possible the standard that they actually use. >The standard they actually use is Big5. > >Perhaps it's worth recalling that the purpose of HZ+ is to serve as a >practical tool for international 7-bit Chinese news and e-mail. Our >primary goal should be practicality and wide popular acceptance, not >compatibility with ROC government standards. > >-- Steve It is very important to distinguish an exchange code and an internal code. I am not proposing using CNS as internal code at all. So it is not Big5 vs CNS, but some form of encoded Big5 vs CNS. Here CNS is just used as a form of encoding for Big5. The users really do not care to read directly the encoded exchange code. What they care is that it can be decoded back to their own internal code. This is a must. Beyond that they want the exchange code to be a widely accepted. And that is where CNS as an offical standard is an advantage. Rather than yet another new exchange code, it is better to take an existing offical exchange code. There is no evidence and no reason why this would alienate any user from Taiwan. So far the reaction I hear from them is that they are using Big5 as both the internal code and exchange code, so their first reaction is why is it necessary to have a 7 bit code. When it is explained that it is for 7 bit email and news, they accept it. They understand this 7 bit code is not Big5, and they are happy as long as it can be converted back and forth from Big5. So the question is that should we take a new non-standard encoding format that takes about 10 instruction to encode, or a standard encoding exchange that may take up to a few hundread instructions to encode. I think with computer technology these days (for that I include even the Mac+ and the PC XT that my kids are still using) the 2nd option is not a problem. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From simpson@math.psu.edu Thu May 5 19:34:55 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11942; Thu, 5 May 94 19:34:55 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id UAA15373; Thu, 5 May 1994 20:27:18 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 5 May 1994 20:27:17 -0400 Date: Thu, 5 May 1994 20:27:17 -0400 Message-Id: <199405060027.UAA10093@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: shin@hermes.ee.nthu.edu.tw, shann@wangwei.math.ncu.edu.tw, ccnet-l@uga.cc.uga.edu Subject: Re: Big5 or CNS ? In-Reply-To: <9405052303.AA02761@apple.com> References: <9405052303.AA02761@apple.com> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > So the question is that should we take a new non-standard encoding > format that takes about 10 instruction to encode, or a standard > encoding exchange that may take up to a few hundread instructions > to encode. By "non-standard encoding format" Ed is referring to 7-bit Big5. By "standard encoding exchange" Ed is referring to 7-bit CNS. However, I don't think 7-bit Big5 is any more non-standard than 7-bit CNS. My actual feeling is just the opposite. Big5 is a very popular standard. CNS is not a popular standard. Standards that have no popular support are inappropriate for the Internet. However, I respect Ed's viewpoint. And I can see the significance of Ed's distinction between "exchange code" and "internal code". If there is a strong consensus that our future HZ+ standard should be based on 7-bit CNS rather than 7-bit Big5, I guess I can live with that consensus. > There is no evidence and no reason why this would alienate any user > from Taiwan. My impression is that there has been rebellion whenever the III tried to impose CNS. Perhaps someone from Taiwan could tell us the history? Shann? Jing-Shin? >From a purely technical point of view, I'm sure CNS has much to recommend it. Perhaps CNS is unpopular for social and political reasons, not technical. So far as I can tell, the III seems to be a fairly hide-bound organization, out of touch with popular computing needs and indifferent to the Internet. -- Steve From bran@monod.biol.mcgill.ca Thu May 5 19:53:51 1994 Return-Path: Received: from sifon.CC.McGill.CA by ifcss.org (4.1/IFCSS-Mailer) id AA12428; Thu, 5 May 94 19:53:51 CDT Received: from monod.biol.mcgill.ca (monod.Biol.McGill.CA [132.206.226.16]) by sifon.CC.McGill.CA (8.6.8/8.6.6) with SMTP id UAA27478 for ; Thu, 5 May 1994 20:51:52 -0400 Received: by monod.biol.mcgill.ca (4.1/SMI-4.1) id AA10091; Thu, 5 May 94 20:52:48 EDT Date: Thu, 5 May 1994 20:52:29 -0400 (EDT) From: Bing Ran Subject: To: soft-authors@ifcss.org Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O sub Bing Ran From mcevilly@netcom.com Thu May 5 23:28:30 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA20190; Thu, 5 May 94 23:28:30 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id VAA25211; Thu, 5 May 1994 21:27:09 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405060427.VAA25211@netcom.com> Subject: Re: CNS and ETen To: soft-authors@ifcss.org Date: Thu, 5 May 1994 21:27:05 -0700 (PDT) Cc: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) In-Reply-To: <9405031116.AA09395@hermes.ee.nthu.edu.tw> from "Jing-Shin Chang" at May 4, 94 09:16:09 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2819 Status: O Jing-Shin Chang wrote: > There are email implementation in Sun's ALE environment which use the > escape sequence: "ESC $ ) 0" to designate chinese codes used in Taiwan. Your message and an earlier message from Ross Paterson helped me figure out that CNS uses this sequence to enter plane 1. When I entered this escape sequence into a CNS file under ETen, however, ETen did not recognize the sequence. It's possible I was doing something wrong, and it's also possible that ETen's support of CNS is not as good as it could be. I would like to propose that we drop the idea of using CNS, since Big5, whether in 8-bit or an HZ 7-bit form, seems to be all that is needed. If we really insisted on using an all-inclusive interchange code, we would be talking about using CCCII or REACC, and we are certainly not at that point. > Since I got all these information from a wide variety of sources > and each pieces of information are very brief, could you do a even more > thorough test for everybody? Thanks. I did try embedding the escape sequence in a CNS file, and it did not work under ETen. In case somebody got something wrong, I tried ( and ); tried putting the escape sequence directly before the character; tried reading the file again (as if any of these things actually would make a difference--but one has to try). The sequence you gave is the same one specified in the CNS 11643 intro; using the notation Ross explained: ESC 2/4 2/9 F where F is 3/0 to 3/15. For F=3/0, this translates to 1B 24 29 30, or ESC $ ) 0. The standard also uses ESC 2/4 2/10 F and ESC 2/4 2/11 F; I tried all of these with various values of F (30 - 3F). I'm aware that use of CNS doesn't necessarily imply its use as an internal code, but I do find the above to suggest that CNS support might not be as good as we thought. > standard way. However, since III is almost the policy-maker of the ROC > information industry, and it is a semi-governmental official, I think this > implementation must follow the national standard strictly. > Other governmental organizations follow the defacto standard, Big5. This includes government offices, the National Central Library, Academia Sinica, Taiwan National University, Taiwan National Normal University, all other universities I'm aware of there, and even the Ministry of Education which administers the Taiwan Academic Network, TANET. I would be very surprised if even III and the national standards bureau (I don't know the official name) do not use Big5 for their day- to-day business. Can anyone give a compelling reason why we should consider the use of CNS as an exchange code, rather than using a 7-bit version of Big5? Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From A7621GAC@AWIUNI11.EDVZ.UNIVIE.AC.AT Fri May 6 03:39:42 1994 Return-Path: Received: from AWIUNI11.EDVZ.UniVie.AC.AT (helios.edvz.univie.ac.at) by ifcss.org (4.1/IFCSS-Mailer) id AA28057; Fri, 6 May 94 03:39:42 CDT Message-Id: <9405060839.AA28057@ifcss.org> Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT by AWIUNI11.EDVZ.UniVie.AC.AT (IBM VM SMTP V2R2) with BSMTP id 8311; Fri, 06 May 94 10:37:04 MEZ Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT (NJE origin A7621GAC@AWIUNI11) by AWIUNI11.EDVZ.UNIVIE.AC.AT (LMail V1.2a/1.8a) with BSMTP id 4196; Fri, 6 May 1994 10:37:03 +0100 Date: Fri, 06 May 94 10:33:27 MEZ From: Werner Lemberg Subject: Re: 7-bit Big5, call for expression of viewpoints To: soft-authors@ifcss.org In-Reply-To: Your message of Thu, 5 May 1994 09:16:05 -0400 Status: O I read my mail under VM CMS, so I can't read Chinese mail :-) In my opinion the proposed standard should be mappable to Unicode. If this is achieved, everybody will be able to read it somehow... Werner From simpson@math.psu.edu Fri May 6 07:52:09 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA04554; Fri, 6 May 94 07:52:09 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA18220; Fri, 6 May 1994 08:49:48 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 08:49:46 -0400 Date: Fri, 6 May 1994 08:49:46 -0400 Message-Id: <199405061249.IAA11234@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re: Big5 or CNS ? In-Reply-To: <9405060739.AA13002@hermes.ee.nthu.edu.tw> References: <9405060739.AA13002@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > I had posted my 2nd B5encode proposal (a new one) to > ccnet-l@uga.uga.edu. However, it seems that my posting was not > broadcasted. (I sent it twice already.) Is there any problem with > the server? I received both of your B5encode proposals. I asked you whether the 2nd one was intended to replace the 1st one. You answered that it wasn't, so I appended it to the 1st one, which I had already stored for anonymous FTP with both of the other proposals, at math.psu.edu, /pub/simpson/chinese/hzps. To the best of my knowledge, all of this mail was seen by everybody on soft-authors and ccnet-l. Could you please join soft-authors? That way we won't have to keep copying everything to you. Also, if you have some additional relevant material concerning your B5encode proposal, could you please make it available by anonymous FTP or some other method? I would be glad to help as needed. > 1. The SeedNet sysadm had informed its users to change the default > code from CNS (7bit protocol) to Big5 in email, news from > 05/09. Is this referring to the "internal code" or the "exchange code" (using Ed's terminology), or both? In any case, I'm very glad to learn that the SeedNet administrator saw the light and switched from CNS to Big5. :-)) What were the reasons for the switch? > 2. I strongly support Lai's (technical) points in using CNS as the standard > exchange code. It is also advocated in my earliest proposal. The > practice is, however, little soft-authors like to write even the > most basic tools for the chinese.* newsgroups. The main reason > could be due to the infamiliarity to the coding schemes. Easy > encoding thus becomes an important criteria in my later posting. > And, I am not rebelliant against CNS -^^-, I am just making compromise. > > 3. My changes (or strictly speaking, my alternative solutions) are leaning > toward the users of the newsgroup, who may not have any technical > background for those encoding stuffs. It is hard to educate > non-technical users those technical stuffs. I'm not sure which "newsgroup" you are referring to. Soft-authors, or ccnet-l? In any case, I think you will find that people in both groups are willing and even eager to be educated. Not only that, but some of us in soft-authors even have some technical background. For example, many of us have PC clones and have written programs in GW-BASIC. :-) Seriously, soft-authors is a group consisting of authors of Chinese software, some of which is widely used. Could you please explain to us, in simple language of course, the exact form of CNS that is (or was) used as an exchange code in SeedNet? What are the escape sequences? And, another question about SeedNet, where is the translation between the "exchange code" and the "internal code" carried out? Is this translation done by system software, e.g. sendmail and nntp daemons, or is it done by user software, e.g. xrn and mailtool? Please enlighten us. > 4. The HZ* proposals we are discussing is actually a bad yet popular > standard, which uses the exchange codes for processing directly. HZ isn't perfect, but a lot of us find it quite useful. :-) > Jing-Shin -^^- -- Steve From simpson@math.psu.edu Fri May 6 09:09:48 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA07175; Fri, 6 May 94 09:09:48 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id KAA18716; Fri, 6 May 1994 10:07:49 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 10:07:47 -0400 Date: Fri, 6 May 1994 10:07:47 -0400 Message-Id: <199405061407.KAA11436@boole.math.psu.edu> To: Werner Lemberg Cc: soft-authors@ifcss.org Subject: Re: 7-bit Big5, call for expression of viewpoints In-Reply-To: <9405060839.AA28057@ifcss.org> References: <9405060839.AA28057@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Werner Lemberg writes: > I read my mail under VM CMS, so I can't read Chinese mail :-) Somebody mentioned recently that he has a friend who is reading HZ Chinese news from a VM/CMS server with no problems. But I also know that VM/CMS is sometimes troublesome, because EBCDIC <-> ASCII translation is not always perfect. > In my opinion the proposed standard should be mappable to Unicode. If this > is achieved, everybody will be able to read it somehow... All of the proposals under discussion are mappable to Unicode. -- Steve From mcevilly@netcom.com Fri May 6 09:39:17 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA08313; Fri, 6 May 94 09:39:17 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id HAA06107; Fri, 6 May 1994 07:38:37 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405061438.HAA06107@netcom.com> Subject: Re: Big5 or CNS ? To: soft-authors@ifcss.org Date: Fri, 6 May 1994 07:38:33 -0700 (PDT) In-Reply-To: <199405061030.DAA14835@mail.netcom.com> from "Roland Wang" at May 5, 94 04:46:18 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2498 Status: O > I second to Ed Lai's approach - use 7 bit HZ+ to encode CNS for the > communication, such as email and news. User end software can be in > any encoding, CNS (only need 7bit to 8bit conversion), Big5 (7bit to > 8bit, then map to Big5), and GB (7bit to 8bit, then CNS2GB mapping). I'm not seeing what the disadvantage of a 7-bit version of Big5 coding is. Can you explain? > I'm not advocating CNS. The reasoning is just for pure technical > perspectives. Could you tell us some of the specific technical points you have in mind? > First of all, we need to clarify a mis-perception about CNS. There is > no doubt about the popularity of Big5 due to the PC popularity (both > IBM PC/compatible and Mac). However, CNS is not just an official standard. > It is used in Taiwan widely even though there is no comparison to Big5. Where? > Supporting CNS is one of the "must" items for Taiwan goverment systems. And yet in practice it is hardly ever done. Which systems are you talking about, by the way? Library systems are required to support CCCII -- my understanding is that this is a contract bidding condition, but it might also be a law (BTW yesterday I said the National Central Library uses Big5--I should clarify that to say they use Big5 for their administration; of course they use CCCII in the catalog). Which governmental units actually use CNS? Can you list any? > That's one of the reasons why most of UNIX vendors support CNS instead of > Big5 (another reason is the technical difficulty to fit Big5 into EUC > scheme). > Could the real reason be that they need a 7-bit coding because some drivers strip the 8th bit? > Second, no matter which encoding we choose (HZ+, or whatever), we need > convertors for the user-end applications (mail, news reader, etc.). It > won't add much complexity for conversion between 7bit HZ+ CNS and Big5, > comparing to the conversion between 7bit HZ+ Big5 to Big5. The convertor > between CNS and Big5 is widely and publicly available. > Converters are available for all the schemes proposed; this isn't special. So what is the advantage of CNS? You say "it won't add much complexity," is that the only 'advantage'? :-) I do not mean to be close-minded about this; I am ready to be convinced of the technical superiority of one scheme over another. Yet although there have been assertions about CNS, there have been no reasons listed to back up those assertions. Given this situation, it's hard to see why CNS should be considered. From simpson@math.psu.edu Fri May 6 09:39:35 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08315; Fri, 6 May 94 09:39:35 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id KAA18925; Fri, 6 May 1994 10:37:01 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 10:37:00 -0400 Date: Fri, 6 May 1994 10:37:00 -0400 Message-Id: <199405061437.KAA11491@boole.math.psu.edu> To: lai@apple.com (Edmund Lai) Cc: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, ccnet-l@uga.cc.uga.edu Subject: Re: Big5 or CNS ? In-Reply-To: <9405052303.AA02761@apple.com> References: <9405052303.AA02761@apple.com> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > It is very important to distinguish an exchange code and an internal code. At first I thought I understood this distinction, but thinking about it some more I got confused. Could you please answer one question. According to your definitions, is HZ (as used in alt.chinese.text) an exchange code or an internal code? I guess my real question is, is one of the properties of an exchange code the fact that it is hidden from users? If so, then HZ is clearly not an exchange code. It is not hidden from users. Users see it all the time. That's why they keep asking questions like "What's this junk on my screen?" and "Why can't I use ETen to read alt.chinese.text?" > I am not proposing using CNS as internal code at all. So it is not > Big5 vs CNS, but some form of encoded Big5 vs CNS. Here CNS is just > used as a form of encoding for Big5. The users really do not care > to read directly the encoded exchange code. What they care is that I think it is misleading to describe CNS as an encoding for Big5. The truth is that CNS and Big5 are two different Chinese character encoding schemes. -- Steve From simpson@math.psu.edu Fri May 6 09:52:56 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08870; Fri, 6 May 94 09:52:56 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id KAA18970; Fri, 6 May 1994 10:50:29 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 10:50:28 -0400 Date: Fri, 6 May 1994 10:50:28 -0400 Message-Id: <199405061450.KAA11513@boole.math.psu.edu> To: "Roland Wang" Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <9405052345.AA23768@federal-excess.apple.com> References: <9405052345.AA23768@federal-excess.apple.com> Reply-To: simpson@math.psu.edu Status: O Roland Wang writes: > I second to Ed Lai's approach - use 7 bit HZ+ to encode CNS for the > communication, such as email and news. User end software can be in > any encoding, CNS (only need 7bit to 8bit conversion), Big5 (7bit to > 8bit, then map to Big5), and GB (7bit to 8bit, then CNS2GB mapping). I'm not clear how this proposal would work in practice. Could you please explain? Would the CNS codes be totally hidden from all users? > First of all, we need to clarify a mis-perception about CNS. There is > no doubt about the popularity of Big5 due to the PC popularity (both > IBM PC/compatible and Mac). We need to clarify a misperception about Big5. Big5 is not only popular on PCs and Macs. It is also popular on Unix. Many people run cxterm in Big5 mode and use it for things like reading and posting to a.c.t.big5, participating in the Chinese poem list, typing documents, TeX, etc. (not to mention communicating with all those PC and Mac users). > However, CNS is not just an official standard. It is used in > Taiwan widely even though there is no comparison to Big5. I haven't heard of even one person who uses CNS in their daily work. Can you name one such person? Are you such a person? > Supporting CNS is one of the "must" items for Taiwan goverment systems. Why should we in the international Internet community care about ROC government imposed standards that do not have popular support? > That's one of the reasons why most of UNIX vendors support CNS instead of > Big5 (another reason is the technical difficulty to fit Big5 into EUC > scheme). What technical difficulty? I think the difficulty is not technical but political, namely the government's insistence on CNS. -- Steve From mcevilly@netcom.com Fri May 6 10:14:47 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA09877; Fri, 6 May 94 10:14:47 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id IAA09422; Fri, 6 May 1994 08:14:11 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405061514.IAA09422@netcom.com> Subject: Re: Big5 or CNS ? To: soft-authors@ifcss.org Date: Fri, 6 May 1994 08:14:10 -0700 (PDT) In-Reply-To: <9405052303.AA02761@apple.com> from "Edmund Lai" at May 5, 94 04:03:47 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2875 Status: O > Beyond that they want the exchange code to be a widely accepted. And > that is where CNS as an offical standard is an advantage. Rather > than yet another new exchange code, it is better to take an existing > offical exchange code. Ed, is this the only reason you support CNS? CNS exists, true, but does it exist in use? Not hardly, compared to Big5. Isn't it better to take an existing code that is actually used and accepted by the people and programmers in Taiwan? > There is no evidence and no reason why this would alienate any user > from Taiwan. So far the reaction I hear from them is that they are And there is no evidence and no reason why 7-bit Big5 would alienate them. But if they aren't able to switch out of plane 1, they will be very alienated. Could you please tell us how ETen users (95% of the Chinese PC environment market in Taiwan) can switch from plane to plane in CNS? Or if you propose the software of the future should convert from CNS to Big5, why not just use 7-bit Big5 in the first place? > using Big5 as both the internal code and exchange code, so their > first reaction is why is it necessary to have a 7 bit code. When > it is explained that it is for 7 bit email and news, they accept > it. They understand this 7 bit code is not Big5, and they are > happy as long as it can be converted back and forth from Big5. This is true, but it is not an advantage for CNS; it is simply a truth about whatever exchange code we choose. > So the question is that should we take a new non-standard encoding > format that takes about 10 instruction to encode, or a standard > encoding exchange that may take up to a few hundread instructions > to encode. I think with computer technology these days (for that > I include even the Mac+ and the PC XT that my kids are still using) > the 2nd option is not a problem. > What reasons do we have to justify the use of these extra CPU cycles? I really do want to see us give CNS a fair shake, but to do so we need to hear specific, concrete, unique advantages it has. If the only advantage is that it is a standard, the truth is that Big5 is in much more common use. In Taiwan it is referred to as "the defacto standard" in English newspapers. I understand it it technically possible to use CNS as the exchange code and Big5 as the internal code, but how does this benefit anyone? Currently CNS is not really in the picture as far as Chinese computing goes; resurrecting it seems like a bad idea to me. For us people on the net it may be easy to ftp a conversion routine and integrate it into our program, but for many programmers in Taiwan the net is still just something they've heard about. It would be good to keep the mapping as simple as possible, ideally something that can be conveyed by word of mouth or in a couple lines on a scratch pad. I think this last point is important. From CAI@neurophys.wisc.edu Fri May 6 10:50:02 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA11405; Fri, 6 May 94 10:50:02 CDT Received: from VMSmail by neurophys.wisc.edu; Fri, 6 May 94 10:45 CST Message-Id: <24050610453909@neurophys.wisc.edu> Date: Fri, 6 May 94 10:45 CST From: CAI@neurophys.wisc.edu Subject: CNS vs Big5 and my vote To: soft-authors@ifcss.org X-Vms-To: IN%"soft-authors@ifcss.org" Status: O 1. CNS vs Big5 There seems to be contradict information regarding the popularity of CNS in Taiwan. Today I read from acc that CNS is used in all government agencies and supporting CNS is a must, while on soft-authors one listed a lot of government agencies that uses Big5 instead of CNS. Another news is that a net is going to change its exchange code from CNS to Big5. To me the main concern is what the end user uses, Big5 or CNS. If what the end user uses most is Big5, then we should go Big5. We all know that CNS<->Big5 is not trial, although not that complex, putting this burden of mapping on MOST end users is not justified if this is just to give convienence to a small number of CNS users, or simply to let the Taiwan government feel happy. On the other hand, if CNS is more popular at the end user, the same logic applies and we should go CNS. By the logic of some CNS supportors, since CNS<->Big5 is not that complex as to keep us away from them, there should be no technical advantage for choosing one over the other, so why CNS is preferred? What I can see now is that almost all the arguments for CNS can be applied to Big5. So I would suggest popularity as our final criterion. 2. For all three proposals, I admire B5encode for simplicity (I original thought both Ya-Gui and Simpson was using this kind of mapping). But the proposals of Ya-Gui and Simpson could reduce the switching between block 1 and block 2. My final vote give to the one that is most efficient in terms of total CPU time when both the switching and mapping are considered (anybody can make an estimate? Has Ross done one? Or we can do an actual test on news.test or similar groups before we finalize, but then we need some extra work). For what escape sequence to use, I prefer ~[ ~] or ~< ~> over ~1 ~2. I am glad that Ya-Gui actually started the implantation. Thanks! cai From khsc@SynOptics.COM Fri May 6 12:11:00 1994 Return-Path: Received: from SynOptics.COM ([134.177.1.95]) by ifcss.org (4.1/IFCSS-Mailer) id AA14584; Fri, 6 May 94 12:11:00 CDT Received: from taichi (taichi.synoptics.com) by SynOptics.COM (4.1/SMI-4.1) id AA17783; Fri, 6 May 94 10:08:30 PDT Received: by taichi (4.1/2.0N) id AA03681; Fri, 6 May 94 10:08:30 PDT Message-Id: <9405061708.AA03681@taichi> Date: Fri, 6 May 94 10:08:30 PDT From: Simon Chow ~{V\9zP[~} To: soft-authors@ifcss.org Subject: Re: Big5 or CNS ? Status: O > Edmund Lai writes: > > So the question is that should we take a new non-standard encoding > > format that takes about 10 instruction to encode, or a standard > > encoding exchange that may take up to a few hundread instructions > > to encode. Thank you for Edmund's clearificatoin of exchange code and internal code, I now have a much better understanding of the whole issue. I think Edmund's email should be compusary reading for "soft-author" and should be archive in ifcss.org (I have a feeling that Edmund has previously explained this in soft-author or some file in ifcss.org already) while most user does not care about the exchange code, *every* program that reads these exchange code need to convert it back to internal code (big5 in this case), while most every application programer has done code conversion base on the scheme based on something similar to that of HZ+S, (where the conversion is based on a simple formular), the conversion between CNS and big5 calls for both a lookup table and a simple formular. I think this will definitely results in confusion in terms of implementation. I am not saying that implementation would be difficult. What I am saying is that not every application developer have access to the CNS-big5 conversion information. It has been a big mistry for quite a while for reader of this group to get precise info on the conversion info between CNS-big5, I am sure if we choose to use CNS as the exchange code, some applicatoin will emerge and get the conversion wrong (eg implement the conversion based on a simple formular instead of taking into account of the "correction" made to CNS standard). If there enough application supporting the wrong conversion scheme, another "standard" similar to Big5-HKU will be born. I am not against using CNS as exchange code, this is just my concern on incorrect implementations. |\/\/\/| =============================== | | ==================================== | | (o)(o) | | Simon Chow C _) UUCP: att!synoptics.com!schow | | SynOptics Communication Inc. | ,___| INTERNET: schow@synoptics.com | | 4401 Great America Pkwy, | / Voice: +1 (408) 764-1327 | | Santa Clara, CA 95052-8185 /____\ Fax: +1 (408) 988-5525 | ============================================================================= #include From jiang Fri May 6 12:45:40 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA16249; Fri, 6 May 94 12:45:40 CDT Date: Fri, 6 May 94 12:45:40 CDT From: jiang (Yuan Jiang) Message-Id: <9405061745.AA16249@ifcss.org> To: lai@apple.com, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, soft-authors Status: O >code the fact that it is hidden from users? If so, then HZ is clearly >not an exchange code. It is not hidden from users. Users see it all I guess HZ was intended to be an exchange code but isused as internal code as well. HZ is not much different from GB. >I think it is misleading to describe CNS as an encoding for Big5. The Ed is suggesting that we use 7-bit CNS in place of HZ+ to encode Big5. Yuan From simpson@math.psu.edu Fri May 6 13:22:17 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA17865; Fri, 6 May 94 13:22:17 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA20314; Fri, 6 May 1994 14:20:17 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 14:20:15 -0400 Date: Fri, 6 May 1994 14:20:15 -0400 Message-Id: <199405061820.OAA11884@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: 7-bit Big5, CPU cycles, Big5 compatibility In-Reply-To: <24050610453909@neurophys.wisc.edu> References: <24050610453909@neurophys.wisc.edu> Reply-To: simpson@math.psu.edu Status: O We at soft-authors are looking at ways to enhance the HZ standard by adding 7-bit Big5 capability. Three competing proposals are under consideration: B5encode, HZ+S, HZ-2. I have called for an expression of viewpoints concerning the merits of the proposals and the desirability of 7-bit Big5. EFFICIENCY IN TERMS OF CPU CYCLES CAI@neurophys.wisc.edu writes: > 2. For all three proposals, I admire B5encode for simplicity (I original > thought both Ya-Gui and Simpson was using this kind of mapping). But the > proposals of Ya-Gui and Simpson could reduce the switching between block 1 > and block 2. My final vote give to the one that is most efficient in terms > of total CPU time when both the switching and mapping are considered > (anybody can make an estimate? Has Ross done one? Or we can do an actual > test on news.test or similar groups before we finalize, but then we need > some extra work). Efficiency in terms of CPU cycles is a property of particular implementations and hardware. The three proposals that we are considering should not be viewed narrowly in terms of particular implementations. Rather, they should be viewed as definitions of an information transmission standard. We will not be voting on an implementation per se. Rather, we will be voting on an HZ+ file format or coding specification. Once the spec is in place, anybody is free to implement in any way he likes. The most efficient implementation will win in open competition, with no need for a vote. I think Yidao Cai's real concern is the speed of possible conversion between Big5 and HZ+ for very large files. By this criterion, I feel that all three proposals are probably about equal. The particular prototype algorithms that I implemented for HZ+S are at a slight disadvantage, because they use multiplication and division by 94 and 157 (one multiplication and one division for each character), but I feel that this speed disadvantage is not very serious and can anyway be overcome by careful programming (implementing multiplication and division by shifts etc.) ANOTHER CRITERION: BIG5 COMPATIBILITY Beyond the issue of CPU cycles, I feel that it is important to consider other criteria for choosing between the various proposals. The criterion that I and apparently many others regard as most significant is Big5 compatibility. And I have to say that, in a very real sense, my HZ+S proposal is more compatible with Big5 than the others. The reason I say this is that HZ+S is the only one of the three proposals that fully respects the natural division of the Big5 code space into blocks more and less frequently used characters. For those who are not closely familiar with Big5, let me mention that Big5 really consists of two character sets in one. The first block or character set consists of the 5401 most frequently used Chinese characters together with miscellaneous alphabets, numerals, symbols, graphical characters, etc. The second block consists of between 7000 and 8000 less frequently used Chinese characters. The first block suffices for daily use. It's interesting to note that early versions of the popular ETen operating environment implemented only the first block. The two blocks are described in more detail in my document /pub/simpson/chinese/hzps/etfonts.txt, at math.psu.edu. All three of our 7-bit Big5 proposals divide the Big5 code space into two blocks. But HZ+S is the only one of the three that puts the dividing line in the right place, so that the natural structure of Big5 blocks 1 and 2 is fully respected. By this criterion, my proposal is the best of the three, HZ-2 is second best, and B5encode is by far the worst. The fact that HZ+S fully respects the natural structure of the Big5 blocks leads to several significant advantages. These advantages are listed in my document /pub/simpson/chinese/hzps/hzps.txt, at math.psu.edu. To quote from that document: (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. (5). HZ+S uses escape sequences sparingly. The ~2 escape sequence is needed only to switch into Big5 block 2, the block containing the less frequently used Chinese characters. Most of the time, we use only ~1 to switch into Big5 block 1, plus ~{ and ~} as in HZ. (6). HZ+S is fully compatible not only with ETen Big5 but also with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less frequently used Chinese characters begin somewhere prior to C940. Thus HZ+S may help to overcome the difficulties associated with these corrupt variants. (7). The way is open for HZ+S to be compatible with a possible future incorporation of Big5 into ISO-2022-JP-2, similar to the present incorporation of CNS-11643. Such an unofficial extension of ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. CONCLUSION In conclusion, I hope that everybody will weigh the three proposals carefully. We need to consider a variety of criteria, not only efficiency in terms of CPU cycles. Although HZ+S may not have an advantage in terms of CPU cycles, it has many other advantages which make it, in my opinion, more desirable than HZ-2 and B5encode. -- Steve From lai@apple.com Fri May 6 13:39:12 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA19133; Fri, 6 May 94 13:39:12 CDT Received: from mac39.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA23438; Fri, 6 May 94 11:36:47 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 11:36:47 -0700 Message-Id: <9405061836.AA23438@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: CCNET-L@uga.cc.uga.edu From: lai@apple.com (Edmund Lai) Subject: Re: CNS vs Big5 and my vote Cc: soft-authors@ifcss.org Status: O >1. CNS vs Big5 > >There seems to be contradict information regarding the popularity of CNS in >Taiwan. Today I read from acc that CNS is used in all government agencies >and supporting CNS is a must, while on soft-authors one listed a lot of >government agencies that uses Big5 instead of CNS. Another news is that >a net is going to change its exchange code from CNS to Big5. > >To me the main concern is what the end user uses, Big5 or CNS. If what >the end user uses most is Big5, then we should go Big5. We all know that >CNS<->Big5 is not trial, although not that complex, putting this >burden of mapping on MOST end users is not justified if this is just to give >convienence to a small number of CNS users, or simply to let the Taiwan >government feel happy. On the other hand, if CNS is more popular at the >end user, the same logic applies and we should go CNS. > I know I am repeating myself, but we are still not getting the facts straight. We need to distinguish between internal code and exchange code. Currently some of the thing we have are 1. Big5 as internal code 2. Big5 as 8 bit exchange code 3. CNS as 7 bit exchange code 4. Modified CNS as internal code 5. Some form of 7 bit encoded Big5 exchange code, there are several different format proposed here, so this means that there is no such thing as a natural 7 bit Big5, and there is no agreed on 7 bit encoded Big5 yet. Of course this does not mean that it is not possible that in future some form of encoded Big5 can become popular, but this is an unknown. As far as popularity right at this moment, there is no question that they are in the order 1, 2, 3, 4, 5. There is no question that (1) > (3) and (2) > (3), but we are talking about 7 bit code here, so we cannot be using (2). Saying Big5 as internal code is more popular than CNS as internal code does not lead to the conclusion any form of encoded Big5 as exchange code is more popular than CNS as exchange code. The end user will not even be aware of the conversion, this is true whether it is some new form of encoded Big5 or CNS, so how can the user be burdened. We can agree or disagree with certain decision, but we should at least get the reasoning straight. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Fri May 6 13:39:49 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA19199; Fri, 6 May 94 13:39:49 CDT Received: from mac39.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA23481; Fri, 6 May 94 11:37:20 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 11:37:20 -0700 Message-Id: <9405061837.AA23481@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: CCNET-L@uga.cc.uga.edu From: lai@apple.com (Edmund Lai) Cc: soft-authors@ifcss.org Status: O The following is code copied from The Workstation Vertical Integration Project, PDD, III. This is to give an idea of how complex it is to convert between Big5 and CNS. There are two routines, one for each direction. This is really the conversion between Big5 ordering and CNS ordering, so this really represents of overhead beyond conversion between Big5 and some form of encoded Big5. Also this routine is only for characters and not for the symbols, but this give you some idea of the complexity. Basically it is just a 64 element data array plus a few lines of code. I know it says IBM and NSC, not Big5 and CNS, but apparently NSC is just CNS spelled in different order, and judgeing from code not shown IBM is same as Big5. I believe this is because Big5 came from the code for IBM5500, which in turn was based on the Japanese character set for IBM5500. char WORDindx[13] = { 0, 0, 3, 3, 4, 15, 22, 22, 23, 26, 26, 36, 50 }; /***************************************************************************** * * * ... IBM2NSC(order) function * * input>. The sequential No. in IBM Set. * * output>. The sequential No. in NSC Set. * * * *****************************************************************************/ int IBM2NSC(order) int order; { int cnt; static short int SeqIBMtoNSC[64][2] = { { 1, 0}, { 1413, 424}, { 1414, -1}, { 1838, 0}, { 3714, 1}, { 4101, -387}, { 4102, 0}, { 4627, 1}, { 4816, -189}, { 4817, 0}, { 4901, 1}, { 4955, 2}, { 4956, 0}, { 4957, 1}, { 5047, -146}, { 5048, 0}, { 5412,-5378}, { 5413, -1}, { 5446, 0}, { 5494, -49}, { 5495, -1}, { 5541, 0}, { 5708, -168}, { 5709, -1}, { 7658, 637}, { 7659, -2}, { 8297, -1}, { 8696, -201}, { 8697, -2}, {10334, -1}, {10480, 0}, {10767, 1}, {10883, -551}, {10884, 0}, {11021, -542}, {11022, -1}, {11126, 0}, {11348, 366}, {11349, -1}, {11547, -422}, {11548, -2}, {11630, -863}, {11631, -3}, {11717, -2}, {11725, 321}, {11726, -3}, {11879, 310}, {11880, -4}, {11936, -3}, {12049, -2}, {12191, -1}, {12236, -304}, {12237, -2}, {12308, -1}, {12502, -196}, {12503, -2}, {12993, -1}, {13036, -45}, {13037, -2}, {13038, 9}, {13039, -3}, {13050, -2}, {13054, -2} }; if ( !(InRange(0, order, 13053)) ) return(SymOffset); cnt = WORDindx[ order >> 10 ]; while ( order >= (SeqIBMtoNSC[cnt][0]-1) ) cnt++; return( order + SeqIBMtoNSC[cnt-1][1] ); } /***************************************************************************** * * * ... NSC2IBM(order) function * * input>. The sequential No. in NSC Set. * * output>. The sequential No. in IBM Set. * * * *****************************************************************************/ int NSC2IBM(order) int order; { int cnt; static short int SeqNSCtoIBM[62][2] = { { 1, 0}, { 1413, 1}, { 1837, -424}, { 1838, 0}, { 3714, 387}, { 3715, -1}, { 4102, 0}, { 4627, 189}, { 4628, -1}, { 4817, 0}, { 4901, 146}, { 4902, -1}, { 4956, 0}, { 4957, -2}, { 4958, -1}, { 5048, 0}, { 5412, 1}, { 5445, 49}, { 5446, 0}, { 5494, 1}, { 5540, 168}, { 5541, 0}, { 5708, 1}, { 7657, 2}, { 8295, -637}, { 8296, 1}, { 8695, 2}, {10332, 551}, {10333, 1}, {10479, 542}, {10480, 0}, {10767, 863}, {10768, -1}, {10884, 0}, {11021, 1}, {11125, 422}, {11126, 0}, {11348, 1}, {11546, 2}, {11628, 3}, {11714, -366}, {11715, 2}, {11723, 3}, {11876, 4}, {11932, 304}, {11933, 3}, {12046, -321}, {12047, 2}, {12189, -310}, {12190, 1}, {12235, 2}, {12306, 196}, {12307, 1}, {12501, 2}, {12991, 45}, {12992, 1}, {13035, 2}, {13036, 3}, {13047, -9}, {13048, 2}, {13052, 2} }; if ( !(InRange(0, order, 13053)) ) return(SymOffset); cnt = WORDindx[ order >> 10 ]; while ( order >= (SeqNSCtoIBM[cnt][0]-1) ) cnt++; return( order + SeqNSCtoIBM[cnt-1][1] ); } /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From simpson@math.psu.edu Fri May 6 13:56:53 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA20153; Fri, 6 May 94 13:56:53 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA04469; Fri, 6 May 94 11:55:31 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA20536; Fri, 6 May 1994 14:46:39 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 14:46:38 -0400 Date: Fri, 6 May 1994 14:46:38 -0400 Message-Id: <199405061846.OAA11925@boole.math.psu.edu> To: jiang@ifcss.org (Yuan Jiang) Cc: lai@apple.com, simpson@math.psu.edu, ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, soft-authors@ifcss.org Subject: Re: Big5 or CNS ? In-Reply-To: <9405061745.AA16249@ifcss.org> References: <9405061745.AA16249@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > I guess HZ was intended to be an exchange code but isused as internal > code as well. HZ is not much different from GB. Yes. And this suggests that the distinction between internal code and exchange code may not make much sense in the environment where HZ operates, and in which the future HZ+ will have to operate. We are talking about an international 7-bit Internet where operating system software is frequently not Chinese-aware. Conversion between Chinese character coding schemes must be handled at the user level, not the operating system level. In these circumstances, it is difficult if not impossible to hide coding schemes from the user. Therefore, the distinction between exchange code and internal code becomes very unclear. If you or Ed can clarify it, I'd be eager to hear. > >I think it is misleading to describe CNS as an encoding for Big5. The > > Ed is suggesting that we use 7-bit CNS in place of HZ+ to encode > Big5. Let's use words precisely. CNS cannot encode Big5. Both CNS and Big5 can encode Chinese characters. CNS can be translated into Big5, and vice versa. The real question is, how would this work in practice? Can you imagine trying to explain to Big5 users why we chose to base our HZ+ on CNS, an unpopular standard that nobody uses? Why didn't we choose Big5? I think that if I were an ordinary Big5 user, I would conclude that the international Internet community does not care about ordinary users, and that would make me mad as hell. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From CAI@neurophys.wisc.edu Fri May 6 14:09:34 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA20750; Fri, 6 May 94 14:09:34 CDT Received: from VMSmail by neurophys.wisc.edu; Fri, 6 May 94 14:05 CST Message-Id: <24050614051609@neurophys.wisc.edu> Date: Fri, 6 May 94 14:05 CST From: CAI@neurophys.wisc.edu Subject: burden To: soft-authors@ifcss.org X-Vms-To: IN%"soft-authors@ifcss.org" Status: O >From: IN%"lai@apple.COM" 6-MAY-1994 13:49:12.10 The end user will not even be aware of the conversion, this is true whether it is some new form of encoded Big5 or CNS, so how can the user be burdened. We can agree or disagree with certain decision, but we should at least get the reasoning straight. ------ By "burden at the end-user" I meant to be the cpu time it takes to do the conversion. I hope I made it clear now. No matter how efficient it is, it's going to cost some, yet no gain to most users. cai From jiang Fri May 6 14:13:22 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA20890; Fri, 6 May 94 14:13:22 CDT Date: Fri, 6 May 94 14:13:22 CDT From: jiang (Yuan Jiang) Message-Id: <9405061913.AA20890@ifcss.org> Subject: Re: CNS vs Big5 and my vote Cc: soft-authors Status: O I've not expressed my preferance because I've been busy catching up the postings. I've read the proposals carefully. Here are my opinions. The principle of judging these proposals is that we should try not to invent new wheels and try to impose as less difficulty to programmers/users as possible. Although an exchange code is not meant to be used by applications, it would make lives simpler if my current application can use it in at least a simple way (eg. using vi to do a simple cut and paste). CNS: I believe Steve misunderstood what Ed said. He suggests using 7-bit CNS in place of a HZ+. It's 7-bit CNS vs HZ+, but not CNS vs Big5. Comparing with a HZ+, 7-bit CNS is not a new wheel and has a user base (more or less). The conversion with Big5 is not difficult (a page of table). So the complete proposal should be something like: ~{ switch to HZ ~G{ and ~H{ switch to 7-bit CNS ~} to ASCII The drawback is that the CNS<->Big5 is not exactly 1-1. But it's not a big problem. It does not make it more difficult to Big5 users than other HZ+ proposals. I don't know much of CNS. If my crude understanding of it serves me, I would vote for this 7-bit CNS. B5encode: It is the most straight forward 7-bit Big5. I like it. However, my North America bias comes in here -- it maps Big5 to the non-printable ASCII as well and I cannot use my English editor to cut and paste or do some simple editing. It's true that we are addressing an international communication problem, however, it's mostly a communication problem between people in Taiwan and others over the world because people in Taiwan consitutes most of the Big5 users and the largest group of current Internet Chinese users. Their opinion on how they like to communicate with the outside world is important. HZ+S&2: The block partition of HZ+2 makes it less easy to use existing font files than with HZ+S. Conclussion: If the 7-bit CNS is as simple as I think, I would vote for it. If not, I would vote for HZ+S. Yuan . From simpson@math.psu.edu Fri May 6 14:15:41 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA21113; Fri, 6 May 94 14:15:41 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA20781; Fri, 6 May 1994 15:13:32 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 15:13:31 -0400 Date: Fri, 6 May 1994 15:13:31 -0400 Message-Id: <199405061913.PAA11973@boole.math.psu.edu> To: Jianqing Hu Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <9405061847.AA19426@iitmax.acc.iit.edu> References: <199405061450.KAA11513@boole.math.psu.edu> <9405061847.AA19426@iitmax.acc.iit.edu> Reply-To: simpson@math.psu.edu Status: O Jianqing Hu writes: > > I think use HZ+ to encode BIG5 doesn't means it is impossible to > use CNS. > > HZ+ <-> BIG5 > HZ+ <-> BIG5 <-> CNS > > As long as an application programmer like, he could do any conversion > > HZ and HZ+, in my understanding, are kind of protocol to solve > the non 8 bit clean nature of the internet, or some segments of it. > > On the other hand, use a HZ+ base on BIG5 will greatly ease the > development of application program in a short run, especially > considering the fact that there is a large number of commercial > software packages using BIG5 as internal code. > > Jianqing Hu | Internet: thssjyh@iitmax.acc.iit.edu > ECE Dept. Illinois Inst. of Tech. | > Chicago, IL 60616 | Office phone: 312-567-6971 > > I agree completely. HZ and HZ+ are intended as popular standards for transmitting Chinese over the Internet. Their only reason for existence is to solve a specific problem, namely the non 8-bit clean nature of the Internet. We should aim to solve this problem in the simplest possible way. We should aim for immediate wide acceptance. The popular Chinese coding schemes are GB and Big5, so let's stick to them as closely as possible. We should *not* try to solve the problem of mediating between the ROC government and Big5 users. This is a problem for the ROC government and the III. It is not our problem. Big5 rules! -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From jiang Fri May 6 14:28:59 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA21643; Fri, 6 May 94 14:28:59 CDT Date: Fri, 6 May 94 14:28:59 CDT From: jiang (Yuan Jiang) Message-Id: <9405061928.AA21643@ifcss.org> Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, lai@apple.com, shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors Status: O >Let's use words precisely. CNS cannot encode Big5. Both CNS and Big5 >can encode Chinese characters. CNS can be translated into Big5, and >vice versa. If the translation is 1<->1, itmay well be "encode". In this case, it's almost 1<->1. We can either use it as an almost encode of Big5 or try to fix it so that there is a 1<->1 (eg. put the deleted Big5 character back). > >The real question is, how would this work in practice? Can you >imagine trying to explain to Big5 users why we chose to base our HZ+ >on CNS, an unpopular standard that nobody uses? Why didn't we choose A new HZ+* does not make Big5 users easier. From ygz@cs.purdue.edu Fri May 6 14:46:20 1994 Return-Path: Received: from arthur.cs.purdue.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22448; Fri, 6 May 94 14:46:20 CDT Received: from ector.cs.purdue.edu (root@ector.cs.purdue.edu [128.10.2.10]) by arthur.cs.purdue.edu (8.6.4/PURDUE_CS-1.3) with ESMTP id for ; Fri, 6 May 1994 14:44:21 -0500 Received: from localhost (ygz@localhost [127.0.0.1]) by ector.cs.purdue.edu (8.6.4/PURDUE_CS-1.3) with SMTP id for ; Fri, 6 May 1994 14:44:19 -0500 Message-Id: <199405061944.OAA29509@ector.cs.purdue.edu> To: soft-authors@ifcss.org Subject: CNS vs Big5 vs ... Date: Fri, 06 May 1994 14:44:16 -0500 From: ygz@cs.purdue.edu (Yongguang Zhang) Status: O We've seen many proposals and arguments on which is good/bad. But we haven't see any real samples. I don't even know what a CNS article looks like. Here I ask you, whoever proposed an encoding, send a page of Chinese to CCNET (and/or post to alt.chinese.computing). It should be meaningful, and about 20-30 lines. It should have a mixture of GB, Big5 (level 1 and level 2), and English. Translate the paragraph in HZ and append it to your post, so that other programmers can see if they like your new encoding. It would also serve as a "test" to see if the encoding is safe enough to pass through the Internet. Yongguang Zhang ygz@cs.purdue.edu From shin@hermes.ee.nthu.edu.tw Fri May 6 14:47:44 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA22481; Fri, 6 May 94 14:47:44 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182055(2)>; Sat, 7 May 1994 03:50:59 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA13745; Sat, 7 May 94 03:44:02 CST Date: Sat, 7 May 1994 17:44:02 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405061944.AA13745@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Status: O > Reply-To: simpson@math.psu.edu > > Jing-Shin Chang writes: > > > I had posted my 2nd B5encode proposal (a new one) to > > ccnet-l@uga.uga.edu. However, it seems that my posting was not > > broadcasted. (I sent it twice already.) Is there any problem with > > the server? > > I received both of your B5encode proposals. I asked you whether the > 2nd one was intended to replace the 1st one. You answered that it > wasn't, so I appended it to the 1st one, ... No. No. I am not referring to the one. What you called 'the 2nd one' here is a combination of your proposed ESC sequences plus my 8-bit to 7-bit mapping process, which, as you know, is simply to clear appropriate high bits. Since the switching between ~1 and ~2 are expected to be frequent, I had worked out another one, which move the most frequently used chars to ~1 and less frequently used chars to ~2 as in your HZ+S and Ya-Gui's HZ-2. In this NEW proposal, I still use bit-clear for the most time, and only a small fraction of the 8-bit codes are moved to the printable area by adding constant offsets to the high bytes or low bytes for codes belonging to a rectangular area. However, my posting seems to "disappear" when it was sent to ccnet-l@uga.uga.edu. In fact, some of my postings was returned afew days ago. I am wondering if there is any problem to post to the above address. The strange thing is that I can use follow-up to post the mails. > Could you please join soft-authors? That way we won't have to keep > copying everything to you. Also, if you have some additional relevant > material concerning your B5encode proposal, could you please make it > available by anonymous FTP or some other method? I would be glad to > help as needed. Thanks for your forwarding services. I would like to joint soft-authors if it helps. But I don't know how. I will announce it on my ftp later. > > > 1. The SeedNet sysadm had informed its users to change the default > > code from CNS (7bit protocol) to Big5 in email, news from > > 05/09. > > Is this referring to the "internal code" or the "exchange code" (using > Ed's terminology), or both? In any case, I'm very glad to learn that > the SeedNet administrator saw the light and switched from CNS to > Big5. :-)) What were the reasons for the switch? Both. Because they use the exchange code for email and news transfer. And, before their BBS, news readers process them, they are converted back to the internal codes so that other external utilities (like the envoked editor) could process it without being annoyed by the ESC's of the exchange code. (This is not the case for HZ*-style encoding -^^-). The main reason is to meet the trend of our popular Big5 environment. By this, I, of course, mean the 8-bit Big5, because 8-bit clean environment is 99% there. (Should we ask the chinese.* to post in 8-bit GB -^^-. Don't flame on me.) > > > 3. My changes (or strictly speaking, my alternative solutions) are leaning > > toward the users of the newsgroup, who may not have any technical > > background for those encoding stuffs. It is hard to educate > > non-technical users those technical stuffs. > > I'm not sure which "newsgroup" you are referring to. Soft-authors, or > ccnet-l? In any case, I think you will find that people in both > groups are willing and even eager to be educated. Not only that, but > some of us in soft-authors even have some technical background. For > example, many of us have PC clones and have written programs in > GW-BASIC. :-) Seriously, soft-authors is a group consisting of > authors of Chinese software, some of which is widely used. Sorry for the confusion. I mean people may not want to use CNS simply because they are not familiar with that. (I, myself, is an example: I am not familiar with GB/HZ in detail, so I am leaning not to hack them -^^-. I guess other programmers may not be so lazy as I am, but some others may be enev lazier than me. -^^-) > > Could you please explain to us, in simple language of course, the > exact form of CNS that is (or was) used as an exchange code in > SeedNet? What are the escape sequences? And, another question about > SeedNet, where is the translation between the "exchange code" and the > "internal code" carried out? Is this translation done by system > software, e.g. sendmail and nntp daemons, or is it done by user > software, e.g. xrn and mailtool? Please enlighten us. In brief: ESC ( F -- designate the ASCII char set as the G0 plane (F=0x30-0x3f) ESC $ ) 0 -- designate the 1st char plane as the G1 plane (freq. chars) ESC $ * 1 -- designate the 2nd char plane to the G2 plane (less freq.) ESC $ + F -- designate the 3rd - 16th char planes to the G3 char plane (F=0x30-0x3f) (I am not sure which F/final char corresponds to which plane). SI (0x0e i.e., ^N) -- use G0, and lock in that mode (just like our '~}') SO (0x0f i.e., ^O) -- use G1, and lock in that mode ('~1') LS2 (ESC+0x6e) -- use G2, and lock ('~2') LS3 (ESC+0x6f) -- use G3, and lock (not currently used) SS2 (ESC+0x8e) -- use G2, but do not lock in the mode, temp. shift (only effective to the immediately following 2bytes) SS3 (ESC+0x8f) -- use G3, unlock (or shift) mode In Sun's email implementation, 'ESC $ ) 0' is used to designate chinese code (EUC) for Taiwan (and 'A', 'B', 'C' for CJK, respectively). In both Sun's and III's implemantation, you will see printable strings quoted by ^N (SO to freq. chars) and ^O (SI to ASCII), and occasionaly, there is temp. shift to less freq. chars with 'ESC N hi-byte lo-byte' (ESC N == SS2). Some of these information is available in the 'chcode.doc' of your downloaded 'chcode' package or my revision uploaded to ifcss.org. Mostly, the exchange code to internal code conversion take places in the application. This process is somewaht like changing an RTF file into its internal data format when your Microsoft Word read a file. In the example for news reader, the appropeate place is to filter the HZ+S format to a temp file, say, before the editor is invoked for following-up. Under this circumstance, the user can then process it with his favorite editor. Only after all processing are done do we need to convert it back to HZ+S for sending to another hosts. (Reading can be done without a temp file.) In Sun's implementation, it remains in its internal code form when the file is accessed by the mail tool (because the use may want to process the text at that time, and she/he does not want to be disturbed by those escapes or any not text elements.) It is in the exchange format when residing in the spool area (because there will no user access at that time and it is ready for sending to another host.) In summary, the places where it should be converted to the internal codes depends on where the user want to access it as an ordinary file for processing. When is archived or ready for transmission, it should be in interchange code form. > > > 4. The HZ* proposals we are discussing is actually a bad yet popular > > standard, which uses the exchange codes for processing directly. > > HZ isn't perfect, but a lot of us find it quite useful. :-) I agree with that too (-^^-), and I think its contribution overwrite its imperfection. Just to indicate that we could do it better if we could clarify the exchange code characteristics and the internal code characteristics when we modify a shareware for chinese processing purposes. In the application for viewing postings, such imperfection may not be so obvious. > > > Jing-Shin -^^- > > -- Steve > From simpson@math.psu.edu Fri May 6 14:56:58 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22867; Fri, 6 May 94 14:56:58 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA21140; Fri, 6 May 1994 15:53:19 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 15:53:17 -0400 Date: Fri, 6 May 1994 15:53:17 -0400 Message-Id: <199405061953.PAA12038@boole.math.psu.edu> To: Edmund Lai Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Big5 vs CNS in Internet newsgroups In-Reply-To: <199405061856.OAA20633@leibniz.math.psu.edu> References: <199405061856.OAA20633@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > We need to distinguish between internal code and exchange code. Could you please define the terms "internal code" and "exchange code"? It seems to me that the distinction is sometimes blurry, because it is not always possible to hide things from the end user. > The end user will not even be aware of the conversion, this is true > whether it is some new form of encoded Big5 or CNS, so how can the > user be burdened. This may be true in certain workstation environments in Taiwan. It is not true on most of the Internet. In the great majority of Internet mail and news servers, Chinese character conversion is not built into the operating system. If you ask your system administrator to install user-transparent Chinese character conversion software, he will laugh in your face. Under such circumstances, conversion cannot take place at the operating system level. It must take place at the user software level. To hide user software from the user is difficult or impossible. HZ is not invisible to the user. If it were, we would never see any questions like "What is that junk on my screen?" If you think that Chinese character conversion is invisible to the user, ask any user in the US who has installed cxterm in his home directory. CHINESE NEWSGROUPS A sharp distincion between internal code and exchange code may make sense in an environment where the conversion between them is hidden from the user. But that is not the environment in which alt.chinese.text operates. So far as news transport software is concerned, alt.chinese.text is a newsgroup like any other. The transport protocol does not perform any Chinese character conversion. All conversion happens at the user level. This issue is becoming more important with the impending creation of a chinese.* newsgroup hierarchy. The chinese.* hierarchy will operate in the same environment where alt.chinese.text now operates. The organizers of the chinese.* newsgroup hierarchy are looking for a simple, practical way to enable Big5 users in Taiwan to participate. Mirror groups like alt.chinese.text.big5 are one possibility, but they have many disadvantages. Another possibility is a future HZ+ standard incorporating 7-bit Big5. The chinese.* organizers have expressed interest in HZ+. Let's not disappoint them. I don't want to be the one to tell them that HZ+ is based not on Big5 but on an ROC government standard that nobody uses. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Fri May 6 15:03:11 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA23131; Fri, 6 May 94 15:03:11 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA21236; Fri, 6 May 1994 16:01:02 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 16:01:00 -0400 Date: Fri, 6 May 1994 16:01:00 -0400 Message-Id: <199405062001.QAA12056@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: CNS <-> Big5 conversion In-Reply-To: <199405061856.OAA20637@leibniz.math.psu.edu> References: <199405061856.OAA20637@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > Also this routine is only for characters and not > for the symbols, but this give you some idea of the complexity. The symbol conversions make it considerably more complicated than is shown here. > I know it says IBM and NSC, not Big5 and CNS, but apparently NSC is > just CNS spelled in different order, and judgeing from code not > shown IBM is same as Big5. I believe this is because Big5 came > from the code for IBM5500, which in turn was based on the Japanese > character set for IBM5500. IBM 5500 is yet another coding scheme, neither Big5 nor CNS. -- Steve From simpson@math.psu.edu Fri May 6 15:15:33 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA23670; Fri, 6 May 94 15:15:33 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA21376; Fri, 6 May 1994 16:13:24 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 16:13:23 -0400 Date: Fri, 6 May 1994 16:13:23 -0400 Message-Id: <199405062013.QAA12076@boole.math.psu.edu> To: jiang@ifcss.org (Yuan Jiang) Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: CNS vs Big5 and my vote In-Reply-To: <9405061913.AA20890@ifcss.org> References: <9405061913.AA20890@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > Comparing with a HZ+, 7-bit CNS is not > a new wheel and has a user base (more or less). Where is this user base? I have never heard of any user who uses CNS. Can you name one CNS user? Big5 has a user base. CNS has no user base. So far as I can determine, CNS is used only as an exchange protocol, where it is completely invisible to users. This would not be the case in the HZ / HZ+ environment. HZ is not invisible to the user. If it is, then why do we keep hearing questions like "What is that junk on my screen?" However, I appreciate Yuan Jiang's thoughtful remarks. -- Steve From shin@hermes.ee.nthu.edu.tw Fri May 6 15:16:45 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA23700; Fri, 6 May 94 15:16:45 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182056(1)>; Sat, 7 May 1994 04:20:16 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA13868; Sat, 7 May 94 04:13:17 CST Date: Sat, 7 May 1994 18:13:17 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405062013.AA13868@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: CNS <-> Big5 conversion Status: O > From simpson@math.psu.edu Sat May 7 04:01:04 1994 > Subject: CNS <-> Big5 conversion > > Edmund Lai writes: > > > I know it says IBM and NSC, not Big5 and CNS, but apparently NSC is > > just CNS spelled in different order, and judgeing from code not > > shown IBM is same as Big5. I believe this is because Big5 came > > from the code for IBM5500, which in turn was based on the Japanese > > character set for IBM5500. > > IBM 5500 is yet another coding scheme, neither Big5 nor CNS. Just to quote a paragraph from the document for the chcode/ccf package: "For the code sets in question, they are divided into three group. the sets in each group have the same sequential order of word. One group, called 'IBM group', includes the BIG5 set, the IBM5550 set and the IBM Host set, the other, called 'NSC group', the NSC set, the NSC with Protocol set, the EUC set and the TCA set, and another, called 'TEL group', the TEL set." To my best knowledge, the Big5 code was affected by IBM5550 code. But I don't remember much. > > -- Steve > From shin@hermes.ee.nthu.edu.tw Fri May 6 15:32:22 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA24251; Fri, 6 May 94 15:32:22 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182049(1)>; Sat, 7 May 1994 04:35:35 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA13930; Sat, 7 May 94 04:28:45 CST Date: Sat, 7 May 1994 18:28:45 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405062028.AA13930@hermes.ee.nthu.edu.tw> To: soft-authors@ifcss.org Subject: ** New B5encode-2 ** Status: O Sorry for posting this message here. My posting to ccnet-l@uga.uga.edu was blocked or returned due to unknown reasons. Could the ifcss.org sysadm check its mail routes to UTARLVM1.UTA.EDU or WSUVM1.CSC.WSU.EDU. The trouble seems to be there. I have worked out a new B5encode revision. I guess you may be interested in that work. If you could receive this email, please give me a message. The new revision could be accessed at the following ftp: hermes.ee.nthu.edu.tw(140.114.28.27):/shin/b5encode I will try to announce this new revision to soft-authors if you can receive this email. Shin -^^- ==================== > Reply-To: simpson@math.psu.edu > > Jing-Shin Chang writes: > > > I had posted my 2nd B5encode proposal (a new one) to > > ccnet-l@uga.uga.edu. However, it seems that my posting was not > > broadcasted. (I sent it twice already.) Is there any problem with > > the server? > > I received both of your B5encode proposals. I asked you whether the > 2nd one was intended to replace the 1st one. You answered that it > wasn't, so I appended it to the 1st one, ... No. No. I am not referring to the one. What you called 'the 2nd one' here is a combination of your proposed ESC sequences plus my 8-bit to 7-bit mapping process, which, as you know, is simply to clear appropriate high bits. Since the switching between ~1 and ~2 are expected to be frequent, I had worked out another one, which move the most frequently used chars to ~1 and less frequently used chars to ~2 as in your HZ+S and Ya-Gui's HZ-2. In this NEW proposal, I still use bit-clear for the most time, and only a small fraction of the 8-bit codes are moved to the printable area by adding constant offsets to the high bytes or low bytes for codes belonging to a rectangular area. However, my posting seems to "disappear" when it was sent to ccnet-l@uga.uga.edu. In fact, some of my postings was returned afew days ago. I am wondering if there is any problem to post to the above address. The strange thing is that I can use follow-up to post the mails. > Could you please join soft-authors? That way we won't have to keep > copying everything to you. Also, if you have some additional relevant > material concerning your B5encode proposal, could you please make it > available by anonymous FTP or some other method? I would be glad to > help as needed. Thanks for your forwarding services. I would like to joint soft-authors if it helps. But I don't know how. I will announce it on my ftp later. > > > 1. The SeedNet sysadm had informed its users to change the default > > code from CNS (7bit protocol) to Big5 in email, news from > > 05/09. > > Is this referring to the "internal code" or the "exchange code" (using > Ed's terminology), or both? In any case, I'm very glad to learn that > the SeedNet administrator saw the light and switched from CNS to > Big5. :-)) What were the reasons for the switch? Both. Because they use the exchange code for email and news transfer. And, before their BBS, news readers process them, they are converted back to the internal codes so that other external utilities (like the envoked editor) could process it without being annoyed by the ESC's of the exchange code. (This is not the case for HZ*-style encoding -^^-). The main reason is to meet the trend of our popular Big5 environment. By this, I, of course, mean the 8-bit Big5, because 8-bit clean environment is 99% there. (Should we ask the chinese.* to post in 8-bit GB -^^-. Don't flame on me.) > > > 3. My changes (or strictly speaking, my alternative solutions) are leaning > > toward the users of the newsgroup, who may not have any technical > > background for those encoding stuffs. It is hard to educate > > non-technical users those technical stuffs. > > I'm not sure which "newsgroup" you are referring to. Soft-authors, or > ccnet-l? In any case, I think you will find that people in both > groups are willing and even eager to be educated. Not only that, but > some of us in soft-authors even have some technical background. For > example, many of us have PC clones and have written programs in > GW-BASIC. :-) Seriously, soft-authors is a group consisting of > authors of Chinese software, some of which is widely used. Sorry for the confusion. I mean people may not want to use CNS simply because they are not familiar with that. (I, myself, is an example: I am not familiar with GB/HZ in detail, so I am leaning not to hack them -^^-. I guess other programmers may not be so lazy as I am, but some others may be enev lazier than me. -^^-) > > Could you please explain to us, in simple language of course, the > exact form of CNS that is (or was) used as an exchange code in > SeedNet? What are the escape sequences? And, another question about > SeedNet, where is the translation between the "exchange code" and the > "internal code" carried out? Is this translation done by system > software, e.g. sendmail and nntp daemons, or is it done by user > software, e.g. xrn and mailtool? Please enlighten us. In brief: ESC ( F -- designate the ASCII char set as the G0 plane (F=0x30-0x3f) ESC $ ) 0 -- designate the 1st char plane as the G1 plane (freq. chars) ESC $ * 1 -- designate the 2nd char plane to the G2 plane (less freq.) ESC $ + F -- designate the 3rd - 16th char planes to the G3 char plane (F=0x30-0x3f) (I am not sure which F/final char corresponds to which plane). SI (0x0e i.e., ^N) -- use G0, and lock in that mode (just like our '~}') SO (0x0f i.e., ^O) -- use G1, and lock in that mode ('~1') LS2 (ESC+0x6e) -- use G2, and lock ('~2') LS3 (ESC+0x6f) -- use G3, and lock (not currently used) SS2 (ESC+0x8e) -- use G2, but do not lock in the mode, temp. shift (only effective to the immediately following 2bytes) SS3 (ESC+0x8f) -- use G3, unlock (or shift) mode In Sun's email implementation, 'ESC $ ) 0' is used to designate chinese code (EUC) for Taiwan (and 'A', 'B', 'C' for CJK, respectively). In both Sun's and III's implemantation, you will see printable strings quoted by ^N (SO to freq. chars) and ^O (SI to ASCII), and occasionaly, there is temp. shift to less freq. chars with 'ESC N hi-byte lo-byte' (ESC N == SS2). Some of these information is available in the 'chcode.doc' of your downloaded 'chcode' package or my revision uploaded to ifcss.org. Mostly, the exchange code to internal code conversion take places in the application. This process is somewaht like changing an RTF file into its internal data format when your Microsoft Word read a file. In the example for news reader, the appropeate place is to filter the HZ+S format to a temp file, say, before the editor is invoked for following-up. Under this circumstance, the user can then process it with his favorite editor. Only after all processing are done do we need to convert it back to HZ+S for sending to another hosts. (Reading can be done without a temp file.) In Sun's implementation, it remains in its internal code form when the file is accessed by the mail tool (because the use may want to process the text at that time, and she/he does not want to be disturbed by those escapes or any not text elements.) It is in the exchange format when residing in the spool area (because there will no user access at that time and it is ready for sending to another host.) In summary, the places where it should be converted to the internal codes depends on where the user want to access it as an ordinary file for processing. When is archived or ready for transmission, it should be in interchange code form. > > > 4. The HZ* proposals we are discussing is actually a bad yet popular > > standard, which uses the exchange codes for processing directly. > > HZ isn't perfect, but a lot of us find it quite useful. :-) I agree with that too (-^^-), and I think its contribution overwrite its imperfection. Just to indicate that we could do it better if we could clarify the exchange code characteristics and the internal code characteristics when we modify a shareware for chinese processing purposes. In the application for viewing postings, such imperfection may not be so obvious. > > > Jing-Shin -^^- > > -- Steve > === Error Messages === >From @UTARLVM1.UTA.EDU:VM1MAIL@WSUVM1.CSC.WSU.EDU Sat May 7 04:11:29 1994 Date: Fri, 06 May 94 13:08:51 PDT To: <@UTARLVM1.UTA.EDU:shin@HERMES.EE.NTHU.EDU.TW> Subject: mail delivery error Batch SMTP transaction log follows: 220 WSUVM1.CSC.WSU.EDU Columbia MAILER R2.10 ptf000 BSMTP service ready. 050 HELO UTARLVM1.UTA.EDU 250 WSUVM1.CSC.WSU.EDU Hello UTARLVM1.UTA.EDU 050 MAIL FROM:<@UTARLVM1.UTA.EDU:shin@HERMES.EE.NTHU.EDU.TW> 250 <@UTARLVM1.UTA.EDU:shin@HERMES.EE.NTHU.EDU.TW>... sender OK. 050 RCPT TO:<25822873@WSUVM1.BITNET> 250 <25822873@WSUVM1.BITNET>... recipient OK. 050 DATA 354 Start mail input. End with . 554-Mail not delivered to some or all recipients: 554 No such local user: 25822873 050 QUIT 221 WSUVM1.CSC.WSU.EDU Columbia MAILER BSMTP service done. Original message follows: Received: from UTARLVM1.UTA.EDU by WSUVM1.CSC.WSU.EDU (Mailer R2.10 ptf000) with BSMTP id 9615; Fri, 06 May 94 13:08:51 PDT Received: from UTARLVM1 (NJE origin SMTP@UTARLVM1) by UTARLVM1.UTA.EDU (LMail V1 .1d/1.7f) with BSMTP id 3850; Fri, 6 May 1994 15:05:34 -0500 Received: from ifcss.org by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Fri, 06 May 94 15:05:30 CDT Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA22481; Fri, 6 May 94 14:47:44 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182055(2)>; Sat, 7 May 1994 03:50:59 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA13745; Sat, 7 May 94 03:44:02 CST Date: Sat, 7 May 1994 17:44:02 +0800 >From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405061944.AA13745@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From simpson@math.psu.edu Fri May 6 15:42:23 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA24763; Fri, 6 May 94 15:42:23 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA21664; Fri, 6 May 1994 16:40:16 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 16:40:14 -0400 Date: Fri, 6 May 1994 16:40:14 -0400 Message-Id: <199405062040.QAA12123@boole.math.psu.edu> To: chinesenet@uclink.berkeley.edu, ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: A possible committee member In-Reply-To: <9405061952.AA02633@econ.Berkeley.EDU> References: <9405061952.AA02633@econ.Berkeley.EDU> Reply-To: simpson@math.psu.edu Status: O The current chairman of the oversight committee for the chinese.* newsgroup hierarchy writes: > By reading CCNET and alt.chinese.computing, it looks like > a unified coding scheme (or whatever it's called) will take > some time to be agreed upon. I too was hoping for an early consensus, but unfortunately CNS has once again raised its ugly head. I wish the ROC government would stop meddling in the computer industry! -- Steve From simpson@math.psu.edu Fri May 6 15:55:50 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA25396; Fri, 6 May 94 15:55:50 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA21846; Fri, 6 May 1994 16:53:04 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 16:53:03 -0400 Date: Fri, 6 May 1994 16:53:03 -0400 Message-Id: <199405062053.QAA12153@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: sample message in HZ+S In-Reply-To: <199405061944.OAA29509@ector.cs.purdue.edu> References: <199405061944.OAA29509@ector.cs.purdue.edu> Reply-To: simpson@math.psu.edu Status: O Responding to Yongguang Zhang's request, this message is a sample of HZ+S containing mixed GB, Big5, and ASCII. Thanks for that suggestion! First, the GB version of the famous "diagonal" poem. ~{6+7g7wCf4_LR@n~}, ~{pNS%Jf3aU9Et3L~}. ~{SqELUU:#OBHH@a~}, ~{SNWS5GL(K<9J3G~}. ~{P]8:F=Iz1(9zV>~}, ~{HKCqS}NRJ$Mr=p~}. ~{7_Fp<1W71(;*OD~}, ~{GR4}IqV]1i5X4:~}. Note that the above is just our old friend HZ. Now for the Big5 version of the same poem. ~10*5z/[5tE28-,X~}, ~1]P^zCU9R7/[\C$~}. ~1(jOuFz8B&7O`=b~}, ~1B-&NBu(;3)3K2S~}. ~1)35Y(N(n@_;P,6~}, ~1&*(b-@,=@:G]1[~}. ~1Ns:E3+:R@_C_6p~}, ~1'c3"90)}I6)c3O~}. I did this using my big2hzps conversion software, in /pub/simpson/chinese/hzps, at math.psu.edu. The escape sequence "tilde 1" will probably be replaced with something else that looks a little nicer, maybe "tilde [" or "tilde >" or "tilde <". This is a minor detail. Apart from this detail, the HZ+S proposal is completely stable. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Fri May 6 16:04:37 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA25911; Fri, 6 May 94 16:04:37 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA21934; Fri, 6 May 1994 17:02:13 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 17:02:12 -0400 Date: Fri, 6 May 1994 17:02:12 -0400 Message-Id: <199405062102.RAA12173@boole.math.psu.edu> To: pinghua@econ.Berkeley.EDU Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: CNS vs Big5 and my vote In-Reply-To: <199405062038.QAA21643@leibniz.math.psu.edu> References: <199405062038.QAA21643@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Pinghua Young writes: > |> The end user will not even be aware of the conversion, this is true > |> whether it is some new form of encoded Big5 or CNS, so how can the > |> user be burdened. > > I don't know much about the technical issues, but I like what > Ed sez in the above, and I know I am not alone. Naturally you like it. Any user would like the idea of not having to be burdened unnecessarily. But the real question is, which scheme will result in the smallest burden for the user? My view is that Big5 <-> CNS conversion is an extra burden that the user doesn't need. -- Steve From shin@hermes.ee.nthu.edu.tw Fri May 6 16:14:37 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA26352; Fri, 6 May 94 16:14:37 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <181977(1)>; Sat, 7 May 1994 05:18:03 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA14190; Sat, 7 May 94 05:11:14 CST Date: Sat, 7 May 1994 19:11:14 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405062111.AA14190@hermes.ee.nthu.edu.tw> To: soft-authors@ifcss.org Subject: ** New ** B5encode-2: 7-bit Big5 encoding Status: O [B5E2/B5encode-2: A New Proposal for 7-bit Big5 Representation] 1994/05/07, Rev. 1.4 Jing-Shin Chang (shin@hermes.ee.nthu.edu.tw) ================================================================ ** Note: If you have received a posting with the same title but was dated ** 1994/05/05, please discard that version. This revision is better. ** ================================================================ This is a NEW proposal for 7-bit Big5, which will be called B5E2. My previous proposal, which uses the 2nd byte to decide which char plane a code should be designated, and whose mapping process is implemented by clearing high bits, will be called B5E1. The current proposal could be considered a compromise between the extremely simple mapping process, negligible processing load and the overhead in file size. The B5encode.c and B5decode.c for this proposal is attached at the end of this posting. In this new proposal, I move those frequently used chars to the first char plane (~1) and those less or rarely used characters to the 2nd char plane (~2). This convention is the same as in Prof. Simpson's proposal (HZ+S) and Ya-Gui's HZ-2. Therefore, switching between ~1 and ~2 is reduced significantly. And the amount of overhead in file size is the same with the two other proposals. (** About 6% in a test over 1 M bytes of file.) In designing the method for moving the codes to the printable area, I divide the code space of the frequently used chars into appropriate blocks, then move a whole block of codes to the (0x21-0x7e,0x21-0x7e) (printable ASCII) area by adding a constant offset vector to their (x,y) coordinates (that is, the values of the high bytes and the low bytes) in the code space. (Ditto to the code space for the less frequently used chars.) As a result, the codes in the same block can be moved to the printable area simply by clearing high bits (== -0x80) or by adding a constant offset vector to the high bytes and/or low bytes during the 8-bit to 7-bit mapping process. Note that: The major operation for the largest blocks F1, F2, R1 and R2 (see the mapping table) is the bit-clear operation, as in my B5E1 proposal. The computation is thus negligible as the B5E1 proposal does. In summary, you only need to implement 8 (or 7) translation vectors for the 8 (or 7) rectangular regions. (Note, since the 8th block "R3.4" as will be mentioned in the mapping tables contains only two user defined codes, this block can be ignored in real implementation.) When chosing the 'appropriate blocks', the philosophy is to retain the geometry of the blocks of continuous codes and their sizes as closely to their BIG5 origins as possible, and larger blocks are fit to the printable area first. With this strategy, only those less frequently used chars, whose low bytes lie in the 0x40-0x7e range are actually divided somewhat differently from their Big5 origins. (See the mapping tables below.) Since I am using the same escape sequence as proposed by Prof. Simpson, all the advantages of his proposal, except for the 8-bit to 7-bit mapping method, are also inherited from his HZ+S -^^- (Thanks Prof. Simpson). In addition, I think this implementation has almost the same processing efficiency as my B5E1. So ... If easy implementation for soft contributors is the main concern, I vote my previous B5E1; if file size is also an important issue, I vote this B5E2 proposal. For processing efficiency, I vote both -^^-. Your comments are highly appreciated. Jing-Shin -^^- =========================================================== [B5E2] 1994/05/07 Notations: Hi: Hi byte ranges (in Big5 code) Lo: Lo byte ranges (in Big5 code) hi: Hi byte ranges (in B5E2 representation) lo: Lo byte ranges (in B5E2 representation) Hop: operations on Hi bytes (Big5->B5E2) Lop: operations on Li bytes (Big5->B5E2) NOP: no operation CLR: clear the 8th bit -YY: add a negative offset 0xYY /* all numbers are HEX numbers */ Principles: 1. The code space for frequently used characters are divided into appropriate rectangles, and moved to the (lower left corner of) the (0x21-0x7e,0x21-0x7e) code space. Two blocks in the original Big5 code space are retained (F1 and F2). These codes will be designated as the plane-1 codes in the 7-bit version. (Switched with the escape ~1). 2. Less frequently (or rarely) used characters are also divided and moved to the (0x21-0x7e,0x21-0x7e) code space. (6 blocks: R1, R2, R3.1, R3.2, R3.3 and R3.4 are used.) These codes will be designated as the plane-2 codes in the 7-bit version. (Switched with the escape ~2). 3. Large blocks are moved first. 4. Use only bit set/clear for moving the blocks whenever possible. 5. Add/Subtract an offset to the high bytes and/or low bytes to move the block if bit set/clear cannot be used to move the blocks to the appropriate code space. The mapping tables are shown as follows: Frequently Used Chars: (geometries are retained) High Bytes a1 c8 L 40 +---------+ o | F2 | w | | 7e +---------+ B a1 +---------+ +---------+ 21 y | F1 | => | F1 | t | | | | 40 +---------+ e | | | | | F2 | s | | | | | | fe +---------+ +---------+ 7e +---------+ 21 48 49 70 ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- F1 a1-c8 a1-fe CLR CLR 21-48 21-7e F2 a1-c8 40-7e -58 NOP 49-70 40-7e ---------------------------------------------------------- Rarely Used Chars : c9 f0 f1 fe 40 +------------------+ +------+ 40 | | |R3.1 | | | | | | | +------+ 5e | R2 | +------+ 5f | | |R3.2 | | | | | | | +------+ 7d 7e +------------------+ +----+++ <- R3.3 + R3.4 ( 0xfd7e, 0xfe7e ) f1 fc fd fe a1 +---------------------------+ | | | | | | | | | | | R1 | | | | | | | | | | | fe +---------------------------+ c9 fe | | V 21 2e 2f 3c 3d 48 21 +------++------++--+ 21 +---------------------------+ 21 (+--+ : R3.3) |R3.1 ||R3.2 |++ 22 | | 22 (++ : R3.4) | || | 3e | | 3f +------++------+ 3f | | 40 +------------------+ | | | | | | | | | R1 | | | | | | R2 | | | | | | | | | | | | | | | 7e +------------------+ +---------------------------+ 21 48 49 7e ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- R1 c9-fe a1-fe CLR CLR 49-7e 21-7e R2 c9-f0 40-7e -a8 NOP 21-48 40-7e R3.1 f1-fe 40-5e -d0 -1f 21-2e 21-3f R3.2 f1-fe 5f-7d -c2 -3e 2f-3c 21-3f R3.3 f1-fc 7e -b4 -5d 3d-48 21 R3.4 fd-fe 7e -c0 -5c 3d-3e 22 ---------------------------------------------------------- =========================================================== === Makefile === # # @(#)$Id: Makefile,v 1.1 1994/05/04 10:39:09 shin Exp $ # SRCS = b5encode.c b5decode.c OBJS = b5encode.o b5decode.o HDRS = hz+.h PROG = b5encode b5decode all: $(PROG) $(PROG): $(@:=.c) $(HDRS) cc -o $@ $(@:=.c) # === hz+.h === /* * @(#)$Id: hz+.h,v 1.2 1994/05/05 15:41:34 shin Exp $ */ #ifndef _hz+_h_ #define _hz+_h_ #define HZ_ESC '~' #define HZ_SI '}' #define HZ_SO_B5_1 '1' #define HZ_SO_B5_2 '2' #define HZ_SO_HZ '{' #define HZ_B5_LS1 HZ_SO_B5_1 #define HZ_B5_LS2 HZ_SO_B5_2 #define HZ_HZ_LS1 HZ_SO_HZ #define MODE_SO_ASC 0 #define MODE_SO_B5_1 1 #define MODE_SO_B5_2 2 #define MODE_SO_HZ 3 /* other encoding schemes */ #define ESC 033 /* 0x2b */ #define SO 016 /* 0x0e */ #define SI 017 /* 0x0f */ #define ALE_LANG HLE /* default character set for Chinese (T) */ #define HLE '0' /* Chinese/Taiwan */ #define CLE 'A' /* Chinese/Mainland */ #define JLE 'B' /* Japanese */ #define KLE 'C' /* Korean */ #define CNS_SS2 0x4e /* ESC 'N' -- to 2nd plane */ #define CNS_SS3 0x4f /* ESC 'O' */ #define EUC_SS2 0x8e /* EUC_SS2 0xa2-b0 for plane 2-16 */ #define EUC_SS3 0x8f #define within(lo,x,hi) (lo<=x&&x<=hi) #define b5cvt(hi,lo,h0,h1,l0,l1,hop,lop) \ \ if ( within(h0,hi,h1) && within(l0,lo,l1) ) { \ hi hop; \ lo lop; \ return; \ } #endif /* * History: * * $Log: hz+.h,v $ * Revision 1.2 1994/05/05 15:41:34 shin * 1. add within() macro * 2. add b5cvt() macro * * Revision 1.1 1994/05/04 07:24:23 shin * Initial revision * */ === b5encode.c (rev. 1.4) (for B5E2) === /* * @(#)$Id: b5encode.c,v 1.4 1994/05/06 16:22:27 shin Exp $ * @(#) encode of a BIG5 text to 7-bit stream */ #include #include "hz+.h" #define STAT_ASC 0 #define STAT_B5_C1L 1 #define STAT_B5_C1H 2 #define STAT_B5_C0 3 #define STAT_B5_C1 4 /* * We can actually set the upper limit down to 0xf9 so that we will not regard * characters of the form `~X' as Big5 char. This could avoid ambiguity * with ocassional `B5B5B5B5~}' stuffs in a big5 file, whose trailing '~}' * is left by an inperfect HZ to BIG5 program. */ /* #define is_high(x) (x>=0xa1&&x<=0xfe) #define is_high(x) (x>=0xa1&&x<0xfe) */ #define is_high(x) (x>=0xa1&&x<=0xf9) #define is_low(x) (!is_high(x)) main() { int state = STAT_ASC; int so = MODE_SO_ASC; int c, c0; /* current and 1st byte */ while ( (c = getchar() ) != EOF ) { switch (state) { case STAT_ASC: if ( is_high(c) ) { c0 = c; state = STAT_B5_C0; } else { if ( so ) { putchar(HZ_ESC); putchar(HZ_SI); so = MODE_SO_ASC; } if ( c == HZ_ESC ) /* '~' => '~~' */ putchar(c); putchar(c); } break; case STAT_B5_C0: b5e(&c0,&c,&so); putchar(c0); putchar(c); state = STAT_B5_C1; break; case STAT_B5_C1: if ( is_high(c) ) { c0 = c; state = STAT_B5_C0; } else { putchar(HZ_ESC); putchar(HZ_SI); putchar(c); so = MODE_SO_ASC; state = STAT_ASC; } break; default: break; } /* switch */ } /* while */ } b5e(hi,lo,so) int *hi, *lo; int *so; { int so0; /* current SO mode */ so0 = ( *hi < 0xc9 ? MODE_SO_B5_1 : MODE_SO_B5_2); if ( so0 != *so ) { putchar(HZ_ESC); putchar( (so0==MODE_SO_B5_1? HZ_B5_LS1 : HZ_B5_LS2) ); *so = so0; } b5cvt(*hi,*lo,0xa1,0xc8,0xa1,0xfe,&=0x7f,&=0x7f); /* F1 */ b5cvt(*hi,*lo,0xa1,0xc8,0x40,0x7e,-=0x58,|=0x00); /* F2 */ b5cvt(*hi,*lo,0xc9,0xfe,0xa1,0xfe,&=0x7f,&=0x7f); /* R1 */ b5cvt(*hi,*lo,0xc9,0xf0,0x40,0x7e,-=0xa8,|=0x00); /* R2 */ b5cvt(*hi,*lo,0xf1,0xfe,0x40,0x5e,-=0xd0,-=0x1f); /* R3.1 */ b5cvt(*hi,*lo,0xf1,0xfe,0x5f,0x7d,-=0xc2,-=0x3e); /* R3.2 */ b5cvt(*hi,*lo,0xf1,0xfc,0x7e,0x7e,-=0xb4,-=0x5d); /* R3.3 */ b5cvt(*hi,*lo,0xfd,0xfe,0x7e,0x7e,-=0xc0,-=0x5c) /* R3.4 */ } /* * History: * * $Log: b5encode.c,v $ * Revision 1.4 1994/05/06 16:22:27 shin * 1. rearranging the blocks of codes to further reduce computation cost * * Revision 1.3 1994/05/05 15:43:08 shin * 1. implement B5E2 proposal * 2. add b5e() * * Revision 1.2 1994/05/04 10:33:16 shin * 1. change ESC seq to HZ+S proposal * * Revision 1.1 1994/05/01 19:01:15 shin * Initial revision * */ === b5decode.c (rev. 1.4) (for B5E2) === /* * @(#)$Id: b5decode.c,v 1.4 1994/05/06 16:26:37 shin Exp $ * @(#) decode of a BIG5 text from 7-bit stream encoded by b5encode */ #include #include "hz+.h" #define STAT_ASC 0 #define STAT_HZ_ESC 1 #define STAT_HZ_SO 2 #define STAT_HZ_C0 3 #define STAT_HZ_C1 4 #define STAT_HZ_AMB 5 /* * the 'dst' parameter is reserved for converting directly from the source to * a destination code. (the so is used for source code) * - Big5/7 => Big5 is implemented in the following macro. */ #define hzPdecode(c0,c,so,dst) \ if ( so != MODE_SO_HZ ) { \ putchar(c0|0x80); \ putchar((so == MODE_SO_B5_2)? c : (c|0x80)); \ } \ else { /* hook up your hz2dst(c0,c) here */ \ putchar(c0); \ putchar(c); \ } \ main() { int state = STAT_ASC; int so = 0; int c, c0; int dst; /* destination code * reserved */ while ( (c = getchar() ) != EOF ) { switch (state) { case STAT_ASC: if ( c == HZ_ESC ) state = STAT_HZ_ESC; else putchar(c); break; case STAT_HZ_ESC: switch ( c ) { case HZ_SO_HZ: so = MODE_SO_HZ; state = STAT_HZ_SO; break; case HZ_SO_B5_1: so = MODE_SO_B5_1; state = STAT_HZ_SO; break; case HZ_SO_B5_2: so = MODE_SO_B5_2; state = STAT_HZ_SO; break; case HZ_SI: state = STAT_ASC; break; default: putchar(c); state = STAT_ASC; break; } break; case STAT_HZ_SO: case STAT_HZ_C1: c0 = c; if ( c == HZ_ESC ) state = STAT_HZ_AMB; else state = STAT_HZ_C0; break; case STAT_HZ_C0: /* hzPdecode(c0,c,so,dst); */ b5d(&c0,&c,so,dst); putchar(c0); putchar(c); state = STAT_HZ_C1; break; case STAT_HZ_AMB: switch ( c ) { case HZ_SO_HZ: so = MODE_SO_HZ; state = STAT_HZ_SO; break; case HZ_SO_B5_1: so = MODE_SO_B5_1; state = STAT_HZ_SO; break; case HZ_SO_B5_2: so = MODE_SO_B5_2; state = STAT_HZ_SO; break; case HZ_SI: state = STAT_ASC; so = MODE_SO_ASC; break; default: /* hzPdecode(c0,c,so,dst); */ b5d(&c0,&c,so,dst); putchar(c0); putchar(c); state = STAT_HZ_SO; } break; default: break; } /* switch */ } /* while */ } b5d(hi,lo,so,dst) int *hi, *lo; int so; int dst; { if ( so == MODE_SO_B5_1 ) { /* ~1 */ b5cvt(*hi,*lo,0x21,0x48,0x21,0x7e,|=0x80,|=0x80); /* F1 */ b5cvt(*hi,*lo,0x49,0x70,0x40,0x7e,+=0x58,|=0x00); /* F2 */ } else if ( so == MODE_SO_B5_2 ) { /* ~2 */ b5cvt(*hi,*lo,0x49,0x7e,0x21,0x7e,|=0x80,|=0x80); /* R 1 */ b5cvt(*hi,*lo,0x21,0x48,0x40,0x7e,+=0xa8,|=0x00); /* R 2 */ b5cvt(*hi,*lo,0x21,0x2e,0x21,0x3f,+=0xd0,+=0x1f); /* R 3.1 */ b5cvt(*hi,*lo,0x2f,0x3c,0x21,0x3f,+=0xc2,+=0x3e); /* R 3.2 */ b5cvt(*hi,*lo,0x3d,0x48,0x21,0x21,+=0xb4,+=0x5d); /* R 3.3 */ b5cvt(*hi,*lo,0x3d,0x3e,0x22,0x22,+=0xc0,+=0x5c); /* R 3.4 */ } /* other non-Big5 modes ... ~{ or ~} */ } /* * History: * * $Log: b5decode.c,v $ * Revision 1.4 1994/05/06 16:26:37 shin * 1. rearranging the blocks of codes to further reduce computation cost * * Revision 1.3 1994/05/05 15:44:59 shin * 1. implement b5d() for B5E2 proposal * * Revision 1.2 1994/05/04 10:31:43 shin * 1. support b5encode mapping scheme for HZ+S proposal * * Revision 1.1 1994/05/01 19:02:46 shin * Initial revision * */ ===== END ====== From shin@hermes.ee.nthu.edu.tw Fri May 6 17:30:14 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA29770; Fri, 6 May 94 17:30:14 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <181978(2)>; Sat, 7 May 1994 06:33:34 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA14403; Sat, 7 May 94 06:26:44 CST Date: Sat, 7 May 1994 20:26:44 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405062226.AA14403@hermes.ee.nthu.edu.tw> To: mcevilly@netcom.com, soft-authors@ifcss.org Subject: Re: CNS and ETen Status: O > From mcevilly@netcom.com Sat May 7 01:53:48 1994 > Subject: Re: CNS and ETen > > Jing-Shin Chang wrote: > > There are email implementation in Sun's ALE environment which use the > > escape sequence: "ESC $ ) 0" to designate chinese codes used in Taiwan. > > Your message and an earlier message from Ross Paterson helped me > figure out that CNS uses this sequence to enter plane 1. When I > entered this escape sequence into a CNS file under ETen, however, > ETen did not recognize the sequence. It's possible I was doing > something wrong, and it's also possible that ETen's support of > CNS is not as good as it could be. Sorry for the confusion. What I had mentioned is for email transmission. That is, the "exchange code" representation of the CNS character set. When we are composing a file (or an email), we are using the "internal code" with the editor. Such codes is what your ETen terminal send to your editor or word processor (like a Chinese Word for Windows.) When the mail is about to be transmmitted to the other hosts, the mailer will convert it to the 7-bit stream with appropriate ESC sequence for indicating which language it was created and which char plane of that language it is used in a particular string. The 7-bit stream is then used for transmission over the network. Such 7-bit streams is the exchange codes for the CNS character set. The internal code for CNS (more exactly, CNS-11643/86) is represented by seting the high bits of both data bytes of of the exchange code if the corresponding char belongs to the 1st plane; if the char belongs to the second plane, the high bit of the first byte of the exchange code is set but the high bit of the 2nd byte is not changed. For instance, the chinese character "one" has the CNS (exchange) code of 0x4421 (which are two printable ASCII chars), and it is commonly used char in the 1st plane. Therefore, its internal code is 0x4421&0x8080=0xc4a1. When you enter the chinese char "one" with whatever input method, in the CNS mode, ETen will send 0xc4a1 to your application (e.g., editor or wordprocessor), and these applications will regard these two bytes as being representaion "one". When you send email, the mailer may adding some ESC chars around 0x4421 to indicate the above mentioned char information. To test whether ETen support the full CNS (internal code), you could generate a table: for (i=0x21; i<= 0x7e; i++) for (j=0x21; j<=0x7e; j++) { fprintf(stdout,"%c%c ", i &0x80, j &0x80); fprintf(stdout,"%c%c ", i &0x80, j); } And inspect such a file in CNS mode (etctl/c:si1,1), and check whether the chars you see on the screen agree with your code book. (Of course, many of the (i,j) pairs in the above combination does not produce legal CNS code. > > Carlos McEvilly | email for info on Bamboo Helper Chinese text > mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly > From lai@apple.com Fri May 6 18:27:53 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA01947; Fri, 6 May 94 18:27:53 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA02378; Fri, 6 May 94 16:25:44 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 16:25:44 -0700 Message-Id: <9405062325.AA02378@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: Big5 or CNS ? Status: O > > I second to Ed Lai's approach - use 7 bit HZ+ to encode CNS for the > > communication, such as email and news. User end software can be in > > any encoding, CNS (only need 7bit to 8bit conversion), Big5 (7bit to > > 8bit, then map to Big5), and GB (7bit to 8bit, then CNS2GB mapping). > >I'm not clear how this proposal would work in practice. Could you >please explain? Would the CNS codes be totally hidden from all users? > Let us take the above statement and change it from CNS to encoded Big5 I second to Stephen G Simpson's approach - use 7 bit HZ+ to encode Encoded Big5 for the communication, such as email and news. User end software can be in any encoding, Big5 (only need decoding), CNS(decode to Big5, then map to CNS), and GB (decode to Big5, then Big52GB mapping). I'm not clear how this proposal would work in practice. Could you please explain? Would the encoded Big5 codes be totally hidden from all users? What are some of the arguments against CNS as an exchange code. CNS is not as popular as Big5, but so is case for the so called encoded Big5. People would wonder what the hell are the characters in the document when they see the CNS code in an undecoded document, but so is the case for the so called encoded Big5. Almost every one of the reason favoring the so called encoded Big5 can be turned around by switching the place between CNS and the so called encoded Big5. There are only two valid argument favoring CNS or encoded Big5. Against CNS: It takes less cycles to encode Big5 into Encoded Big5. For CNS: It is already an established 7 bit standard, at least it is established compared to the so called encoded Big5. This is a judgement call. I keep hear burdening the user with all these extra cycles. When the Mac first come up, that is what they say too. If you write a single character to character terminal, this is one cycle, but the Mac choose to "burden" the user by drawing it as a graphic, what a waste of cycle. Now you do it in every low end system. Who cares if it takes one hundred time more cycles if the user does not even notice it. > > > That's one of the reasons why most of UNIX vendors support CNS instead of > > Big5 (another reason is the technical difficulty to fit Big5 into EUC > > scheme). > >What technical difficulty? I think the difficulty is not technical >but political, namely the government's insistence on CNS. > If the only reason is only political, then how come only workstation vendor yields but not the PC vendors? /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Fri May 6 18:36:36 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA02349; Fri, 6 May 94 18:36:36 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA03239; Fri, 6 May 94 16:32:46 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 16:32:46 -0700 Message-Id: <9405062332.AA03239@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: Big5 or CNS ? Status: O >We are talking about an international 7-bit Internet where operating >system software is frequently not Chinese-aware. Conversion between >Chinese character coding schemes must be handled at the user level, >not the operating system level. In these circumstances, it is >difficult if not impossible to hide coding schemes from the user. >Therefore, the distinction between exchange code and internal code >becomes very unclear. If you or Ed can clarify it, I'd be eager to >hear. > If you have software that does the automatic decoding then it would be transparent to you whether you use CNS or encoded Big5. If you don't have software that does it automatically, then it would not be transparent to you and you would see something that look like garbage, but that would be true whether you use CNS or encoded Big5. So in this aspect CNS and encoded Big5 is completely even, and this is no reason to favored encoded Big5 over CNS. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Fri May 6 18:41:36 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA02602; Fri, 6 May 94 18:41:36 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA03864; Fri, 6 May 94 16:39:20 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 16:39:20 -0700 Message-Id: <9405062339.AA03864@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: Big5 or CNS ? Status: O >Yuan Jiang writes: > > I guess HZ was intended to be an exchange code but isused as internal > > code as well. HZ is not much different from GB. > >Yes. And this suggests that the distinction between internal code and >exchange code may not make much sense in the environment where HZ >operates, and in which the future HZ+ will have to operate. > HZ in an exchange code. This does not mean that exchange code cannot be used as an internal code. RTF is an exchange code, PostScript is an exchange code, but occasionally they are used as internal code. >Yes. And this suggests that the distinction between internal code and >exchange code may not make much sense in the environment where HZ >operates, and in which the future HZ+ will have to operate. You mean that people is going to start using encoded Big5 as internal code? I doubt that. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From mcevilly@netcom.com Fri May 6 18:46:19 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA02723; Fri, 6 May 94 18:46:19 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id QAA04013; Fri, 6 May 1994 16:45:19 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405062345.QAA04013@netcom.com> Subject: Re: CNS <-> Big5 conversion To: soft-authors@ifcss.org Date: Fri, 6 May 1994 16:45:16 -0700 (PDT) Cc: ccnet-l@uga.bitnet In-Reply-To: <199405062328.QAA17651@mail2.netcom.com> from "Jing-Shin Chang" at May 7, 94 06:13:17 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 678 Status: O > > IBM 5500 is yet another coding scheme, neither Big5 nor CNS. > > Just to quote a paragraph from the document for the chcode/ccf package: > "For the code sets in question, they are divided into three group. > the sets in each group have the same sequential order of word. One > group, called 'IBM group', includes the BIG5 set, the IBM5550 set Just to prevent any possible confusion, it is true that IBM 5500 code is quite different from Big5. Therefore a table that serves to convert CNS<-->IBM code is useless for CNS<-->Big5. Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From mcevilly@netcom.com Fri May 6 19:02:34 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA03394; Fri, 6 May 94 19:02:34 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id RAA05882; Fri, 6 May 1994 17:01:19 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405070001.RAA05882@netcom.com> Subject: Big5 <--> 7-bit Big5 is simple To: ccnet-l@uga.bitnet Date: Fri, 6 May 1994 17:01:18 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405062325.AA02378@apple.com> from "Edmund Lai" at May 6, 94 04:25:44 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 125 Status: O > What are some of the arguments against CNS as an exchange code. 1. unpopular 2. unsupported 3. unfamiliar 4. unnecessary From shin@hermes.ee.nthu.edu.tw Fri May 6 19:16:28 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA04005; Fri, 6 May 94 19:16:28 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182030(2)>; Sat, 7 May 1994 08:19:58 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA14517; Sat, 7 May 94 08:13:08 CST Date: Sat, 7 May 1994 22:13:08 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405070013.AA14517@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility Status: O Sorry that my ** New ** B5encode can not be posted to ccnet-l due to some unknown mailer problems. Under such circumstances, people may not compare my first B5encode proposal, hereinafter the [B5E1] proposal, and the new revision: [B5E2], with the other proposals. You can access my new proposal at: hermes.ee.nthu.edu.tw(140.114.28.27):/shin/b5encode In summary, my new proposal also divide the character planes according to whether the characters are frequently used or less frequently used. The frequently used chars in (Hi: 0xa1-0xc8, Lo: 0x40-0x7e) and (Hi: 0xa1-0xc8, Lo: 0xa1-0xfe) are moved to the first plane (~1), and those less frequently used chars in (Hi: 0xc9-0xfe, Lo: 0x40-0x7e) and (Hi: 0xc9-0xfe, Lo: 0xa1-0xfe) are moved to (~2). This is the same as in the HZ+S and HZ-2. Therefore, the frequency of switching from ~1 to ~2 are the same as the other two proposals. Furthermore, the computation efficiency is still retained since most characters can be moved to the printable area by simple bit-clear operations. This is done by dividing the code space of the frequently used chars into two blocks, each blocks can be shifted, AS A WHOLE, by clearing the high bits or by adding a constant offset vector to the "coordinates" (i.e., the high byte values and the low byte values) of the blocks of codes. (Note: clear the high bit is equivalent to subtracting 0x80 in the x/y direction in the code space.) The less frequently used char area is moved to the 2nd plane in the same manner. Therefore, each code can be transformed al most by using two additions. (And only 8 offset vectors are required for such translation operations for 8 blocks of codes. Since one of the blocks contains only two codes in the User-Defined char area, we can actually discard this block.) In addition, lager blocks are moved first, and when bit-clearing is OK for shifting the codes, such an operation is adopted. Furthermore, the geometry and size of the code are retained since the subblocks are moved as a whole without changing the relative positions of the codes. Therefore, this implementation both save processing time and file size. At the same time, the relative positions of the codes are preserved. I guess this mapping process could be very useful. I personally does not have any strong prefernce over the ESC sequences. In my implementation, the ESC sequence proposed by Prof. Simpson is used. However, you can change the header file to a preferred sequence without much difficulty. Both of my proposals [B5E1] and [B5E2] has their relative advantages, so I would like to hear comments to these two proposals. Any comments are welcomed and highly appreciated. Shin -^^- === a few comments on the follwoing posting === > From simpson@math.psu.edu Sat May 7 02:19:57 1994 > Reply-To: simpson@math.psu.edu > > ... deleted ... > > EFFICIENCY IN TERMS OF CPU CYCLES > > CAI@neurophys.wisc.edu writes: > > 2. For all three proposals, I admire B5encode for simplicity (I original > > thought both Ya-Gui and Simpson was using this kind of mapping). But the > > proposals of Ya-Gui and Simpson could reduce the switching between block 1 > > and block 2. My final vote give to the one that is most efficient in terms My B5E2 can do it now. So HZ+S, HZ-2 and B5E2 are as good as one another in this regard. > > of total CPU time when both the switching and mapping are considered > > (anybody can make an estimate? Has Ross done one? Or we can do an actual > > test on news.test or similar groups before we finalize, but then we need > > some extra work). > > Efficiency in terms of CPU cycles is a property of particular > implementations and hardware. The three proposals that we are > considering should not be viewed narrowly in terms of particular > implementations. Rather, they should be viewed as definitions of an > information transmission standard. We will not be voting on an > implementation per se. Rather, we will be voting on an HZ+ file > format or coding specification. Once the spec is in place, anybody is > free to implement in any way he likes. The most efficient The coding specification, however, is just the mapping methods proposed in all our previous discussions. > implementation will win in open competition, with no need for a vote. > > I think Yidao Cai's real concern is the speed of possible conversion > between Big5 and HZ+ for very large files. By this criterion, I feel > that all three proposals are probably about equal. The particular > prototype algorithms that I implemented for HZ+S are at a slight > disadvantage, because they use multiplication and division by 94 and > 157 (one multiplication and one division for each character), but I > feel that this speed disadvantage is not very serious and can anyway > be overcome by careful programming (implementing multiplication and > division by shifts etc.) > I also indicated such possibility when Ya-Gui posted his coments on HZ+S's current implementation. It's true that the current HZ+S implementaion could be optimized. But I think an addition or bit operation has the best achievable efficiency. -^^- > > ANOTHER CRITERION: BIG5 COMPATIBILITY > > Beyond the issue of CPU cycles, I feel that it is important to > consider other criteria for choosing between the various proposals. > The criterion that I and apparently many others regard as most > significant is Big5 compatibility. And I have to say that, in a very > real sense, my HZ+S proposal is more compatible with Big5 than the > others. > > The reason I say this is that HZ+S is the only one of the three > proposals that fully respects the natural division of the Big5 code > space into blocks more and less frequently used characters. > > For those who are not closely familiar with Big5, let me mention > that Big5 really consists of two character sets in one. The first > block or character set consists of the 5401 most frequently used > Chinese characters together with miscellaneous alphabets, numerals, > symbols, graphical characters, etc. The second block consists of > between 7000 and 8000 less frequently used Chinese characters. The > first block suffices for daily use. It's interesting to note that > early versions of the popular ETen operating environment implemented > only the first block. The two blocks are described in more detail in > my document /pub/simpson/chinese/hzps/etfonts.txt, at math.psu.edu. > > All three of our 7-bit Big5 proposals divide the Big5 code space into > two blocks. But HZ+S is the only one of the three that puts the > dividing line in the right place, so that the natural structure of > Big5 blocks 1 and 2 is fully respected. By this criterion, my > proposal is the best of the three, HZ-2 is second best, and B5encode > is by far the worst. In my B5E2, the Freq. used & Less Freq. used division is also fully respected. In addition, the relative code positions are retained. Further more, in a strict mathematical sense, the "distance" between the original code and the mapped code has a constant of two (or even one). So I think, the criteria of "natural division" and "big5 compatibility" of the codes in my B5E2 proposal is competitive with the other proposals. -^^- > > The fact that HZ+S fully respects the natural structure of the Big5 > blocks leads to several significant advantages. These advantages are > listed in my document /pub/simpson/chinese/hzps/hzps.txt, at > math.psu.edu. To quote from that document: > > (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. > To use exsisting fonts, you may have to add an appropriate offset as an index to the font file. I think both of my B5E1 and B5E2 also have the same simplicity to do that. However, the current discussion is to have a good exchange code; therefore, the relation with the font may not be so closely related. > (5). HZ+S uses escape sequences sparingly. The ~2 escape sequence is > needed only to switch into Big5 block 2, the block containing the less > frequently used Chinese characters. Most of the time, we use only ~1 > to switch into Big5 block 1, plus ~{ and ~} as in HZ. The number of switching for any algorithm that maps the Freq. and Less Freq. words into two different planes will be the same for any input file. Presently, HZ+S, HZ-2, and B5E2 all have the same power in this aspect. My privite testing over 1M bytes of chinese file show that the file size is increased by about 6%, which is attributed to the ESC chars. The B5E1 implementation do have a larger overhead in file size. > > (6). HZ+S is fully compatible not only with ETen Big5 but also with > corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less > frequently used Chinese characters begin somewhere prior to C940. > Thus HZ+S may help to overcome the difficulties associated with these > corrupt variants. Sorry, I am not familiar with such a problem. Could you give an example? > > (7). The way is open for HZ+S to be compatible with a possible future > incorporation of Big5 into ISO-2022-JP-2, similar to the present > incorporation of CNS-11643. Such an unofficial extension of > ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. > I have posted an article which gives some comments on the meanings of ISO-compatibility. Interested people could check it in alt.chinese.computing. In brief, ISO-compatibility is better defined by (1) do not use non-control chars as ESC chars. (2) follow the general syntax of the ESC sequences for designating a character set or a char. plane in that set. So, as long as we agree on the same ESC sequence, we can reach the same degree of compatibility. We can therefore regard this task and the mapping task as two independent tasks. > > CONCLUSION > > In conclusion, I hope that everybody will weigh the three proposals > carefully. We need to consider a variety of criteria, not only Now, we have four proposals. > efficiency in terms of CPU cycles. Although HZ+S may not have an > advantage in terms of CPU cycles, it has many other advantages which > make it, in my opinion, more desirable than HZ-2 and B5encode. > > -- Steve > I am personally very appreciate of Prof. Simpson's initiation and consideration of this very interesting and very important work. So I tried very hard (really !!-^^-) to propose my alternatives and hope that they are as competitive as the other proposals. Probably there are other persons who have better ideas. Would you netters like to contribute better alternatives? Jing-Shin > From lai@apple.com Fri May 6 19:37:20 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA04718; Fri, 6 May 94 19:37:20 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA10166; Fri, 6 May 94 17:35:06 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 17:35:06 -0700 Message-Id: <9405070035.AA10166@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: CCNET-L@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: Big5 <--> 7-bit Big5 is simple Status: O I can play the same game too. >> What are some of the arguments against encoded Big5 as an exchange code. > >1. unpopular >2. unsupported >3. unfamiliar >4. unnecessary /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Fri May 6 19:54:13 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA05379; Fri, 6 May 94 19:54:13 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA11934; Fri, 6 May 94 17:52:09 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 17:52:09 -0700 Message-Id: <9405070052.AA11934@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: Big5 or CNS ? Status: O > > No, for a user with BIG5 software, they will expect a software that > somehow let their BIG5 package usefule, certain kind of filter. > > If CNS is used, the filter will be very complicated and the speed > will be slow. The big difference is here. > I have posted code from Taiwan that was in pulbic domain, it can be considered to be clever but hardly complicated. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Fri May 6 19:58:10 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA05508; Fri, 6 May 94 19:58:10 CDT Received: from mac103.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA12243; Fri, 6 May 94 17:56:04 -0700 for soft-authors@ifcss.org Date: Fri, 6 May 94 17:56:04 -0700 Message-Id: <9405070056.AA12243@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l%uga.BITNET@cunyvm.cuny.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: CNS <-> Big5 conversion Status: O >> > IBM 5500 is yet another coding scheme, neither Big5 nor CNS. >> >> Just to quote a paragraph from the document for the chcode/ccf package: >> "For the code sets in question, they are divided into three group. >> the sets in each group have the same sequential order of word. One >> group, called 'IBM group', includes the BIG5 set, the IBM5550 set > >Just to prevent any possible confusion, it is true that IBM 5500 >code is quite different from Big5. Therefore a table that serves >to convert CNS<-->IBM code is useless for CNS<-->Big5. > > >Carlos McEvilly | email for info on Bamboo Helper Chinese text >mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly Let us copied some more from the code int (*Transform[8][8])() = { /* BIG5 IBM NSC EUC TCA TEL IBMH NSCP */ /*BIG5*/ IDent, IDent, IBM2NSC,IBM2NSC,IBM2NSC,TCA2TEL,IDent, IBM2NSC, /*IBM */ IDent, IDent, IBM2NSC,IBM2NSC,IBM2NSC,TCA2TEL,IDent, IBM2NSC, /*NSC */ NSC2IBM,NSC2IBM,IDent, IDent, IDent, TCA2TEL,NSC2IBM,IDent, /*EUC */ NSC2IBM,NSC2IBM,IDent, IDent, IDent, TCA2TEL,NSC2IBM,IDent, /*TCA */ NSC2IBM,NSC2IBM,IDent, IDent, IDent, TCA2TEL,NSC2IBM,IDent, /*TEL */ TEL2TCA,TEL2TCA,TEL2TCA,TEL2TCA,TEL2TCA,IDent, TEL2TCA,TEL2TCA, /*IBMH*/ IDent, IDent, IBM2NSC,IBM2NSC,IBM2NSC,TCA2TEL,IDent, IBM2NSC, /*NSCP*/ NSC2IBM,NSC2IBM,IDent, IDent, IDent, TCA2TEL,NSC2IBM,IDent }; So Big5 and IBM to NSC uses the same code. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From mcevilly@netcom.com Fri May 6 20:19:52 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA06436; Fri, 6 May 94 20:19:52 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id SAA14299; Fri, 6 May 1994 18:19:13 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405070119.SAA14299@netcom.com> Subject: Re: Big5 <--> 7-bit Big5 is simple To: ccnet-l@uga.bitnet Date: Fri, 6 May 1994 18:19:13 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405070035.AA10166@apple.com> from "Edmund Lai" at May 6, 94 05:35:06 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 895 Status: O Edmund Lai wrote: > I can play the same game too. I am sincere in stating that CNS is unpopular, unsupported, unfamiliar, and unnecessary. Big5 is none of these things. Of course this is my opinion and you are free to think otherwise. You have made valuable contributions to the net in the past, so I know game-playing is not the only way we can carry out this discussion! :-) It would be helpful if you could explain your ideas about CNS more so we could more fully understand your position. Many of my own questions about CNS were encapsulated in two messages to CCNET-L (a.c.c) earlier today, one responding to Roland Wang and one to yourself. Perhaps you would be so kind as to sincerely address the questions raised in those messages? -- Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From simpson@math.psu.edu Fri May 6 20:58:53 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA07752; Fri, 6 May 94 20:58:53 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA23537; Fri, 6 May 1994 21:56:50 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 21:56:49 -0400 Date: Fri, 6 May 1994 21:56:49 -0400 Message-Id: <199405070156.VAA12675@boole.math.psu.edu> To: Jing-Shin Chang Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <199405062145.RAA22226@leibniz.math.psu.edu> References: <199405062145.RAA22226@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > > > 1. The SeedNet sysadm had informed its users to change the default > > > code from CNS (7bit protocol) to Big5 in email, news from > > > 05/09. > > > > Is this referring to the "internal code" or the "exchange code" (using > ... > Both. Because they use the exchange code for email and news transfer. > ... > The main reason is to meet the trend of our popular Big5 environment. So, do I understand you correctly? You seem to be saying that SeedNet no longer uses CNS for anything. As of May 9, 1994, SeedNet will switch to Big5. Is that correct? Do you know of any network or agency in Taiwan that uses CNS? > > Could you please explain to us, in simple language of course, the > > exact form of CNS that is (or was) used as an exchange code in > > SeedNet? What are the escape sequences? And, another question about > > In brief: > ESC ( F -- designate the ASCII char set as the G0 plane (F=0x30-0x3f) > ESC $ ) 0 -- designate the 1st char plane as the G1 plane (freq. chars) > ESC $ * 1 -- designate the 2nd char plane to the G2 plane (less freq.) ...... SeedNet no longer uses the above escape sequences, is that correct? Who uses them? > In Sun's email implementation, 'ESC $ ) 0' is used to designate > chinese code (EUC) for Taiwan (and 'A', 'B', 'C' for CJK, You are talking about a Sun Chinese environment in Taiwan. However, in the international setting, I found that Sun's support for Chinese is minimal to nonexistent. Some months ago I read an article in a Sun trade publication about internationalization, EUC, and the Sun Chinese environment. I phoned 1-800-USA-4SUN and asked them to send me all available information about the Chinese environment. They never sent me anything. So far as I can determine, people outside Taiwan who want to do Chinese on Suns are on their own. They can buy some extremely expensive commercial software, or they can use free software such as cxterm. So far as I can tell, the US computer company that is doing the most for Chinese is Apple. Does Apple use CNS? I am under the impression that Apple uses only GB and Big5. Maybe Ed and Roland can answer this. > In summary, the places where it should be converted to the internal codes > depends on where the user want to access it as an ordinary file for > processing. When is archived or ready for transmission, it should be > in interchange code form. >From this it appears that, contrary to what has been claimed, the exchange code is not invisible to the user. -- Steve From simpson@math.psu.edu Fri May 6 21:15:28 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08401; Fri, 6 May 94 21:15:28 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id WAA23631; Fri, 6 May 1994 22:13:29 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 6 May 1994 22:13:27 -0400 Date: Fri, 6 May 1994 22:13:27 -0400 Message-Id: <199405070213.WAA12700@boole.math.psu.edu> To: John Delacour Cc: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, ccnet-l@uga.cc.uga.edu Subject: Re: Big5 or CNS ? In-Reply-To: <199405062306.TAA22776@leibniz.math.psu.edu> References: <199405062306.TAA22776@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O John Delacour writes: > Jing-shin, > > Maybe I'm being simplistic but I think you're muddying the waters, and much > though I love Ed Lai's Cihui etc. [ ... stuff omitted ...] > I don't see the point of this branch of the > discussion. > > At present users of GB-based software regularly use GB<->HZ translation for > email and posting to newsgroups such as ACT. There is no problem at all > that I have experienced. > > HZ+, as proposed by Steve Simpson, provides exactly the same facility for > users of Big5-based software or mixed GB and Big5. > [... stuff omitted ...] > that's what HZ+ is for and it works, so what is the point of > discussing CNS and what not at the moment. Thanks, John. You are grasping exactly what HZ+ is all about. I can't understand why everybody doesn't just get on the HZ+ bandwagon. Why in the world is there all this noise about CNS? :-) -- Steve From jiang Fri May 6 21:33:07 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA09009; Fri, 6 May 94 21:33:07 CDT Date: Fri, 6 May 94 21:33:07 CDT From: jiang (Yuan Jiang) Message-Id: <9405070233.AA09009@ifcss.org> Subject: Re: Big5 <--> 7-bit Big5 is simple Cc: soft-authors Status: O I see the disadvantage listed against CNS is still a CNS vs Big5 argument. Again, it's not a CNS vs Big5 debate. The major problem of 7-bit CNS is its inperfect mapping with Big5. The other possible drawback is that it may contain non-printable ASCII. My north American bias (dislike of non-printables) may tell me to withdraw my vote for 7-bit CNS. Among the other HZ+, the new B5encode has the same principle of HZ+S -- clearing the high bits and shift the code to printable ASCII. The advantages of HZ+S apply to B5encode too. The current code impletementation does not have multiple operations. So until HZ+s has a new code impletementation, I may vote for B5encode. Actually, I don't see any scheme has the absolute advantage over the others. The chinese.* groups have to go on. If we cannot reach a consensus, a result of vote would not hurt. The problem is whether the vote drawing should be restricted to soft-authors or to CCNET. Yuan From yawei Fri May 6 21:39:34 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA09186; Fri, 6 May 94 21:39:34 CDT Date: Fri, 6 May 94 21:39:34 CDT From: yawei (Ya-Gui Wei) Message-Id: <9405070239.AA09186@ifcss.org> To: simpson@math.psu.edu Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility Cc: soft-authors, yawei Status: O Looks like Steve is not being completely fair, :-) so I guess a rebutal is needed. An "advantage" would commonly referred to strong point over competing proposals, but Steve doesn't seem to be using the word in this way. Advantages (1) and (2) are shared by all competing proposals. (I think they refer to compatibility with HZ and Big5.) Advantage (3): >The fact that HZ+S fully respects the natural structure of the Big5 >blocks leads to several significant advantages. These advantages are HZ-2 moves a few infrequently-used characters into the first block. But this could only _reduce_ the need to switch into the second block. It looks sorta' like an "advantage" to me, albeit small. > (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. True. But same is true for all competing proposals. > (5). HZ+S uses escape sequences sparingly. The ~2 escape sequence is > needed only to switch into Big5 block 2, the block containing the less > frequently used Chinese characters. Most of the time, we use only ~1 > to switch into Big5 block 1, plus ~{ and ~} as in HZ. With the new "B5encode", this is true for all 3 proposals. Actually HZ-2 has a very tiny advantage in this regard, see my rebutal to advantage (3) above. :-) > (6). HZ+S is fully compatible not only with ETen Big5 but also with > corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less > frequently used Chinese characters begin somewhere prior to C940. > Thus HZ+S may help to overcome the difficulties associated with these > corrupt variants. Irrelevant. We are concerned with mapping between Big5 and a 7-bit code, not with mapping between the code and a dictionary file. Done properly, implementations of all three proposals will be able to use the HKU font, and _properly_. > (7). The way is open for HZ+S to be compatible with a possible future > incorporation of Big5 into ISO-2022-JP-2, similar to the present > incorporation of CNS-11643. Such an unofficial extension of > ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. Not sure what this entails, so no comment for now. >CONCLUSION The conclusion is that all 3 proposals possess the same 7 advantages listed by Steve, though HZ-2 leads slightly at points (3) and (5). :-) To be fair, I think HZ+S's packing of all the Big5 characters into two tight blocks without leaving unused gaps is aesthetically desirable. I don't know why Steve forgot to mention this advantage. With everything else being equal, this just might be the winning edge. :-) Unfortunately, this advantage brings with it an disadvantage, which is the increased in CPU time (true no matter how you implement it). There's no free lunch, you know. One disadvantage to the new B5encode (as I understood it) would be that the codes don't come in the same order as the Big5 codes. This could be a problem if one decides to use an ASCII sorter to sort a file and expect the output to be in Big5 order. My conclusion is that the 3 proposals are extremely similar in merit (and very similar in detail), so I am quite indifferent to which one is adopted. Ya-Gui From mcevilly@netcom.com Fri May 6 21:58:48 1994 Return-Path: Received: from netcom.com (netcom6.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA09908; Fri, 6 May 94 21:58:48 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id TAA15717; Fri, 6 May 1994 19:58:08 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405070258.TAA15717@netcom.com> Subject: Coming soon: 7-bit Big5! To: ccnet-l@uga.bitnet Date: Fri, 6 May 1994 19:58:08 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405070042.AA10838@apple.com> from "Edmund Lai" at May 6, 94 05:42:07 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1932 Status: O Edmund Lai said: > [Carlos McEvilly said:] > >Just to prevent any possible confusion, it is true that IBM 5500 > >code is quite different from Big5. Therefore a table that serves > >to convert CNS<-->IBM code is useless for CNS<-->Big5. > OK, let us copy more code from the conversion progarm and see what it says. > int (*Transform[8][8])() = { > /* BIG5 IBM NSC EUC TCA TEL IBMH NSCP */ > /*BIG5*/ IDent, IDent, IBM2NSC,IBM2NSC,IBM2NSC,TCA2TEL,IDent, IBM2NSC, > /*IBM */ IDent, IDent, IBM2NSC,IBM2NSC,IBM2NSC,TCA2TEL,IDent, IBM2NSC, ... < deleted > ... > IDent(order) > int order; > { > return (order); > } Interesting. They said this code had bugs. It looks like you have found one! In case there is still any doubt in anyone's mind that Big5 and IBM 5500 are different, the following should put the matter to rest: Codepoints for hanzi numbers 1 - 10, in Big5 (left) and IBM 5500 (right) ======================================================================== 1. A440 != 8C40 2. A447 != 8C47 3. A454 != 8C54 4. A57C != 8D5D 5. A4AD != 8C8C 6. A4BB != 8C9A 7. A443 != 8C43 8. A44B != 8C4B 9. A445 != 8C45 10. A451 != 8C51 Since the III conversion code that contains CNS conversion functions is over 350K, I haven't had time to look through all of it. It really shouldn't be necessary to bring the complexity of CNS into this. (I can just imagine shareware authors contemplating adding 350K of code to their programs :-) (the truth is that the 350K is not all CNS code--there are other conversion routines in there as well--the point is, with no CNS it is not necessary to use *any* of this code. So let's get rid of CNS!). It might be good to try to steer this back to HZ+ now. We need to build support for 7-bit Big5. The HZ+S proposal has advantages over and above the other proposals (HZ+2 and b5encode), so let's make it happen! From lai@apple.com Fri May 6 22:48:25 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA11411; Fri, 6 May 94 22:48:25 CDT Received: from mac19.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA25585; Fri, 6 May 94 20:46:16 -0700 for soft-authors@ifcss.org Message-Id: <9405070346.AA25585@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Fri, 6 May 1994 20:50:53 -0800 To: CCNET-L@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Ed Lai) Subject: Re: Big5 <--> 7-bit Big5 is simple Status: O >> >>I am sincere in stating that CNS is unpopular, unsupported, >>unfamiliar, and unnecessary. Big5 is none of these things. Of >> >Again, Ed is not arguing CNS' advantage over Big5. He is >comparing 7-bit CNS with HZ+S, HZ-2 and B5encode. >At least, CNS is used by some workstations and SeedNet >and is requiredome government biddings. On the other hand >HZ+S, HZ-2 and B5encode are just in the process of invention -- >totally unpopular, .... > >Yuan Yes, so I have not been wasting my time after all. Technical I have no problem with any of these proposals, if I want I can propose a dozen more. I have no particular preference for CNS, I have not heard of CNS until a few days ago. I was the first one to admit that CNS would take up more cycles, only that I don't think it is significant. My point is that when there can be multiple proposals and none is natuarally and clearly superior, it is easiest to take some existing standard and just use it. Let say tomorrow some of the major manufacturer in Taiwan announces a new 7 bit Big5 standard. I would be the first to support it over CNS. However I would not then say that the reason I cannot support CNS is that it takes 355K code to convert between Big5 and CNS, or even 35.5K. There are something I value more than winning or losing an argument. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From simpson@math.psu.edu Fri May 6 23:10:56 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12130; Fri, 6 May 94 23:10:56 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id AAA24179; Sat, 7 May 1994 00:08:58 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 00:08:56 -0400 Date: Sat, 7 May 1994 00:08:56 -0400 Message-Id: <199405070408.AAA12887@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: comment on Jing-Shin's latest new proposal In-Reply-To: <9405062111.AA14190@hermes.ee.nthu.edu.tw> References: <9405062111.AA14190@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > [B5E2/B5encode-2: A New Proposal for 7-bit Big5 Representation] > > 1994/05/07, Rev. 1.4 > Since I am using the same escape sequence as proposed by Prof. Simpson, > all the advantages of his proposal, except for the 8-bit to 7-bit mapping > method, are also inherited from his HZ+S -^^- (Thanks Prof. Simpson). Apparently Jing-Shin's view is that the advantages of HZ+S derive mainly from the use of ~1 and ~2 as escape sequences. He thinks that by choosing the same escape sequences, his B5E2 proposal inherits most of the advantages of HZ+S. I disagree. The choice of escape sequences is only a minor detail. The truth is that, although one or two of the advantages of HZ+S are partially inherited by Jing-Shin's B5E2 proposal, most of the advantages are not inherited at all. The following advantages of HZ+S are not inherited by B5E2: (1). The proposed HZ+S or HZ+ standard is based firmly on two existing standards which are already very popular: HZ and Big5. This should enable HZ+ to win a degree of immediate acceptance. [ Jing-Shin by his own admission is unfamiliar with HZ and GB. ] (2). HZ+S is very easy to implement. I have already hacked up some Big5 <-> HZ+S conversion filters. This experimental software is based on Fung F. Lee's GB <-> HZ conversion filters and is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps/. (3). HZ+S fully respects the natural partition of the Big5 code range into blocks of more and less frequently used characters. This leads to several additional advantages, listed in (4)-(7) below. (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. .... (6). HZ+S is fully compatible not only with ETen Big5 but also with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less frequently used Chinese characters begin somewhere prior to C940. Thus HZ+S may help to overcome the difficulties associated with these corrupt variants. (7). The way is open for HZ+S to be compatible with a possible future incorporation of Big5 into ISO-2022-JP-2, similar to the present incorporation of CNS-11643. Such an unofficial extension of ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. -- Steve From mcevilly@netcom.com Fri May 6 23:12:38 1994 Return-Path: Received: from netcom.com (netcom6.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA12151; Fri, 6 May 94 23:12:38 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id VAA22282; Fri, 6 May 1994 21:12:01 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405070412.VAA22282@netcom.com> Subject: Re: Big5 <--> 7-bit Big5 is simple To: ccnet-l@uga.bitnet Date: Fri, 6 May 1994 21:12:00 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405070242.AA09246@ifcss.org> from "Yuan Jiang" at May 6, 94 09:42:11 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2292 Status: O > >I am sincere in stating that CNS is unpopular, unsupported, > >unfamiliar, and unnecessary. Big5 is none of these things. Of > > > Again, Ed is not arguing CNS' advantage over Big5. He is Yes, I understand he is not trying to convince all users in Taiwan to change to CNS. He is proposing CNS be used as an exchange code. My question about that is, why is it desirable? What are the benefits? How can these benefits justify the extra complexity? As programmers we should not be afraid of complexity. But when it is unecessary, let's do away with it. After all, we hope that many other programmers will be able to quickly learn (even by word of mouth, or on the back of an envelope) how to enable 7-bit Big5 in their applications. I've worked with programming teams in Taiwan, and know how they work. They are smart enough that they'd just as soon invent their own code, rather than tracking down some source code on the net. If it's too complex to write on the back of a cigarette pack during a smoking break, Taiwanese programmers will just write a new one--or not write it at all-- that's what we should be trying to avoid. To benefit users, we need to make it as simple as possible even for programmers. If we do this more programs will get adapted for 7-bit Big5. Keeping it simple means keeping CNS out of the picture and benefitting users. > comparing 7-bit CNS with HZ+S, HZ-2 and B5encode. > At least, CNS is used by some workstations and SeedNet > and is requiredome government biddings. On the other hand > HZ+S, HZ-2 and B5encode are just in the process of invention -- > totally unpopular, .... > I would argue that 1) Nobody has named even a single instance of an organization that uses CNS in its day to day operations, with the exception of Seednet, which is phasing it out. 2) A direct mapping from Big5 to 7-bit is better than a convoluted mapping from CNS. 3) CNS, since it is hardly used now, should not be encouraged--if we encourage it, it might slow the adoption of Unicode in the future. 4) Yes, HZ+S et al. are in the process of invention, and that gives the inventors control of them. CNS, on the other hand, is under the control of the Taiwanese government, and could be revised at any time, as it has been in the past, or even abandoned. From butta1@crsa.bu.edu Fri May 6 23:21:08 1994 Return-Path: Received: from crsa.bu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12358; Fri, 6 May 94 23:21:08 CDT Received: from ithake.bu.edu by crsa.bu.edu (8.6.4/Spike-2.1) id AAA27231; Sat, 7 May 1994 00:19:11 -0400 Received: from localhost by ithake.bu.edu (8.6.4/Spike-2.1) id AAA07103; Sat, 7 May 1994 00:18:57 -0400 Date: Sat, 7 May 1994 00:18:56 -0400 (EDT) From: C I I Project Subject: Re: Big5 <--> 7-bit Big5 is simple To: Ed Lai Cc: soft-authors@ifcss.org In-Reply-To: <9405070346.AA25585@apple.com> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O On Fri, 6 May 1994, Ed Lai wrote: > >Again, Ed is not arguing CNS' advantage over Big5. He is > >comparing 7-bit CNS with HZ+S, HZ-2 and B5encode. > My point is that when there can be multiple proposals and > none is natuarally and clearly superior, it is easiest to > take some existing standard and just use it. are you saying to use purely 7-bit CNS w/o any escape sequence? such as i assume has been done on Seednet, or to incorporate 7-bit CNS instead of 7-bit BIG5 into the three proposals, or something else? nelson From simpson@math.psu.edu Fri May 6 23:26:56 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12549; Fri, 6 May 94 23:26:56 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id AAA24231; Sat, 7 May 1994 00:24:55 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 00:24:53 -0400 Date: Sat, 7 May 1994 00:24:53 -0400 Message-Id: <199405070424.AAA12916@boole.math.psu.edu> To: lai@apple.com (Edmund Lai) Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <9405062323.AA02064@apple.com> References: <9405062323.AA02064@apple.com> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > There are only two valid argument favoring CNS or encoded Big5. > Against CNS: It takes less cycles to encode Big5 into Encoded Big5. > For CNS: It is already an established 7 bit standard, at least it is > established compared to the so called encoded Big5. No. There are more than two valid arguments. The principal argument against CNS is one that you did not mention, namely that it is an unpopular standard which nobody uses. Do you know even one person who uses CNS in his or her daily work? There are millions who use Big5. I know that you have an argument for CNS based on your distinction between internal codes and exchange codes. There are two problems with that argument. First, according to Jing-Shin's information, networks such as SeedNet, which in the past used to use CNS as an exchange code, no longer do so. They now use Big5 instead. Second, the distinction between internal and exchange codes will break down in an environment such as alt.chinese.text, where the so-called exchange code is not hidden from the user. -- Steve From simpson@math.psu.edu Fri May 6 23:38:22 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12910; Fri, 6 May 94 23:38:22 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id AAA24326; Sat, 7 May 1994 00:36:16 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 00:36:15 -0400 Date: Sat, 7 May 1994 00:36:15 -0400 Message-Id: <199405070436.AAA12938@boole.math.psu.edu> To: Edmund Lai Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Cc: Multiple recipients of list CCNET-L Subject: Re: Big5 or CNS ? In-Reply-To: <199405062336.TAA22906@leibniz.math.psu.edu> References: <199405062336.TAA22906@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > If you have software that does the automatic decoding then it would > be transparent to you whether you use CNS or encoded Big5. > > If you don't have software that does it automatically, then it would > not be transparent to you and you would see something that look like > garbage, but that would be true whether you use CNS or encoded Big5. Even after this explanation, your key distinction between "internal code" and "exchange code" is still unclear to me. Could you please clarify these concepts? Is HZ as used in alt.chinese.text an internal code or an exchange code? Are you proposing to use CNS as an internal code or an exchange code? Apart from these semantical issues, the real question is: Why are you opposed to basing HZ+ on Big5? Why do you want to base it on CNS? In my view, the principal argument in favor of basing HZ+ on Big5 is that Big5 is an extremely popular standard. The principal argument against basing HZ+ on CNS is that CNS is not a popular standard, in fact nobody uses it. What do you say about those arguments? -- Steve From simpson@math.psu.edu Fri May 6 23:46:15 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA13180; Fri, 6 May 94 23:46:15 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id AAA24349; Sat, 7 May 1994 00:44:10 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 00:44:08 -0400 Date: Sat, 7 May 1994 00:44:08 -0400 Message-Id: <199405070444.AAA12950@boole.math.psu.edu> To: Edmund Lai Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <199405062342.TAA22937@leibniz.math.psu.edu> References: <199405062342.TAA22937@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > You mean that people is going to start using encoded Big5 as internal > code? I doubt that. No, that's not what I mean. I don't accept your sharp distinction between internal code and exchange code. Maybe your real question is, do I think that someday there could be software that would work directly with 7-bit Big5, rather than translate it into 8-bit Big5 first? The answer is, yes, I think that could happen. It's possible. Similar things have happened in the past. An example is ZWDOS. -- Steve From roland_wang@powertalk.apple.com Fri May 6 23:53:25 1994 Return-Path: Received: from guardian.apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA13269; Fri, 6 May 94 23:53:25 CDT Received: from federal-excess.apple.com by guardian.apple.com with SMTP (5.65/8-Oct-1993-eef) id AA24632; Fri, 6 May 94 11:13:28 -0700 Received: from powertalk.apple.com by federal-excess.apple.com with SMTP (5.64/26-Sep-1993-eef) id AA17025; Fri, 6 May 94 11:12:30 PDT for soft-authors@ifcss.org Message-Id: <9405061812.AA17025@federal-excess.apple.com> Date: 6 May 1994 11:13:09 -0700 From: "Roland Wang" Subject: Re(2): Big5 or CNS ? To: "simpson@math.psu.edu" Cc: "ccnet-l@uga.cc.uga.edu" , "shin@hermes.ee.nthu.edu.tw" , "soft-authors@ifcss.org" Status: O We may be looking into the same issue from different angels. Because of my background and experience in international system development (UNIX, X Window, and now Macintosh), my view may be different from yours. The technical difficulty of supporting Big5 in UNIX is because most of international UNIX systems are using EUC (Extended UNIX Code). Big5 can't fit into EUC scheme. Big5 encoding is (1xxxxxxx 1xxxxxxx) and (1xxxxxxx 0xxxxxxx), which is conflicting with EUC (please refer to UNIX book for details). When I worked for a leading workstation vendor, we tried hard to support Big5 at system level and gave up because it requires major changes in system design and implementation. And, we implemented a utility which can deal with Big5. If a UNIX is not using EUC, that's another story. I know there is a workstation UNIX supporting Big5. Cxterm is a self-contained application which has its own Chinese input (dictionary, etc) and output (fonts, etc.). Running cxterm doesn't need international system at all. Is anybody here working with III and CCL in Taiwan? They have better answer about the popularity of CNS. I used to read a.c.t on a Sparc. I hacked about 50 lines code in a GUI news tool to do conversion between HZ and GB. And, I hacked another 10 lines (to call the Big5-GB mapping routine) to make it capable to read a.c.t.big5 just for fun. The point is - as long as I have convertors, it's fine to use any encoding. I have no interest in advocating CNS or Big5. I just want to provide more info to make the discussion more thorough. The only interest I participate this group is to learn from your guys about Chinese computing, (if any chance) contribute my knowledge and experience and make network friends. _Roland Wang /* Disclaimer - All statements and opinions above are my own. */ ------ From: simpson@math.psu.edu, Fri, May 06, 1994 ------ Roland Wang writes: > I second to Ed Lai's approach - use 7 bit HZ+ to encode CNS for the > communication, such as email and news. User end software can be in > any encoding, CNS (only need 7bit to 8bit conversion), Big5 (7bit to > 8bit, then map to Big5), and GB (7bit to 8bit, then CNS2GB mapping). I'm not clear how this proposal would work in practice. Could you please explain? Would the CNS codes be totally hidden from all users? > First of all, we need to clarify a mis-perception about CNS. There is > no doubt about the popularity of Big5 due to the PC popularity (both > IBM PC/compatible and Mac). We need to clarify a misperception about Big5. Big5 is not only popular on PCs and Macs. It is also popular on Unix. Many people run cxterm in Big5 mode and use it for things like reading and posting to a.c.t.big5, participating in the Chinese poem list, typing documents, TeX, etc. (not to mention communicating with all those PC and Mac users). > However, CNS is not just an official standard. It is used in > Taiwan widely even though there is no comparison to Big5. I haven't heard of even one person who uses CNS in their daily work. Can you name one such person? Are you such a person? > Supporting CNS is one of the "must" items for Taiwan goverment systems. Why should we in the international Internet community care about ROC government imposed standards that do not have popular support? > That's one of the reasons why most of UNIX vendors support CNS instead of > Big5 (another reason is the technical difficulty to fit Big5 into EUC > scheme). What technical difficulty? I think the difficulty is not technical but political, namely the government's insistence on CNS. From simpson@math.psu.edu Sat May 7 00:33:40 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14833; Sat, 7 May 94 00:33:40 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA24569; Sat, 7 May 1994 01:31:42 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 01:31:40 -0400 Date: Sat, 7 May 1994 01:31:40 -0400 Message-Id: <199405070531.BAA13037@boole.math.psu.edu> To: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility In-Reply-To: <9405070013.AA14517@hermes.ee.nthu.edu.tw> References: <9405070013.AA14517@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > You can access my new proposal at: > > hermes.ee.nthu.edu.tw(140.114.28.27):/shin/b5encode OK, thanks for making your stuff available in this way. I'll get rid of what I have from you at my site. > In my B5E2, the Freq. used & Less Freq. used division is also fully > respected. In addition, the relative code positions are retained. How is that? In HZ+S, the relative code positions are indeed retained, because the codes are simply packed in order onto an initial segment of the graphical ASCII range. In your proposal, the order is changed, right? Or at least the codes are not packed tightly, i.e. there are gaps in the range of the mapping. > However, the current discussion is to have a good exchange code; > therefore, the relation with the font may not be so closely > related. No. The current discussion is *not* about how to have a good exchange code. The current discussion is about how to incorporate 7-bit Big5 into HZ resulting in HZ+. Right now, in 1994, HZ is used not only as an exchange code but also as an internal code. There is every reason to believe that HZ+ will continue to be used in exactly the same way as HZ+ is now used. Ya-Gui has already indicated his plan to support HZ+ in ZWDOS. ZWDOS has screen and printing capabilities, so fonts will be needed. The fact that Big5 fonts can be used essentially unchanged is a significant advantage of HZ+S which is not shared by the other HZ+ proposals. Jing-Shin, I'd suggest you familiarize yourself with HZ. I have the feeling that you don't know what considerations motivated it and how it has developed. I notice that your proposals do not include HZ. Is this omission intentional? > In brief, ISO-compatibility is better defined by (1) do not use > non-control chars as ESC chars. (2) follow the general syntax of > the ESC sequences for designating a character set or a char. plane > in that set. So, as long as I disagree. The choice of escape sequences is only a minor detail. True ISO-2022 compatibility means more than a certain syntax for escape sequences. True ISO-2022 compatibility means: to handle mixed character sets using the same kind of mappings as in ISO-2022. HZ+S has this kind of ISO-2022 compatibility. The other 7-bit Big5 proposals do not have it. > Now, we have four proposals. Great! :-) > So I tried very hard (really !!-^^-) to propose my alternatives and > hope that they are as competitive as the other proposals. Probably > there are other persons who have better ideas. Would you netters > like to contribute better alternatives? Let a hundred flowers bloom! The only problem is that, by the time we reach a consensus on HZ+, we may be well into the next century. -- Steve From simpson@math.psu.edu Sat May 7 00:39:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15082; Sat, 7 May 94 00:39:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA24588; Sat, 7 May 1994 01:37:36 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 01:37:34 -0400 Date: Sat, 7 May 1994 01:37:34 -0400 Message-Id: <199405070537.BAA13048@boole.math.psu.edu> To: lai@apple.com (Edmund Lai) Cc: CCNET-L@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: Big5 <--> 7-bit Big5 is simple In-Reply-To: <9405070035.AA10166@apple.com> References: <9405070035.AA10166@apple.com> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > I can play the same game too. > > >> What are some of the arguments against encoded Big5 as an exchange code. > > > >1. unpopular > >2. unsupported > >3. unfamiliar > >4. unnecessary Ed, what you are saying is simply not true. Big5 is extremely popular as both an exchange code and an internal code (pretending for a moment that I understand these concepts). For example, SeedNet recently switched to Big5. CNS is unpopular. Big5 is popular. End of story. -- Steve From simpson@math.psu.edu Sat May 7 00:50:14 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15334; Sat, 7 May 94 00:50:14 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA24642; Sat, 7 May 1994 01:48:07 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 01:48:05 -0400 Date: Sat, 7 May 1994 01:48:05 -0400 Message-Id: <199405070548.BAA13076@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: I wish CNS would die and go away Reply-To: simpson@math.psu.edu Status: O Pinghua Young writes: > BTW, I am very happy to see the active discussion of a unified > encoding scheme. As always, it takes time to work out any > differences and make compromises. The discussion has been quite vigorous. The current state is that we were almost ready to agree on a very simple 7-bit Big5 implementation, when suddenly the waters got muddied by the CNS issue. Big5 is the popular coding standard, used by everybody in Taiwan and many people elsewhere for their daily work. CNS is a competing standard that has no popular support whatsoever but is endorsed by the ROC governnment. If this CNS business goes on much longer, I think I am going to give up. I am not very patient with committees. -- Steve From simpson@math.psu.edu Sat May 7 03:51:12 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA19203; Sat, 7 May 94 03:51:12 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id EAA25252; Sat, 7 May 1994 04:49:12 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 04:49:11 -0400 Date: Sat, 7 May 1994 04:49:11 -0400 Message-Id: <199405070849.EAA13396@boole.math.psu.edu> To: yawei@ifcss.org (Ya-Gui Wei) Cc: simpson@math.psu.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Comparison of the three proposals In-Reply-To: <9405070239.AA09186@ifcss.org> References: <9405070239.AA09186@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes: > Looks like Steve is not being completely fair, :-) so I guess > a rebutal is needed. An "advantage" would commonly referred to > strong point over competing proposals, but Steve doesn't seem > to be using the word in this way. The word "advantage" has more than one meaning. DEFINITION 0 1ad-van-tage \ed-'vant-ij\ n [ME avantage, fr. MF, fr. avant before, fr. L abante] (14c) 1: superiority of position or condition 2a: BENEFIT, GAIN; esp: benefit resulting from some course of action 2b obs: INTEREST 2a 3: a factor or circumstance of benefit to its possessor 4: the first point won in tennis after deuce -- to advantage: so as to produce a favorable impression or effect I'm using "advantage" in sense 2 or 3, namely a benefit or desirable feature. I carefully listed what I consider to be the principal benefits of HZ+S. I wish that the proponents of the other proposals would list the benefits of their proposals. This would make comparison easier. > Advantages (1) and (2) are shared by all competing proposals. > (I think they refer to compatibility with HZ and Big5.) Big5 compatibility is shared to some degree by all the proposals, but it is not shared equally. As I pointed out in lengthy mail to soft-authors, HZ+S is much more compatible with Big5 than the other proposals. This is because the two blocks used by HZ+S are identical to the two blocks into which the Big5 character set naturally divides itself (more frequent block vs. less frequent block). Furthermore, HZ+S maps these blocks in order with no gaps. None of the other proposals has this degree of Big5 compatibility. The structure of the Big5 character set is documented in my /pub/simpson/chinese/hzps/etfonts.txt at math.psu.edu. On another matter, am I the only one who has noticed that the various B5encode proposals do not incorporate HZ? By contrast, HZ+S and HZ-2 embrace HZ whole-heartedly. Advantage (2) referred to a sample implementation of HZ+S. It is available for FTP at math.psu.edu, /pub/simpson/chinese/hzps. It is not rocket science, but it is good enough to immediately begin posting 7-bit mixed Big5, GB, and ASCII to the Internet. John Delacour and I have already used it for this purpose. I don't think any of the other proposals have yet provided any kind of implementation. > HZ-2 moves a few infrequently-used characters into the first > block. A few? More than 1000! And this implies that the dividing line between the blocks is more than 1000 characters away from where it ought to be for true Big5 compatibility. By contrast, HZ+S puts the dividing line at exactly the right place. > But this could only _reduce_ the need to switch into > the second block. It looks sorta' like an "advantage" to me, > albeit small. You are sacrificing full Big5 compatibility for a few CPU cycles. To me, this seems "penny-wise and pound-foolish". (Does this idiom have a Chinese equivalent?) > > (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. > True. But same is true for all competing proposals. It depends on what you mean by "essentially unchanged". For HZ+S, you have to change only two lines in the HBF header file. For the other proposals, you have to make a lot of adjustments, because those proposals are not so completely compatible with Big5. > > (5). HZ+S uses escape sequences sparingly. The ~2 escape sequence is > > needed only to switch into Big5 block 2, the block containing the less > > frequently used Chinese characters. Most of the time, we use only ~1 > > to switch into Big5 block 1, plus ~{ and ~} as in HZ. > With the new "B5encode", this is true for all 3 proposals. True. This is the one advantage that all the proposals now share. > Actually HZ-2 has a very tiny advantage in this regard, > see my rebutal to advantage (3) above. :-) See my rebuttal to your rebuttal, above. That tiny advantage in CPU cycles costs HZ-2 a lot in terms of Big5 compatibility. > > (6). HZ+S is fully compatible not only with ETen Big5 but also with > > corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which the less > > frequently used Chinese characters begin somewhere prior to C940. > > Thus HZ+S may help to overcome the difficulties associated with these > > corrupt variants. > > Irrelevant. We are concerned with mapping between Big5 and a > 7-bit code, not with mapping between the code and a dictionary > file. It's not irrelevant. The spurious "HKU-Big5 standard" has caused endless problems and confusion. It is still causing problems and confusion right up to the present day, in Mule and Mosaic-L10N for example. (I'm not sure what you mean by a dictionary file.) HZ+S provides a way to neutralize the "HKU-Big5" problem, because the less frequently used characters begin exactly at the beginning of block 2. This advantage of HZ+S is not shared by HZ-2. Maybe it is shared to some degree by b5encode, I'm not sure. > Done properly, implementations of all three proposals will be > able to use the HKU font, and _properly_. It's not only a matter of using a font. It's a matter of respecting the structure of the Big5 character set. HZ+S respects this structure fully, while the other proposals respect it less than fully. > > (7). The way is open for HZ+S to be compatible with a possible future > > incorporation of Big5 into ISO-2022-JP-2, similar to the present > > incorporation of CNS-11643. Such an unofficial extension of > > ISO-2022-JP-2 is already implemented in Mule, the multilingual Emacs. > > Not sure what this entails, so no comment for now. If you look into Mule, you will see what I am talking about. HZ+S incorporates Big5 into HZ in *exactly* the same way that Mule incorporates Big5 into ISO-2022-JP-2. This is another advantage of HZ+S that is not shared by HZ-2 and b5encode. > The conclusion is that all 3 proposals possess the same 7 advantages > listed by Steve, though HZ-2 leads slightly at points (3) and (5). :-) No. HZ+S has a commanding lead in Big5 compatibility. Less important but still significant is HZ+S's lead in Mule / ISO-2022-JP-2 compatibility. > To be fair, I think HZ+S's packing of all the Big5 characters > into two tight blocks without leaving unused gaps is aesthetically > desirable. I don't know why Steve forgot to mention this advantage. > With everything else being equal, this just might be the winning > edge. :-) This aesthetic aspect of HZ+S is very pleasing to me personally. I didn't really forget to mention it, I just thought that aesthetics would be considered irrelevant. But, given that you also appreciate it, I should probably add it to my list. :-) > Unfortunately, this advantage brings with it an disadvantage, which > is the increased in CPU time (true no matter how you implement it). > There's no free lunch, you know. Sure, there's no such thing as a free lunch. But if I can achieve beauty and harmony and full Big5 compatibility at the cost of only a few CPU cycles, why not go for it? That's what I did in HZ+S. > One disadvantage to the new B5encode (as I understood it) would be > that the codes don't come in the same order as the Big5 codes. > This could be a problem if one decides to use an ASCII sorter > to sort a file and expect the output to be in Big5 order. > > My conclusion is that the 3 proposals are extremely similar > in merit (and very similar in detail), so I am quite indifferent > to which one is adopted. I still like HZ+S better. But I can live with any reasonable proposal that combines HZ with 7-bit Big5. -- Steve From shin@hermes.ee.nthu.edu.tw Sat May 7 10:03:25 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA26675; Sat, 7 May 94 10:03:25 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182224(1)>; Sat, 7 May 1994 23:06:55 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA16054; Sat, 7 May 94 23:00:03 CST Date: Sun, 8 May 1994 13:00:03 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405071500.AA16054@hermes.ee.nthu.edu.tw> To: lai@APPLE.COM, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, soft-authors@ifcss.org Status: O > From simpson@math.psu.edu Sat May 7 12:43:54 1994 > Return-Path: > > Edmund Lai writes: > > > You mean that people is going to start using encoded Big5 as internal > > code? I doubt that. > > No, that's not what I mean. I don't accept your sharp distinction > between internal code and exchange code. > > Maybe your real question is, do I think that someday there could be > software that would work directly with 7-bit Big5, rather than > translate it into 8-bit Big5 first? The answer is, yes, I think that > could happen. It's possible. Similar things have happened in the > past. An example is ZWDOS. It is of course POSSIBLE. However, even this is possible I don't think it is acceptable to the Big5 users if we implement it in the incoming chinese.* newsgroup in this way. The HZ users are not using the same computing environment as the Big5 users. The major distinction is: Big5 users - Use internal codes for information processing and also for communication (this is possible since our computing environment is almost, say 95%, 8-bit clean. Using 7-bit Big5 or variants is simply a method to accomodate with the HZ users' 7-bit convention for the new chinese.* groups. I guess they would like to use the internal codes directly if possible. In fact, what we are discussing is only among the soft authors based on our hacker's interests and characteristics, not among the users.) HZ users - Use exchange codes for information processing and also for communication (I am leaning to consider HZ as an exchange code. When you process a HZ file, you need be careful not to delete those ESC's. Normally, a user should not be disturbed by such stuffs, and should be hide by the applications, like editors. Of course, the HZ users already live with such computing environment for a long time.) So, in the future inplementation of news readers, we should provide both kinds of conversions so that HZ users can process HZ as usually but the Big5 users should be given the Big5/8 streams. I am not familiar with the environment of HZ users, ZWDOS is the only share I knew. Did they process GB files frequently? If this is the case, it is also appropriate to convert it back to GB and leave the users with a file without those ESC's. > > -- Steve > From simpson@math.psu.edu Sat May 7 10:22:25 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA27194; Sat, 7 May 94 10:22:25 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA26373; Sat, 7 May 1994 11:20:20 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 11:20:18 -0400 Date: Sat, 7 May 1994 11:20:18 -0400 Message-Id: <199405071520.LAA14054@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, chinesenet@uclink.berkeley.edu Newsgroups: alt.chinese.text Subject: an explanation and an example of HZ+ Reply-To: simpson@math.psu.edu Status: O WHAT IS HZ+ ? HZ+ is based on HZ, the popular 7-bit mixed GB and ASCII file format which is currently used in alt.chinese.text and elsewhere. HZ+ is a plan to enhance HZ by incorporating 7-bit Big5. The HZ+ plan will enable 7-bit Chinese e-mail and news throughout the Internet. Various proposals for HZ+ are now under discussion via the soft-authors@ifcss.org mail alias. HZ+S is my particular proposal. I think it is fair to say that HZ+S is the most highly developed and best articulated of all the competing HZ+ proposals. The formal HZ+S proposal and a sample implementation of HZ+S are available for your inspection at math.psu.edu, /pub/simpson/chinese/hzps. This software can be used now to transmit 7-bit mixed Big5, GB, and ASCII to alt.chinese.text and elsewhere on the Internet. AN EXAMPLE OF HZ+ Here is an example of 7-bit mixed Big5 / GB / ASCII as implemented in HZ+ (actually HZ+S). A common English expression is "penny-wise and pound-foolish," referring to someone who carefully saves his pennies but makes bad decisions by losing sight of the larger context. In HZ+, the GB Chinese for "penny-wise and pound-foolish" looks like ~{RrP!J'4s~}, just as in HZ. The Big5 Chinese for the same expression looks like ~1)_&R(C&L~}. The capability of representing Big5 Chinese is present in HZ+ but not in HZ. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From CAI%MVF.DecNet@wigate.nic.wisc.edu Sat May 7 11:40:34 1994 Return-Path: Received: from vms.macc.wisc.edu by ifcss.org (4.1/IFCSS-Mailer) id AA29382; Sat, 7 May 94 11:40:34 CDT Received: from VMSmail by wircs1.DecNet; Sat, 07 May 94 11:38 CDT Message-Id: <24050711380011@wircs1.DecNet> Date: Sat, 07 May 94 11:38 CDT From: CAI%MVF.DecNet@wigate.nic.wisc.edu Subject: preference To: soft-authors@ifcss.org X-Vms-To: WIRCS1::IN%"soft-authors@ifcss.org" Status: O After spending some time studying the three proposals (four if include B5E1), I found that they are very close and it is hard to tell which one is better. But I am leaning to Ya-Gui's original proposal right now for reasons Ya-Gui pointed out in his mail (by the way, I figured that out myself, just to make sure everybody not think otherwise). I propose the three proposals to be voted if no more questions coming up. cai From shin@hermes.ee.nthu.edu.tw Sat May 7 12:06:02 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA00256; Sat, 7 May 94 12:06:02 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182259(2)>; Sun, 8 May 1994 01:09:17 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA16526; Sun, 8 May 94 01:01:30 CST Date: Sun, 8 May 1994 15:01:30 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405071701.AA16526@hermes.ee.nthu.edu.tw> To: simpson@math.psu.edu, yawei@ifcss.org Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility Cc: ccnet-l%uga.uga.edu@twnmoe10.edu.tw, soft-authors@ifcss.org Status: O There is a concern on the "natural structure" of Big5 codes when we divide the codes in the different mapping methods. Here is my view points in a personal communication to Carlos. I think such "naturness" is somewhat subjective and do not have much siginificance especially for communication purposes. Ya-Gui did hit one point about sorting. This has been briefly explained in the mail to Carlos. Professor Simpson mentioned that we might use HZ+ (or other proposals) for information processing (that is, as the internal code.) I don't think it is acceptable for Big5 users and such implementations will be against their habits; the same is true for HZ users: if they live happily using HZ for information processing, we should not force them to accept its GB form for such processing. This also partially answers Ya-Gui's question on sorting. There are also other related issues. Here are the quotations: (I hope that Carlos will not mind to use the materials in this personal communication.) |> =============================== |> Dear Carlos, |> |> ... deleted ... |> |> I am very exciting on the current issues we are discussing, since |> the results of such discussions may create the first multi-lingual |> newsgroups(s) in the net. |> |> .... deleted ... please believe me, I am not trying to |> ignore Professor Simpson's contribution in all this discussion process. |> |> Even if my proposal (in Big5/8 <-> Big5/7 mapping) is considered more |> appropriate according to some critera and was adopted by the software |> authors, my proposal is just a follow-up of Professor Simpson's HZ+S |> scheme with the Big5/8<->Big5/7 engine replaced by my scheme. So it |> is still adequate to call all the mixed char set scheme as |> HZ+S and most of the credits are due to Professor Simpson. I have noticed that several persons are considering that I am proposing a completely new mixed character encoding schemes. If you are referring to my very first posting which asked people not to invent a new wheel, and introducing the CNS/7 bit codes, and the ESC sequences they used for communication, then this is a kind of such proposal. But, later there are opinions disfavoring ISO compatibility and in favor of HZ compatibility. I then post my first B5encode/decode.c to make it clear: I am not insisting on any such stuffs, it is just a kind of preference, and the change is more than trivial. (So why we mentioned ISO-compatibility again? -^^-) So what I am now concentrated is the Big5 8<->7 bit mapping algorithm. The answer to Professor's question that my B5encoding did not mentioned HZ at all is based on the same situation. If you had inspected my codes, you will find that I had leave the entry for HZ converters; any existing converter could be hooked up without much change. So my implementation is already a HZ+Sencoder/decoder. Waht I am concered is to propose (probably better) alternatives to Professor Simpson's original implementation and a different mapping method. So, when comparing my proposals it may be pointless to compare the other points. |> |> I did once propose to use appropriate non-private ESC sequence |> to make it 99% ISO-2022 compatible , but I did not insist on such |> issues since I know that the users of the newsgroups may have to |> change their installation too much. To follow the HZ-style escape, |> however, does not imply I technically have a stronger support over |> that. We are just fitting ourselves to user's need. So I don't |> think many people will say that the basic sprit of the |> mixed char set scheme is changed when we choose to use ~1 and ~2 |> instead of ~G and ~H or similar. What I am interested in simply |> in the Big5/8<->Big5/7 engine. I will not say that I am proposing |> a complete mixed char set scheme. |> |> Personally, I am very appreciate of Professor Simpson's initiation |> any valuable contribution to the HZ+S encoding schem. I had expressed |> this point in my first posting concerning 7-bit encoding of Chinese chars. |> And I am also very clear that Professor Simpson is considering not |> to exclude the Big5 users from the chinese.* hierarchy. I had also |> provide some of the information or known implementations as possible |> as I can. |> |> So, in this sense, I think we are in cooperation, not competition. |> |> Howervr, for the different mapping schemes, there do have different |> criteria for judgement and I hope that we can finally come out with |> an optimal or suboptimal solution. So I just tried to address this |> problem from other perspectives. I have also some different comments |> about the "naturness" of the mapping schemes. Such comments, however, |> does not implies that I dislike a particular scheme. The most enjoyable |> things I can get from the net is to share other's ideas and probably |> post some of my ideas if possible. |> |> As a Big5 user, I wish my paticipation in the discussion would reflect |> some of the (technical) points of view concerning our daily-used code |> and not be silent about this. |> |> As a technical person with good academic records (really -^^-), I think |> I know how to respect the contribution of the original author(s) of an |> idea. I am just trying to give my proposal to a part of the scheme. |> Please forgive me if I somehow did not express myself very well. |> |> Best wishes. |> |> Jing-Shin -^^- |> |> P.S. ... |> As usual, I have a few comments one the technical part |> of the following posting. Hope that my view points are |> not drastically different from yours. -^^- |> |> > > [B5E2/B5encode-2: A New Proposal for 7-bit Big5 Representation] |> > > 1994/05/07, Rev. 1.4 |> > > Jing-Shin Chang |> > > (shin@hermes.ee.nthu.edu.tw) |> > |> > Dear Dr. Chang: |> > |> > ... deleted ... |> > |> > It is of great concern to me as a programmer that the |> > encoding scheme we choose for 7-bit transmission be the |> > correct one. The most correct scheme, in my opinion, is |> > that which follows most closely the Big5 character code. |> > |> |> I agree with this to some extent. However, it is vague how we |> tell a scheme is more close to Big5 than another. Under my background |> I would consider one scheme as being closer than the other if we have |> a distance measure, and this distance measure tells me that |> distance(X,Big5) < distance(Y,Big5). My distance measure is |> obviously the bit-offset in my B5E1 proposal or the byte-offset |> in my B5E2 proposal. There are subject distance measures like |> to preserve the sorting order of the char set. This property, however, |> does not seem to have a good measure to tell us which is better, especially |> for chinese chars. In my B5E1 proposal, you can inspect the Big5 order by |> scanning two large code blocks, In my B5E2 proposal, you could traverse it |> in 8 (actually 7) continuous blocks. This is probably also a measure |> for such somewhat unclear criterion. But I tends to clarify things with |> more objective measures. |> |> In English text, you could say that I am sorting a file alphabetically. |> In chinese char (sets), however, you may have to say that I am sorting |> the file in Big5/8 order or CNS order and the like. But we rarely |> use such sorting for practical purposes. The best we can is to use such |> sorting order as part of our internal data structure for storing data |> records. |> |> Probably there are other better uses of such information. But, as a |> natural language researcher for so many years, I did not have a strong |> feeling on this point. To add some points: the code designers of the chinese codes has long arguing to consider the "proper" sorting sequence in designing the codes. However, there is no agreed criteria in this aspect up to the present time. In looking up an English dictionary, your primary searching key is the letters from 'a' to 'z'; in looking up a chinese dictionary, on the contrary, you have several choices (it also implies that you cannot do it in a unique and simple way.) You may index the words by number of strokes, you may index it by radical elements; you can index it with BoPoMoFo (or ping-yin in mainland china). So, such a sorting sequence rarely has any practical use in chinese information processing. |> |> > |> > Now we are discussing the establishment of a 7-bit Big5 encoding. |> > ... |> > At some point, however, we need to all agree on one proposal-- |> > ... |> |> I agree that we need to make a decision at some point, and I am not sure |> whether people is ready for making the decisions. But I don't think I am |> trying to keep up with any others. When I found out that I can make some |> compromise in the computation and encoding efficiency with the overhead |> of char plane switching, I, of course, wish others to know that there is |> such a solution. |> |> What I am thinking is that we make our decision based on several |> potentially usable schemes. And we should encourage proposals to |> the prospective soft-contributors before a formal vote is called. |> I didn't know whether the soft-authors@ifcss.org had made the final |> decision. If it is decided, I apologize for posting it at this point. |> |> BTW, my impression, from Professor Simpson's forwarded email of the |> soft-authors discussion list, seems to suggest that my B5E2 is not known |> to the ccnet-l (and soft-authors?) due to the mailer problems to ccnet-l |> these two days. Anybody (especially ccnet-l netters) received that |> proposal? |> |> > |> > I believe the HZ+ proposal was Steve's idea in the beginning. |> |> Absolutly agree. |> |> > Of course you did not copy his idea, but instead of offering |> > suggestions you offered an entire new proposal. This would |> > be fine if the first proposal was seriously flawed, but I |> > don't believe it was seriously flawed. Adding another |> > proposal was probably the right thing for you to do at the |> > time, but at this point having more proposals leads to more |> > confusion. |> |> As mentioned above, I was only trying to propose an alternative of |> the 7<->8 bit conversion part of the mixed char set scheme. I was not |> trying to propose an entirely new proposal. And I don't think I had |> annouced any details about the whole mixed char scheme. |> I do not think the original proposal was seriously flawed either. |> What we see is that these proposals are in some sense similar and have |> their relative advantages in the other aspects. If the soft contributors |> have supported one scheme, nobody will against that. |> But I don't think that I will force the soft contributors to consider |> a single proposal in the discussion stage. |> I even think that Ya-Gui could give us better advices from his working |> experiences on his HZ scheme and the HZ-2 proposal before making a serious |> decision. I don't think it would make any greater confusion. |> |> > |> > Steve's initiative and hard work also deserve respect, as do |> > anyone's. I could also add my proposal to the collection, |> > but it seems better to help with the work that has already |> > been done. Now both of your proposals have converged to the |> > point that they are very close. However, one is still a more |> > natural mapping to the Big5 code set. Maybe you could consider |> > combining your proposal with Steve's? If you would be willing |> > to do this, there would be less confusion, and the work could |> > continue in a more cooperative (instead of competitive) fashion. |> > Since Steve's proposal is probably the most ideal one, would |> > you be willing to consider giving it your support? |> |> I of course am willing to help if I can afford to. In fact, in my |> b5encode.c and b5decode.c, I am SUPPORTING the mixed char set scheme. |> You could easily remove my 7<->8 conversion codes (just a few lines) |> and replace it with the preferred and accepted 7/8 conversion algorithm, |> and hook up the HZ part, they will then [be] the HZ+Sencoder and HZ+Sdecoder. |> Probably this is what my support up to the current stage. |> |> I had also indicated that Professor's original implementation could be |> optimized by changing the multiplication and division operation with |> bit-shifts when Ya-Gui have some comments on computation efficiency of |> Professor's original implementation. I guess this is also a kind of support. |> |> In fact, it is easy to optimize Professor's original implementation so |> that the mapping could be done on-the-fly; you only need to construct a table |> (probably much larger than my B5E2, where 8 vectors are used) of offsets |> for the various strips of code blocks and add/subtract for encoding/decoding. |> That's what I could think of up to the present. I think I will not hide |> anything when I could contribute and make things better. |> But I will not try to confine myself to a single solution or consider any |> solution as the "final solution". |> |> > |> > ............................ I hope you will consider merging your |> > proposal with Steve's HZ+S. This will improve Steve's proposal by |> > putting more weight behind it, and will eliminate some of the |> > confusion and get the process moving forward. I know you deserve |> > to get credit for the work you have done. You will also get some |> > credit and a good feeling by supporting the winning candidate when |> > defeat is clear. This is the spirit of cooperation, and I hope |> > you agree with me that it is a good way to work. |> > |> |> I am sorry if I did not show a good manner in expressing my ideas. |> I hope that people would think that I am just too exciting on such issues. |> I am not sure whether the soft-authors had made their final decision. |> If this is the case, my posting at this time point is obviously not |> appropriate. In addition, I don't think I will have any feeling of winning |> or defeating no matter which mapping algorithm is accepted by the wide |> variety of soft contributors. We have our own personal preference, but |> it depends on those prospective soft contributors who are willing to |> write applications based on the selected mapping algorithm. |> |> > |> > ........... It would be a difficult and brave (gan tsui) step |> > for you to support Steve's proposal, but I think there are benefits |> > in 1) having a better natural mapping to Big5 and 2) working in |> > the spirit of cooperation. I hope you will be willing to |> > consider this suggestion. |> |> I agree with these two points. Of course, we may have different criteria |> on natural mapping. I am not sure whether other people think to pack |> a noncontinuous blocks codes into a region is more nature than just to |> move them as a whole in the code space. We need further comments to |> know their feeling. |> |> And, as mentioned above, such naturness does not have much significance for |> information processing; we are dealing with information exchange and we |> are not suppose to handle our text in this space. |> |> In know implementations, I did not notice any requirements for such |> natuarness either. |> |> > |> > Regards, |> > |> > Carlos McEvilly |> > |> > |> Best wishes. |> |> Jing-Shin -^^- |> ================================ > From yawei@ifcss.org Sat May 7 10:45:29 1994 > Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility > > > Looks like Steve is not being completely fair, :-) so I guess > a rebutal is needed. An "advantage" would commonly referred to > ... deleted ... > > Advantage (3): > >The fact that HZ+S fully respects the natural structure of the Big5 > >blocks leads to several significant advantages. These advantages are > > HZ-2 moves a few infrequently-used characters into the first > block. But this could only _reduce_ the need to switch into > the second block. It looks sorta' like an "advantage" to me, > albeit small. > > > (5). HZ+S uses escape sequences sparingly. The ~2 escape sequence is > > needed only to switch into Big5 block 2, the block containing the less > > frequently used Chinese characters. Most of the time, we use only ~1 > > to switch into Big5 block 1, plus ~{ and ~} as in HZ. > > With the new "B5encode", this is true for all 3 proposals. > Actually HZ-2 has a very tiny advantage in this regard, > see my rebutal to advantage (3) above. :-) > > >CONCLUSION > > To be fair, I think HZ+S's packing of all the Big5 characters > into two tight blocks without leaving unused gaps is aesthetically > desirable. I don't know why Steve forgot to mention this advantage. > With everything else being equal, this just might be the winning > edge. :-) > > Unfortunately, this advantage brings with it an disadvantage, which > is the increased in CPU time (true no matter how you implement it). > There's no free lunch, you know. > > One disadvantage to the new B5encode (as I understood it) would be > that the codes don't come in the same order as the Big5 codes. > This could be a problem if one decides to use an ASCII sorter > to sort a file and expect the output to be in Big5 order. > > My conclusion is that the 3 proposals are extremely similar > in merit (and very similar in detail), so I am quite indifferent > to which one is adopted. > > Ya-Gui > > From simpson@math.psu.edu Sat May 7 12:24:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA00629; Sat, 7 May 94 12:24:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA26845; Sat, 7 May 1994 13:22:36 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 13:22:35 -0400 Date: Sat, 7 May 1994 13:22:35 -0400 Message-Id: <199405071722.NAA14253@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Jing-Shin, please read the a.c.t FAQ In-Reply-To: <9405071500.AA16054@hermes.ee.nthu.edu.tw> References: <9405071500.AA16054@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O I wrote: > > do I think that someday there could be software that would work > > directly with 7-bit Big5, rather than translate it into 8-bit > > Big5 first? The answer is, yes, I think that could happen. It's > > possible. Similar things have happened in the past. An example > > is ZWDOS. Jing-Shin Chang replied: > It is of course POSSIBLE. However, even this is possible I don't think > it is acceptable to the Big5 users if we implement it in the incoming > chinese.* newsgroup in this way. What is not acceptable? Your comment makes no sense. Big5 users will be free to use any software they wish to access the chinese.* newsgroups. They will not be required to use ZWDOS or any other particular software. The only requirement is that their software will have to understand the HZ+ file format (or at least the 7-bit Big5 part of it). This requirement will be unavoidable, because HZ+ will be the standard medium for postings in the chinese.* hierarchy (assuming everything unfolds in the way we now envision). Jing-Shin, I think you mentioned earlier that are not very familiar with how HZ and alt.chinese.text work. The level of this discussion might be improved if you would familarize yourself with those things. A good starting point might be to read the alt.chinese.text FAQ list. As an additional point of information, I should mention that Ya-Gui Wei, the author of ZWDOS, has indicated that he is ready to incorporate 7-bit Big5 support into ZWDOS, as soon as the HZ+ standard is in place. > Big5 users - Use internal codes for information processing > and also for communication (this is possible since > our computing environment is almost, say 95%, 8-bit clean. I think you are saying "Big5 users" when what you really mean is "Big5 users in Taiwan". The Internet internationally (in the United States, for example) is far from 8-bit clean. We cope with this by using workarounds such as uuencode, MIME, and HZ. In the United States, the most common method of mailing a Big5 file is to uuencode it. > Using 7-bit Big5 or variants is simply a method to accomodate with > the HZ users' 7-bit convention for the new chinese.* groups. No. This is a serious misunderstanding on your part. Using 7-bit Big5 is *not* a method to accommodate the preferences of HZ users. Rather, it is a method to cope with the fact that the Internet is not 8-bit clean. > I guess they would like to use the internal codes directly if > possible. I'm sure you are correct in saying that Big5 users would *like* to use 8-bit Big5 *if possible*. What you are overlooking is that, in the international Internet environment, this is usually *not possible*. The reason is that the Internet is not 8-bit clean. > In fact, what we are discussing is only among the soft authors > based on our hacker's interests and characteristics, not among the > users.) Why do you keep criticizing soft-authors? Everybody in soft-authors have written Chinese software. Some of this software is used by many thousands of people. Soft-authors is well in touch with the needs of users. Please read the alt.chinese.text FAQ. > HZ users - Use exchange codes for information processing > and also for communication (I am leaning to consider > HZ as an exchange code. When you process a HZ file, > you need be careful not to delete those ESC's. Normally, What are you talking about? HZ does not use the Escape character. Instead, it uses ~, a graphical ASCII byte. The danger of deleting a ~ is not greater than the danger of deleting an A or a B. Please inform yourself about the nature of HZ. Please read the HZ specification. Please read the alt.chinese.text FAQ. Please get a copy of ZWDOS and read the documentation. > So, in the future inplementation of news readers, we should provide > both kinds of conversions so that HZ users can process HZ as usually > but the Big5 users should be given the Big5/8 streams. Yes, of course. My sample implementation of HZ+ produces 8-bit Big5 streams. If you would like to test my sample implementation, it is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. > I am not familiar with the environment of HZ users There, you said it yourself. > Did they process GB files frequently? Yes, they do. > If this is the case, it is also appropriate to convert it back to > GB and leave the users with a file without those ESC's. HZ users have software which does this. -- Steve From simpson@math.psu.edu Sat May 7 13:09:16 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA02042; Sat, 7 May 94 13:09:16 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA27032; Sat, 7 May 1994 14:07:11 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 14:07:09 -0400 Date: Sat, 7 May 1994 14:07:09 -0400 Message-Id: <199405071807.OAA14320@boole.math.psu.edu> To: John Delacour Cc: chinesenet@uclink.berkeley.edu, ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: I wish CNS would die and go away In-Reply-To: <9405071610.aa25003@hinge.demon.co.uk> References: <9405071610.aa25003@hinge.demon.co.uk> Reply-To: simpson@math.psu.edu Status: O To: John Delacour John, Your comments are encouraging. I'm *very* glad to learn there are such excellent prospects of incorporating HZ and HZ+ support into existing Macintosh mail and news software. In the DOS world also, things will move quickly. It's significant that Ya-Gui Wei, the author of ZWDOS, seems willing to commit to HZ+. As for the Unix world, it won't take me long to hack up some Emacs Lisp code for convenient HZ+ news and mail interfaces. The existing Emacs MIME interface will serve as a prototype. The only missing piece is the HZ+ standard itself. I had no idea that opening a discussion on soft-authors and ccnet would turn into such a shambles. Let's hope that an HZ+ consensus is still possible. > As to the CNS thing, sure Big5 could be easily converted to CNS > 7-bit but I don't see what the point would be. At least, unlike Ed > Lai, I have been aware of CNS and its relationship to Big5 for > several months and not just for a day or two. I'm sure Ed is a > very nice guy but ... Not only is Ed a nice guy, but he has made some nice contributions to Chinese computing on the Internet. In particular, he had a lot to do with the creation of the HZ standard. Having said that, I still insist that Ed's views on CNS are completely wrong. I have been aware of CNS for years. CNS has always been a thorn in the side of Chinese computing. It's current role is that of of a red herring which distracts us from the task of creating a viable HZ+ standard. -- Steve From shin@hermes.ee.nthu.edu.tw Sat May 7 13:27:48 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA02379; Sat, 7 May 94 13:27:48 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182320(1)>; Sun, 8 May 1994 02:29:45 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA16766; Sun, 8 May 94 02:19:41 CST Date: Sun, 8 May 1994 16:19:41 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405071819.AA16766@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: comment on Jing-Shin's latest new proposal Cc: CCNET-L%UGA.Bitnet@TWNMOE10.Edu.TW Status: O |> From: Stephen G Simpson |> |> Jing-Shin Chang writes: |> > [B5E2/B5encode-2: A New Proposal for 7-bit Big5 Representation] |> > |> > 1994/05/07, Rev. 1.4 |> > Since I am using the same escape sequence as proposed by Prof. Simpson, |> > all the advantages of his proposal, except for the 8-bit to 7-bit mapping |> > method, are also inherited from his HZ+S -^^- (Thanks Prof. Simpson). |> |> Apparently Jing-Shin's view is that the advantages of HZ+S derive |> mainly from the use of ~1 and ~2 as escape sequences. He thinks that Of course not. But I feel that some of the advantages you have mentioned has more or less related to your proposal in the particular ESC seq. For instance, you had mentioned ISO compatibility, which has once been agreed not to be in favored over HZ-compatibility. But it is mentioned again. So I mentioned this mainly to refer to such issues. |> by choosing the same escape sequences, his B5E2 proposal inherits most |> of the advantages of HZ+S. I disagree. The choice of escape |> sequences is only a minor detail. Yes. So we should concentrate on the design of 8<->7 mapping. In fact, in my B5encode/decode, I am appreciating these parts of proposals in your HZ+S proposal. We know that it is trivial. |> |> The following advantages of HZ+S are not inherited by B5E2: |> |> (1). The proposed HZ+S or HZ+ standard is based firmly on two |> existing standards which are already very popular: HZ and Big5. This |> should enable HZ+ to win a degree of immediate acceptance. I am not sure why this is an issue. I thought that all of us has considered to use the existing HZ in the HZ part of HZ+S or other variants and not to invent another HZ standard. That's why I never mentioned the HZ part. So I think all the proposals are based on HZ and Big5. |> |> [ Jing-Shin by his own admission is unfamiliar with HZ and GB. ] |> |> (2). HZ+S is very easy to implement. I have already hacked up some |> Big5 <-> HZ+S conversion filters. This experimental software is based |> on Fung F. Lee's GB <-> HZ conversion filters and is available for |> anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps/. True. I didn't hack up any GB/HZ/Big5 converter. But I used them in various applications. (hztty, cxterm, ...) I think I could also hook Lee's converter (and any public domain converters) in hours (or minutes). But it does not seem to be an issues when we are discussing the Big5 7-bit mapping. |> |> (3). HZ+S fully respects the natural partition of the Big5 code |> range into blocks of more and less frequently used characters. |> This leads to several additional advantages, listed in (4)-(7) |> below. Natural partition is somewaht vague to define. And more importantly, we need to know whether a partition that we feed nature in some sense has any practical application for such naturness. Here are some of my view points in a communication to Carlos: ... deleted ... ................................ I have also some different comments about the "naturness" of the mapping schemes. Such comments, however, does not implies that I dislike a particular scheme. ... deleted ... ........................................., it is vague how we tell a scheme is more close to Big5 than another. Under my background I would consider one scheme as being closer than the other if we have a distance measure, and this distance measure tells me that distance(X,Big5) < distance(Y,Big5). My distance measure is obviously the bit-offset in my B5E1 proposal or the byte-offset in my B5E2 proposal. There are subject distance measures like to preserve the sorting order of the char set. This property, however, does not seem to have a good measure to tell us which is better, especially for chinese chars. In my B5E1 proposal, you can inspect the Big5 order by scanning two large code blocks, In my B5E2 proposal, you could traverse it in 8 (actually 7) continuous blocks. This is probably also a measure for such somewhat unclear criterion. But I tends to clarify things with more objective measures. In English text, you could say that I am sorting a file alphabetically. In chinese char (sets), however, you may have to say that I am sorting the file in Big5/8 order or CNS order and the like. But we rarely use such sorting for practical purposes. The best we can is to use such sorting order as part of our internal data structure for storing data records. Probably there are other better uses of such information. But, as a natural language researcher for so many years, I did not have a strong feeling on this point. ---- To add some points: the code designers of the chinese codes has long arguing to consider the "proper" sorting sequence in designing the codes. However, there is no agreed criteria in this aspect up to the present time. In looking up an English dictionary, your primary searching key is the letters from 'a' to 'z'; in looking up a chinese dictionary, on the contrary, you have several choices (it also implies that you cannot do it in a unique and simple way.) You may index the words by number of strokes, you may index it by radical elements; you can index it with BoPoMoFo (or ping-yin in mainland china). So, such a sorting sequence rarely has any practical use in chinese information processing. --- ... Of course, we may have different criteria on natural mapping. I am not sure whether other people think to pack a noncontinuous blocks codes into a region is more nature than just to move them as a whole in the code space. We need further comments to know their feeling. -- Is such feeling an important issues? -- And, as mentioned above, such naturness does not have much significance for information processing; we are dealing with information exchange and we are not suppose to handle our text in this space. In know implementations, I did not notice any requirements for such natuarness either. ... end of message ... |> |> (4). Existing Big5 fonts can be used essentially unchanged for HZ+S. |> I don't think it needs any significant changes to use the existing fonts, because the efforts to find the index of the font of a characters is essentially the same. |> .... |> |> (6). HZ+S is fully compatible not only with ETen Big5 but also |> with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which |> the less frequently used Chinese characters begin somewhere prior |> to C940. Thus HZ+S may help to overcome the difficulties |> associated with these corrupt variants. I am not familiar with this problem. Could you gives an example? Thanks. |> |> (7). The way is open for HZ+S to be compatible with a possible |> future incorporation of Big5 into ISO-2022-JP-2, similar to the |> present incorporation of CNS-11643. Such an unofficial extension |> of ISO-2022-JP-2 is already implemented in Mule, the multilingual |> Emacs. When I proposed not to use private ESC's in my first posting, I had suggested to use a (99%) ISO compatible schemes, but people think that ISO compatibility is not as important as HZ compatiblity. So I didn't insist on that. I think it is trivial to switch back to that status. |> |> -- Steve |> From simpson@math.psu.edu Sat May 7 14:27:54 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA04207; Sat, 7 May 94 14:27:54 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA27333; Sat, 7 May 1994 15:25:52 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 15:25:50 -0400 Date: Sat, 7 May 1994 15:25:50 -0400 Message-Id: <199405071925.PAA14434@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re: comment on Jing-Shin's latest new proposal In-Reply-To: <9405071819.AA16766@hermes.ee.nthu.edu.tw> References: <9405071819.AA16766@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > But I feel that some of the advantages you have mentioned has more > or less related to your proposal in the particular ESC seq. For > instance, you had mentioned ISO compatibility, which has once been > agreed not to be in favored over HZ-compatibility. But it is > mentioned again. No. Let me repeat. The advantages of HZ+S have nothing to do with the choice of escape sequences. The kind of ISO compatibility that I am talking about has nothing to do with the choice of escape sequences. The choice of escape sequences is a very minor, relatively unimportant detail. The kind of ISO compatibility that I am talking about is concerned not with the choice of escape sequences but with the natural structure of the Big5 character set, in particular the natural division into two blocks and the ordering withing each block. If you don't perceive any such structure, then you will never be able to understand my kind of ISO compatibility. In that case, I urge you to first read my document describing the Big5 character set, math.psu.edu, /pub/simpson/chinese/hzps/etfonts.txt. If you take the time to read that document and look at a Big5 table, you will surely see that the Big5 character set does indeed have a structure. After that, I urge you to look at Mule, the Multilingual Emacs. You will see that Mule's method of incorporating Big5 into ISO-2022-JP-2 is exactly the same as my method of incorporating Big5 into HZ+. Both both methods fully respect the structure of the Big5 character set. Incidentally, the author of Mule was also a member of the ISO-2022-JP-2 committee. > We know that it is trivial. Sorry, but I don't think it is so trivial. > |> (1). The proposed HZ+S or HZ+ standard is based firmly on two > |> existing standards which are already very popular: HZ and Big5. This > |> should enable HZ+ to win a degree of immediate acceptance. > > I am not sure why this is an issue. ... > I think all the proposals are based on HZ and Big5. All of them except yours. Your documents on B5encode never mention HZ. Moreover, you have said that you are not familiar with HZ and do not understand the reasons for HZ or the environment in which HZ users operate. > |> (3). HZ+S fully respects the natural partition of the Big5 code > |> range into blocks of more and less frequently used characters. > |> This leads to several additional advantages, listed in (4)-(7) > |> below. > > Natural partition is somewaht vague to define. > ........................................., it is vague how we > tell a scheme is more close to Big5 than another. Under my background > I would consider one scheme as being closer than the other if we have > a distance measure, and this distance measure tells me that > distance(X,Big5) < distance(Y,Big5). My distance measure is Your remarks show that you do not appreciate or value the natural structure of the Big5 character set. Presumably that is why your b5encode does not respect this structure. If you are interested in learing about the structure of the Big5 character set, please read my document on this subject, referenced above. > |> (6). HZ+S is fully compatible not only with ETen Big5 but also > |> with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which > |> the less frequently used Chinese characters begin somewhere prior > |> to C940. Thus HZ+S may help to overcome the difficulties > |> associated with these corrupt variants. > > I am not familiar with this problem. Could you gives an example? > Thanks. An example is cxterm's Big5 support. If you study cxterm, you may see what I am talking about. > |> (7). The way is open for HZ+S to be compatible with a possible > |> future incorporation of Big5 into ISO-2022-JP-2, similar to the > |> present incorporation of CNS-11643. Such an unofficial extension > |> of ISO-2022-JP-2 is already implemented in Mule, the multilingual > |> Emacs. > > When I proposed not to use private ESC's in my first posting, I had > suggested to use a (99%) ISO compatible schemes, but people think > that ISO compatibility is not as important as HZ compatiblity. So I > didn't insist on that. I think it is trivial to switch back to that > status. You seem to think that ISO-2022 compatibility is nothing but a matter of using ISO-2022 style escape sequences. In reality, the choice of escape sequences is a minor, unimportant point. The really important point is the structure of character sets. Please look at Mule. -- Steve From jiang Sat May 7 14:28:28 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA04343; Sat, 7 May 94 14:28:28 CDT Date: Sat, 7 May 94 14:28:28 CDT From: jiang (Yuan Jiang) Message-Id: <9405071928.AA04343@ifcss.org> Subject: Re: Big5 <--> 7-bit Big5 is simple Cc: soft-authors Status: O > >are you saying to use purely 7-bit CNS w/o any escape sequence? such as i >assume has been done on Seednet, or to incorporate 7-bit CNS instead of > With escape sequence of course. Steve is right that we should put the decision on which sequence, ~1 or ~+ etc, after the encoding method to chosen. From shin@hermes.ee.nthu.edu.tw Sat May 7 14:58:07 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA05224; Sat, 7 May 94 14:58:07 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182341(1)>; Sun, 8 May 1994 04:00:58 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA16843; Sun, 8 May 94 03:50:54 CST Date: Sun, 8 May 1994 17:50:54 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405071950.AA16843@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility Cc: CCNET-L%UGA.Bitnet@TWNMOE10.Edu.TW Status: O |> From simpson@math.psu.edu Sat May 7 13:31:06 1994 |> |> Jing-Shin Chang writes: |> > You can access my new proposal at: |> > |> > hermes.ee.nthu.edu.tw(140.114.28.27):/shin/b5encode |> |> > In my B5E2, the Freq. used & Less Freq. used division is also fully |> > respected. In addition, the relative code positions are retained. |> |> How is that? In HZ+S, the relative code positions are indeed |> retained, because the codes are simply packed in order onto an initial |> segment of the graphical ASCII range. In your proposal, the order is |> changed, right? Or at least the codes are not packed tightly, |> i.e. there are gaps in the range of the mapping. In HZ+S, you are preserving the character order in the big5 char order more explicitly in the sense as mentioned above. In my B5E2, a whole block of codes are shifted as a whole in the code space. The sequential order is implicitly preserved, but there is no much difficulty to find out the big5 char order. The most important point is, you rarely need to process you data in the transformed code space. So I never consider it a problem. In fact, the sorting sequence in the various chinese codes are designed based on all kinds of criteria which are subject to arguing. You simply cannot use it for indexing purpose as simple as looking up an English dictionary. (Don't you think the blocks of codes in the transformed code space an exact copy of their counterpart in the original Big5 code space? Is it important to pack the codes tightly. -^^-) |> |> > However, the current discussion is to have a good exchange code; |> > therefore, the relation with the font may not be so closely |> > related. |> |> No. The current discussion is *not* about how to have a good exchange |> code. The current discussion is about how to incorporate 7-bit Big5 |> into HZ resulting in HZ+. Right now, in 1994, HZ is used not only as |> an exchange code but also as an internal code. There is every reason |> to believe that HZ+ will continue to be used in exactly the same way |> as HZ+ is now used. Ya-Gui has already indicated his plan to support |> HZ+ in ZWDOS. ZWDOS has screen and printing capabilities, so fonts |> will be needed. |> I guess you are right for the HZ users, but not for the Big5 users. Even if Ya-Gui is willing to support (his revised version of ZWDOS?), I would suggest him to transform the HZ+* codes to Big5 internal codes before any information processing. You cannot force the Big5 users to change drastically to accept a different computing schemes and habits. It's true that HZ users live happily with the current status, i.e., processing their text with the ESC characters. I don't think this is a normal way for Big5 users. Do you see any famous utilities that process you documents and not hide you those non-text elements like those ESC codes? I agree that we are working on a standard that is acceptable by those potential users. But I don't think that we are to change their habits. (Just like that we are not willing to force Big5 Users to become CNS users.) |> The fact that Big5 fonts can be used essentially unchanged is a |> significant advantage of HZ+S which is not shared by the other HZ+ |> proposals. Why the other proposals cannot use it directly? I had a few comments on this point in a reply to Ya-Gui's comments on your posting. |> |> Jing-Shin, I'd suggest you familiarize yourself with HZ. I have the |> feeling that you don't know what considerations motivated it and how |> it has developed. Thanks for your suggestion. But I am wondering if it is helpful for me to design the mapping schemes for 7-bit Big5. |> |> I notice that your proposals do not include HZ. Is this omission |> intentional? Don't we all agree to use the existing HZ standard when we switched to '~{' (or whatever HZ ESC)? Do I need to mention anything in addition to this fact? In fact, my .c codes have some "holes" there for hooking up any HZ related converters. I had also mentioned that they could become HZ+* encoder/decoder by plugging-in related filter. I think it is quite explicit. In all my proposals (except the very early posting which ask people not to invent new wheels), I always used B5encoding/decoding to emphasize this undecide part. So I think there should be no misunderstanding. |> |> > In brief, ISO-compatibility is better defined by (1) do not use |> > non-control chars as ESC chars. (2) follow the general syntax of |> > the ESC sequences for designating a character set or a char. plane |> > in that set. So, as long as |> |> I disagree. The choice of escape sequences is only a minor detail. |> True ISO-2022 compatibility means more than a certain syntax for |> escape sequences. True ISO-2022 compatibility means: to handle mixed |> character sets using the same kind of mappings as in ISO-2022. HZ+S |> has this kind of ISO-2022 compatibility. The other 7-bit Big5 |> proposals do not have it. Are you referring to the use of the 94x94 characters for coding multibyte characters? If you are referring to this fact, (1) is just an alternative statement of this rule, and (2) is related to the tasks of designating the mixing character sets. I think we are referring to the same rules. Then by using your proposed ESC's (or any other's, like the one proposed in my very early posting for using CNS), plus an HZ encoding/decoding standard, puls a Big5 encoding/decoding standard, I think ISO compatibility is not a big problem. (If we need to go back to my early proposal for using those ESC sequences for CNS, I think it is easy.) |> |> > Now, we have four proposals. |> |> Great! :-) |> |> > So I tried very hard (really !!-^^-) to propose my alternatives and |> > hope that they are as competitive as the other proposals. Probably |> > there are other persons who have better ideas. Would you netters |> > like to contribute better alternatives? |> |> Let a hundred flowers bloom! The only problem is that, by the time we |> reach a consensus on HZ+, we may be well into the next century. Not so bad. -^^- |> |> -- Steve |> From ygz@cs.purdue.edu Sat May 7 15:08:32 1994 Return-Path: Received: from arthur.cs.purdue.edu by ifcss.org (4.1/IFCSS-Mailer) id AA05623; Sat, 7 May 94 15:08:32 CDT Received: from ector.cs.purdue.edu (root@ector.cs.purdue.edu [128.10.2.10]) by arthur.cs.purdue.edu (8.6.4/PURDUE_CS-1.3) with ESMTP id for ; Sat, 7 May 1994 15:06:32 -0500 Received: from localhost (ygz@localhost [127.0.0.1]) by ector.cs.purdue.edu (8.6.4/PURDUE_CS-1.3) with SMTP id for ; Sat, 7 May 1994 15:06:29 -0500 Message-Id: <199405072006.PAA18388@ector.cs.purdue.edu> To: soft-authors@ifcss.org Subject: War on encoding scheme Date: Sat, 07 May 1994 15:06:27 -0500 From: ygz@cs.purdue.edu (Yongguang Zhang) Status: O I know the water is muddy enough, but let me throw another stone: How about 7-bit Unicode, another mail-safe standard? (See the example at the end of the document.) -ygz ============================================================================= INTERNET-DRAFT D. Goldsmith draft-goldsmith-mime-utf7-03.txt M. Davis Expires: October 1994 Taligent, Inc. April 1994 UTF-7 A Mail-Safe Transformation Format of Unicode Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months. Internet-Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet- Drafts as reference material or to cite them other than as a ``working draft'' or ``work in progress.'' To learn the current status of any Internet-Draft, please check the 1id-abstracts.txt listing contained in the Internet-Drafts Shadow Directories on ds.internic.net, nic.nordu.net, ftp.isi.edu, or munnari.oz.au. Abstract The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993(E) jointly define a 16 bit character set (hereafter referred to as Unicode) which encompasses most of the world's writing systems. However, Internet mail (STD 11, RFC 822) currently supports only 7- bit US ASCII as a character set. MIME (RFC 1521 and RFC 1522) extends Internet mail to support different media types and character sets, and thus could support Unicode in mail messages. MIME neither defines Unicode as a permitted character set nor specifies how it would be encoded, although it does provide for the registration of additional character sets over time. This document describes a new transformation format of Unicode that contains only 7-bit ASCII characters and is intended to be readable by humans in the limiting case that the document consists of characters from the US-ASCII repertoire. It also specifies how this transformation format is used in the context of RFC 1521, RFC 1522, Goldsmith & Davis [Page 1] Internet Draft UTF-7 April 1994 and the document "Using Unicode with MIME". Motivation Although other transformation formats of Unicode exist and could conceivably be used in this context (most notably UTF-1 and UTF-8, also known as UTF-2 or UTF-FSS), they suffer the disadvantage that they use octets in the range decimal 128 through 255 to encode Unicode characters outside the US-ASCII range. Thus, in the context of mail, those octets must themselves be encoded. This requires putting text through two successive encoding processes, and leads to a significant expansion of characters outside the US-ASCII range, putting non-English speakers at a disadvantage. For example, using UTF-FSS together with the Quoted-Printable content transfer encoding of MIME represents US-ASCII characters in one octet, but other characters may require up to nine octets. Overview UTF-7 encodes Unicode characters as US-ASCII, together with shift sequences to encode characters outside that range. For this purpose, one of the characters in the US-ASCII repertoire is reserved for use as a shift character. Many mail gateways and systems cannot handle the entire US-ASCII character set (those based on EBCDIC, for example), and so UTF-7 contains provisions for encoding characters within US-ASCII in a way that all mail systems can accomodate. UTF-7 should normally be used only in the context of 7 bit transports, such as mail and news. In other contexts, straight Unicode or UTF-8 is preferred. See the document "Using Unicode with MIME" for the overall specification on usage of Unicode transformation formats with MIME. Definitions First, the definition of Unicode: The 16 bit character set Unicode is defined by "The Unicode Standard, Version 1.1". This character set is identical with the character repertoire and coding of the international standard ISO/IEC 10646-1:1993(E); Coded Representation Form=UCS-2; Subset=300; Implementation Level=3. Note. Unicode 1.1 further specifies the use and interaction of these character codes beyond the ISO standard. However, any valid Goldsmith & Davis [Page 2] Internet Draft UTF-7 April 1994 10646 BMP (Basic Multilingual Plane) sequence is a valid Unicode sequence, and vice versa; Unicode supplies interpretations of sequences on which the ISO standard is silent as to interpretation. Next, some handy definitions of US-ASCII character subsets: Set D (directly encoded characters) consists of the following characters (derived from RFC 1521, Appendix B): the upper and lower case letters A through Z and a through z, the 10 digits 0-9, and the following nine special characters (note that "+" and "=" are omitted): Character ASCII & Unicode Value (decimal) ' 39 ( 40 ) 41 , 44 - 45 . 46 / 47 : 58 ? 63 Set O (optional direct characters) consists of the following characters (note that "\" and "~" are omitted): Character ASCII & Unicode Value (decimal) ! 33 " 34 # 35 $ 36 % 37 & 38 * 42 ; 59 < 60 = 61 > 62 @ 64 [ 91 ] 93 ^ 94 _ 95 ` 96 { 123 | 124 } 125 Goldsmith & Davis [Page 3] Internet Draft UTF-7 April 1994 Rationale. The characters "\" and "~" are omitted because they are often redefined in variants of ASCII. Set B (Modified Base 64) is the set of characters in the Base64 alphabet defined in RFC 1521, excluding the pad character "=" (decimal value 61). Rationale. The pad character = is excluded because UTF-7 is designed for use within header fields as set forth in RFC 1522. Since the only readable encoding in RFC 1522 is "Q" (based on RFC 1521's Quoted- Printable), the "=" character is not available for use (without a lot of escape sequences). This was very unfortunate but unavoidable. The "=" character could otherwise have been used as the UTF-7 escape character as well (rather than using "+"). Note that all characters in US-ASCII have the same value in Unicode when zero-extended to 16 bits. UTF-7 Definition A UTF-7 stream represents 16-bit Unicode characters in 7-bit US-ASCII as follows: Rule 1: (direct encoding) Unicode characters in set D above may be encoded directly as their ASCII equivalents. Unicode characters in Set O may optionally be encoded directly as their ASCII equivalents, bearing in mind that many of these characters are illegal in header fields, or may not pass correctly through some mail gateways. Rule 2: (Unicode shifted encoding) Any Unicode character sequence may be encoded using a sequence of characters in set B, when preceded by the shift character "+" (US-ASCII character value decimal 43). The "+" signals that subsequent octets are to be interpreted as elements of the Modified Base64 alphabet until a character not in that alphabet is encountered. Such characters include control characters such as carriage returns and line feeds; thus, a Unicode shifted sequence always terminates at the end of a line. As a special case, if the sequence terminates with the character "-" (US-ASCII decimal 45) then that character is absorbed; other terminating characters are not absorbed and are processed normally. Rationale. A terminating character is necessary for cases where the next character after the Modified Base64 sequence is part of character set B. It can also enhance readability by delimiting encoded sequences. Goldsmith & Davis [Page 4] Internet Draft UTF-7 April 1994 Also as a special case, the sequence "+-" may be used to encode the character "+". A "+" character followed immediately by any character other than members of set B or "-" is an ill-formed sequence. Unicode is encoded using Modified Base64 by first converting Unicode 16-bit quantities to an octet stream (with the most significant octet first). Text with an odd number of octets is ill-formed. Rationale. ISO/IEC 10646-1:1993(E) specifies that when characters in the UCS-2 form are serialized as octets, that the most significant octet appear first. This is also in keeping with common network practice of choosing a canonical format for transmission. Next, the octet stream is encoded by applying the Base64 content transfer encoding algorithm as defined in RFC 1521, modified to omit the "=" pad character. Instead, when encoding, zero bits are added to pad to a Base64 character boundary. When decoding, any bits at the end of the Modified Base64 sequence that do not constitute a complete 16-bit Unicode character are discarded. If such discarded bits are non-zero the sequence is ill-formed. Rationale. The pad character "=" is not used when encoding Modified Base64 because of the conflict with its use as an escape character for the Q content transfer encoding in RFC 1522 header fields, as mentioned above. Rule 3: The space (decimal 32), tab (decimal 9), carriage return (decimal 13), and line feed (decimal 10) characters may be directly represented by their ASCII equivalents. However, note that MIME content transfer encodings have rules concerning the use of such characters. Usage that does not conform to the restrictions of RFC 822, for example, would have to be encoded using MIME content transfer encodings other than 7bit or 8bit, such as quoted-printable, binary, or base64. Given this set of rules, Unicode characters which may be encoded via rules 1 or 3 take one octet per character, and other Unicode characters are encoded on average with 2 2/3 octets per character plus one octet to switch into Modified Base64 and an optional octet to switch out. Example. The Unicode sequence "A." (hexadecimal 0041,2262,0391,002E) may be encoded as follows: A+ImIDkQ. Goldsmith & Davis [Page 5] Internet Draft UTF-7 April 1994 Example. The Unicode sequence "Hi Mom !" (hexadecimal 0048, 0069, 0020, 004D, 006F, 004D, 0020, 263A, 0021) may be encoded as follows: Hi Mom +Jjo-! Example. The Unicode sequence representing the Han characters for the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E) may be encoded as follows: +ZeVnLIqe- Use of Character Set UTF-7 Within MIME Character set UTF-7 is safe for mail transmission and therefore may be used with any content transfer encoding in MIME (except where line length and line break restrictions are violated). Specifically, the 7 bit encoding for bodies and the Q encoding for headers are both acceptable. The MIME character set identifier is UNICODE-1-1-UTF7. Example. Here is a text portion of a MIME message containing the Unicode sequence "Hi Mom !" (hexadecimal 0048, 0069, 0020, 004D, 006F, 004D, 0020, 263A, 0021). Content-Type: text/plain; charset=UNICODE-1-1-UTF7 Hi Mom +Jjo-! Example. Here is a text portion of a MIME message containing the Unicode sequence representing the Han characters for the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E). Content-Type: text/plain; charset=UNICODE-1-1-UTF7 +ZeVnLIqe- Example. Here is a text portion of a MIME message containing the Unicode sequence "A." (hexadecimal 0041,2262,0391,002E). Content-Type: text/plain; charset=UNICODE-1-1-UTF7 A+ImIDkQ. Example. Here is a text portion of a MIME message containing the Unicode sequence "Item 3 is 1." (hexadecimal 0049, 0074, 0065, 006D, 0020, 0033, 0020, 0069, 0073, 0020, 00A3, 0031, 002E). Goldsmith & Davis [Page 6] Internet Draft UTF-7 April 1994 Content-Type: text/plain; charset=UNICODE-1-1-UTF7 Item 3 is +AKM-1. Note that to achieve the best interoperability with systems that may not support Unicode or MIME, when preparing text for mail transmission line breaks should follow Internet conventions. This means that lines should be short and terminated with the proper SMTP CRLF sequence. Unicode LINE SEPARATOR (hexadecimal 2028) and PARAGRAPH SEPARATOR (hexadecimal 2029) should be converted to SMTP line breaks. Ideally, this would be handled transparently by a Unicode-aware user agent. This preparation is not absolutely necessary, since UTF-7 and the appropriate MIME content transfer encoding can handle text that does not follow Internet conventions, but readability by systems without Unicode or MIME will be impaired. See RFC 1521 for an in-depth discussion of mail interoperability issues. Lines should never be broken in the middle of a UTF-7 shifted sequence, since such sequences may not cross line breaks. Therefore, UTF-7 encoding should take place after line breaking. If a line containing a shifted sequence is too long after encoding, a MIME content transfer encoding such as Quoted Printable can be used to encode the text. Another possibility is to perform line breaking and UTF-7 encoding at the same time, so that lines containing shifted sequences already conform to length restrictions. Discussion In this section we will motivate the introduction of UTF-7 as opposed to the alternative of using the existing transformation formats of Unicode (e.g. UTF-8) with MIME's content transfer encodings. Before discussing this, it will be useful to list some assumptions about character frequency within typical natural language text strings that we use to estimate typical storage requirements: 1. Most Western European languages use roughly 7/8 of their letters from US-ASCII and 1/8 from Latin 1 (ISO-8859-1). 2. Most non-European alphabet-based languages (e.g., Greek) use about 1/6 of their letters from ASCII (since white space is in the 7-bit area) and the rest from their alphabets. 3. East Asian ideographic-based languages (including Japanese) use essentially all of their characters from the Han or CJK syllabary area. Goldsmith & Davis [Page 7] Internet Draft UTF-7 April 1994 4. Non-directly encoded punctuation characters do not occur frequently enough to affect the results. Notice that current 8 bit standards, such as ISO-8859-x, require use of a content transfer encoding. For comparison with the subsequent discussion, the costs break down as follows (note that many of these figures are approximate since they depend on the exact composition of the text): 8859-x in Base64 Text type Average octets/character All 1.33 8859-x in Quoted Printable Text type Average octets/character US-ASCII 1 Western European 1.25 Other 2.67 Note also that Unicode encoded in Base64 takes a constant 2.67 octets per character. For purposes of comparison, we will look at UTF-8 in Base64 and Quoted Printable, and UTF-7. UTF-1 gives results substantially similar to UTF-8. Also note that fixed overhead for long strings is relative to 1/n, where n is the encoded string length in octets. UTF-8 in Base64 Text type Average octets/character US-ASCII 1.33 Western European 1.5 Some Alphabetics 2.44 All others 4 UTF-8 in Quoted Printable Text type Average octets/character US-ASCII 1 Western European 1.63 Some Alphabetics 5.17 All others 7-9 Goldsmith & Davis [Page 8] Internet Draft UTF-7 April 1994 UTF-7 Text type Average octets/character Most US-ASCII 1 Western European 1.5 All others 2.67+2/n We feel that the UTF-8 in Quoted Printable option is not viable due to the very large expansion of all text except Western European. This would only be viable in texts consisting of large expanses of US- ASCII or Latin characters with occasional other characters interspersed. We would prefer to introduce one encoding that works reasonably well for all users. We also feel that UTF-8 in Base64 has high expansion for non- Western-European users, and is less desirable because it cannot be read directly, even when the content is largely US-ASCII. The base encoding of UTF-7 gives competitive results and is readable for ASCII text. UTF-7 gives results competitive with ISO-8859-x, with access to all of the Unicode character set. We believe this justifies the introduction of a new transformation format of Unicode. As an alternative to use of UTF-7, it is possible to intermix Unicode characters with other character sets using an existing MIME mechanism, the multipart/mixed content type (thanks to Nathaniel Borenstein for pointing this out). For instance (repeating an earlier example): Content-type: multipart/mixed; boundary=foo --foo Content-type: text/plain; charset=us-ascii Hi Mom --foo Content-type: text/plain; charset=UNICODE-1-1 Content-transfer-encoding: base64 Jjo= --foo Content-type: text/plain; charset=us-ascii ! --foo-- Theoretically, this removes the need for UTF-7 in message bodies Goldsmith & Davis [Page 9] Internet Draft UTF-7 April 1994 (multipart may not be used in header fields). However, we feel that as use of the Unicode character set becomes more widespread, intermittent use of specialized Unicode characters (such as dingbats and mathematical symbols) will occur, and that text will also typically include small snippets from other scripts, such as Cyrillic, Greek, or East Asian languages (anything in the Roman script is already handled adequately by existing MIME character sets). Although the multipart technique works well for large chunks of text in alternating character sets, we feel it does not adequately support the kinds of uses just discussed, and so we still believe the introduction of UTF-7 is justified. Goldsmith & Davis [Page 10] Internet Draft UTF-7 April 1994 Summary The UTF-7 encoding allows Unicode characters to be encoded within the US-ASCII 7 bit character set. It is most effective for Unicode sequences which contain relatively long strings of US-ASCII characters interspersed with either single Unicode characters or strings of Unicode characters, as it allows the US-ASCII portions to be read on systems without direct Unicode support. UTF-7 should only be used with 7 bit transports such as mail and news. In other contexts, use of straight Unicode or UTF-8 is preferred. Acknowledgements Many thanks to the following people for their contributions, comments, and suggestions. If we have omitted anyone it was through oversight and not intentionally. Glenn Adams Harald T. Alvestrand Nathaniel Borenstein Lee Collins Jim Conklin Dave Crocker Steve Dorner Dana S. Emery Ned Freed Kari E. Hurtta John H. Jenkins John C. Klensin Valdis Kletnieks Keith Moore Masataka Ohta Einar Stefferud Erik M. van der Poel Goldsmith & Davis [Page 11] Internet Draft UTF-7 April 1994 Appendix A -- Examples Here is a longer example, taken from a document originally in Big5 code. It has been condensed for brevity. There are two versions: the first uses optional characters from set O (and thus may not pass through some mail gateways), and the second uses no optional characters. Content-type: text/plain; charset=unicode-1-1-utf7 Below is the full Chinese text of the Analects (+itaKng-). The sources for the text are: "The sayings of Confucius," James R. Ware, trans. +U/BTFw-: +ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation) +Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990. "The Chinese Classics with a Translation, Critical and Exegetical Notes, Prolegomena, and Copius Indexes," James Legge, trans., Taipei: Southern Materials Center Publishing, Inc., 1991. (Chinese text with English translation) Big Five and GB versions of the text are being made available separately. Neither the Big Five nor GB contain all the characters used in this text. Missing characters have been indicated using their Unicode/ISO 10646 code points. "U+-" followed by four hexadecimal digits indicates a Unicode/10646 code (e.g., U+-9F08). There is no good solution to the problem of the small size of the Big Five/GB character sets; this represents the solution I find personally most satisfactory. (omitted...) I have tried to minimize this problem by using variant characters where they were available and the character actually in the text was not. Only variants listed as such in the +XrdxmVtXUXg- were used. (omitted...) John H. Jenkins +TpVPXGBG- John_Jenkins@taligent.com 5 January 1993 Goldsmith & Davis [Page 12] Internet Draft UTF-7 April 1994 (omitted...) Content-type: text/plain; charset=unicode-1-1-utf7 Below is the full Chinese text of the Analects (+itaKng-). The sources for the text are: +ACI-The sayings of Confucius,+ACI- James R. Ware, trans. +U/BTFw-: +ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation) +Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990. +ACI-The Chinese Classics with a Translation, Critical and Exegetical Notes, Prolegomena, and Copius Indexes,+ACI- James Legge, trans., Taipei: Southern Materials Center Publishing, Inc., 1991. (Chinese text with English translation) Big Five and GB versions of the text are being made available separately. Neither the Big Five nor GB contain all the characters used in this text. Missing characters have been indicated using their Unicode/ISO 10646 code points. +ACI-U+-+ACI- followed by four hexadecimal digits indicates a Unicode/10646 code (e.g., U+-9F08). There is no good solution to the problem of the small size of the Big Five/GB character sets+ADs- this represents the solution I find personally most satisfactory. (omitted...) I have tried to minimize this problem by using variant characters where they were available and the character actually in the text was not. Only variants listed as such in the +XrdxmVtXUXg- were used. (omitted...) John H. Jenkins +TpVPXGBG- John+AF8-Jenkins+AEA-taligent.com 5 January 1993 (omitted...) Goldsmith & Davis [Page 13] Internet Draft UTF-7 April 1994 Security Considerations This document raises no security issues. References [UNICODE 1.1] "The Unicode Standard, Version 1.1": Version 1.0, Volume 1 (ISBN 0-201-56788-1), Version 1.0, Volume 2 (ISBN 0- 201-60845-6), and "Unicode Technical Report #4, The Unicode Standard, Version 1.1" (available from The Unicode Consortium, and soon to be published by Addison- Wesley). [ISO 10646] ISO/IEC 10646-1:1993(E) Information Technology--Universal Multiple-octet Coded Character Set (UCS). [MIME/UNICODE] "Using Unicode with MIME", Goldsmith, D. and M. Davis. [US-ASCII] Coded Character Set--7-bit American Standard Code for Information Interchange, ANSI X3.4-1986. [ISO-8859] Information Processing -- 8-bit Single-Byte Coded Graphic Character Sets -- Part 1: Latin Alphabet No. 1, ISO 8859-1:1987. Part 2: Latin alphabet No. 2, ISO 8859-2, 1987. Part 3: Latin alphabet No. 3, ISO 8859-3, 1988. Part 4: Latin alphabet No. 4, ISO 8859-4, 1988. Part 5: Latin/Cyrillic alphabet, ISO 8859-5, 1988. Part 6: Latin/Arabic alphabet, ISO 8859-6, 1987. Part 7: Latin/Greek alphabet, ISO 8859-7, 1987. Part 8: Latin/Hebrew alphabet, ISO 8859-8, 1988. Part 9: Latin alphabet No. 5, ISO 8859-9, 1990. [RFC822] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982. [RFC-1521] Borenstein N., and N. Freed, "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, Bellcore, Innosoft, September 1993. [RFC-1522] Moore, K., "Representation of Non-Ascii Text in Internet Message Headers" RFC 1522, University of Tennessee, September 1993. [UTF-8] X/Open Company Ltd., "File System Safe UCS Transformation Format (FSS_UTF)", X/Open Preliminary Specification, Document Number: P316. This information also appears in Unicode Technical Report #4, and in a forthcoming annex Goldsmith & Davis [Page 14] Internet Draft UTF-7 April 1994 to ISO/IEC 10646. Authors' Addresses David Goldsmith Taligent, Inc. 10201 N. DeAnza Blvd. Cupertino, CA 95014-2233 Phone: 408-777-5225 Fax: 408-777-5081 EMail: david_goldsmith@taligent.com Mark Davis Taligent, Inc. 10201 N. DeAnza Blvd. Cupertino, CA 95014-2233 Phone: 408-777-5116 Fax: 408-777-5081 EMail: mark_davis@taligent.com Expiration This document expires on October 8, 1994. Goldsmith & Davis [Page 15] From mcevilly@netcom.com Sat May 7 15:38:45 1994 Return-Path: Received: from netcom.com (netcom2.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA06588; Sat, 7 May 94 15:38:45 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id NAA21016; Sat, 7 May 1994 13:36:30 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405072036.NAA21016@netcom.com> Subject: Re: Big5 <--> 7-bit Big5 is simple To: ccnet-l@uga.bitnet Date: Sat, 7 May 1994 13:36:26 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <199405072016.NAA19363@mail2.netcom.com> from "Yuan Jiang" at May 7, 94 03:17:10 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1063 Status: RO > Again, Ed is not comparing CNS with Big5. He is comparing > 7-bit CNS with the other HZ+ proposals. Yuan, you have not answered the point about unnecessary complexity. Although you have made the above statement several times, you are missing this point. Again, CNS introduces *unnecessary* complexity. I was not comparing CNS with Big5 when I pointed out CNS is unpopular, unsupported, unfamiliar, and unnecessary. The comparison is between ways to map Big5 to a 7-bit scheme. Using CNS to do this is unnecessarily complex. The comparison is between A + B + C and A + C Where A = 7-bit Big5, B = CNS, and C = HZ+. HZ+ and Big5 do not need CNS. And if we have an HZ+ mapping that is virtually the same as Big5 in terms of block partioning and order, then we don't need to worry about creating a new standard. We can adhere to the already existing, popular coding structure of Big5. HZ+S does this. Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From jiang Sat May 7 16:16:44 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA07851; Sat, 7 May 94 16:16:44 CDT Date: Sat, 7 May 94 16:16:44 CDT From: jiang (Yuan Jiang) Message-Id: <9405072116.AA07851@ifcss.org> To: ccnet-l@uga.bitnet, mcevilly@netcom.com Subject: Re: Big5 <--> 7-bit Big5 is simple Cc: soft-authors Status: RO > >Yuan, you have not answered the point about unnecessary I was trying to clear up what we were debating. If we were talking about difrerent things, it's irrelevent to talk about anything else. Of course, CNS' complexity is a disadavantage. >complexity. Although you have made the above statement >several times, you are missing this point. > >Again, CNS introduces *unnecessary* complexity. I was not >comparing CNS with Big5 when I pointed out CNS is unpopular, >unsupported, unfamiliar, and unnecessary. The comparison >is between ways to map Big5 to a 7-bit scheme. Using CNS >to do this is unnecessarily complex. > >The comparison is between > > A + B + C > >and > > A + C > >Where A = 7-bit Big5, B = CNS, and C = HZ+. I see the issue as A<->B vs C<->B, in which B is 8-bit B5, A is HZ+ and C is 7-bit CNS. >HZ+ and Big5 do not need CNS. And if we have an HZ+ mapping >that is virtually the same as Big5 in terms of block partioning >and order, then we don't need to worry about creating a new Ok, HZ+S has the advantage in partition order, complexity, etc, but not popularity, support or famililarity. > From shin@hermes.ee.nthu.edu.tw Sat May 7 16:16:51 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA07856; Sat, 7 May 94 16:16:51 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182352(1)>; Sun, 8 May 1994 05:20:18 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA16997; Sun, 8 May 94 05:10:11 CST Date: Sun, 8 May 1994 19:10:11 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405072110.AA16997@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Status: O |> From simpson@math.psu.edu Sat May 7 09:56:29 1994 |> |> Jing-Shin Chang writes: |> > > > 1. The SeedNet sysadm had informed its users to change the default |> > > > code from CNS (7bit protocol) to Big5 in email, news from |> > > > 05/09. |> > > |> > > Is this referring to the "internal code" or the "exchange code" (using |> > ... |> > Both. Because they use the exchange code for email and news transfer. |> > ... |> > The main reason is to meet the trend of our popular Big5 environment. |> |> So, do I understand you correctly? You seem to be saying that SeedNet |> no longer uses CNS for anything. As of May 9, 1994, SeedNet will |> switch to Big5. Is that correct? I think it is true. The default char selection in their PD freeware are changed. |> |> Do you know of any network or agency in Taiwan that uses CNS? I don't know. But I remember that some official purchase projects require that the CNS code be supported in the computers because they have practical needs for a larger character set. For instance, people may have very strange first names or last names, the police offices and some government offices than need such chars be supported. I think this is not due to any political reason as guessed in some previous postings. There is such need. The CNS chars commonly used (and implemented in workstations, like Sun and IBM workstations) is the revision announced in '86; it has the same char set as Big5. However, it is more than 400K now. |> |>> > Could you please explain to us, in simple language of course, the |>> > exact form of CNS that is (or was) used as an exchange code in |>> > SeedNet? What are the escape sequences? And, another question about |>> |> ...... |> SeedNet no longer uses the above escape sequences, is that correct? |> Who uses them? I guess they are still using it for mail transfer, but not for news. I make this guess because their announcement said that you can still get your chinese mail correctly if your party use the same code selection. And one example is CNS to CNS. So they still support it for mails. The above sequences is a national standard, so it will not disappear simply because SeedNet does not use it for email transfer. SeedNet is an instance that uses it for mail transfer. Sun's workstations implement CNS in the form of EUC, other localized UNIX OS (SVR4) also support EUC (or, we can say CNS.) |> |> > In Sun's email implementation, 'ESC $ ) 0' is used to designate |> > chinese code (EUC) for Taiwan (and 'A', 'B', 'C' for CJK, |> |> You are talking about a Sun Chinese environment in Taiwan. However, |> in the international setting, I found that Sun's support for Chinese |> is minimal to nonexistent. Some months ago I read an article in a Sun |> trade publication about internationalization, EUC, and the Sun Chinese |> environment. I phoned 1-800-USA-4SUN and asked them to send me all |> available information about the Chinese environment. They never sent |> me anything. |> |> So far as I can determine, people outside Taiwan who want to do |> Chinese on Suns are on their own. They can buy some extremely |> expensive commercial software, or they can use free software such as |> cxterm. The ALE (Asia Language Environments, including HLE/CLE/JLE/KLE,) is an important localization project for CJK users, the US users may not be familiar with that simply because they do need to. You should be able to get such localized version from Sun if you have such a request. Such special version is target at CJK contries, so I don't think the sales will talk it to you in your campus. Also, I think the major saling department might be established in these countries. So, it is likely that you need to buy it, say, from Taiwan. Maybe you don't know that Taiwan is a big buyer of the Suns. So far as I know, such US workstation companies are more and more willing to do such localization because their market share in the US is becoming smaller. |> |> So far as I can tell, the US computer company that is doing the most |> for Chinese is Apple. Does Apple use CNS? I am under the impression |> that Apple uses only GB and Big5. Maybe Ed and Roland can answer |> this. |> |> > In summary, the places where it should be converted to the internal codes |> > depends on where the user want to access it as an ordinary file for |> > processing. When is archived or ready for transmission, it should be |> > in interchange code form. |> |> From this it appears that, contrary to what has been claimed, the |> exchange code is not invisible to the user. Exchange codes is of course visible. However, it is not appropriate to present it in that form to your user. A good word processor, for example, will only show you the chars you have in the documents with an appropreate presentation form to show it according to the attributes associated with that chars (like high-lighting, underlining). It will not present you something like ^[4mABC^[0m for an underlined string. These ESC's, no matter in what form, are not intended for processing (e.g., editing). The distinction is not so obvious in the news application because our processing is minimal. But it does not implies that we should not design a better interface to the users. |> |> -- Steve |> |> From shin@hermes.ee.nthu.edu.tw Sat May 7 16:26:33 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA08077; Sat, 7 May 94 16:26:33 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182352(1)>; Sun, 8 May 1994 05:29:51 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA17023; Sun, 8 May 94 05:18:12 CST Date: Sun, 8 May 1994 19:18:12 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405072118.AA17023@hermes.ee.nthu.edu.tw> To: JD@EREMITA.DEMON.CO.UK, simpson@math.psu.edu Subject: Re: Big5 or CNS ? Cc: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, soft-authors@ifcss.org Status: O Sorry, I did not remember that I had post anything related to this stuffs. Could you please send me the original copy? I am not clear what are the isseus in this posting. Shin -^^- ============== > From simpson@math.psu.edu Sat May 7 10:12:57 1994 > > John Delacour writes: > > Jing-shin, > > > > Maybe I'm being simplistic but I think you're muddying the waters, and much > > though I love Ed Lai's Cihui etc. [ ... stuff omitted ...] > > I don't see the point of this branch of the > > discussion. > > > > At present users of GB-based software regularly use GB<->HZ translation for > > email and posting to newsgroups such as ACT. There is no problem at all > > that I have experienced. > > > > HZ+, as proposed by Steve Simpson, provides exactly the same facility for > > users of Big5-based software or mixed GB and Big5. > > [... stuff omitted ...] > > that's what HZ+ is for and it works, so what is the point of > > discussing CNS and what not at the moment. > > Thanks, John. You are grasping exactly what HZ+ is all about. I > can't understand why everybody doesn't just get on the HZ+ bandwagon. > Why in the world is there all this noise about CNS? :-) > > -- Steve > From simpson@math.psu.edu Sat May 7 18:13:12 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11088; Sat, 7 May 94 18:13:12 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id TAA28167; Sat, 7 May 1994 19:11:12 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 19:11:11 -0400 Date: Sat, 7 May 1994 19:11:11 -0400 Message-Id: <199405072311.TAA14741@boole.math.psu.edu> To: jiang@ifcss.org (Yuan Jiang) Cc: soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, ccnet-l@uga.cc.uga.edu Subject: we need more HZ+ proposals :-) In-Reply-To: <9405071928.AA04343@ifcss.org> References: <9405071928.AA04343@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > >are you saying to use purely 7-bit CNS w/o any escape sequence? such as i > >assume has been done on Seednet, or to incorporate 7-bit CNS instead of > With escape sequence of course. If you and Ed are serious about wanting to use CNS in HZ+, could you please write a specific proposal and make it available for anonymous FTP? This will make it possible to compare your proposal with my proposal, Ya-Gui's proposal, and Jing-Shin's four or five proposals. Your proposal should specify what kind or style of escape sequences you want to use. Are you going to use ~ as in HZ, or Esc as in the now abandoned SeedNet exchange code, or what? I think we should postpone a vote on HZ+ until we have at least 20 proposals and have had plenty of time to weigh all of their advantages. :-) -- Steve From simpson@math.psu.edu Sat May 7 18:40:07 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11665; Sat, 7 May 94 18:40:07 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id TAA28271; Sat, 7 May 1994 19:38:06 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 19:38:04 -0400 Date: Sat, 7 May 1994 19:38:04 -0400 Message-Id: <199405072338.TAA14794@boole.math.psu.edu> To: Yuan Jiang , lai@apple.com Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 or CNS ? In-Reply-To: <199405071951.PAA27418@leibniz.math.psu.edu> References: <199405071951.PAA27418@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > Again, we have to remind ourselves that we are comparing CNS > with the other HZ+, not Big5. Popularity is an argument when > comparing to Big5. Ed Lai uses "encoded Big5" to refer to > the other HZ+ proposals. It may be the source of misunderstanding. Ed's term "encoded Big5" makes no sense. Big5 is itself a Chinese character coding scheme. Any form of Big5 is already encoded. Encoded Big5 can be nothing else but Big5. I feel that you and Ed are twisting yourselves into logical pretzels trying to prove that CNS is somehow more desirable or popular than Big5, when this is obviously not the case. I can't understand why you want to do this. You are obstructing the creation of a viable 7-bit HZ+ standard based on HZ and Big5. > Comparing 7-bit CNS with the other HZ+ proposals, it fairs better > in terms of popularity, support and impletementation. They are > precisely the only advantages of CNS, no others. The logic of what you are saying is as follows. Nobody has yet used any HZ+ proposal, because HZ+ is not yet standardized. Therefore, any coding scheme which has already been used by even one person or in one software package is more "popular", or "supported", or "implemented" than any HZ+ proposal. Therefore, any such coding scheme is better than any HZ+ proposal. In particular, CNS is better than any HZ+ proposal, because some obscure government agencies in Taiwan may have used CNS as an exchange code at some time in the past. Therefore, CNS will make HZ+ more popular. Therefore, HZ+ should be based on CNS. I dissent from this logic. It is up to us to design a new HZ+ standard for maximum compatibility with already popular standards. I designed my HZ+S proposal to be maximally compatible with two extremely popular standards, HZ and Big5. I think my logic makes more sense than your logic. Could you and Ed please look at my HZ+S proposal and sample implementation and tell me what you think is wrong with it, or how you would improve on it? My stuff is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. -- Steve From butta1@cs.bu.edu Sat May 7 19:26:55 1994 Return-Path: Received: from cs.bu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12845; Sat, 7 May 94 19:26:55 CDT Received: from csa.bu.edu by cs.bu.edu (8.6.4/Spike-2.1) id UAA03469; Sat, 7 May 1994 20:24:27 -0400 Received: from localhost by csa.bu.edu (8.6.4/Spike-2.1) id UAA07760; Sat, 7 May 1994 20:24:50 -0400 Date: Sat, 7 May 1994 20:24:50 -0400 (EDT) From: C I I Project Subject: let's move on to vote!! To: soft-authors@ifcss.org Cc: roland_wang@POWERTALK.APPLE.COM Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O Ok guys, let's move on to vote on a proposal, and spend the effort and energy on the voted proposal as a team, instead everyone doing everything individual, which will no go as far as if we had combined the minds of everyone together to focus on one proposal. I really want to log on and see something fresh concerning this discussion. As far as the cns/big5 deal, let's stick with big5 because of to-date i've not heard from one CNS loyalist user from ROC to defend this code, whether on this list, CCNET-L, acc, or act, or at least they don't seem to care to propogate its use via the Internet. The cns/big5 debate can carry once we create a chinese.comp.cns-vs-big5 :) btw, 7-bit BIG5 has been around as brought up by Simpson as accomodated by ISO-2022-JP-2 as used in mule. FYI, see documents in ifcss.org:/software/mule/* Therefore this 7-bit BIG5 thing isn't just pop out of nowhere, if there were complaints on it, then Mule should have addressed this problem by now, and mule has been around for quite some time for users to try it out, and mule still are accomodating BIG5, and the code is news and mail safe in 7-bit environment. nelson From simpson@math.psu.edu Sat May 7 19:50:55 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA13481; Sat, 7 May 94 19:50:55 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id UAA28513; Sat, 7 May 1994 20:48:55 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 20:48:53 -0400 Date: Sat, 7 May 1994 20:48:53 -0400 Message-Id: <199405080048.UAA14901@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility In-Reply-To: <9405071950.AA16843@hermes.ee.nthu.edu.tw> References: <9405071950.AA16843@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > In HZ+S, you are preserving the character order in the big5 char order > more explicitly in the sense as mentioned above. Yes. That's why my HZ+S proposal is better. The Big5 order is very explicitly preserved. If you look at a Big5 table, you will see that within each block the characters are are linearly ordered by radicals and stroke count. I consider it important to explicitly preserve this natural order. The fact that HZ+S explicitly preserves this order is a significant advantage of HZ+S over all of the other proposals. > The most important point is, you rarely need to process you data > in the transformed code space. So I never consider it a problem. You never *need* to process it there, but in some circumstances it may be *desirable* to process it there. That has been our experience with HZ. I am thinking of ZWDOS. If you don't consider it a problem, that may be because you are not familiar with HZ or how people use it. > (Don't you think the blocks of codes in the transformed code space an > exact copy of their counterpart in the original Big5 code space? I don't think this is necessary or desirable. The code block structure is *not* a significant structural element in Big5. Characters with the same radical are *not* grouped in the same code block. The most significant structural elements of Big5 are, (1) the division into more and less frequently used characters, (2) the ordering of the characters by radical and stroke count. > Is it important to pack the codes tightly. -^^-) Ya-Gui and I regard tight packing as elegant and beautiful. Tight packing also has some practical advantages, for instance in the use of fonts. > You cannot force the Big5 users to change drastically to accept a > different computing schemes and habits. It's true that HZ users live > happily with the current status, i.e., processing their text with the ESC > characters. I don't think this is a normal way for Big5 users. Nobody is going to force Big5 users to use any particular software. Big5 users, like GB users, will be free to use any software they like. I can well imagine that many or most Big5 users, just like many or most GB users, will prefer software that hides the escape sequences. > |> Jing-Shin, I'd suggest you familiarize yourself with HZ. I have the > |> feeling that you don't know what considerations motivated it and how > |> it has developed. > > Thanks for your suggestion. But I am wondering if it is helpful for > me to design the mapping schemes for 7-bit Big5. HZ+ is one unified proposal. It doesn't make sense to design the 7-bit Big5 mapping scheme for HZ+ in ignorance of HZ. If that's what you are trying to do, I think you should abandon the attempt. -- Steve From simpson@math.psu.edu Sat May 7 20:08:11 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14026; Sat, 7 May 94 20:08:11 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA28591; Sat, 7 May 1994 21:06:11 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 21:06:09 -0400 Date: Sat, 7 May 1994 21:06:09 -0400 Message-Id: <199405080106.VAA14938@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw, yawei@ifcss.org Subject: Thank you, Jing-Shin! In-Reply-To: <9405071950.AA16843@hermes.ee.nthu.edu.tw> References: <9405071950.AA16843@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O I wrote: |> I notice that your proposals do not include HZ. Is this omission |> intentional? and > |> Jing-Shin, I'd suggest you familiarize yourself with HZ. Jing-Shin replied: > Thanks for your suggestion. But I am wondering if it is helpful for > me to design the mapping schemes for 7-bit Big5. Ah, now I understand. Thanks very much for that clarification! You are only interested in designing mapping schemes for 7-bit Big5. You are not interested in the HZ part of HZ+. Your various b5encode proposals were never intended as stand-alone proposals for HZ+ (defining HZ+ to consist of HZ plus some form of 7-bit Big5). Rather, your b5encode proposals were only intended as suggestions for me and Ya-Gui. You were suggesting to me and Ya-Gui how we might improve our own HZ+ proposals by using a superior 7-bit Big5 mapping scheme. Thank you very much for your excellent suggestions. In that light, I think we now have only two HZ+ proposals, my HZ+S and Ya-Gui's HZ-2. Good! Life is suddenly much simpler! Thank you, Jing-Shin! Ya-Gui, let's you and I think about it for a day or two and decide whether we want to change our respective HZ+ proposals to use one of Jing-Shin's mapping schemes. After that, we can perhaps finally call for a vote on HZ+. With only two proposals on the table, it may be relatively easy to reach a consensus. -- Steve From yawei Sat May 7 20:15:09 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA14255; Sat, 7 May 94 20:15:09 CDT Date: Sat, 7 May 94 20:15:09 CDT From: yawei (Ya-Gui Wei) Message-Id: <9405080115.AA14255@ifcss.org> To: simpson@math.psu.edu Subject: Rebuttal and Summary of Big5 transformation debate Cc: chinesenet@uclink.berkeley.edu, lee@simd.stanford.edu, soft-authors, yawei Status: O There may be different definition for "advantage", but when one used the word in a comparison context, it could be perceived by the reader that those are features unique to one proposal. Just yesterday one netter on soft-authors was saying "since HZ+S had so many advantages, I'd vote for it." Obviously the netter has perceived that similar "advantages" are lacking on competing proposals. Since all the proposals are supposedly 100% Big5 compatible, I don't see how one proposal can be "more compatible" with Big5 than others. Let's not over-emphasize the one aesthetic nicety I pointed out that HZ+S had. If CPU time is not a concern, why don't we just settle on CNS. Seednet's ditching of CNS may be more than just a statement against CNS, it may be a statement against all 7-bit schemes. I think we are wasting time debating on how to choose among essentially similar schemes, and forgot the most important question that we should be asking: how are we going to get all the Big5 users to switch to a 7-bit scheme? There are only one thing that really matters: user acceptance. To get there one needs to get it into softwares. But even if we are going to implement it in a few software packages, I still don't see how to convert the vast Big5 users in Taiwan into using it. This is the prime reason why I had not wanted to publish the HZ-2 proposal even though it was drafted (and implemented in a converter) almost a year ago and circulated and discussed among a few people (I don't remember hearing any objection from you after I sent the proposal to you last Sept., Steve.) Even if we can settle on a proposal here, it still wouldn't matter unless we can come up with a few reasons to _convince_ Big5 users to use it. In this regard, I don't see how these proposals will resolve the dilemmas faced by the chinese.* hiarachy. Maybe we ought to forget all these proposals completely. :-) As far as chinese.* hiarachy is concerned, I think it is best that it operates with parallel mirrored newsgroups for the time being. Maybe we instead ought to include 8-bit Big5 in HZ by assigning a set of escape sequence for it. (Hmm, there's another proposal.) Fung Fung, you have been very silent, what do you think of expanding HZ to include 8-bit stuff? (Can somebody check to see if Fung Fung Lee is still on soft-authors?) For symmetricity we can put in 8-bit GB as well. (Some people are already doing this, and ZWDOS can already decode it.) For people who want to send e-mail through 7-bit channels, we can include BOTH a 7-bit scheme and 8-bit Big5 (Oops, yet another proposal!). It doesn't take much programming to support both, after all, it only takes a couple lines of C code to convert from one to the other. Frankly I don't think these are matters to be settled by a vote, since most of us don't have the time to carefully look at the merits of each proposal. Plus I don't have the time to wage a campaign to try to influence the votes. Sure we can argue all over and vote to our heart's content. But remember that committee standards don't neccessarily mean anything. CNS is a nice-looking committee standard and we are trying desperately to resist it. :-) Ya-Gui -- Here's a recap of all the proposals on how to twist Big5 into mixing with HZ. I bet we can come up with a dozen more. There are way too many choices. 1 Use CNS: Involves table lookup for some out-of-place characters. Seperate table-lookup mechanism is needed, and may be more CPU intensive than HZ+S. Without compelling reasons, we should refrain from inventing new wheels. However, at this point I still don't know enough about CNS to cast my vote for or against it. 2 HZ+S: parse Big5 into two continuous linear blocks, then refit them into two 94x94 blocks. CPU intensive since parsing involves multiplication and refitting involves division. Rewriting *157 and /94 with logical operators may be possible, but would take too many of them and still won't be efficient. 3 HZ-2: fold each Big5 row (157 cols each) into two adjacent rows (94 and 63 col each). 4 New B5encode: fold each Big5 row into two disadjacent rows. Too similar to HZ-2. :-) 5 Just Use straight Big5: least CPU intensive. Mixing of GB/HZ and Big5 is _really_ achieved, but can't go through 7-bit channels. 6 Old B5encode. This has been withdrawn. 7 Use straight Big5 AND one of the 7-bit schems together. These are just what to do with Big5 codes. There can be many proposals on the escape sequences as well. 94x94 of them to be exact. So many proposals, so little time. From simpson@math.psu.edu Sat May 7 20:25:05 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14545; Sat, 7 May 94 20:25:05 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA28658; Sat, 7 May 1994 21:23:03 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sat, 7 May 1994 21:23:02 -0400 Date: Sat, 7 May 1994 21:23:02 -0400 Message-Id: <199405080123.VAA14965@boole.math.psu.edu> To: Yuan Jiang , lai@apple.com Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, shin@hermes.ee.nthu.edu.tw Subject: Re: Big5 <--> 7-bit Big5 is simple In-Reply-To: <199405072016.QAA27527@leibniz.math.psu.edu> References: <199405072016.QAA27527@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Ed Lai said: >> What are some of the arguments against encoded Big5 as an exchange code. Yuan Jiang said: > Again, Ed is not comparing CNS with Big5. He is comparing > 7-bit CNS with the other HZ+ proposals. No. Ed didn't mention HZ+ proposals. He mentioned Big5. If Ed wants to compare CNS with other HZ+ proposals, let him do so in a straightforward way. My HZ+ proposal is available for anonymous FTP. Have you and Ed looked at it? You said, "He is comparing 7-bit CNS with the other HZ+ proposals." This means that, in your logic, CNS is an HZ+ proposal. I object to this. CNS is not an HZ+ proposal. CNS is an unpopular coding scheme in Taiwan, that's all it is. If you and Ed want to create your own HZ+ proposal in opposition to other HZ+ proposals, please create it. -- Steve From mcevilly@netcom.com Sat May 7 21:56:31 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA16841; Sat, 7 May 94 21:56:31 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id TAA19376; Sat, 7 May 1994 19:54:35 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405080254.TAA19376@netcom.com> Subject: Re: Rebuttal and Summary of Big5 transformation debate To: ccnet-l@uga.bitnet Date: Sat, 7 May 1994 19:54:35 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405080115.AA14255@ifcss.org> from "Ya-Gui Wei" at May 7, 94 08:15:09 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2579 Status: O Ya-Gui made several important points, one I would like to reemphasize and agree with being that we have to keep in mind the end users--whatever we do will be irrelevant if it is not used. Making sure that this happens is largely dependent on software development after the HZ+ code discussion is finalized, and the best way we can facilitate that future effort now is by adopting a coding method that is likely to be used by programmers. > But remember that committee standards don't neccessarily mean > anything. CNS is a nice-looking committee standard and we are > trying desperately to resist it. :-) You are the first person to say CNS is nice looking. Once you open it up and look inside, you find a tangle of escape sequences and seven separate planes all of different sizes, some of them non- contiguous even within the same plane! > 1 Use CNS: Involves table lookup for some out-of-place > characters. Seperate table-lookup mechanism is needed, and > may be more CPU intensive than HZ+S. Without compelling > reasons, we should refrain from inventing new wheels. > However, at this point I still don't know enough about > CNS to cast my vote for or against it. And after several days of discussion on CNS, there just haven't been any messages posted which answered the basic questions: who uses it? who supports it? why is it necessary? If nobody provides convincing answers to these questions, it is hard to see how anyone could be sure it is worthy of support. It is very understandable that you should be unsure. > 2 HZ+S: parse Big5 into two continuous linear blocks, then > refit them into two 94x94 blocks. CPU intensive since parsing > involves multiplication and refitting involves division. > Rewriting *157 and /94 with logical operators may be > possible, but would take too many of them and still won't be > efficient. Then just use two small tables, either hard coded or computed once on program startup. These tables would be really tiny (around 80 and 140 elements each). No multiplication, no division. Was that your only objection? > 3 HZ-2: fold each Big5 row (157 cols each) into two adjacent > rows (94 and 63 col each). > 4 New B5encode: fold each Big5 row into two disadjacent rows. > Too similar to HZ-2. :-) > So many proposals, so little time. Maybe we should designate escape codes for each proposal, and be done with it! :-) Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From simpson@math.psu.edu Sun May 8 00:30:47 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA21210; Sun, 8 May 94 00:30:47 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA29570; Sun, 8 May 1994 01:28:48 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 01:28:46 -0400 Date: Sun, 8 May 1994 01:28:46 -0400 Message-Id: <199405080528.BAA15394@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: 7-bit Unicode ? In-Reply-To: <199405072006.PAA18388@ector.cs.purdue.edu> References: <199405072006.PAA18388@ector.cs.purdue.edu> Reply-To: simpson@math.psu.edu Status: O Yongguang Zhang writes: > > I know the water is muddy enough, but let me throw another stone: > How about 7-bit Unicode, another mail-safe standard? > (See the example at the end of the document.) This UTF-7 spec an interesting document. UTF-7 uses Unicode together with a MIME-like base64 coding to achieve an efficient 7-bit mail-safe representation of multilingual text. Not bad at all. If Unicode becomes popular, it may solve a lot of our problems. However, until that time, Chinese Internet community will still need simple, practical, limited solutions such as HZ. The nice thing about HZ+ is that it is only a very simple combination of two existing standards which are already extremely popular, HZ and Big5. Unicode is, unfortunately, not so widespread. -- Steve From simpson@math.psu.edu Sun May 8 00:57:12 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA21964; Sun, 8 May 94 00:57:12 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA29656; Sun, 8 May 1994 01:55:14 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 01:55:13 -0400 Date: Sun, 8 May 1994 01:55:13 -0400 Message-Id: <199405080555.BAA15446@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: let's move on to vote!! In-Reply-To: References: Reply-To: simpson@math.psu.edu Status: O C I I Project writes: > As far as the cns/big5 deal, let's stick with big5 because of to-date > i've not heard from one CNS loyalist user from ROC to defend this code, Yes! Let's limit our discussion to HZ+, defined as a plan to enhance HZ by adding in 7-bit Big5. Let's leave the CNS debate for another place and another time. > let's move on to vote on a proposal, and spend the effort and energy on > the voted proposal as a team, Right on, Nelson! The international Chinese Internet community needs HZ+ now. Let's stop arguing and get to work! > btw, 7-bit BIG5 has been around as brought up by Simpson as accomodated > by ISO-2022-JP-2 as used in mule. FYI, see documents in > ifcss.org:/software/mule/* > Therefore this 7-bit BIG5 thing isn't just pop out of nowhere, if there > were complaints on it, then Mule should have addressed this problem by > now, and mule has been around for quite some time for users to try it Nelson is absolutely correct. Mule has been doing 7-bit Big5 for some time now, with great success! Not only that, but the HZ+S road to 7-bit Big5 is exactly the same as Mule's road !!!! This is what I was trying to say in my long-winded discussion of ISO-2022 compatibility. Nelson has shown me a way to say it shorter, clearer, and better. Thanks, Nelson! -- Steve From simpson@math.psu.edu Sun May 8 01:05:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22285; Sun, 8 May 94 01:05:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id CAA29689; Sun, 8 May 1994 02:03:38 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 02:03:36 -0400 Date: Sun, 8 May 1994 02:03:36 -0400 Message-Id: <199405080603.CAA15460@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Use HZ+S to save CPU cycles! In-Reply-To: <199405080254.WAA28991@leibniz.math.psu.edu> References: <9405080115.AA14255@ifcss.org> <199405080254.WAA28991@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Ya-Gui said: > > 2 HZ+S: parse Big5 into two continuous linear blocks, then > > refit them into two 94x94 blocks. CPU intensive since parsing > > involves multiplication and refitting involves division. > > Rewriting *157 and /94 with logical operators may be > > possible, but would take too many of them and still won't be > > efficient. Carlos answered: > Then just use two small tables, either hard coded or computed once > on program startup. These tables would be really tiny (around 80 > and 140 elements each). No multiplication, no division. Was that > your only objection? Carlos is absolutely right! You can do these multiplications and divisions very fast using table lookup. The tables are tiny. I hope Ya-Gui and Yidao will now stop saying that HZ+S is inefficient! -- Steve From simpson@math.psu.edu Sun May 8 01:50:09 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA23757; Sun, 8 May 94 01:50:09 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id CAA29812; Sun, 8 May 1994 02:48:11 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 02:48:09 -0400 Date: Sun, 8 May 1994 02:48:09 -0400 Message-Id: <199405080648.CAA15530@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: anectodal history of HZ-2 and HZ+S In-Reply-To: <9405080115.AA14255@ifcss.org> References: <9405080115.AA14255@ifcss.org> Reply-To: simpson@math.psu.edu Status: O yawei@ifcss.org (Ya-Gui Wei) writes: > it. This is the prime reason why I had not wanted to publish the > HZ-2 proposal even though it was drafted (and implemented in a > converter) almost a year ago and circulated and discussed among a > few people (I don't remember hearing any objection from you after > I sent the proposal to you last Sept., Steve.) My first knowledge of this was when I started posting some questions and half-baked ideas here on soft-authors, about HZ and 7-bit Big5. That was a year ago, in April 1993. (I have saved all the correspondence, in case anybody wants to see it.) After a few weeks, Ya-Gui replied by private e-mail saying that he and Fung were working on this and he was testing a better method for efficiency. I replied saying I was excited about this, but I heard nothing more. In September 1993, I again posted to soft-authors suggesting to use a Mule/ISO-2022-JP-2 style method for 7-bit Big5. Ya-Wei replied privately by sending me a draft of HZ-2 dated May 13. In November I asked Ya-Wei why he had not published his HZ-2 proposal. He replied that he and Fung were reluctant because of problems involving possible future extensions of GB. I continued to meditate on my ISO-2022 ideas. I heard nothing more until last month, when Ya-Gui asked me to send him a copy of his own HZ-2 proposal, which he had misplaced. I immediately sent him a copy, then repeated my Mule/ISO-2022-JP-2 ideas in private e-mail to Ya-Wei and Fung. I never phrased my ideas as an objection to HZ-2. Rather I presented them as suggestions for improvement, the advantage being ISO-2022 compatibility. The next day, Yidao posted about 7-bit Big5 to soft-authors. I immediately followed up by stating what is essentially my current HZ+S proposal. That was the start of the present public discussion. The discussion received further impetus because of the chinese.* newsgroup hierarchy proposal. -- Steve From yawei Sun May 8 01:52:01 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA23777; Sun, 8 May 94 01:52:01 CDT Date: Sun, 8 May 94 01:52:01 CDT From: yawei (Ya-Gui Wei) Message-Id: <9405080652.AA23777@ifcss.org> To: simpson@math.psu.edu, soft-authors Subject: Re: let's move on to vote!! Status: O I for one would like to find out a bit more about CNS, just to see how flawed a design it really is. Somebody mentioned some convertion code, but somehow I missed it. Can someone send it to be again? I think that, if CNS is only slightly flawed, then we may want to refrain from inventing new wheels. While we are at this, I hope that each of us can give some thought to one question raised by one netter (J.Chen?) on alt.chinese.computing: he invested a lot of money buying commercial Big5 softwares, and can already use them to read and post to alt.chinese- .text.big5. why would anyone think that he should not post in big5 but should instead convert to some HZ+ scheme first? Indeed, why? I know this is a seperate question from the coding scheme debate. But if we can't convince users like Mr. Chen, then even if everyone on soft-authors supports a new scheme, it still doesn't eliminate the need for Big5 newsgroups, and users of the new scheme will probably be mostly users of today's HZ. But then, these users don't need the new scheme. See, we are without a user base! One scheme that might help with getting Big5 users is the straight Big-5 proposal I mentioned in my last mail. If these users are reading some of the contents with their current software, they may wonder what the rest of the stuff says and find the way to join in. I hope that we can consider this scheme for a few minutes as well. I should say that each of the proposals under consideration has its merits, and I have hinted that ZWDOS will support whatever comes out of it. Ya-Gui From yawei Sun May 8 02:03:03 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA24134; Sun, 8 May 94 02:03:03 CDT Date: Sun, 8 May 94 02:03:03 CDT From: yawei (Ya-Gui Wei) Message-Id: <9405080703.AA24134@ifcss.org> To: simpson@math.psu.edu, soft-authors Subject: Re: Use HZ+S to save CPU cycles! Status: O >I hope Ya-Gui and Yidao will now stop saying that HZ+S is inefficient! What Carlos said is true. But then by the same token maybe we should stop saying that CNS is inefficient as well. :-) Ya-Gui From butta1@crsa.bu.edu Sun May 8 02:12:06 1994 Return-Path: Received: from crsa.bu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA24346; Sun, 8 May 94 02:12:06 CDT Received: from ithake.bu.edu by crsa.bu.edu (8.6.4/Spike-2.1) id DAA10506; Sun, 8 May 1994 03:10:07 -0400 Received: from localhost by ithake.bu.edu (8.6.4/Spike-2.1) id DAA10121; Sun, 8 May 1994 03:09:54 -0400 Date: Sun, 8 May 1994 03:09:52 -0400 (EDT) From: C I I Project Subject: Re: let's move on to vote!! To: Ya-Gui Wei Cc: soft-authors@ifcss.org In-Reply-To: <9405080652.AA23777@ifcss.org> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O On Sun, 8 May 1994, Ya-Gui Wei wrote: > > I for one would like to find out a bit more about CNS, > just to see how flawed a design it really is. Somebody > mentioned some convertion code, but somehow I missed > it. Can someone send it to be again? see ifcss.org:/software/unix/convert/ChnCodeFilter-1.4.* official versions -> unix/convert/chcode.* w/bugs are -> dos/convert/chcode.* > .text.big5. why would anyone think that he should > not post in big5 but should instead convert to some if he's posting locally in taiwan, then this would be irrelevant to him but if he post internationally, he must educate himself, with a FAQ or some guideline, that in order to reach his audience, he must use a 7-bit exchange code as some part of Internet isn't like 8-bit clean taiwan. if he is in a 7-bit environment, then this would be a useful exchange code. > One scheme that might help with getting Big5 users > is the straight Big-5 proposal I mentioned in my > last mail. If these users are reading some of the info distribution is not efficient this way, you're dealing w/a 7-bit code and a 8-bit code... users limited 7-bit software will never be able to read the 8-bit stuff, as it's stripped and therefore corrupted.. rather have it mirrored than mixed like that, or go for hz+. nelson From simpson@math.psu.edu Sun May 8 02:51:17 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA25054; Sun, 8 May 94 02:51:17 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id DAA00111; Sun, 8 May 1994 03:49:13 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 03:49:10 -0400 Date: Sun, 8 May 1994 03:49:10 -0400 Message-Id: <199405080749.DAA15627@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu Subject: No more proposals! Let's get down to work! In-Reply-To: <9405080115.AA14255@ifcss.org> References: <9405080115.AA14255@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes: > question that we should be asking: how are we going to get all > the Big5 users to switch to a 7-bit scheme? There are several things we can do. As individuals, we can write software that supports HZ+. As a group, we can work e.g. to convince the organizers of the chinese.* newsgroup hierarchy to make HZ+ their official standard for postings. On the software issue, I have already written prototype HZ+S <-> Big5 filters. They have been tested and they work fine. Ya-Gui Wei seems willing to support HZ+ in his program, ZWDOS. John Delacour says that Macintosh support for HZ+ e-mail and news is on the way. Emacs support can happen very soon. Are there any more programmers here on soft-authors who can support HZ+? Let's get to work! By the way, Ya-Wei spoke of Big5 users switching to 7-bit, but this will never happen. Eight-bit Big5 is much too well entrenched, e.g. in Taiwan. But that's OK. People don't have to completly switch over to 7-bit Big5. All we aim for is to make 7-bit Big5 (along with 7-bit GB) available as a convenient tool for international e-mail and news. This is realistic. This is the proper role for HZ+. > Maybe we ought to forget all these proposals completely. :-) Sorry, but I'm not ready to give up. The world needs HZ+. > far as chinese.* hiarachy is concerned, I think it is best that > it operates with parallel mirrored newsgroups for the time being. The problem there is that many users cannot receive 8-bit news, so they won't be able to read the *.big5 groups. These users *need* 7-bit Big5, and we know how to give it to them! HZ+ is the answer. (I know that I was saying a few weeks ago that all news feeds are 8-bit clean, but I was *wrong*. Thanks to Jianqing Hu and Yongguang Zhang for enlightening me.) > Maybe we instead ought to include 8-bit Big5 in HZ by assigning a > set of escape sequence for it. (Hmm, there's another proposal.) No more proposals, please! > e-mail through 7-bit channels, we can include BOTH a 7-bit scheme > and 8-bit Big5 (Oops, yet another proposal!). Arrgh! Please, please, please stop making proposals! :-) > Frankly I don't think these are matters to be settled by a vote, > since most of us don't have the time to carefully look at the > merits of each proposal. Plus I don't have the time to wage a > campaign to try to influence the votes. Well then, let's just agree on HZ+S. Is this OK with everybody? Speak up! > Sure we can argue all over and vote to our heart's content. But > remember that committee standards don't neccessarily mean > anything. CNS is a nice-looking committee standard and we are > trying desperately to resist it. :-) HZ+S is not a committee standard! I designed it all by myself, with acknowledged input from HZ and HZ-2 and Mule. All the parts of HZ+S fit together perfectly. > Here's a recap of all the proposals on how to twist Big5 into > mixing with HZ. I bet we can come up with a dozen more. Can we please declare a moratorium on new proposals? There's nothing wrong with HZ+S. Instead of making new proposals, let's get down to some constructive work toward making the HZ+ standard a reality. -- Steve From simpson@math.psu.edu Sun May 8 03:21:38 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA25626; Sun, 8 May 94 03:21:38 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id EAA00248; Sun, 8 May 1994 04:19:22 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 04:19:19 -0400 Date: Sun, 8 May 1994 04:19:19 -0400 Message-Id: <199405080819.EAA15698@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: CNS sucks! In-Reply-To: <9405080703.AA24134@ifcss.org> References: <9405080703.AA24134@ifcss.org> Reply-To: simpson@math.psu.edu Status: O > we should stop saying that CNS is inefficient as well. :-) I don't care how "efficient" or "inefficient" CNS may be. The more serious problem about CNS is that it has no popular support. None. Millions and millions of people use Big5, but nobody uses CNS. There is not one single person anywhere on earth who uses CNS. Despite some misleading statements made by various people, CNS is not a method of "encoding" Big5. CNS is rather a completely different Chinese character coding scheme, in direct competition to Big5. The competition was decided long ago. Big5 won, CNS lost. Can we please just forget about CNS? However, if you absolutely must learn about CNS, look at iiidns.iii.org.tw:/pub/seed/chcode. That archive provides conversion routines between Big5 and X, where X is any one of five archaic, useless, unpopular coding schemes. One of the X's is CNS, except it is not called CNS, instead it is called NSC for some reason, I guess somebody made a typographical error. -- Steve From shin@hermes.ee.nthu.edu.tw Sun May 8 05:27:48 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA28032; Sun, 8 May 94 05:27:48 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182061(1)>; Sun, 8 May 1994 18:31:18 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA01421; Sun, 8 May 94 18:20:32 CST Date: Mon, 9 May 1994 08:20:32 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405081020.AA01421@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org, yawei@ifcss.org Subject: Re: Thank you, Jing-Shin! Cc: CCNET-L%UGA.bitnet@TWNMOE10.EDU.TW Status: O Dear Professor, You seems to be very exciting when you finally noticed what my main concern is!! -^^- Somehow, I feel that my proposal(s) are not treated fairly enough!! -^^- I think the Big5 mapping is the major topic in designing a 7-bit coding schemes for communication purpose. And that all us had agreed on the HZ part. And the ESC's can be choosen based on preference (or XXX-compatibility). Then why you do not think that my B5E1 and B5E2 are qualified as complete proposals? Do I need to say that I am going to use the current HZ standard to make it a complete proposal? (How angry I am!! -^^-) BTW, according to Nelson, a 7-bit Big5 encoding method is used in Mule. Do you have a statistics on the user group of such Big5 Mule? Is that mapping method the same as your currnt proposal? If this is the case, why wasted our time and why not just go straight on how to make the users switch transparently. If it is not, should we use that mapping scheme? or keep on inventing our new wheels? What's the advantages of that Mule Big5/7 mapping schemes? Jing-Shin -^^- P.S. Since there are so many mails, I will responde to your posting in a LIFO fashion. If you find the order is somewhat strange, don't be angry. -^^- ========== a few comments follow: ======== |> From simpson@math.psu.edu Sun May 8 10:55:51 1994 |> Subject: Thank you, Jing-Shin! |> |> I wrote: |> |> I notice that your proposals do not include HZ. Is this omission |> |> intentional? |> and |> > |> Jing-Shin, I'd suggest you familiarize yourself with HZ. |> |> Jing-Shin replied: Seems to lost something that I though important: ... Don't we all agree to use the existing HZ standard when we switched to '~{' (or whatever HZ ESC)? Do I need to mention anything in addition to this fact? In fact, my .c codes have some "holes" there for hooking up any HZ related converters. I had also mentioned that they could become HZ+* encoder/decoder by plugging-in related filter. I think it is quite explicit. In all my proposals (except the very early posting which ask people not to invent new wheels), I always used B5encoding/decoding to emphasize this undecide part. So I think there should be no misunderstanding. ... |> > Thanks for your suggestion. But I am wondering if it is helpful for |> > me to design the mapping schemes for 7-bit Big5. |> |> Ah, now I understand. Thanks very much for that clarification! You |> are only interested in designing mapping schemes for 7-bit Big5. You |> are not interested in the HZ part of HZ+. Your various b5encode |> proposals were never intended as stand-alone proposals for HZ+ |> (defining HZ+ to consist of HZ plus some form of 7-bit Big5). Rather, |> your b5encode proposals were only intended as suggestions for me and |> Ya-Gui. You were suggesting to me and Ya-Gui how we might improve our |> own HZ+ proposals by using a superior 7-bit Big5 mapping scheme. |> |> Thank you very much for your excellent suggestions. In that light, I |> think we now have only two HZ+ proposals, my HZ+S and Ya-Gui's HZ-2. |> Good! Life is suddenly much simpler! Thank you, Jing-Shin! |> There are still four. OK? (-^^-) In fact, I have a new idea to incorporate Ya-Gui's idea of moveing some (* as many as possible) less frequently used chars to the first plane with the same method as for B5E2 so that switching to the 2nd plane is even less frequent. I think Ya-Gui's idea is great. If this scheme comes out, it may be called YAB5E (Yet Another Big5 Encoding or Ya-Gui's value Added Big5 Encoding). (Don't flame on me to muddle the water again!! -^^- I am not sure whether I had time for such stuffs, although it requires only a few hours -^^-. How do you like this idea, Ya-Gui? ) |> |> Ya-Gui, let's you and I think about it for a day or two and decide |> whether we want to change our respective HZ+ proposals to use one of |> Jing-Shin's mapping schemes. After that, we can perhaps finally call Sound reasonable. |> for a vote on HZ+. With only two proposals on the table, it may |> be relatively easy to reach a consensus. |> |> -- Steve |> |> From shin@hermes.ee.nthu.edu.tw Sun May 8 07:19:33 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA00459; Sun, 8 May 94 07:19:33 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182082(2)>; Sun, 8 May 1994 20:22:23 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA01558; Sun, 8 May 94 20:15:04 CST Date: Mon, 9 May 1994 10:15:04 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405081215.AA01558@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: 7-bit Big5, CPU cycles, Big5 compatibility Status: O |> From simpson@math.psu.edu Sun May 8 10:59:06 1994 |> |> Jing-Shin Chang writes: |> |> > In HZ+S, you are preserving the character order in the big5 char order |> > more explicitly in the sense as mentioned above. |> |> Yes. That's why my HZ+S proposal is better. The Big5 order is very |> explicitly preserved. |> |> If you look at a Big5 table, you will see that within each block the |> characters are are linearly ordered by radicals and stroke count. I |> consider it important to explicitly preserve this natural order. The |> fact that HZ+S explicitly preserves this order is a significant |> advantage of HZ+S over all of the other proposals. |> |> > The most important point is, you rarely need to process you data |> > in the transformed code space. So I never consider it a problem. |> |> You never *need* to process it there, but in some circumstances it may |> be *desirable* to process it there. That has been our experience with |> HZ. I am thinking of ZWDOS. If you don't consider it a problem, that |> may be because you are not familiar with HZ or how people use it. |> Explicitly preserving such order may make people feel comfortable. But it has little significance for information processing. My judgement is: "in some circumstances" has a probability of near zero. So I don't think it is a "significant" advantage over the other proposals. |> > (Don't you think the blocks of codes in the transformed code space an |> > exact copy of their counterpart in the original Big5 code space? |> |> I don't think this is necessary or desirable. The code block |> structure is *not* a significant structural element in Big5. |> Characters with the same radical are *not* grouped in the same code |> block. The most significant structural elements of Big5 are, (1) the |> division into more and less frequently used characters, (2) the |> ordering of the characters by radical and stroke count. You will see all these information in the Big5/8 code space, and people are using this code space for information processing. Not the Big5/7 code space. The same is true for the use of existing font. We don't expect that Big5 users are willing to process in two different code space such that they either have two index files to the fonts or two set of fonts. |> |> > Is it important to pack the codes tightly. -^^-) |> |> Ya-Gui and I regard tight packing as elegant and beautiful. Tight |> packing also has some practical advantages, for instance in the use of |> fonts. I remember that Ya-Gui said: > To be fair, I think HZ+S's packing of all the Big5 characters > into two tight blocks without leaving unused gaps is aesthetically > desirable. I don't know why Steve forgot to mention this advantage. > With everything else being equal, this just might be the winning > edge. :-) "Aesthetically desirable" however just make one feel natural. And we could say that such feeling is an "aesthetic advantage". That's all. (How do you think, Ya-Gui?) As mentioned above, unless you want to process information in the 7-bit code space, there is no obvious gain to do such packing. Eventually, we will choose one code space for information processing. The fact is: the Big5 users are processing their documents in the untransformed code space (Big5/8). This fact has a very little chance to be reverted in the near future. If that is reverted, we could said that we had beat the de factor standard of the Big5/8 code. The probability is near zero. (Just like the CNS vs Big5 war -^^-). |> |> > You cannot force the Big5 users to change drastically to accept a |> > different computing schemes and habits. It's true that HZ users live |> > happily with the current status, i.e., processing their text with the ESC |> > characters. I don't think this is a normal way for Big5 users. |> |> Nobody is going to force Big5 users to use any particular software. |> Big5 users, like GB users, will be free to use any software they like. |> I can well imagine that many or most Big5 users, just like many or |> most GB users, will prefer software that hides the escape sequences. The above statement is to discuss the following point: You wrote: *|> *|> No. The current discussion is *not* about how to have a good exchange ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *|> code. The current discussion is about how to incorporate 7-bit Big5 ^^^^^^^^ *|> into HZ resulting in HZ+. Right now, in 1994, HZ is used not only as ^^^^^^^^^^^^^^^^^^^^^^ *|> an exchange code but also as an internal code. There is every reason ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ *|> to believe that HZ+ will continue to be used in exactly the same way ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *|> as HZ+ is now used. Ya-Gui has already indicated his plan to support *|> HZ+ in ZWDOS. ZWDOS has screen and printing capabilities, so fonts *|> will be needed. *|> What I feel from the above statements is that you are considering that we use HZ+ for exchange code as well as internal code even for the Big5 users. (Probably only "in some circumstances" ? -^^-) If this is the case, I will suggest that we do not go in this direction. My picture is: HZ Users ~{~} Big5 Users XXX +-------------+ +-------------+ | Application | (news, mail) | Application | (HZ+ supporting AP's) +-------------+ +-------------+ | | +-------------+ +-------------+ | HZ+ <-> HZ | | HZ+<->Big5/8| *NOT* Big5/7 +-------------+ +-------------+ | | +-------------+ +-------------+ | HZ+ | | HZ+ | +-------------+ +-------------+ | communication | ========================================================= ** So I think that HZ+ should still be regarded as an exchange code. ** It's nature should not be defined to be the same as HZ, which acts both as an exchange code and an internal code in the past. We should not confuse HZ with HZ+ in these aspects; remember that we are to integrate the HZ+BIG5. So it is probably not adequate to say that HZ+ will go in the same way as HZ. I don't expect that there will be many HZ soft-authors to write dedicated AP's for Big5 users. So the best transparent way is to go back to Big5/8 and let the Big5 soft-authors or commercial products developers handled the rest. In summary, I would give HZ/Big5/HZ+ the following attributes: - HZ+: exchange code - HZ : exchange code (e.g., in act) + internal code (daily life) - Big5: exchange code (e.g., in tw newsgroups, etc.) + internal code (daily life) Just don't put too much load to HZ+. Do you also think the above configuration is what we are seeking for? How about your idea, Ya-Gui? |> |> > |> Jing-Shin, I'd suggest you familiarize yourself with HZ. I have the |> > |> feeling that you don't know what considerations motivated it and how |> > |> it has developed. |> > |> > Thanks for your suggestion. But I am wondering if it is helpful for |> > me to design the mapping schemes for 7-bit Big5. |> |> HZ+ is one unified proposal. It doesn't make sense to design the |> 7-bit Big5 mapping scheme for HZ+ in ignorance of HZ. If that's what |> you are trying to do, I think you should abandon the attempt. Yes, HZ+ is a unified proposal. However, I did consider the HZ part and BIG5 part in the HZ+ scheme as two independent components (and the ESC sequence is a third and independent part). There is already existing HZ standard, so it seems that I do not need to worry about this part other than finding appropriate HZ filters. (I had once hacked up the HZ standard but it is too far from my daily life. But it doesn't hurt.) As I know from one of your latest posting, I think you had already understand that I was concentrating on the B5encoding part and I don't explicitly express my opinion on the HZ part because it is already there. So the answer here is probably redundant. |> |> -- Steve |> From simpson@math.psu.edu Sun May 8 08:00:00 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA01291; Sun, 8 May 94 08:00:00 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA00931; Sun, 8 May 1994 08:58:01 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 08:57:59 -0400 Date: Sun, 8 May 1994 08:57:59 -0400 Message-Id: <199405081257.IAA16132@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Jing-Shin's proposals In-Reply-To: <199405081024.GAA00579@leibniz.math.psu.edu> References: <199405081024.GAA00579@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > Somehow, I feel that my proposal(s) are not treated fairly enough!! -^^- Jing-Shin, I am truly sorry. If I have made a mistake, I will be happy to correct it. You certainly have the right to make as many proposals as you wish. I want to treat all of your proposals fairly. > I think the Big5 mapping is the major topic in designing a 7-bit coding > schemes for communication purpose. I feel that it is one topic, but not the only topic. > And that all us had agreed on the HZ part. And the ESC's can be > choosen based on preference (or XXX-compatibility). Yes, we have all agreed on those principles. > Then why you do not think that my B5E1 and B5E2 are qualified as complete > proposals? Because they do not support HZ. > Do I need to say that I am going to use the current HZ standard to > make it a complete proposal? Yes, you do. The way I see it, we are debating and discussing various proposals for HZ+. HZ+ is defined as a modified or enhanced version of HZ which incorporates 7-bit Big5. Your current B5E1 and B5E2 proposals do not include HZ and therefore do not qualify as complete HZ+ proposals. > (How angry I am!! -^^-) I'm truly sorry if I have made you angry. I want to treat you fairly. I'm quite willing to work with you to correct any problem. Let me make the following "proposal" to correct this problem. :-) You will first need to familiarize yourself with HZ. I hope this will not be too much trouble for you. I know that the time you can spend on HZ+ is very limited. You can start by reading the HZ specification. It is available in many places, for instance math.psu.edu, /pub/simpson/chinese/hzps/hz.txt. After that, you may want to spend a little time learning about the environment in which HZ users operate. You can do this by testing HZ software and reading its documentation. Some of your questions may be answered in the FAQ list for alt.chinese.text. FYI, a.c.t is a newsgroup which uses HZ as its coding scheme. If you have any questions or concernes about HZ, there are many people on soft-authors who can help you. I would be happy to help in any way I can. The originators of HZ are still active on soft-authors, so you can also contact them directly. After you become acquainted with HZ, you can rewrite your proposals so as to incorporate HZ. You may find that you want to revise your mapping schemes in this light. If two proposals are not enough, you are welcome to write as many proposals as you wish. The only thing I would ask is that you present complete HZ+ proposals. After that, your HZ+ proposals can be considered on an equal basis with other HZ+ proposals. To give you time to prepare your proposals, I'm willing to postpone the voting. Soft-authors, is there any objection to this plan? I think we ought to be completely fair to Jing-Shin, who has made many valuable contributions to this discussion. -- Steve From simpson@math.psu.edu Sun May 8 08:12:18 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA01573; Sun, 8 May 94 08:12:18 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id JAA00969; Sun, 8 May 1994 09:10:07 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 09:10:05 -0400 Date: Sun, 8 May 1994 09:10:05 -0400 Message-Id: <199405081310.JAA16150@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Cc: handa@etlken.etl.go.com Subject: Re: Thank you, Jing-Shin! In-Reply-To: <199405081024.GAA00579@leibniz.math.psu.edu> References: <199405081024.GAA00579@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin writes: > BTW, according to Nelson, a 7-bit Big5 encoding method is used in Mule. > Do you have a statistics on the user group of such Big5 Mule? Sorry, no, I don't. Maybe we can ask Kenichi Handa? Kenichi is the author of Mule. He was here on soft-authors a few days ago. I am copying this mail to him. Kenichi, do you have any statistics on how many people use Mule? > Is that mapping method the same as your currnt proposal? Yes, it is. > If this is the case, why wasted our time and why not just go > straight on how to make the users switch transparently. Sorry, I did not intend to waste your valuable time. The reason I do not advocate switching to Mule's coding scheme is that Mule's escape sequences are not entirely suitable for the purposes that we are discussing. Mule's escape sequences are perfect for Mule and other ISO-2022 applications, but not for HZ+. > If it is not, should we use that mapping scheme? or keep on that we > are discussing.inventing our new wheels? What's the advantages of > that Mule Big5/7 mapping schemes? Mule is an excellent software package. The advantages of the Mule mapping schemes are discussed in the Mule documentation. -- Steve From simpson@math.psu.edu Sun May 8 08:16:10 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA01754; Sun, 8 May 94 08:16:10 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id JAA00985; Sun, 8 May 1994 09:14:10 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 09:14:08 -0400 Date: Sun, 8 May 1994 09:14:08 -0400 Message-Id: <199405081314.JAA16160@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Jing-Shin's proposals, part 2 In-Reply-To: <199405081024.GAA00579@leibniz.math.psu.edu> References: <199405081024.GAA00579@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > |> Thank you very much for your excellent suggestions. In that light, I > |> think we now have only two HZ+ proposals, my HZ+S and Ya-Gui's HZ-2. > |> Good! Life is suddenly much simpler! Thank you, Jing-Shin! > |> > > There are still four. OK? (-^^-) Sorry. As of now there are only two, but there will be (at least) four when you have finished turning your proposals into complete HZ+ proposals. OK? -- Steve From shin@hermes.ee.nthu.edu.tw Sun May 8 08:18:38 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA01776; Sun, 8 May 94 08:18:38 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182086(1)>; Sun, 8 May 1994 21:22:08 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA01657; Sun, 8 May 94 21:15:08 CST Date: Mon, 9 May 1994 11:15:08 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405081315.AA01657@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: comment on Jing-Shin's latest new proposal Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Status: O |> From simpson@math.psu.edu Sun May 8 03:26:35 1994 |> |> Jing-Shin Chang writes: |> > But I feel that some of the advantages you have mentioned has more |> > or less related to your proposal in the particular ESC seq. For |> > instance, you had mentioned ISO compatibility, which has once been |> > agreed not to be in favored over HZ-compatibility. But it is |> > mentioned again. |> |> No. Let me repeat. The advantages of HZ+S have nothing to do with |> the choice of escape sequences. The kind of ISO compatibility that I |> am talking about has nothing to do with the choice of escape |> sequences. The choice of escape sequences is a very minor, relatively |> unimportant detail. |> |> The kind of ISO compatibility that I am talking about is concerned not |> with the choice of escape sequences but with the natural structure of |> the Big5 character set, in particular the natural division into two |> blocks and the ordering withing each block. If you don't perceive any I think that dividing the codes into freq. chars and less freq. chars so that we can designate the two blocks with some final chars (like G, H for CNS char set.) is probably what you are referring to as ISO-comp. here. Is this what you mean? You had add "...and the ordering within each block" as a criteria for ISO-comp.. Do you think it is a must? If this is not a requirement, I think such kind of compatibility is not unique to HZ+S. My knowledge on ISO-comp. comes from an article which discusses coding principle and rules for multibyte char sets. Such requirement is not emphasized. Do you have more references? |> blocks and the ordering withing each block. If you don't perceive any |> such structure, then you will never be able to understand my kind of |> ISO compatibility. |> Are you referring to such a map? I guess I know such structires. /* ASCII: 0x00 - 0x1f - control characters 0x20 - 0x2f - space and symbols 0x30 - 0x39 - digits (0 - 9) 0x3a - 0x40 - symbols (0x20- 0x40 - symbols and digits) 0x41 - 0x5a - upper case 0x5b - 0x60 - symbols 0x61 - 0x7a - lower case 0x7b - 0x7e - symbols 0x7f (127) - DEL Big5: High Bytes: 0x81-0x8d,0x8e-0xa0,0xa1-0xfe (0x01-0x0d,0x0e-0x20,0x21-0x7e, when high bit stripped) Low Bytes: 0x40-0x7e,0xa1-0xfe a0: ' ' | 0x80, a1: '!' | 0x80, fe: '~' | 0x80 High Bytes 00 7e 80 81 8d 8e a0 a1 a3 a4 c6 c9 f9 fa fe ff 00 +----------------------------------------------------------------+ | | | | 3f +----------------------------------------------------------------+ 40 +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+ : | 40-7e: 63 chars * U * . U . * S * # S # ! A ! x U x | : | * 3 * . 2 . * P * # T # ! L ! x 1 x | : | * * . . * C * # D # ! T ! x x | 7e +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+ 80 (NULL|0x80) 81 +----------------------------------------------------------------+ a1 +--------------------*----*-.-------.-*---*-#---#--!----!-x---x--+ : | a1-fe: 94 chars * U * . U . * bf* # S # ! A ! x U x | : | * 3 * . 2 . * c0$ # T # ! L ! x 1 x | : | * * . . * e0$ # D # ! Td5! x x | fe +--------------------*----*-.-------.-*-----#---#--!------x---x--+ ff (DEL) */ |> In that case, I urge you to first read my document describing the Big5 |> character set, math.psu.edu, /pub/simpson/chinese/hzps/etfonts.txt. |> If you take the time to read that document and look at a Big5 table, |> you will surely see that the Big5 character set does indeed have a |> structure. |> |> After that, I urge you to look at Mule, the Multilingual Emacs. You |> will see that Mule's method of incorporating Big5 into ISO-2022-JP-2 |> is exactly the same as my method of incorporating Big5 into HZ+. Both |> both methods fully respect the structure of the Big5 character set. |> |> Incidentally, the author of Mule was also a member of the |> ISO-2022-JP-2 committee. |> |> > We know that it is trivial. |> |> Sorry, but I don't think it is so trivial. |> |> > |> (1). The proposed HZ+S or HZ+ standard is based firmly on two |> > |> existing standards which are already very popular: HZ and Big5. This |> > |> should enable HZ+ to win a degree of immediate acceptance. |> > |> > I am not sure why this is an issue. ... |> > I think all the proposals are based on HZ and Big5. |> |> All of them except yours. Your documents on B5encode never mention |> HZ. Moreover, you have said that you are not familiar with HZ and do |> not understand the reasons for HZ or the environment in which HZ users |> operate. I guess I had answered the "never mention HZ" question in a previous reply. I mentioned that I was concentrating on Big5/7 <-> Big5/8 conversion. So the B5E1 and B5E2 are just the Big5 part. It does not prevent these proposals from becoming HZ+Big5 based when combined with your or Ya-Gui's more detailed spec. about HZ. |> |> > |> (3). HZ+S fully respects the natural partition of the Big5 code |> > |> range into blocks of more and less frequently used characters. |> > |> This leads to several additional advantages, listed in (4)-(7) |> > |> below. |> > |> > Natural partition is somewaht vague to define. |> > ........................................., it is vague how we |> > tell a scheme is more close to Big5 than another. Under my background |> > I would consider one scheme as being closer than the other if we have |> > a distance measure, and this distance measure tells me that |> > distance(X,Big5) < distance(Y,Big5). My distance measure is |> |> Your remarks show that you do not appreciate or value the natural |> structure of the Big5 character set. Presumably that is why your |> b5encode does not respect this structure. |> |> If you are interested in learing about the structure of the Big5 |> character set, please read my document on this subject, referenced |> above. |> You are referring to the "natural" structure, which, by your definition, is to preserve "the sorting order of the original char set". My "natural" structure is defined by the translation of the blocks of codes to another space without changing the geometry of the block of code. So we are based on different criteria. If the sorting order is important, your mapping is slightly better (but to find the sorting order from the transformed code space of B5E1 or B5E1 is not difficult, they are just a kind of scanning of the two or eight blocks). If we are seeking for easy mapping, my B5E1 and B5E2 is extremly easy. My personal experiences are: such sorting sequences are rarely used in chinese information processing. On the contrary, the mapping operation is a must for HZ+. So, probably we are not appreciate the same things. |> > |> (6). HZ+S is fully compatible not only with ETen Big5 but also |> > |> with corrupt variants of Big5 (e.g. so-called "HKU-Big5") in which |> > |> the less frequently used Chinese characters begin somewhere prior |> > |> to C940. Thus HZ+S may help to overcome the difficulties |> > |> associated with these corrupt variants. |> > |> > I am not familiar with this problem. Could you gives an example? |> > Thanks. |> |> An example is cxterm's Big5 support. If you study cxterm, you may see |> what I am talking about. I used HKU-Big5 once, and create my own ETBig5 dict. Yet, I am still not clear about such problems. More explicit example? |> |> > |> (7). The way is open for HZ+S to be compatible with a possible |> > |> future incorporation of Big5 into ISO-2022-JP-2, similar to the |> > |> present incorporation of CNS-11643. Such an unofficial extension |> > |> of ISO-2022-JP-2 is already implemented in Mule, the multilingual |> > |> Emacs. |> > |> > When I proposed not to use private ESC's in my first posting, I had |> > suggested to use a (99%) ISO compatible schemes, but people think |> > that ISO compatibility is not as important as HZ compatiblity. So I |> > didn't insist on that. I think it is trivial to switch back to that |> > status. |> |> You seem to think that ISO-2022 compatibility is nothing but a matter |> of using ISO-2022 style escape sequences. In reality, the choice of |> escape sequences is a minor, unimportant point. The really important |> point is the structure of character sets. Please look at Mule. |> |> -- Steve |> |> From simpson@math.psu.edu Sun May 8 10:02:56 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA04124; Sun, 8 May 94 10:02:56 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA01328; Sun, 8 May 1994 11:00:56 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 11:00:55 -0400 Date: Sun, 8 May 1994 11:00:55 -0400 Message-Id: <199405081500.LAA16346@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: How many HZ+ proposals are there? In-Reply-To: <199405081024.GAA00579@leibniz.math.psu.edu> References: <199405081024.GAA00579@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > |> Thank you very much for your excellent suggestions. In that light, I > |> think we now have only two HZ+ proposals, my HZ+S and Ya-Gui's HZ-2. > |> Good! Life is suddenly much simpler! Thank you, Jing-Shin! > > There are still four. OK? (-^^-) Sorry. Right now there are only two, HZ+S and HZ-2. Your B5E1 and B5E2 documents do not qualify as HZ+ proposals. After you turn them into HZ+ propsals, that will make four. Is this OK with you? > In fact, I have a new idea to incorporate > ... > If this scheme comes out, it may be called YAB5E (Yet > Another Big5 Encoding > ... > Don't flame on me to muddle the water again!! -^^- Sorry, instead of four I should have said 5. Or maybe 6, or even more. But don't worry, nobody is going to flame you. :-) -- Steve From mcevilly@netcom.com Sun May 8 10:37:33 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA05050; Sun, 8 May 94 10:37:33 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id IAA22026; Sun, 8 May 1994 08:35:37 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405081535.IAA22026@netcom.com> Subject: Re: Use HZ+S to save CPU cycles! To: ccnet-l@uga.bitnet Date: Sun, 8 May 1994 08:35:37 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405080703.AA24134@ifcss.org> from "Ya-Gui Wei" at May 8, 94 02:03:03 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 782 Status: O > >I hope Ya-Gui and Yidao will now stop saying that HZ+S is inefficient! > What Carlos said is true. But then by the same token maybe > we should stop saying that CNS is inefficient as well. :-) No, CNS is much worse because the tables will be huge. With HZ+S, the tables have two excellent properties: first, they are tiny, second, they can be easily generated with a simple mathematical function (addition). With CNS they would have to be hard coded. This gives CNS a disadvantage, because (I've already adressed this, but I'll say it again): it's the "it won't fit on the back of a cigarette pack" argument. It is an *unnecessary* layer of complexity between the users, and this will discourage development of compatible programs. This is why CNS will hurt the end users. From simpson@math.psu.edu Sun May 8 10:43:17 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA05177; Sun, 8 May 94 10:43:17 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA01467; Sun, 8 May 1994 11:41:14 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 11:41:12 -0400 Date: Sun, 8 May 1994 11:41:12 -0400 Message-Id: <199405081541.LAA16409@boole.math.psu.edu> To: chinesenet@uclink.berkeley.edu, ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: reasons for using HZ+ In-Reply-To: <9405080652.AA23777@ifcss.org> References: <9405080652.AA23777@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Ya-Gui Wei writes > I hope that each of us can give some thought to one question raised > by one netter (J.Chen?) on alt.chinese.computing: Jeffrey Chen's posting had a somewhat angry tone, but he made some very good points. I posted a long reply. If anybody would like to see Jeffrey Chen's posting or my reply, please let me know. > he invested a lot of money buying commercial Big5 softwares, and > can already use them to read and post to alt.chinese- > .text.big5. why would anyone think that he should not post in big5 > but should instead convert to some HZ+ scheme first? Indeed, why? This is a good question. Right now, the best reason to post news in HZ+ (instead of or in addition to 8-bit Big5) would be that you will reach additional readers. This is because many people have no way to receive 8-bit news. Later, if the chinese.* newsgroup hierarchy standardizes on HZ+, you will have no choice but to post in HZ+. This is a matter for the chinese.* organizers to decide. The chinese.* organizers are not going to allow direct 8-bit postings, and they are very reluctant to use set up Big5 mirror newsgroups. But they have strongly indicated that they will allow 7-bit Big5 postings, provided we can come up with a workable standard. This is a wonderful opportunity for HZ+. Let's go for it! I am copying this mail to the chinese.* organizers. Another reason to use HZ+ rather than 8-bit Big5 is that e-mail is often not 8-bit clean. Furthermore, many people use e-mail to post news, so the e-mail situation affects the news situation as well. > One scheme that might help with getting Big5 users is the straight > Big-5 proposal I mentioned in my last mail. A problem with this idea is that many people have no way to receive 8-bit news. A UUCP news feed plus an old tty driver will strip the 8th bits, leaving nothing but garbage. And this argument applies all the more to e-mail. > I should say that each of the proposals under consideration has its > merits, and I have hinted that ZWDOS will support whatever comes > out of it. Thanks, Ya-Gui! We are counting on you to help make HZ+ a reality. -- Steve From lai@apple.com Sun May 8 13:48:32 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA09987; Sun, 8 May 94 13:48:32 CDT Received: from mac23.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA07667; Sun, 8 May 94 11:46:11 -0700 for soft-authors@ifcss.org Message-Id: <9405081846.AA07667@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Sun, 8 May 1994 11:50:54 -0800 To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Ed Lai) Subject: CNS is 7 bit Big5 with mistake corrected Cc: shin@hermes.ee.nthu.edu.tw Status: O >Jing-Shin Chang writes: > > > In HZ+S, you are preserving the character order in the big5 char order > > more explicitly in the sense as mentioned above. > >Yes. That's why my HZ+S proposal is better. The Big5 order is very >explicitly preserved. > >If you look at a Big5 table, you will see that within each block the >characters are are linearly ordered by radicals and stroke count. I >consider it important to explicitly preserve this natural order. The >fact that HZ+S explicitly preserves this order is a significant >advantage of HZ+S over all of the other proposals. > lines deleted > >I don't think this is necessary or desirable. The code block >structure is *not* a significant structural element in Big5. >Characters with the same radical are *not* grouped in the same code >block. The most significant structural elements of Big5 are, (1) the >division into more and less frequently used characters, (2) the >ordering of the characters by radical and stroke count. > A Big5 expert should not be making statement like that. Big5 ordering is supposed to be stroke count and radical, not the other way around. i.e. within the same block characters are sorted by stroke count, and for characters with the same stroke count, they are ordered by radical. At least that was the intent. Except that something funny happened, for some characters they count the stroke count wrong, so some of them end up at the wrong place. I am not a CNS expert, in fact I just learn about CNS a few days ago here by asking what it is. So my knowledge is only based on Ross Peterson's posting and the chcode. From my understanding CNS character set is just the Big5 character set with the ordering corrected. That is why the chcode can do the ordering conversion with only a 63 elements array. Then why are there 13053 character in Big5 and only 13051 characters in CNS. This has been mentioned as a short coming of CNS. Actually these two characters are duplicates in Big5. It is like having two "A" ascii, then do you want to eliminate the duplicate? So really CNS is in essence two things, it takes Big5 character set and clean up the ordering and some mistakes, and then it remaps it to 7 bit printable ascii range. That is why I say just use the CNS scheme for the extension of HZ. This propsal is no differnt from other HZ extension scheme except for the order correction. The order correction code has been published, it is very simple and in public domain. It would take some extra cycles, but it would be worth it. And then it is also a standard. I keep hearing that CNS is not very popular, but our friends in Taiwan have confirmed again and again that there is not a lot of demand for 7 bit standard in Taiwan, so 7 bit standard cannot be popular, and CNS is certainly a leading 7 bit standard. It is so funny when I keep hearing about great Big5 and how bad CNS is. The truth is that CNS is just a slightly improved version of Big5, so if one says the character set and ordering of CNS is bad, then one is also critizing Big5. If one say the character set aspect is OK, it is the mapping to 7 bit that is wrong, then one is critizing HZ extension to Big5. Some may say that Ed is trying to raise the reputation of CNS by associating with Big5. Let me say that I hardly consider Big5 to be a badge of honor, so CNS's reputation is hardly enhanced because of this. CNS is not my favorite code, but it would fit the Big5 HZ extension. As an exchange code, it should be as clean as possible, and according to Ross, CNS includes the radicals and I think that it is big plus. Remember a document is read by many types of machine with different internal codes, GB, unicode etc, the more complete and cleaner the exchange code is, the better it is. CNS is an improvement. But what about CCCII and unicode based exchange code. The conversion would be more costly, but if we accept that, that would even be better. So my proposal is that we use CNS as an exchange code so that the exchange code can be clean, follows an established 7 bit standard. As for the escape code, I don't care which ones are used, pick any things proposed with HZ extension. In other words, in my own opinon, my proposal is not that differnt from others, except to correct for the mistakes in the Big5 (which ends up as CNS) and to add in the radicals (again ends up with CNS). It may be critized that the error correction is not worth the effort. But at least I am one of the know everything expert who can at the same time think that Big5 is the greatest and CNS is the worst. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Sun May 8 14:05:11 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA10553; Sun, 8 May 94 14:05:11 CDT Received: from mac23.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA08666; Sun, 8 May 94 12:01:51 -0700 for soft-authors@ifcss.org Message-Id: <9405081901.AA08666@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Sun, 8 May 1994 12:06:57 -0800 To: CCNET-L@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Ed Lai) Subject: Re: Use HZ+S to save CPU cycles! Cc: shin@HERMES.EE.NTHU.EDU.TW Status: O >> >I hope Ya-Gui and Yidao will now stop saying that HZ+S is inefficient! >> What Carlos said is true. But then by the same token maybe >> we should stop saying that CNS is inefficient as well. :-) > >No, CNS is much worse because the tables will be huge. >With HZ+S, the tables have two excellent properties: first, >they are tiny, second, they can be easily generated with a >simple mathematical function (addition). With CNS they would >have to be hard coded. This gives CNS a disadvantage, because >(I've already adressed this, but I'll say it again): it's the >"it won't fit on the back of a cigarette pack" argument. >It is an *unnecessary* layer of complexity between the >users, and this will discourage development of compatible >programs. This is why CNS will hurt the end users. Today I tried not to respond to every inaccuracies, it seem that some people's philosophy is that if you keep repeating lies, then it becomes truth. But I cannot resist answering to this one. There are conversion code from Taiwan available to the public. I have posted extract from it. The conversion table has 63 elements and takes up 63*4=252 bytes in each direction. There is also a 13 number index to speed up the search into the conversion table, that takes up another 13*2=26 bytes. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From mcevilly@netcom.com Sun May 8 14:49:17 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA11858; Sun, 8 May 94 14:49:17 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id MAA12819; Sun, 8 May 1994 12:47:07 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405081947.MAA12819@netcom.com> Subject: Re: Use HZ+S to save CPU cycles! To: lai@apple.com (Ed Lai) Date: Sun, 8 May 1994 12:47:06 -0700 (PDT) Cc: ccnet-l@uga.bitnet, soft-authors@ifcss.org In-Reply-To: <9405081901.AA08666@apple.com> from "Ed Lai" at May 8, 94 12:06:57 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1653 Status: O (Mail to Edmund Lai, also copied to ccnet-l and soft-authors) Ed Lai writes: > Today I tried not to respond to every inaccuracies, it seem > that some people's philosophy is that if you keep repeating > lies, then it becomes truth. > But I cannot resist answering to this one. If you would be so kind as to answer some of the sincere doubts about your views on CNS, we would not have to keep repeating them. There have been repetitions on all sides, btw. Please, join our discussion in a spirit of cooperation. > There are conversion code from Taiwan available to the public. > I have posted extract from it. The conversion table has 63 > elements and takes up 63*4=252 bytes in each direction. > There is also a 13 number index to speed up the search into the > conversion table, that takes up another 13*2=26 bytes. > Now this is the kind of positive, informative, contribution we need here. I admit I am not an expert on CNS either, and haven't studied the chcode closely. Can this conversion table be easily expressed as a mathematical function, or must it be hard coded? It was my impression that the latter was true--please correct me if I'm wrong. Having to hard code the table would be a slight disadvantage for CNS, because of the complexity argument (OK, I won't repeat it now). In case anyone missed the point, tables are not *necessary* for Big5 to HZ+ conversion. They would speed things up, but the fact that they are not necessary means HZ+ will be less complex to implement. Regards, Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From simpson@math.psu.edu Sun May 8 15:19:50 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA13021; Sun, 8 May 94 15:19:50 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA02549; Sun, 8 May 1994 16:17:40 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 16:17:39 -0400 Date: Sun, 8 May 1994 16:17:39 -0400 Message-Id: <199405082017.QAA16891@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Big5 protocol In-Reply-To: <199405081923.PAA16783@boole.math.psu.edu> References: <199405081541.LAA16409@boole.math.psu.edu> <9405081753.AA19910@iitmax.acc.iit.edu> <199405081923.PAA16783@boole.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Jianqing Hu writes: > From this point, I think what we should invent is a protocol to > bridge the 8 bit BIG5 and the 7 bit internet. It is an interface > rather than an encoding scheme. With software supporting this > protocol, BIG5 users can use their favorite BIG5 package to send 7 bit > safe code to internet. This sounds like an interesting idea. I'm not exactly sure what it means, but this is probably due to my ignorance. I am familiar in a general way with protocols such as FTP, HTTP, NNTP, but I don't know what a Big5 protocol would be or what it would do. Could you please explain how a Big5 protocol would work in practice? Is it a piece of user software, a daemon, or what? Would we need a system administrator's help to install it? -- Steve From simpson@math.psu.edu Sun May 8 16:20:46 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14982; Sun, 8 May 94 16:20:46 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA02795; Sun, 8 May 1994 17:18:45 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 17:18:43 -0400 Date: Sun, 8 May 1994 17:18:43 -0400 Message-Id: <199405082118.RAA16984@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: CNS is 7 bit Big5 with mistake corrected Reply-To: simpson@math.psu.edu Status: O I wrote: > >If you look at a Big5 table, you will see that within each block the > >characters are are linearly ordered by radicals and stroke count. Ed Lai replied: > A Big5 expert should not be making statement like that. I'm not a Big5 expert. I'm just an ordinary guy who happens to like to spend his leisure time gazing at the Big5 character table. People get their kicks in different ways .... :-) > Big5 ordering is supposed to be stroke count and radical, not the > other way around. i.e. within the same block characters are sorted > by stroke count, and for characters with the same stroke count, > they are ordered by radical. At least that was the intent. I guess I should have said it the other way around. That's what I meant. Stroke count and radical, not radical and stroke count. Sorry for that mistake on my part. > Except that something funny happened, for some characters they > count the stroke count wrong, so some of them end up at the wrong > place. True. The people who created the Big5 ordering miscounted some strokes. So what? Everybody can make mistakes. :-) Probably most Chinese dictionaries have mistakes in them. They are still extremely useful. Big5 is something like that. It has some mistakes in it, but it is very, very useful. The main reason it is very, very useful is because it is very, very popular. The mistakes are annoying to some people, but they don't detract from the usefulness. > CNS character set is just the Big5 character set with the ordering > corrected. Plus some rearrangement of the non-Chinese-character parts. But, did you ever stop to think, maybe not all the miscounts were found. Maybe the CNS ordering still has some miscounts. Do you know that it doesn't? I don't. Maybe the CNS ordering will have to be "corrected" again next year. :-) > So really CNS is in essence two things, it takes Big5 character set > and clean up the ordering and some mistakes, and then it remaps it > to 7 bit printable ascii range. Here I must seriously disagree. The 7-bit aspect is *not* an essential feature of CNS. The chcode stuff has two versions of CNS, a 7-bit version and an 8-bit version. The user implementation of CNS in ETen is 8-bit, not 7-bit. (Of course no users use *any* version of CNS, but that's another story.) I would desribe the essence of CNS by saying that it is a Chinese character coding scheme. It is somewhat like Big5, but with the order changed (or "cleaned up" as you put it), and some duplicate characters deleted, and with the non-Chinese-character part of Big5 twisted completely out of shape. These little "corrections" make CNS in practice a completely different coding system from Big5. In practice, people don't like it, because whenever they try to use it, garbage shows up on their computer screens and printouts. > I keep hearing that CNS is not very popular, but our friends in > Taiwan have confirmed again and again that there is not a lot of > demand for 7 bit standard in Taiwan, so 7 bit standard cannot be > popular, I don't follow your reasoning here. Are you saying that CNS *would* be popular, if only there were a large demand for 7-bit? If that's what you're saying, I question it. My hunch is that, if there were a large demand for 7-bit, people would use some 7-bit form of Big5. They would not use CNS. The reason is that Big5 is already extremely popular, while CNS is not popular. The interface between 8-bit Big5 and 7-bit Big5 would be simpler than the interface between 8-bit Big5 and 7-bit CNS. People always choose the simplest solution to their problem. > CNS is certainly a leading 7 bit standard. I question this. Can you name one person who uses it? Can you name one network that uses it for an exchange code? So far as I know, there isn't *any* leading 7-bit standard in Taiwan. 8-bit Big5 swamps everything else. > It is so funny when I keep hearing about great Big5 and how > bad CNS is. The truth is that CNS is just a slightly improved > version of Big5, so if one says the character set and ordering I don't think CNS is "bad" in any absolute sense. From a purely technical viewpoint, CNS probably has a lot to recommend it. The only problem about CNS that I can see is that it is not popular. And in my opinion, the popularity factor overrides everything else. If you want to convince people to use CNS instead of Big5, you ought to give some very, very compelling reasons. You haven't given any compelling reasons. Correcting a few stroke counts is not a compelling reason. > Some may say that Ed is trying to raise the reputation of > CNS by associating with Big5. I wouldn't say that, at least not any more. I'm glad to see that you are no longer desribing CNS misleadingly as "an encoding method for Big5". In general, I think the level of this posting is much higher than what you said before. At least now you are stating openly that you prefer CNS to Big5, and you are giving some reasons for your preference. Stroke count corrections are not a compelling reason, but at least they are a reason. > So my proposal is that we use CNS as an exchange code so that the > exchange code can be clean, Why is CNS "clean" while Big5 is "dirty"? Both of them are made of bits and bytes. :-) -- Steve From simpson@math.psu.edu Sun May 8 16:31:17 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15365; Sun, 8 May 94 16:31:17 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA02873; Sun, 8 May 1994 17:29:18 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 17:29:17 -0400 Date: Sun, 8 May 1994 17:29:17 -0400 Message-Id: <199405082129.RAA17008@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Re: reasons for using HZ+ In-Reply-To: <199405082033.QAA02622@leibniz.math.psu.edu> References: <199405082033.QAA02622@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Forwarded posting from John Delacour on CCNET. --------- >Ya-Gui Wei writes > > I hope that each of us can give some thought to one question raised > > by one netter (J.Chen?) on alt.chinese.computing: > [ ] > > he invested a lot of money buying commercial Big5 softwares, and > > can already use them to read and post to alt.chinese- > > .text.big5. why would anyone think that he should not post in big5 > > but should instead convert to some HZ+ scheme first? Indeed, why? I too have good software for both GB and Big5 and very nice truetype and postscript fonts to make working with Chinese a real pleasure, and I don't feel my "investment" is is any way reduced by the very slight inconvenience of having to convert Internet mail and news postings to 7-bit. In an ideal world this would not be necessary but until everything is 8-bit clean it is simply inconsiderate to post stuff in 8-bit code. This is the whole point of HZ. I can send 8-bit mail to anyone I like, but who can read it? Eudora for Mac and Windows, for example, facilitates this using MIME, but it's not that simple at all and the encoding/decoding is done on the fly and is beyond the control of the user. On the other hand with HZ a simple translation is performed at user level and there is no mystery involved at all -- quite simply 128 is subtracted from every GB byte and the result is placed between ~{ and ~}. The same goes for HZ+ except that about a quarter of the characters need to be treated differently, but the principle is the same. Forgive me if this doesn't sound too technical or hi-falutin but I don't see the point of making something so fundamentally simple sound like a great scientific theory. The tables are all there in the public domain and it doesn't take a genius to evaluate the merits of the various coding systems. >From the point of view of the Mac user very few things are needed in order to make HZ+ a working model:- Already for HZ there are the very basic I/O gb2hz and hz2gb applications based on Fung F.Lee's code. It probably took about ten minutes to compile and link each of these, so the same can be said of creating applications from Steve Simpson's code. Such little applications, though, are not very convenient since it means importing a file created in GB and exporting it to another file as HZ. Far better to have the gb<->hz(+) built into a text editor in the way that Tie Zeng at Apple has done with Tex-edit. I was hoping he'd be reading all this but perhaps he's signed off, so I'll write him. That is the second step, ie. to have a text-editor into which one can input the Chinese and then by a simple switch translate the text into 7-bit HZ(+). The third step is to have the authors of the newsreaders and mailers, eg. Eudora, rnMac, TheNews, NewsWatcher, Nuntius etc. build the 728/827 translators into their programmes and I don't think that is going to be any problem at all. I have send the relevant code to several of the Mac authors concerned with an clear explanation of the principle of the thing and including Ed Lai's Subtitle and cclib.16 so that they can see the pretty pictures and be encouraged :-) In short I'll say that I sympathise very deeply with Steve Simpson, given the quantity of irrelevant discussion that has gone on here in the past two weeks and admire his persistence. Nevertheless, even the irrelevant stuff has been very interesting and informative and the whole discussion has been thoroughly positive and constructive (except for the email I got from some idiot with a double-barrelled name pleading with me to sign off -- I notice he has made no contribution at all to the debate!) It would be encouraging to see the slightest evidence that anyone using UNIX or DOS or whatever had actually put HZ+ to practical use and tested fact against opinion. I have a test for you all. I will post a bit of ZhuangZi to ACC using HZ+ and you will tell me which fish I got wrong and which fish it ought to have been. When you have answered, I will know how well you know ZhuangZi and how well you understand HZ+ (The article will be entitled HZ+FISH) John From simpson@math.psu.edu Sun May 8 16:46:16 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15910; Sun, 8 May 94 16:46:16 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA02966; Sun, 8 May 1994 17:44:15 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 17:44:14 -0400 Date: Sun, 8 May 1994 17:44:14 -0400 Message-Id: <199405082144.RAA17035@boole.math.psu.edu> To: John Delacour Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: testing HZ+ In-Reply-To: <199405082033.QAA02622@leibniz.math.psu.edu> References: <199405082033.QAA02622@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O To: John Delacour John, once again, thanks for your enthusiasm about HZ+. I really appreciate your kind words. > It would be encouraging to see the slightest evidence that anyone using > UNIX or DOS or whatever had actually put HZ+ to practical use and tested > fact against opinion. My HZ+S prototype implementation has been available for anonymous FTP for about a week now. So far only a few people have bothered to test it. Maybe everybody else knows me too well to think that my programs might contain bugs .... :-) Seriously, my HZ+S <-> Big5 filters are straightforward modifications of Fung Fung Lee's HZ <-> GB filters. I'm quite sure they will work on any machine that has a C compiler, but more testing would be welcome. Everybody can get the stuff at math.psu.edu, /pub/simpson/chinese/hzps/. > I have a test for you all. I will post a bit of ZhuangZi to ACC using HZ+ > and you will tell me which fish I got wrong and which fish it ought to have > been. > When you have answered, I will know how well you know ZhuangZi and how well > you understand HZ+ > (The article will be entitled HZ+FISH) I also used the filters to post some mixed Big5 / GB / ASCII stuff. I urge everybody to try it. It's fun! -- Steve From mcevilly@netcom.com Sun May 8 17:31:35 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA17146; Sun, 8 May 94 17:31:35 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id PAA27699; Sun, 8 May 1994 15:29:39 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405082229.PAA27699@netcom.com> Subject: Expert opinion on CNS To: ccnet-l@uga.bitnet Date: Sun, 8 May 1994 15:29:39 -0700 (PDT) Cc: soft-authors@ifcss.org X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2938 Status: O Here is the opinion of a Chinese character coding expert from Taiwan on the CNS 11643 coding standard. I've put it into several seven-bit formats so everyone can understand it. Reference: Huang Da-Yi, "Tongyong Hanzi Biaozhun Jiaohuanma, CNS 11643," in _Zhongwenzima_, Changsung Wenhua Shiye Youxian Gongsi Chuban, 1990, page 53. [The below text is the intro- ductory paragraph for his comments on CNS 11643, and follows a three-page technical introduction. The short excerpt conveys his feelings and conclusions about CNS; there are 15 pages of detailed criticism of specific parts of CNS which explain why he came to this conclusion. My Chinese typing is way too slow, otherwise I would type in more. For the hanzi- impaired :-), there is pinyin at the end.] The ISBN code of this book is 957-9064-00-8. Huang Da-Yi has written at least two books on Chinese computing. Here's his opinion on CNS: (HZ encoded GB) ~{;(AKUbwa6`5DF*7y@4=iI\UbwaR;8vSVFFSV@C5D6+Nw#,J5TZJG@K7Q~} ~{1JD+!#2;9}#,9CDnK|:C4uR2JG8v9z5Daa~} ~{12Q'WSQ'Ko#,U}?IR}RTN*V?VM{&"68&59$&5\v0|0**j!"JS)d3S8T0|;C^V!"~} ~1(8)p2/44Ag[E!')R7m!"&?(,'i^j&L7')p)pD$Q"?V68;P7'O:Fn0|68~} ~1L"&c)Z>"&g?)!$X#BT!"(G-e,u*5&pM{JS3A&90|+4(oN$6-!"(,3S!"~} ~1Wd&c(C445Y5tJF8JQ/KO:V3PT*M:>^I:%#I*.Y3*%!J6BIP*6FIFZD[Z1=K$^MT[#J MKF&\T+?'H4&R;Z]!J.RJ:[+.JKJPW<7IH4$-"J5UIFZK:JRPM,'#^*%&IE"N MR:%!I%VE::5(Q?VD:JYAIFZF;K7[O=>S;ZW3L.JN8;S0M\>JNJW3#0JZV*2C MIEBR>J2GLT*A0\'VM4VA0:6FJ,.H4Z:SI+"[\KGJJ_RD5ZJZIU&ESKOYKA0\/VI-^^XZW3 ML.JN8:2DI.6Y<;BCJV6S?JJZJ^$-"KWZOL>D;+['KEVA0:6_I6FDWJ5(K+#% ?LJ%!L4ZHT\'7ITNEQZ90O,NJNK_YNWZA0PT*#0H-"@H- ` end (pinyin generated by Bamboo Helper:) Hua1 le5 zhe4me5 duo1 de5 pian1fu2 lai2 jie4shao4 zhe4me5 yi1 ge5 you4 po4 you4 lan4 de5 dong1xi1, shi2zai4 shi4 lang4fei4 bi3mo4. Bu2guo4, gu1 nian4 ta1 hao3dai3 ye3shi4 ge5 guo2jia1 biao1zhun3, qian1she4 dao4 fa3 tong3 de5 wen4ti3, zhi3hao3 mian3wei2qi2nan2; tong2shi2, ye3ke3yi3 rang4 da4jia1 hao3 hao3 ping2lun4 zhe4ge5 guo2jia1 biao1zhun3 de5 ge4zhong3 bu4 he2li3 zhi1 chu4. Sui1ran2, ta1 bing4 mei2you3 shen2me5 shi2 zhi3 shang4 de5 li4yong4 jia4zhi2, ke3shi4, hai2 bu4 shi1 wei2 fu4 mian4 jiao4yu4 de5 zui4hao3 jiao4cai2. Guan1xin1 zheng3ge5 guo2jia1 zhong1wen2 dian4nao3 qian2tu2 de5 hou4bei4 xue2 zi5 xue2 sun1, zheng4 ke3 yin3 yi3wei2 jian4, jiang1lai2 bi4mian3 fan4 tong2yang4 de5 cuo4wu4. From simpson@math.psu.edu Sun May 8 20:10:59 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22394; Sun, 8 May 94 20:10:59 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA03598; Sun, 8 May 1994 21:08:51 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Sun, 8 May 1994 21:08:48 -0400 Date: Sun, 8 May 1994 21:08:48 -0400 Message-Id: <199405090108.VAA17399@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: Big5 or CNS ? In-Reply-To: <9405082337.AA19205@ifcss.org> References: <9405082337.AA19205@ifcss.org> Reply-To: simpson@math.psu.edu Status: RO Yuan Jiang writes: > You misunderstand what Ed and I try to say. This misunderstanding > makes this debate so upsetting and unnecessary. OK, sorry, I'll calm down. There, I'm calm now. :-) > Again, we are not saying that CNS is more popular than Big5. NO! > No! We are saying that 7-bit CNS is more popular than any of the > other HZ+(minus HZ1). Ed and I have always explicitly stated the > two objects we compare although we might have used the wrong terms, > such as "encoded Big5". When talking about CNS not being popular, > you and John rarely state what you are comparing to. When I said CNS is not popular, I was always comparing CNS with Big5. I think I always stated that explicitly. I think that kind of comparison is valid and makes sense. I am comparing one Chinese coding standard to another Chinese coding standard. When you and Ed say that CNS is popular, what are you comparing CNS to? Apparently you are comparing it to a nonexistent standard, HZ+. I don't think this comparison makes much sense. If you compare a "something" to a "nothing", the "something" will always be more popular than the "nothing". Fine, but so what? If you want to discuss CNS, let's try to focus on the practical benefits of specific ways to use CNS. My current concern is the emerging HZ+ standard. Once the standard is in place, our biggest problem will be to lure Big5 users into using it. This requires a clear, simple message. We have to explain what HZ+ is, why they don't have to be afraid of it, and what it can do for them. If Big5 users hear that HZ+ is a 7-bit form of Big5, they won't be afraid. They will still want to know what 7-bit Big5 is good for, but this question has an easy answer, namely it is for transmitting Big5 over 7-bit networks. If Big5 users hear that HZ+ is CNS, they may stick around long enough to ask what CNS is. But if the answer to that question is, "CNS is an ROC government-approved coding scheme that is similar to Big5 but incompatible with Big5", they will run like hell. Nobody wants incompatibility problems. We'll never get a chance to tell them what HZ+ is good for. This is the essence of why I want to base HZ+ on Big5, not CNS. -- Steve From g53150@sakura.kudpc.kyoto-u.ac.jp Sun May 8 20:59:48 1994 Return-Path: Received: from kudpc.kudpc.kyoto-u.ac.jp by ifcss.org (4.1/IFCSS-Mailer) id AA24511; Sun, 8 May 94 20:59:48 CDT Received: from sakura.kudpc.kyoto-u.ac.jp (sakura.kudpc.kyoto-u.ac.jp [130.54.9.11]) by kudpc.kudpc.kyoto-u.ac.jp (8.6.5/8.6.5) with SMTP id KAA10264; Mon, 9 May 1994 10:57:45 +0900 Received: by sakura.kudpc.kyoto-u.ac.jp (4.1/6.4J.6) id AA12560; Mon, 9 May 94 10:58:06 JST Date: Mon, 9 May 94 10:58:06 JST From: "christian wittern" Message-Id: <9405090158.AA12560@sakura.kudpc.kyoto-u.ac.jp> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: CNS, BIG5 etc Status: O Returning from a couple of days off, I just finished wading through megabytes of discussion on this topic. The discussion startet very exciting, but towards the end got somehow beside the point IMHO. Steve wrote: > The discussion has been quite vigorous. The current state is that we > were almost ready to agree on a very simple 7-bit Big5 implementation, > when suddenly the waters got muddied by the CNS issue. Big5 is the > popular coding standard, used by everybody in Taiwan and many people > elsewhere for their daily work. CNS is a competing standard that has > no popular support whatsoever but is endorsed by the ROC governnment. > If this CNS business goes on much longer, I think I am going to give > up. I am not very patient with committees. > > -- Steve > I recommend that you take a rest, Steve. :-) In the meantime, I would like to explain, why I am strongly in favour of using CNS for the 7 bit part: 1. The topic CNS vs BIG5 is a misnamer. What we are discussing is the use of which code for exchange in email and news. This can, as has been established, be only a 7 bit code, so BIG5 is out. What is left over is CNS and an "ad hoc generated mapping of BIG5 to a 7bit code". For an enduser it will make *absolutely no difference* which of these codes is used: If he does not have the right software to read it, there will be only junk on his screen. In this point both codes are equal. 2. For the programmer 7 bit mapping of BIG5 can be done algorythmically, CNS has to be done by table lookup. The table lookup for CNS can be done with a pure reference table, without any analysis of the relationship - This will be simpler, but more MM=UI MAs every computer science book points out, table lookup is much faster than any algorythmic operation. Of course, the table would have to be held in memory, which would be at most 27kB. If it is held in memory only for the first block of 5401 characters, this can be reduced to 11k, with some penalty in speed for the other block of rarely used characters, which would then have to be read from disk on request. Anyway, the complexity of the task would be in the same category in both cases, far from overloading a program with 350k of code. 3. We have been talking of CNS 11643/86, which has been superseded by CNS 11643/92. The first two planes, which have roughly the same characterset as BIG5 (13051 chars) are unchanged. What has been level 14 in the older standard has been renamed to level 3. Level 4 includes those 7298 characters from the Unicode Han Character Repertoire, which have not yet shown up. It thus makes the first for levels of CNS a proper superset of this characterset. If CNS is used for the 7 bit part of HZ+ and if additional escape sequences are defined for the other levels of CNS, we would have a 7 bit encoding of the UniHan characters without additional effort. This might not be necessarry now (and of course need not be implemented immediately), but might come in handy later. There are three more levels in CNS so that CNS 11643/92 defines a total of 48027 characters. If we define escape sequences for all seven levels, somebody could even write a software that lets the people on CHPOEM-L exchange Tang poetry without fiddlling around with kludges to get some rare characters. I think, this would be terrific. :-) To arrive to a conclusion, and to clear the muddy waters: 1. CNS as 7 bit exchange code does not compete against BIG5 as a 8 bit internal code. Setting it up as CNS against BIG5 is muddying the waters. 2. To an enduser, 7bit CNS and "remapped 7bit BIG5" looks alike, certainly *not* like the familiar BIG5. Reconversion to BIG5 will always have to take place, either internally in the application, or externally with a filter. 3. "remapped 7 bit BIG5" would mean reinventing the wheel, whereas CNS is an existing standard. 4. Programming differences can be neglected. 5. CNS as a well defined code with an interesting potential for future extension. Both might be equal at (2) and (4), CNS is superior in (3) and (5). I recommend CNS 11643/92 as an exchange code for email and news on the internet. Christian Wittern, Kyoto From mcevilly@netcom.com Sun May 8 22:59:13 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA28992; Sun, 8 May 94 22:59:13 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id UAA28141; Sun, 8 May 1994 20:57:14 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405090357.UAA28141@netcom.com> Subject: Re: CNS, BIG5 etc To: ccnet-l@uga.bitnet Date: Sun, 8 May 1994 20:57:13 -0700 (PDT) Cc: soft-authors@ifcss.org, g53150@sakura.kudpc.kyoto-u.ac.jp In-Reply-To: <199405090159.SAA19373@mail.netcom.com> from "christian wittern" at May 9, 94 10:58:06 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 6593 Status: O Christian Wittern writes: > 1. The topic CNS vs BIG5 is a misnamer. What we are discussing is > the use of which code for exchange in email and news. This can, as > has been established, be only a 7 bit code, so BIG5 is out. Big5 is *not* out as an internal code, regardless of what is used as an exchange code. It's important to keep this in mind because we need to map from whatever 7-bit code we choose to Big5. We have a choice between a simple, straightforward mapping from Big5 to 7-bit, and a complex, uncertain mapping which is subject to change at any time at the whim of a bureau of the Taiwanese government. > For an enduser it will make *absolutely no difference* which of > these codes is used: At first glance this would appear so. Thinking through the way software gets written, however, changes the picture. Programmers must decide whether to write software for the new HZ+. The choice of exchange codes will influence their decisions. Therefore it makes a very big difference for the end user. > 2. For the programmer 7 bit mapping of BIG5 can be done > algorythmically, CNS has to be done by table lookup. The table > lookup for CNS can be done with a pure reference table, without > any analysis of the relationship - This will be simpler, but more Both exchange codes (CNS and HZ+S) can use tables. With CNS, the use of tables is mandatory (unless the programmer uses a case- statement-from-hell algorithm). Therefore one method, HZ+S, has more flexibility and is easier to convey. Both exchange codes have the speed advantage; only one has the simplicity advantage. > Anyway, the complexity of the task would be in the same category > in both cases, far from overloading a program with 350k of code. > You don't really think anyone believed that 350K of code was needed, do you? I mentioned it to show what our current CNS-compatible author must contemplate dealing with. Of course it could be done in a small table, although we haven't seen the source code to do so. It was my mistake to bring in that image, but let me now say let's stay with real issues. To address your point above: the complexity difference is there, though I admit it is subtle. This must be why some netters are missing it or brushing it off. If you say there is no difference between needing a table and not, perhaps you could tell me why Taiwan readers don't just use their Big5 software to read the HZ (GB 7-bit) newsgroups now? We both know the answer to this rhetorical question: because the programmers haven't taken the time to track down all the tables and write the software. How can we know they will go to track down the CNS tables? Remember, CNS changes all the time. I went to the Bureau of Standards in Taiwan last August to buy a copy. They said to me: "Are you sure you want to buy this? There will be a new one coming out sometime soon ('zui4jin')." I bought it then because having a copy for reference seemed better than nothing. > 3. We have been talking of CNS 11643/86, which has been superseded > by CNS 11643/92. The first two planes, which have roughly the same Well at least some of us have been talking about the 1992 version. It's been sitting next to my keyboard for the last few days, taking up space! Do you want to buy it? BTW acc. to Huang Da-Yi, if you use the first two planes in CNS to represent Big5, you lose EBCDIC compatibility. We might not care so much about this, but it might matter to a lot of users at twnmoe10.edu.tw, who are on a VM/CMS system. > Level 4 includes those 7298 characters from the Unicode Han > Character Repertoire, which have not yet shown up. It thus makes > the first for levels of CNS a proper superset of this > characterset. > If CNS is used for the 7 bit part of HZ+ and if additional escape > sequences are defined for the other levels of CNS, we would have a > 7 bit encoding of the UniHan characters without additional effort. > This might not be necessarry now (and of course need not be > implemented immediately), but might come in handy later. > > There are three more levels in CNS so that CNS 11643/92 defines a > total of 48027 characters. If we define escape sequences for all > seven levels, somebody could even write a software that lets the > people on CHPOEM-L exchange Tang poetry without fiddlling around > with kludges to get some rare characters. I think, this would be > terrific. :-) Interesting. The logical thing for the Taiwan standards bureau to do would be to evolve CNS to be equivalent to Unicode, and I suspect this is what they are doing. That's great. In the meantime, while the bugs are still being worked out of that, we need a good seven- bit mapping for Big5. Something that doesn't require a complex mapping, something that won't change often. > To arrive to a conclusion, and to clear the muddy waters: (1. Big5 vs. CNS vs. 7-bit Big5 vs CNS) (2. Users won't know the difference) > 3. "remapped 7 bit BIG5" would mean reinventing the wheel, whereas > CNS is an existing standard. I am wondering whether the wheel which has already been invented is broken in this case. Steve and I have asked others to name anyone who is still using this wheel; no one has come forward. One Chinese coding expert from Taiwan has given his opinion in a book, that CNS is an example of what to avoid. Why doesn't anybody use CNS? Lots of money has been spent on CNS, yet still it doesn't roll--maybe it is square? If it had no disadvantages, I wouldn't feel this way, but since it does, CNS needs some hard evidence to back it up. The evidence hasn't arrived. > 4. Programming differences can be neglected. ...at the peril of end users, who need applications. HZ+S makes programming easy. > 5. CNS as a well defined code with an interesting potential for > future extension. True, it is interesting... but this might turn out to be in the sense of the oft-repeated Chinese curse, "May you live in interesting times." I wonder why the III, which invented this wheel, is abandoning it? It may appear that I have my mind set against CNS. Actually this is not so--I just don't see a need for it at this point, and many of the doubts raised against it have not been dispelled. Can you dispel any of the doubts against CNS? (unused, unpopular, unnecessary, unfamiliar, unsupported?) (If anyone has any of these doubts about encoded Big5, let me point out that encoded Big5 has advantages not shared by CNS). Carlos McEvilly | email for info on Bamboo Helper Chinese text mcevilly@netcom.com | tools, or ftp to ftp.netcom.com:/pub/mcevilly From shann@wangwei.math.ncu.edu.tw Mon May 9 03:50:37 1994 Return-Path: Received: from wangwei.math.ncu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA06638; Mon, 9 May 94 03:50:37 CDT Received: from qingzhao.math.ncu.edu.tw by wangwei.math.ncu.edu.tw (4.1/SMI-4.1) id AA15753; Mon, 9 May 94 16:48:46 CST Date: Mon, 9 May 94 16:48:46 CST From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Message-Id: <9405090848.AA15753@wangwei.math.ncu.edu.tw> To: soft-authors@ifcss.org Subject: exchange and internal codes Cc: chinesenet@uclink.berkeley.edu Status: O Some discussions are about HZ+ as exchange or internal code. I think it might be both. Here is an example. Imagine I post the following message, which quote somebody else's directly, to chinese.test: =================================== Someone said: > This is a test ~{UbJGR;8vJTQi!#~} But I reply for ~1?V3S&"68H]^T!$~} =================================== Or, I might send a message to a friend asking for: =================================== ~1EA~}, ~1Py;CX|^V)t(o~} "~{:sCf~}" ~1/Q3S~} "~{aaCf~}" ~1G6+#~} behind? =================================== (The ~1...~} was made from HZ+S's implementation big2hzps.) If HZ+ would be an internal code (or process code, whatever) for some program with appropriate fonts, it would be wonderful. -Shann From simpson@math.psu.edu Mon May 9 07:40:19 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11547; Mon, 9 May 94 07:40:19 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA06089; Mon, 9 May 1994 08:38:13 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 9 May 1994 08:38:11 -0400 Date: Mon, 9 May 1994 08:38:11 -0400 Message-Id: <199405091238.IAA22200@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Cc: chinesenet@uclink.berkeley.edu Subject: chinese.* needs HZ+ now Reply-To: simpson@math.psu.edu Status: O To: soft-authors@ifcss.org, ccnet-l@uga.uga.edu Cc: chinesenet@uclink.berkeley.edu One of the organizers of the proposed chinese.* Internet newsgroup hierarchy has just released a new draft of the proposal. The draft contains a paragraph on encoding methods, which I append below. I present this in order to show you that there is a pressing need for HZ+, where HZ+ is defined as a 7-bit standard building on HZ and incorporating Big5. Let's work together to reach a consensus and create a viable HZ+ standard as soon as possible. My own particular proposal for HZ+ is called HZ+S. I feel that HZ+S is ready now to fulfill the need for HZ+, provided we on soft-authors and ccnet-l can reach a consensus. HZ+S is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps/. A sample implementation is included. Please look at HZ+S and send me your comments. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ --------------------- appended material follows ---------------------- PROPOSAL TO CREATE A NEW HIERARCHY CHINESE.* -------------------------------------------- [material omitted] Encoding Method --------------- The default encoding method would be HZ (hanzi) because it is a 7-bit protocol and is thus easiest and least trouble-some to transmit on the Internet. Cross-mirroring groups using different encoding methods (e.g. BIG5) may be created (if that turns out to be necessary) and named with an indicative suffix (e.g. chinese.***.big5). There are several proposals to incorporate Big5 into the 7-bit protocal HZ, and an active and vigorous discussion of these proposals are being conducted by interested parties on alt.chinese.computing. When the new protocol to incorporate both GB and Big5 is accepted and used widely, the chinese.* hierarchy will use that protocol so that users can use their favorite Chinese encoding scheme (be it GB, Big5 or CNS etc) on their computers and still be able to communicate with each other in chinese.* using the new protocol. [material omitted] ---------------------- end of appended material --------------------- From cyl Mon May 9 08:33:44 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA12985; Mon, 9 May 94 08:33:44 CDT Date: Mon, 9 May 94 08:33:44 CDT From: cyl (Yilong Chen) Message-Id: <9405091333.AA12985@ifcss.org> To: soft-authors Subject: (fwd) Re: Expert opinion on CNS Newsgroups: alt.chinese.computing Organization: IFCSS Chinese Community Info. Center Status: O finally someone new from taiwan said something about CNS, what do you guys think? nelson >From: r1503185@hp850s.ntu.edu.tw (Bow-Yaw Wang) Newsgroups: alt.chinese.computing Subject: Re: Expert opinion on CNS Date: 9 May 1994 13:10:31 GMT Organization: National Taiwan University Lines: 25 Message-ID: <2qlco8$9n8@netnews.ntu.edu.tw> References: <199405082229.RAA26588@cs.utexas.edu> NNTP-Posting-Host: r1503185%@hp850s.ntu.edu.tw X-Newsreader: TIN [version 1.1 PL9] To be fair you should also quote Dr. Timothy Huang's opinions on Big-5. Or better, quote also his latest opnions to Unicode/ISO-10646 if you can find his writings after the Unicode/ISO-10646 merge. Dr. Huang is the major advocate of CCCII. He always portrait anything other than CCCII as garbages. I believe he has his valid reason to promote CCCII, but I don't think you can quote his arguments for this Big-5-vs-CNS debate unless you quote for both CNS and Big-5. As for my own opinions: I have no strong feelings on either of the standard, but I can testify to the technical validity on Ed Lai's points. As the translation between Big-5 and CNS does not take much of the computing resources, why not use an existingg standardard instead of a surgeon'ed one? Maybe because of the mandatory from the government, maybe because of the ease of adaption to the 7-bit world, CNS has gained a group of firm supporters in Taiwan, in particular outside the personal computing world or the IBM mainframe world. It is the native encoding method used in all all the computers from DEC. Yes, CNS is not as popular as Big-5, but CNS is also not a dying standard, and is certainly more popular than the 7-bit surgery of Big-5. Hao-yang Wang Pai Technology, Inc. Taipei -- ³¯ Nelson Chin, Chen Yilong ====102 Brook Street Quincy MA 02170 1508 024 USA ¶h CIIP: nchin@hnt.com USENET: bu.mule CICC: cyl@ifcss.org PH:+1 617 472-2851 Às PRODIGY: GWBV10E@prodigy.com BUTTA: ttennis@bu.edu GIS: butta1@crsa.bu.edu From simpson@math.psu.edu Mon May 9 10:29:27 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA17017; Mon, 9 May 94 10:29:27 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA07150; Mon, 9 May 1994 11:27:03 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 9 May 1994 11:27:01 -0400 Date: Mon, 9 May 1994 11:27:01 -0400 Message-Id: <199405091527.LAA22503@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Cc: r1503185@hp850s.ntu.edu.tw (Bow-Yaw Wang) Subject: (fwd) Re: Expert opinion on CNS Newsgroups: alt.chinese.computing In-Reply-To: <9405091333.AA12985@ifcss.org> References: <9405091333.AA12985@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yilong Chen writes: > finally someone new from taiwan said something about CNS, what do > you guys think? Excellent! I'm glad somebody in Taiwan finally spoke up for CNS. It's very informative. I have to admit that I was wrong in saying that CNS is totally unpopular. I apologize, and I'm glad to be enlightened. > Maybe because of the mandatory from the government, maybe because > of the ease of adaption to the 7-bit world, CNS has gained a group > of firm supporters in Taiwan, in particular outside the personal > computing world or the IBM mainframe world. It is the native > encoding method used in all all the computers from DEC. Yes, CNS > is not as popular as Big-5, but CNS is also not a dying standard, Let's have some more information. One question I'd like an answer to: how close together are the Taiwan DEC version of CNS and the registered ISO-2022 version of CNS? If they are the same, I'd like to know that too. I'm somewhat confused about the relationships between 7-bit CNS, 8-bit CNS, and Big5. For example, ETen has a Big5 mode (the default mode) and an 8-bit Big5 mode, but no 7-bit Big5 mode. More questions: Does Taiwan DEC use a 7-bit or an 8-bit form of CNS, or both? Do Taiwan DEC text editors handle only CNS files, or only Big5 files, or both? If the text editors deal with CNS files, is that 8-bit or 7-bit CNS? What are the Taiwan DEC interfaces between Big5, 7-bit CNS, and 8-bit CNS, if any? Please pardon my ignorance, and enlighten me. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From mcevilly@netcom.com Mon May 9 11:24:29 1994 Return-Path: Received: from netcom.com (netcom11.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA18912; Mon, 9 May 94 11:24:29 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id JAA12591; Mon, 9 May 1994 09:22:06 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405091622.JAA12591@netcom.com> Subject: Re: (fwd) Re: Expert opinion on CNS To: ccnet-l@uga.bitnet Date: Mon, 9 May 1994 09:22:05 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <9405091333.AA12985@ifcss.org> from "Yilong Chen" at May 9, 94 08:33:44 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1606 Status: O > From: r1503185@hp850s.ntu.edu.tw (Bow-Yaw Wang) > To be fair you should also quote Dr. Timothy Huang's opinions on Big-5. Or > better, quote also his latest opnions to Unicode/ISO-10646 if you can find > his writings after the Unicode/ISO-10646 merge. > > Dr. Huang is the major advocate of CCCII. He always portrait anything other > than CCCII as garbages. I believe he has his valid reason to promote CCCII, > but I don't think you can quote his arguments for this Big-5-vs-CNS debate > unless you quote for both CNS and Big-5. I was not really interested in doing a comparison between CNS and Big5. Big5 is a fact of life no matter what; we can avoid CNS. HZ+ gives us a direct mapping of Big5 in 7-bits. We have no choice but to support Big5, so Dr. Huang's opinion on it would not affect the situation. > computers from DEC. Yes, CNS is not as popular as Big-5, but CNS is also not > a dying standard, and is certainly more popular than the 7-bit surgery of > Big-5. Thanks for telling us more about CNS. We need more technical facts in this discussion. Can you tell us, is it true that Seednet is abandoning CNS? We are wondering why the inventors of CNS (III) might be withdrawing it as an exchange code on Seednet. The advantage of a 7-bit version of Big5 is that it is tied to Big5 (which most users in Taiwan use), and not tied to an unfamiliar, unaccepted standard that doesn't map directly to Big5 and could change at any time. Let's give Big5 a 7-bit facelift with HZ+S, not a 7-bit multiple personality disorder with CNS. > Hao-yang Wang > Pai Technology, Inc. > Taipei From A7621GAC@AWIUNI11.EDVZ.UNIVIE.AC.AT Mon May 9 14:08:42 1994 Return-Path: Received: from AWIUNI11.EDVZ.UniVie.AC.AT (helios.edvz.univie.ac.at) by ifcss.org (4.1/IFCSS-Mailer) id AA25450; Mon, 9 May 94 14:08:42 CDT Message-Id: <9405091908.AA25450@ifcss.org> Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT by AWIUNI11.EDVZ.UniVie.AC.AT (IBM VM SMTP V2R2) with BSMTP id 3428; Mon, 09 May 94 21:06:12 MEZ Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT (NJE origin A7621GAC@AWIUNI11) by AWIUNI11.EDVZ.UNIVIE.AC.AT (LMail V1.2a/1.8a) with BSMTP id 8225; Mon, 9 May 1994 21:06:10 +0100 Date: Mon, 09 May 94 20:54:18 MEZ From: Werner Lemberg Subject: Future 7-bit standard To: soft-authors@ifcss.org Status: O If I have understood the discussion about HZ+ etc. right, it is intended to have a 7-bit encoding standard quickly. A few days before I said it doesn't matter to me which encoding scheme should be used. Then came Unicode UTF-7. I believe this will be the standard of the future, because almost every language of the world can be addressed. I know, I know, this is a bit outside of the discussion, but my mother tongue is German, and we all here suffer under missing Umlauts ... How realistic is an acceptance of Unicode in the Chinese speaking community? IMHO UTF-7 is simple to implement, and mapping tables for Big5, GB, CNS are all available (at unicode.org). A question still remain: how to access all CCCII characters not included in Unicode. Werner From A7621GAC@AWIUNI11.EDVZ.UNIVIE.AC.AT Mon May 9 14:09:11 1994 Return-Path: Received: from AWIUNI11.EDVZ.UniVie.AC.AT (helios.edvz.univie.ac.at) by ifcss.org (4.1/IFCSS-Mailer) id AA25475; Mon, 9 May 94 14:09:11 CDT Message-Id: <9405091909.AA25475@ifcss.org> Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT by AWIUNI11.EDVZ.UniVie.AC.AT (IBM VM SMTP V2R2) with BSMTP id 3332; Mon, 09 May 94 20:49:56 MEZ Received: from AWIUNI11.EDVZ.UNIVIE.AC.AT (NJE origin A7621GAC@AWIUNI11) by AWIUNI11.EDVZ.UNIVIE.AC.AT (LMail V1.2a/1.8a) with BSMTP id 7901; Mon, 9 May 1994 20:49:53 +0100 Date: Mon, 09 May 94 20:45:57 MEZ From: Werner Lemberg Subject: GB/Big5 conversion To: soft-authors@ifcss.org Status: O Are there some intelligent converters between GB/Big5 in the public domain ? With intelligent I mean doing conversion word by word (or better phrase by phrase) instead of character by character. Werner From simpson@math.psu.edu Mon May 9 14:34:24 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA26487; Mon, 9 May 94 14:34:24 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA09309; Mon, 9 May 1994 15:32:24 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 9 May 1994 15:32:23 -0400 Date: Mon, 9 May 1994 15:32:23 -0400 Message-Id: <199405091932.PAA22929@boole.math.psu.edu> To: Werner Lemberg Cc: soft-authors@ifcss.org Subject: GB/Big5 conversion In-Reply-To: <9405091909.AA25475@ifcss.org> References: <9405091909.AA25475@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Werner Lemberg writes: > Are there some intelligent converters between GB/Big5 in the public domain ? > With intelligent I mean doing conversion word by word (or better phrase by > phrase) instead of character by character. > > > Werner I asked the same question here a few weeks ago. Somebody told me that TianMa has it, but that's commercial, not free software. -- Steve From jiang Mon May 9 22:41:00 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA13047; Mon, 9 May 94 22:41:00 CDT Date: Mon, 9 May 94 22:41:00 CDT From: jiang (Yuan Jiang) Message-Id: <9405100341.AA13047@ifcss.org> To: ccnet-l@uga.bitnet, mcevilly@netcom.com Subject: Re: CNS, BIG5 etc Cc: g53150@sakura.kudpc.kyoto-u.ac.jp, soft-authors Status: O >We have a choice between a simple, straightforward mapping from Big5 >to 7-bit, and a complex, uncertain mapping which is subject to change >at any time at the whim of a bureau of the Taiwanese government. The problem is that a simple, straight forward mapping (in the fashing of GB->HZ mapping) from B5 to 7-bit does not exist. The simpliest and most straight forward one, proposed so far is B5encode v.1, and B5encode v.2 is next. However, I don't see you promoting them if you love simplicity and straghtforwardness so much. There is no absolute winner if you emphysize simplicity. But 7-bit CNS is the clear winner in terms of popularity and support. From rap@doc.ic.ac.uk Tue May 10 05:07:08 1994 Return-Path: Received: from santos.doc.ic.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA24285; Tue, 10 May 94 05:07:08 CDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0q0ogP-00070fC; Tue, 10 May 94 11:05 BST Message-Id: Date: Tue, 10 May 94 11:05 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Big5/CNS conversion Status: O I've written a little program to generate fairly efficient C functions to convert back and forth between Big5 and CNS. It's currently in incoming/b5cns.tar.gz on ifcss.org, but probably belongs in /software/unix/convert/. It's just meant as a prototype, and is probably buggy. The C functions it generates are not for the faint-hearted, but they should do the conversions with an average of fewer than 3 comparisons on top of the arithmetic that most implementations of a 7-bit form require anyway. From simpson@math.psu.edu Tue May 10 06:20:19 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA26124; Tue, 10 May 94 06:20:19 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id HAA15601; Tue, 10 May 1994 07:18:19 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 10 May 1994 07:18:18 -0400 Date: Tue, 10 May 1994 07:18:18 -0400 Message-Id: <199405101118.HAA24680@boole.math.psu.edu> To: Ross Paterson Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Big5<->CNS conversion; adding CNS to HZ+ ? In-Reply-To: References: Reply-To: simpson@math.psu.edu Status: O Ross Paterson writes: > I've written a little program to generate fairly efficient C functions > to convert back and forth between Big5 and CNS. It's currently in > > incoming/b5cns.tar.gz > > on ifcss.org, but probably belongs in /software/unix/convert/. Thanks, Ross! I haven't yet looked at your program, but I assume it uses your Big5<->CNS mapping generated from the Unicode table, as in your earlier posting. How much does that mapping differ from the mapping used in iiidns.iii.org.tw:/pub/seed/chcode? I have no idea how the chcode mapping was generated. There has been a suggestion to employ 7-bit CNS as an "exchange code" for Big5 in HZ+. One problem I foresee is that Big5<->CNS conversion may not be completely standardized. If two Big5 users use different Big5<->CNS filters, confusion may result. I intend no criticism of your program. As we learned yesterday, there is a group of people in Taiwan who actually use CNS in their work. For this reason if no other, the world needs Big5<->CNS conversion, and your idea of basing it on the Unicode table is excellent. I'm only pointing out that we perhaps can't expect Big5<->CNS conversion to be an exact science. What would you think if I were to add CNS support to HZ+S, with different escapes, in addition to but not replacing the Big5 support that is already there? Before doing this, I'd need a little more information on CNS. [ Backgound: HZ+ is defined as HZ plus some form of 7-bit Big5, for use on the Internet. HZ+S is my specific proposal for HZ+. HZ+S is in math.psu.edu, /pub/simpson/chinese/hzps. ] -- Steve From simpson@math.psu.edu Tue May 10 12:44:00 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08155; Tue, 10 May 94 12:44:00 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA18065; Tue, 10 May 1994 13:42:00 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 10 May 1994 13:41:59 -0400 Date: Tue, 10 May 1994 13:41:59 -0400 Message-Id: <199405101741.NAA25358@boole.math.psu.edu> To: chinesenet@uclink.berkeley.edu Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re:Revised Proposal(first attemp In-Reply-To: <9405101157.tn350463@aol.com> References: <9405101157.tn350463@aol.com> Reply-To: simpson@math.psu.edu Status: O mpei@aol.com writes: > (1) How will you or us decide if a new scheme is "accepted and used > widely"? Some people may think there is a hidden agenda to > exclude... > > (2) Do we allow a new scheme on the chinese.* hierarchy BEFORE it > is used widely? If we do not, then you have a catch-22 > situation, because there is no place for the new scheme to > try and win wide acceptance of the users, ... Professor Pei, thanks for your understanding. I don't want the chinese.* hierarchy to adopt a totally unproven standard. But keep in mind that, no matter how good HZ+ is from a technical standpoint, wide acceptance will depend on many factors. Timely adoption by the chinese.* hierarchy would give HZ+ a big boost. A new communication standard is always to some extent a chicken-and-egg situation -- nobody wants to use it unless everybody else is already using it. After we finalize the HZ+ technical spec, there will be an initial testing period where people simply use HZ+ informally on the Internet. Hopefully we can get some of our Taiwanese friends involved and they will make good used of the Big5 support. After this testing period, we can consider whether to adopt HZ+ as an official chinese.* standard. > (3) If someone proposes an odd scheme, let him and his followers try > to promote it. If it is no good, eventually, it will wither and > die due to lack of public support. I agree. And even if chinese.* adopts HZ+ as a standard, I'm not 100 percent certain that HZ+ will become widely accepted. As I said, many factors are involved. The chinese.* volunteers should monitor the situation closely. Coding schemes are a technical issue, but they also affect people's perceptions. > > The participants of the new hierarchy may include overseas Chinese, > > people in Mainland China (when it finally joins the Internet at some > > future point), Hongkong, and Taiwan, Singapore, Malaysia and just > > about anybody else... > > I suggest deleting the word "may". We want to "include", not just > "may include", which implies the possibility of "may not include". That's a good idea. Or maybe even better, "will include", which stresses our intention or desire to include. > How about adding an opening statement like: > > The purpose of the new hieraracy is to provide a forum for anyone > who writes in the Chinese language - both traditional and simplified. > > This defines a broad charter for the new hierarachy. I agree, that's very good. It stresses the all-inclusive nature of the chinese.* hierarchy as an open forum based on language only, not nationality, not physical location -- not even ethnicity! :-) I especially like the idea of mentioning both simplified and traditional characters. Because of technical problems (7-bit vs 8-bit, HZ vs Big5, etc), some traditional character users may get the impression that we are trying to exclude them. By mentioning traditional characters explicitly, we can help to correct that impression. About the word "writes", there may also be some people whose Chinese is not very good, who want to read a little but not post. (For example, my knowledge of Chinese is very limited, not enough to write much, but I sometimes read a little using some simple software that I developed -- cchelp.) Also, there are lots of people who can write Chinese very well but don't like to type it into the computer. Such readers should be welcome, even if they don't post anything. (But maybe this doesn't have to be stated explicitly -- it's implicit in the idea of a language-based newsgroup hierarchy.) -- Steve From jiang Tue May 10 13:19:18 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA09693; Tue, 10 May 94 13:19:18 CDT Date: Tue, 10 May 94 13:19:18 CDT From: jiang (Yuan Jiang) Message-Id: <9405101819.AA09693@ifcss.org> To: mcevilly@netcom.com, ccnet-l@uga.bitnet Subject: Re: (fwd) Re: Expert opinion on CNS In-Reply-To: Mail from 'mcevilly@netcom.com (Carlos McEvilly)' dated: Mon, 9 May 1994 09:22:05 -0700 (PDT) Cc: soft-authors@ifcss.org Status: O >> >> Dr. Huang is the major advocate of CCCII. He always portrait anything other >> than CCCII as garbages. I believe he has his valid reason to promote CCCII, >> but I don't think you can quote his arguments for this Big-5-vs-CNS debate >> unless you quote for both CNS and Big-5. > >I was not really interested in doing a comparison between CNS and Big5. However, the fact that Dr. Huang has more critical comments on Big5, we should expect him trashing the other HZ+ more than CNS. From simpson@math.psu.edu Tue May 10 13:56:27 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11129; Tue, 10 May 94 13:56:27 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA18121; Tue, 10 May 1994 13:49:29 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 10 May 1994 13:49:27 -0400 Date: Tue, 10 May 1994 13:49:27 -0400 Message-Id: <199405101749.NAA25370@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Taiwan, Big5, and the chinese.* newsgroup hierarchy In-Reply-To: <199405101741.NAA25358@boole.math.psu.edu> References: <9405101157.tn350463@aol.com> <199405101741.NAA25358@boole.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O mpei@aol.com writes: > (1) How will you or us decide if a new scheme is "accepted and used > widely"? Some people may think there is a hidden agenda to > exclude... > > (2) Do we allow a new scheme on the chinese.* hierarchy BEFORE it > is used widely? If we do not, then you have a catch-22 > situation, because there is no place for the new scheme to > try and win wide acceptance of the users, ... Professor Pei, thanks for your understanding. I don't want the chinese.* hierarchy to adopt a totally unproven standard. But keep in mind that, no matter how good HZ+ is from a technical standpoint, wide acceptance will depend on many factors. Timely adoption by the chinese.* hierarchy would give HZ+ a big boost. A new communication standard is always to some extent a chicken-and-egg situation -- nobody wants to use it unless everybody else is already using it. After we finalize the HZ+ technical spec, there will be an initial testing period where people simply use HZ+ informally on the Internet. Hopefully we can get some of our Taiwanese friends involved and they will make good used of the Big5 support. After this testing period, we can consider whether to adopt HZ+ as an official chinese.* standard. > (3) If someone proposes an odd scheme, let him and his followers try > to promote it. If it is no good, eventually, it will wither and > die due to lack of public support. I agree. And even if chinese.* adopts HZ+ as a standard, I'm not 100 percent certain that HZ+ will become widely accepted. As I said, many factors are involved. The chinese.* volunteers should monitor the situation closely. Coding schemes are a technical issue, but they also affect people's perceptions. > > The participants of the new hierarchy may include overseas Chinese, > > people in Mainland China (when it finally joins the Internet at some > > future point), Hongkong, and Taiwan, Singapore, Malaysia and just > > about anybody else... > > I suggest deleting the word "may". We want to "include", not just > "may include", which implies the possibility of "may not include". That's a good idea. Or maybe even better, "will include", which stresses our intention or desire to include. > How about adding an opening statement like: > > The purpose of the new hieraracy is to provide a forum for anyone > who writes in the Chinese language - both traditional and simplified. > > This defines a broad charter for the new hierarachy. I agree, that's very good. It stresses the all-inclusive nature of the chinese.* hierarchy as an open forum based on language only, not nationality, not physical location -- not even ethnicity! :-) I especially like the idea of mentioning both simplified and traditional characters. Because of technical problems (7-bit vs 8-bit, HZ vs Big5, etc), some traditional character users may get the impression that we are trying to exclude them. By mentioning traditional characters explicitly, we can help to correct that impression. About the word "writes", there may also be some people whose Chinese is not very good, who want to read a little but not post. (For example, my knowledge of Chinese is very limited, not enough to write much, but I sometimes read a little using some simple software that I developed -- cchelp.) Also, there are lots of people who can write Chinese very well but don't like to type it into the computer. Such readers should be welcome, even if they don't post anything. (But maybe this doesn't have to be stated explicitly -- it's implicit in the idea of a language-based newsgroup hierarchy.) -- Steve From dchan@cisco.com Tue May 10 14:17:08 1994 Return-Path: Received: from lager.cisco.com by ifcss.org (4.1/IFCSS-Mailer) id AA12131; Tue, 10 May 94 14:17:08 CDT Received: (dchan@localhost) by lager.cisco.com (8.6.8+c/CISCO.SERVER.1.1) id MAA29775 for soft-authors@ifcss.org; Tue, 10 May 1994 12:15:08 -0700 From: Derek Chan Message-Id: <199405101915.MAA29775@lager.cisco.com> Subject: please delete me from the list To: soft-authors@ifcss.org Date: Tue, 10 May 94 12:15:08 PDT X-Mailer: ELM [version 2.3 PL11] Status: O Please delete me from the list. I want to be out of it temparatly. Thanks. Derek -- From lai@apple.com Tue May 10 15:06:47 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA14158; Tue, 10 May 94 15:06:47 CDT Received: from mac231.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA10571; Tue, 10 May 94 13:04:41 -0700 for soft-authors@ifcss.org Date: Tue, 10 May 94 13:04:41 -0700 Message-Id: <9405102004.AA10571@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu From: lai@apple.com (Edmund Lai) Subject: Re: Big5<->CNS conversion Status: O > > I've written a little program to generate fairly efficient C functions > > to convert back and forth between Big5 and CNS. It's currently in > > > > incoming/b5cns.tar.gz > > > > on ifcss.org, but probably belongs in /software/unix/convert/. > >Thanks, Ross! > >I haven't yet looked at your program, but I assume it uses your >Big5<->CNS mapping generated from the Unicode table, as in your earlier >posting. How much does that mapping differ from the mapping used in >iiidns.iii.org.tw:/pub/seed/chcode? I have no idea how the chcode >mapping was generated. > >There has been a suggestion to employ 7-bit CNS as an "exchange code" >for Big5 in HZ+. One problem I foresee is that Big5<->CNS conversion >may not be completely standardized. If two Big5 users use different >Big5<->CNS filters, confusion may result. > I can answer this question. I have not look at Ross's program yet but I have his previous posting of Big5<->CNS as deduced from unicode. I also have look at the chcode source, during the weekend I translate the Big5->CNS portion into PASCAL because that is the only compiler I got on my portable at home. To test out the program I go through Ross's posting, and the result matches. Since the two conversion came from two completely independent source, this is very reassuring. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From simpson@math.psu.edu Tue May 10 18:33:04 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22336; Tue, 10 May 94 18:33:04 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id TAA22046; Tue, 10 May 1994 19:30:57 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 10 May 1994 19:30:56 -0400 Date: Tue, 10 May 1994 19:30:56 -0400 Message-Id: <199405102330.TAA25932@boole.math.psu.edu> To: lai@apple.com (Edmund Lai) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Why convert to CNS? In-Reply-To: <9405102004.AA10571@apple.com> References: <9405102004.AA10571@apple.com> Reply-To: simpson@math.psu.edu Status: O OK everybody, I admit I was wrong. CNS is not as unpopular as I had thought. However, I still have some serious concerns about the way Ed is proposing to use CNS. And I have a reason to worry about this, because I want HZ+ and the chinese.* newsgroups to be a success. I wrote: > There has been a suggestion to employ 7-bit CNS as an "exchange code" > for Big5 in HZ+. One problem I foresee is that Big5<->CNS conversion > may not be completely standardized. If two Big5 users use different > Big5<->CNS filters, confusion may result. Ed Lai replied: > I can answer this question. I have not look at Ross's program yet > but I have his previous posting of Big5<->CNS as deduced from unicode. > I also have look at the chcode source, during the weekend I translate > the Big5->CNS portion into PASCAL because that is the only compiler > I got on my portable at home. To test out the program I go through > Ross's posting, and the result matches. Since the two conversion > came from two completely independent source, this is very reassuring. Ed, I don't understand, how does this answer the question? What test did you actually do here? The only conclusive test would be to generate complete Big5 and CNS tables (the entire code range, not just the standard 13051 characters) and then convert them using both converters and see if the results match exactly. Did you do that? Even if you did, that would only prove that *these two* converters agree. It would not prove that *all* Big5<->CNS converters agree. There are good reasons to suspect that all Big5<->CNS converters do not agree. According to Christian Wittern's recent posting, some of Big5 (apart from the standard 13053 characters) is not very well standardized, that is it may depend on which vendor's implementation you have, whether you are in Taiwan or Hong Kong, etc. I can well believe what Christian is saying. Big5 is a popular standard, not an official standard. Now, if Big5 is not completely standardized, how can Big5<->CNS conversion be completely standardized? I strongly suspect it is not. My concern is the following. Suppose we use CNS as an "exchange code" for Big5 in an international setting. What would this mean in practice? Suppose X is in Hong Kong and Y is in Canada and X wants to send e-mail to Y. X types his message in Big5, then presses the "send" key. X's Hong Kong software translates it into CNS and sends it. Y's Taiwanese software receives it and translates it back to Big5, except Y doesn't know it's a Taiwanese translation, because all this is supposed to be transparent to the user. Now X and Y get upset because they find that some of the non-Chinese characters don't match -- what they see on the screen isn't the same. They complain to Ed Lai and Steve Simpson in the U.S.A. Ed and Steve can't tell whether the problem is because X and Y are using different versions of Big5, or different Big5<->CNS converters, or both. X and Y conclude that the "user transparent exchange code" (HZ+ in this case) is junk. My point is, Big5 is already complicated enough. Why introduce an extra layer of complication by translating to CNS and back? It seems better to use an extremely simple mapping from the Big5 codes to the exchange codes, as in my HZ+S proposal. The exchange standard guarantees that this mapping is 100 percent invertible, so we know X and Y have the same bytes. If their screens don't look exactly the same, they must be using different versions of Big5. We don't have to wonder whether the problem is in Big5<->CNS conversion. I know you are a CNS fan, so maybe your answer is that, people should stop using Big5 and switch to CNS. But, for better or worse, Big5 is extremely popular. People will not give it up easily. If HZ+ doesn't work well with Big5, they will give up HZ+ rather than give up Big5. -- Steve From simpson@math.psu.edu Tue May 10 20:49:11 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA26815; Tue, 10 May 94 20:49:11 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id VAA22998; Tue, 10 May 1994 21:45:04 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 10 May 1994 21:45:03 -0400 Date: Tue, 10 May 1994 21:45:03 -0400 Message-Id: <199405110145.VAA26155@boole.math.psu.edu> To: Roland Wang Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: why should we use CNS as an exchange code for Big5 ? In-Reply-To: <199405110057.UAA22787@leibniz.math.psu.edu> References: <199405110057.UAA22787@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Roland Wang writes: > 3. character set != internal code (encoding scheme) != exchange code > (They can have same value, but they are different concepts) > > there are still misunderstandings/confusions on #3. And, some > discussions are repeating. > ... > The better we understand #3, the easier for discussion (and > utimately reach a good solution for chinese.* encoding scheme). If you think I'm confused, please enlighten me. Please explain the terms "exchange code" and "internal code" as they will apply to the chinese.* hierarchy, and as they now apply to alt.chinese.text. After that, please tell me why Ed Lai wants chinese.* to use CNS as an exchange code for Big5, even though CNS<->Big5 translation may not be 100 percent well defined. Have you looked at my proposed HZ+S standard for the chinese.* hierarchy? Have you any suggestions for improving it? If you look at it, you will notice that it is based on Big5. Ed Lai thinks I should throw out Big5 and use CNS instead. Is that what you think? Why? I'm thinking of possibly adding CNS support to HZ+S. But I'm not willing to throw out Big5 support. -- Steve From simpson@math.psu.edu Wed May 11 07:07:25 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14846; Wed, 11 May 94 07:07:25 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id IAA26352; Wed, 11 May 1994 08:05:23 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 11 May 1994 08:05:22 -0400 Date: Wed, 11 May 1994 08:05:22 -0400 Message-Id: <199405111205.IAA27194@boole.math.psu.edu> To: ftang@roc.aux.apple.com (Frank Tang) Cc: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Big5 vs CNS, a summary In-Reply-To: <9405102224.AA08581@roc.aux.apple.com> References: <9405102224.AA08581@roc.aux.apple.com> Reply-To: simpson@math.psu.edu Status: O Frank Tang writes: >From my point of view, the Big5 and CNS > relationship is just the same as the replationship between > Shift-JIS and JIS in Japan. The CNS and JIS is the character set is > more complicated standard. The Big5 and Shift-JIS are the de factor > standard for its encodeing scheme. Unfortunatlly, Eten also add > several user area code in the Big5 . That is all. Thanks for your viewpoint on this. >From the purely technical aspect, the relationship between CNS levels 1 and 2 and Big5 is not as simple as between JIS and shift-JIS. Actually, the JIS and shift-JIS codesets are identical except for conventions about setting or unsetting the high bit. The relationship between CNS and Big5 is much more complicated than that. First, the mapping of the standard set of Chinese characters is quite complicated. Second and more seriously, there are many different versions of Big5 with additional characters and special symbols (not only Eten's). Therefore, Big5<->CNS conversion would seem difficult if not impossible to standardize completely. > We should use Big5 for Internal encoding for PC. But use CNS as a > external code for information exchange. Look at the X window 11 > Release 6. The CNS even is list there in the ICCCM 2.0 spec !!!! In addition to technical problems of exact interconvertibility, I see other problems involving popular acceptance and perceptions. I fear that these problems would make it very awkward to use CNS as an exclusive standard for the chinese.* newsgroup hierarchy and similar Internet situations. In my proposed standard now known as HZ+S, excellent Big5 support and downward compatibility with HZ are the paramount considerations. I am thinking of perhaps adding CNS support, but I am not willing to drop Big5 support. Because of the international nature of the chinese.* hierarchy, many chinese.* users will not have access to a Chinese operating system. Nevertheless, let me ask you as a point of factual information, do the Apple Chinese operating systems support CNS? Apple strongly supports Big5, of course. -- Steve From simpson@math.psu.edu Wed May 11 08:06:34 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA16487; Wed, 11 May 94 08:06:34 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id JAA26746; Wed, 11 May 1994 09:04:35 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 11 May 1994 09:04:33 -0400 Date: Wed, 11 May 1994 09:04:33 -0400 Message-Id: <199405111304.JAA27297@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: how about a vote on CNS ? Reply-To: simpson@math.psu.edu Status: O I was hoping that the soft-authors group would be able to quickly reach a consensus on an HZ+ standard, HZ+ being defined as HZ plus some form of 7-bit Big5. The discussion became bogged down on two issues: (1) What shall be the mapping from the Big5 code space to the graphical ASCII range, 0x2121-0x7E7E? (2) What about CNS? There are currently two HZ+ proposals on the table, my HZ+S and and Ya-Gui Wei's HZ-2. In addition, we are waiting for Jing-Shin Chang to convert his B5E1 and B5E2 mappings into HZ+ proposals. That will make four HZ+ proposals altogether. I anticipate that the four proposals will be in essence the same, except for issue (1). I am thinking that we perhaps ought to take a vote now on issue (2). I feel that we have enough information about CNS to come to a decision on that aspect of HZ+. All of the current HZ+ proposals currently support Big5 and do not support CNS. Thus we can phrase the vote as a choice from the following options: A. Add CNS support, in addition to the existing Big5 support. B. Drop Big5 support. Support CNS instead. C. Forget about CNS for the time being. D. Wait for more information on CNS. I myself am strongly opposed to option B. If we choose option A, what I have in mind for HZ+S is to use ~G and ~H as escape sequences for CNS levels 1 and 2. I choose G and H because they are the registered ISO-2022 final characters for CNS. Before we vote on issue (2), would anybody like to air any additional views or information? -- Steve From shann@wangwei.math.ncu.edu.tw Wed May 11 08:31:42 1994 Return-Path: Received: from wangwei.math.ncu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA17130; Wed, 11 May 94 08:31:42 CDT Received: from qingzhao.math.ncu.edu.tw by wangwei.math.ncu.edu.tw (4.1/SMI-4.1) id AA20782; Wed, 11 May 94 21:29:55 CST Date: Wed, 11 May 94 21:29:55 CST From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Message-Id: <9405111329.AA20782@wangwei.math.ncu.edu.tw> To: soft-authors@ifcss.org Subject: more info on CNS Status: O John Delacour wrote: > I'm not sure it is clear to anyone what is the nature of CNS.... > There are three "planes" (is that the right word?) 1,2 & E. E is for user > characters. CNS is supposed to be a standard information exchange code in my country. But I learned it two weeks ago on this net. Then I asked a student to visit the Bureau of Standard for any information. He brought back six 1.44MB disketts with 40x40 bitmap files, and a puvlication that list all the characters defined in the code. The current version of CNS is CNS 11643, dated May 21, 1992. But the one-page manual together with the bitmap files is dated March 9, 1994. Two earlier versions are called CNS 5205 and CNS 7654. This new version is said compatible with old ones. But I have no info about the old versions. The first announcement of the CNS stuff was on Aug 4, 1986. There are 16 "planes", so to speak. In this version planes 1--7 are defined, 12--16 are designated for user-defined characters. They didn't mention what to do with planes 8--11. The two 7-bit bytes are between 21--7E. Thus there are 94x94=8836 characters on each plane. Plane 1: (1) 684 symbols defined in the range of 2121--427E, including punctuation, Arabic, Chinese and Roman numerals, BoPoMoFo symbols, English and Greek alphabets, Chinese BuShou, etc. (2) 5401 frequently used characters are defined in 4421--7D4B. Plane 2: 7650 less-frequently used characters defined in 2121--7244. Plane 3: The first part of 6148 characters which was on Plane 14, version of 1988. In the range of 2121--6246. I found at least one character that is repeated in previous Planes. But this particular one is assigned in different BuShou than the previous one. Plane 4: The rarely used characters, especially those appear in names. Includes the Han characters in ISO D IS 10646 (I don't know what is this) and the second part of 171 characters which was on Plane 14. Totally 7298 characters, 2121--6E5C. Plane 5: Rarely used characters defined by Ministry of Education, said to be disjoint from the previous planes. 8603 characters, 2121--7C51. Plane 6: Yi4 Ti3 Zi4, characters in different ways of writing, meant the ancient writings instead of the simplified writings, defined by Ministry of Education, and disjoint from previous planes. Number of strokes <= 14. 6388 characters, 2121--647A. Plane 7: Same source as those in Plane 6, with number of stokes >= 15. 6539 characters, 2121--6655. There is a kind of profession, usually like a fortune teller or a medicine man, in designing names according to ones luck, birth time, wind-water, etc. That's why there are some very rarely used characters in names. They are included in Plane 4. I think you really have to be a Chinese scholar to know the characters on higher planes. I am not one. -Shann From CAI@neurophys.wisc.edu Wed May 11 10:03:48 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA02874; Wed, 11 May 94 10:03:48 CDT Received: from VMSmail by neurophys.wisc.edu; Wed, 11 May 94 09:58 CST Message-Id: <24051109584814@neurophys.wisc.edu> Date: Wed, 11 May 94 09:58 CST From: CAI@neurophys.wisc.edu Subject: CNS dilemma To: soft-authors@ifcss.org X-Vms-To: IN%"soft-authors@ifcss.org" Status: O Steve is calling for a vote on CNS, I don't know how to respond. I have the feeling that this round of discussion is going to be like the previous one if we don't take any actions now. For people favoring CNS, I think it's worth to repeat that even if CNS is adapted, those with CNS support could not read CNS message directly, as pointed out in the "last message" of (sorry, I don't have the name here). As I can see, CNS is equal to other proposals in the sense that all are used to "encode" 8-bit Big5 (Steve, sorry for my abuse of the word "encode"). So the question left is efficiency. Unless we can come up with a unique and authentic (not from guessing) CNS<->Big5 conversion, I would say that CNS is in a disadvantagous position as compared to other proposals, in the sense that it is not yet a complete proposal at all. For other proposals, I think they are close (no essential difference) and any one would work. cai From simpson@math.psu.edu Wed May 11 11:27:10 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA06589; Wed, 11 May 94 11:27:10 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA28183; Wed, 11 May 1994 12:24:52 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 11 May 1994 12:24:50 -0400 Date: Wed, 11 May 1994 12:24:50 -0400 Message-Id: <199405111624.MAA27691@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: CNS and 7-bit Big5 (long explanation) In-Reply-To: <24051109584814@neurophys.wisc.edu> References: <24051109584814@neurophys.wisc.edu> Reply-To: simpson@math.psu.edu Status: O CAI@neurophys.wisc.edu writes: > As I can see, CNS is equal to other proposals in the sense that all > are used to "encode" 8-bit Big5 (Steve, sorry for my abuse of the > word "encode"). That's OK, I have probably abused lots of words, too, especially on the topic of CNS. I have been very confused and misinformed about CNS I apologize to all of the CNS partisans. Let me now present my current understanding of all the issues connected to CNS. WHAT IS CNS? My present view of CNS levels 1 and 2 is that it is simply a different coding system. It is similar in concept but by no means the same as Big5. If you look at a character table, CNS levels 1 and 2 look very similar to Big5 blocks 1 and 2 (normally coded as 0xA140-0xC87E and 0xC940-0xFEFE respectively), with characters arranged by stroke count and radical. However, the order is different because of corrected stroke counts, also two characters are deleted, and also it differs from Big5 in the placement of some of the alphabets and special symbols and additional Chinese characters. A significant sociological difference between Big5 and CNS is that CNS is an official standard while Big5 is a popular standard. As for implementations of CNS, there are two versions, 7-bit CNS and 8-bit CNS. ETen's CNS mode and apparently Taiwan DEC user applications use the 8-bit form. Some people have said that 7-bit CNS is used as an exchange code, at least it used to be used on SeedNet, but now it has been replaced by Big5. It seems hard to get information on this. The Unicode tables refer to the 7-bit form of CNS and the 8-bit form of Big5. WHAT IS 7-BIT BIG5? Big5 also has two forms, a 7-bit form and an 8-bit form. The 8-bit form is extremely popular and widely used, for instance in ETen, cxterm, Chinese OS/2, Chinese MS-Windows, Mac, etc. 8-bit Big5 is much more more popular than any form of CNS. The 7-bit form of Big5 is used in at least one application, namely Mule. This is part of Mule's (admittedly somewhat nonstandard) implelementation of ISO-2022-JP-2. For our HZ+ standard, we need 7-bit Big5, and that's exactly what I did in my HZ+S proposal, in the simplest and most straightforward way. ADVANTAGES OF HZ+S OVER OTHER PROPOSALS > For other proposals, I think they are close (no essential difference) and > any one would work. I disagree. They would all work, but HZ+S is the best, because it implements 7-bit Big5 in the best possible way. Let me explain why. Some people have criticized some HZ+ proposals for "reinventing the wheel" or "inventing a new wheel". I'd like to point out that my HZ+S did not invent or reinvent any wheels. HZ+S's 7-bit Big5 is code-for-code identical to Mule's 7-bit Big5, which is part of Mule's (admittedly somewhat nonstandard) implementation of ISO-2022-JP-2. When I wrote my original HZ+ proposal, I had Mule firmly in mind. Furthermore, the Mule/HZ+S form of 7-bit Big5 is in an important sense parallel to 7-bit CNS. What I mean by parallel is that Big5 block 1 and CNS level 1 are both packed tightly into exactly the same 7-bit code space and are in fact very similar except for the CNS rearrangements for corrected stroke counts, etc. Likewise for Big5 block 2 and CNS level 2. These features of HZ+S's form of 7-bit Big5 (compatibility with Mule, parallelism with 7-bit CNS) are advantages of HZ+S which are not shared by any of the other HZ+ proposals. Some people have criticized HZ+S for being inefficient. To answer this, Carlos McEvilly and I have devised a fast HZ+S <-> 8-bit Big5 conversion using very small tables. With this algorithm, HZ+S is as efficient as the other proposals. CNS CANNOT BE HZ+ > Unless we can come up with a unique and authentic (not from > guessing) CNS<->Big5 conversion, I would say that CNS is in a > disadvantagous position as compared to other proposals, in the > sense that it is not yet a complete proposal at all. I agree. CNS by itself is in no way a complete HZ+ proposal. In order to make CNS into a complete HZ+ proposal, one of the things we would have to do is to define a complete, standardized, Big5<->CNS conversion map. This implies that we would have to standardize Big5, something that I think is far beyond our power to do. Industrial standards committees in Taiwan have tried to do this and failed. Big5 is a popular standard, but it will probably never be an official standard. ACTION SUMMARY CNS and Big5 are two different coding systems. Big5 is very popular. CNS is far less popular, but it is an official standard. Therefore, we may want to add CNS support to HZ+, in addition to the Big5 support that is already there. However, it would be a big mistake for HZ+ to get rid of Big5 and replace it with CNS. (Ed Lai's proposal to use CNS as an "exchange code" for Big5 would amount to this.) -- Steve From simpson@math.psu.edu Wed May 11 13:29:16 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA11649; Wed, 11 May 94 13:29:16 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA29092; Wed, 11 May 1994 14:26:53 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 11 May 1994 14:26:51 -0400 Date: Wed, 11 May 1994 14:26:51 -0400 Message-Id: <199405111826.OAA27897@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: Why convert to Big5? Newsgroups: alt.chinese.computing,comp.software.international In-Reply-To: <2qppnl$41s@ifcss.org> References: <9405102004.AA10571@apple.com> <199405102332.SAA25720@cs.utexas.edu> <2qpnj0$chi@gap.cco.caltech.edu> <2qppnl$41s@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yilong Chen writes: > There is the standard section of big5, but there's also the vender added > stuff, i think any b5-cns conversion will involve the standard section > only, and leave out the vender added section, so there won't be any > misinterpretation of characters, it's just that the vendor-added stuff > won't get transmitted. Correct me if i'm wrong. If this is right, it means that Ed Lai's proposal of using CNS as the exchange code for Big5 will result in the vendor-added stuff getting lost. This seems like a very bad idea. But then again, I'm not sure what Big5<->CNS conversion does to the vendor-added stuff. Certainly the answer to that question cannot be very simple. In my opinion, a minimal requirement for an exchange code for Big5 is guaranteed 100 percent accurate round-trip conversion over the entire Big5 code space, 0xA140-0xFEFE. Ed hasn't explained how his proposal will meet this criterion. I don't think it can do so. > btw, i think the gb-b5 conversion is more misinterpreted than any > characters that may involved in b5-cns conversion, and we've been coping > with reading inacurately-converted stuff in act.b5 and act x-mirrors, for > a while... what do people feel about that... if we do x-mirrors in > chinese.* with hz and big5, definitely, our audience on the mirror group > won't receive all the characters interpreted 100% correctly. Yes. This is one of the disadvantages of mirror groups. The other disadvantage is wastefulness of duplicated content. Nevertheless, the experience of a.c.t.big5 and the tw.*.hz mirrors show that mirroring may have at least a temporary role to play. >From a technical standpoint, inaccuracy in Big5 <-> HZ mirroring is something we have to accept, because of the imperfect correspondence between GB and Big5. However, inaccuracy in transmitting Big5 from one place to another is *not* something we have to accept. We can easily design our exchange standard to insure 100 percent accuracy in transmitting Big5. My HZ+S proposal and other HZ+ proposals insure this. -- Steve From mcevilly@netcom.com Wed May 11 18:07:48 1994 Return-Path: Received: from netcom.com (netcom13.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA23822; Wed, 11 May 94 18:07:48 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id QAA09659; Wed, 11 May 1994 16:05:43 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405112305.QAA09659@netcom.com> Subject: Re: 8-bit CNS To: simpson@math.psu.edu Date: Wed, 11 May 1994 16:05:42 -0700 (PDT) Cc: ccnet-l@uga.bitnet, soft-authors@ifcss.org In-Reply-To: <199405111806.OAA27864@boole.math.psu.edu> from "Stephen G Simpson" at May 11, 94 02:06:24 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 775 Status: O > Carlos, could you please look at the 8-bit CNS mode of ETen and see > whether it does things the same way? If this is right, it means the > routine to convert from 8-bit CNS to HZ+S is just to unset the high > bits. But could you make sure that the above is how the high bits > are, both for Level 1, only the first for level 2? For my test set of ten codepoints that represent the hanzi numbers 1 through 10, this is correct. The characters shown on the screen under ETen in CNS mode are indeed the ones for the corresponding characters in level two at the same codepoints, when only the first byte has the high bit set. We still have not learned how to enable any of the other planes using escape sequences. Perhaps ETen chose not to support the remaining planes. From lai@apple.com Wed May 11 19:45:16 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA27178; Wed, 11 May 94 19:45:16 CDT Received: from [17.203.80.11] by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA06071; Wed, 11 May 94 17:42:38 -0700 for soft-authors@ifcss.org Date: Wed, 11 May 94 17:42:38 -0700 Message-Id: <9405120042.AA06071@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu From: lai@apple.com (Edmund Lai) Subject: Re: CNS and 7-bit Big5 (long explanation) Status: O > >In order to make CNS into a complete HZ+ proposal, one of the things >we would have to do is to define a complete, standardized, Big5<->CNS >conversion map. This implies that we would have to standardize Big5, >something that I think is far beyond our power to do. Industrial >standards committees in Taiwan have tried to do this and failed. Big5 >is a popular standard, but it will probably never be an official >standard. > >In my opinion, a minimal requirement for an exchange code for Big5 is >guaranteed 100 percent accurate round-trip conversion over the entire >Big5 code space, 0xA140-0xFEFE. >However, inaccuracy in transmitting Big5 from >one place to another is *not* something we have to accept. We can >easily design our exchange standard to insure 100 percent accuracy in >transmitting Big5. My HZ+S proposal and other HZ+ proposals insure >this. We are told each Big5 vendor has its own vendor specific extension, and it cannot be standardized. So what happen when you use software from one vendor, and send document using HZ+S to another vendor? >practice? Suppose X is in Hong Kong and Y is in Canada and X wants to >send e-mail to Y. X types his message in Big5, then presses the >"send" key. X's Hong Kong software translates it into CNS and sends HZ+ >it. Y's Taiwanese software receives it and translates it back to >Big5, except Y doesn't know it's a Taiwanese translation, because all >this is supposed to be transparent to the user. Now X and Y get upset >because they find that some of the non-Chinese characters don't match >-- what they see on the screen isn't the same. Yes, that is what will happen with HZ+S, user of HZ+S will get very upset. We need to do something about it. Can we have a standard Big5 that every body can convert to and from? No, we are told that Big5 cannot be standardized, and furthermore any translation that requires content specific table look up like CNS is not acceptable to the authors of HZ+S. So I guess Steve has just declared that HZ+S is not acceptable by his own standard. So I guess Steve is telling us that we have no choice but to pick a broad based standard that can cover almost everything and so that everyone can translate to and from. That would be CNS, CCCII, Unicode, some extension of GB. Despite claims that I am a CNS advocate, anyone of the above is fine with me. So pick one now that HZ+S is out. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From simpson@math.psu.edu Wed May 11 22:32:12 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA03608; Wed, 11 May 94 22:32:12 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id XAA02146; Wed, 11 May 1994 23:29:58 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 11 May 1994 23:29:56 -0400 Date: Wed, 11 May 1994 23:29:56 -0400 Message-Id: <199405120329.XAA28879@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Ed's campaign against Big5 vendor codes In-Reply-To: <9405120042.AA06071@apple.com> References: <9405120042.AA06071@apple.com> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > We are told each Big5 vendor has its own vendor specific extension, and > it cannot be standardized. So what happen when you use software > from one vendor, and send document using HZ+S to another vendor? The answer to this question is, it's exactly the same as what happens when you send a Big5 document. If you use software from two different Big5 vendors, the vendor-specific characters that you see on your screen may be different. Big5 users are familiar with this fact of life. Big5 is a popular standard, not an official standard. > We need to do something about it. I disagree. It's not up to us to standardize Big5. It's not up to us to eliminate vendor-specific characters from Big5. The best thing we can do is provide a guaranteed 100 percent accurate exchange standard, so that all Big5 codes (even the vendor-specific ones) are transmitted accurately, byte for byte. This way at least, two ETen users will see the same ETen-specific characters on their screens. > Can we have a standard Big5 that every body can convert to and > from? No, we are told that Big5 cannot be standardized That's right. Vendor-specific characters -- even user-specific characters! -- are a fact of life in the Big5 world. If you don't like that fact, use GB or CNS or Unicode instead. Yet despite your outrage, millions of Big5 users will continue to use Big5. > and furthermore any translation that requires content specific > table look up like CNS is not acceptable to the authors of HZ+S. The problem about using CNS in this way is that Big5 -> CNS -> Big5 is not completely accurate. ------------------- --- ---------- -------- On the other hand, Big5 -> HZ+S -> Big5 is 100 percent accurate, byte for byte. -------------------- -- --- ------- -------- ---- --- ---- Not only that, but ETen -> HZ+S -> ETen is 100 percent accurate character for character, even the ETen-specific characters. That's one of the things I like about HZ+S. > So I guess Steve has just declared that HZ+S is not acceptable by his > own standard. So I guess Steve is telling us that we have no choice but > to pick a broad based standard that can cover almost everything and so > that everyone can translate to and from. That would be CNS, CCCII, Ed, you are pretending not to understand me. Why? The matter is really very simple. Big5 and GB are far and away the world's most popular Chinese character coding schemes. The Internet needs a 7-bit standard for mixed Big5, GB, and ASCII. HZ+S can be that standard. I'm willing to incorporate CNS support into HZ+S, but I'm not willing to throw out excellent Big5 support. Listen, I freely admit that Big5 is far from perfect. I might even go so far as to say that Big5 is covered with warts. Nevertheless, warts and all, Big5 is extremely popular. We must provide excellent support for Big5, including the warts. That's all there is to it. -- Steve From mcevilly@netcom.com Wed May 11 23:05:15 1994 Return-Path: Received: from netcom.com (netcom11.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA04755; Wed, 11 May 94 23:05:15 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id VAA13584; Wed, 11 May 1994 21:03:06 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405120403.VAA13584@netcom.com> Subject: Re: CNS and 7-bit Big5 (long explanation) To: ccnet-l@uga.bitnet Date: Wed, 11 May 1994 21:03:05 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <199405120137.SAA18871@mail.netcom.com> from "Edmund Lai" at May 11, 94 05:42:38 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 2233 Status: O Ahh, the water is getting muddy again. Let me try to explain a few simple points. Big5 is a fact of life in Taiwan. Most users use it. We can't change that. It is a known fact. All possible coding schemes, present and future, have the same problem with the *Big5* user-defined characters, and some vendor defined characters [the same problem with user defined characters is also present with CNS, even moreso since the CNS user-defined space is so huge--but let's talk about Big5]. Incompatible user-defined characters is a problem in document interchange in Taiwan. Most users don't run into the problem however, except in rare instances (as the one pointed out where a character in the Chinese name for Acer is not defined in the base set; users who want to use that character would have to define it themselves). Since different people would define it at different places, there is a mapping problem in such cases. This mapping problem is from a Big5 codepoint (whether it be vendor-specific or user-defined) to the user's input, output, or storage device. The last point above is crucial. It is a fact of life even with the CNS, GB, CCCII, and Unicode schemes, not something to blame HZ+S for. Once we convert a character from one scehme to another, we will have to deal with all the problems of the target scheme. In this case the target scheme is Big5; it would not matter which scheme we were coming from, the problem would still exist. However if we use CNS we will add a *new* problem. This new problem is different--it *is* in the 7-bit to Big5 mapping. This problem is twofold: 1) relative complexity and 2) instability. There is good reason to believe that CNS (even in the lower two planes) is unstable and will change. Even worse, we even have disagreement about which version of CNS is the real version. The good news about this *new* problem is that we can eliminate it by using HZ+. We still have a problem to solve. The problem is Big5 to 7-bit mapping for the chinese.* newsgroup hierarchy. Fortunately, there is a solution for this one: HZ+. The only scheme that guarantees a consistent mapping from Big5 to 7-bit is HZ+ (HZ+S being, in my opinion, the best of the HZ+ proposals). From roland_wang@powertalk.apple.com Thu May 12 00:25:07 1994 Return-Path: Received: from guardian.apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA07568; Thu, 12 May 94 00:25:07 CDT Received: from federal-excess.apple.com by guardian.apple.com with SMTP (5.65/8-Oct-1993-eef) id AA24078; Wed, 11 May 94 18:43:18 -0700 Received: from powertalk.apple.com by federal-excess.apple.com with SMTP (5.64/26-Sep-1993-eef) id AA27655; Wed, 11 May 94 18:42:14 PDT for soft-authors@ifcss.org Message-Id: <9405120142.AA27655@federal-excess.apple.com> Date: 11 May 1994 18:42:52 -0700 From: "Roland Wang" Subject: Re: why should we use CNS as an exchange code for Big5 ? To: "simpson@math.psu.edu" Cc: "ccnet-l@uga.cc.uga.edu" , "soft-authors@ifcss.org" Status: O Steve writes: > If you think I'm confused, please enlighten me. Please explain the > terms "exchange code" and "internal code" as they will apply to the > chinese.* hierarchy, and as they now apply to alt.chinese.text. Let's take a look at an example in the simplified Chinese support in Sun's Solaris. GB 2312-80 (character set) defines a pair of hex values 52 (101 0010)/ 3b (011 1011) as the Chinese character "Yi" (one). Solaris uses 0xd2bb (11010010 10111011) to represent (or say, encode) Chinese character "Yi" defined in GB 2312-80. 0xd2bb is the internal encoding value. Another system may use another value to represent "Yi", for example, 0xd23b which resembles Big5 encoding scheme. When you send an Chinese email (which only has "Yi" as the contend) on Solaris, "Yi" will be encoded as "ESC $ ) A 0f 52 3b 0e" - this is exchange encoding conforming to ISO 2022. > After that, please tell me why Ed Lai wants chinese.* to use CNS as an > exchange code for Big5, even though CNS<->Big5 translation may not be > 100 percent well defined. There is no perfect solution for vender-defined Big5 chars across platforms unless all platforms have all possible mappings. Vender X defines xx as "ABC" while vender Y defines xx as "XYZ". Only those systems from the same vender can understand their own defined chars. No matter the exchange code is CNS or Big5. So, this is not the ground to reject using CNS. > Have you looked at my proposed HZ+S standard for the chinese.* > hierarchy? Have you any suggestions for improving it? If you look at > it, you will notice that it is based on Big5. Ed Lai thinks I should > throw out Big5 and use CNS instead. Is that what you think? Why? Yes, I have followed the discussion from the very beginning. After we have clarified mis-perception about CNS (compare to Big5), the discussion (real issue) should be the trade-off between the popularity of Big5 and the technical superiority (both linguistics and computers) of CNS. Due to my background and experience, my opinion may have strong bias to commercial systems. Nevertheless, if you ask for my opinion, I'd say to use 7bit ISO 2022 compliant CNS. Pros: CNS is a standard (getting popular), and technically superior, contains more characters, and ISO 2022 is a standard, too. Cons: (so, pros for using Big5) Add complexity for conversion from/to Big5 (which is so popular) Just for your info - CNS support is the standard listed in alomst every X/UNIX systems (X11R6 ICCCM, COSE/CDE, and so on). > I'm thinking of possibly adding CNS support to HZ+S. But I'm not > willing to throw out Big5 support. Using Big5 in HZ+ is not a bad idea (considering its popularity). But, please don't make things more complicated by adding CNS support as another encoding. If there is a strong requirement for supporting CNS, dropping Big5 is the best solution. _Roland Wang /* Disclaimer - All statements and opinions above are my own. */ From simpson@math.psu.edu Thu May 12 00:56:26 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08423; Thu, 12 May 94 00:56:26 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id BAA02825; Thu, 12 May 1994 01:53:05 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 12 May 1994 01:53:03 -0400 Date: Thu, 12 May 1994 01:53:03 -0400 Message-Id: <199405120553.BAA29147@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Why can't CNS and Big5 coexist? Newsgroups: alt.chinese.computing In-Reply-To: <2qrpcg$ch8@netnews.ntu.edu.tw> References: <199405111642.LAA14917@cs.utexas.edu> <2qrpcg$ch8@netnews.ntu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Hao-yang Wang writes: > The proper, formal way does exist. (Just feed a text file > consisting of all the Big-5 characters into the conversion utility > provided by ETen, and you will get a "proper" table of CNS.) It is > by no means a guessingwork. Does this method of defining Big5->CNS conversion (via some utility in ETen) give exactly the same results as chcode? Does it give exactly the same results as Ross's conversion based on the Unicode table? If you take a table of the entire Big5 code space, translate it into CNS by the ETen utility, then translate it back to Big5 by chcode, can you guarantee that the table you get will be identical? I don't think you can guarantee this. > 3. You can argue that, although a proper way of conversion exists, > it is not widely known. This is very true. What is the proper way of conversion? Does it guarantee 100 percent accuracy for Big5 -> CNS -> Big5 over the entire Big5 code space? If it doesn't, why do you think it's better than methods like HZ+S, which are much simpler and guarantee 100 percent accuracy? > 4. Either Big-5 or CNS, but not both. I strongly disagree with the > inclusive approach. Why do CNS partisans want to drop a hydrogen bomb on Big5? What's wrong with letting HZ+ support both Big5 and CNS, side by side? > Will all implementation be required to support both Big-5 and CNS? No. > Otherwise what will (and should) happen when the program encounters > unrecognised encodings? That's up to the program. If I were a program and I encountered unrecognized codings, I'd probably ignore them and look for recognized codings. I have written a sample implementation of HZ+S. It consists of the filters: big2hzp, hzp2big, gb2hzp, hzp2gb. When hzp2big encounters 7-bit GB codes, it passes them through unconverted. It converts only 7-bit Big5 codes. This approach seems to work rather well. If I add CNS support, I plan to follow the same approach. -- Steve From simpson@math.psu.edu Thu May 12 09:47:30 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA22103; Thu, 12 May 94 09:47:30 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id KAA08511; Thu, 12 May 1994 10:45:12 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 12 May 1994 10:45:10 -0400 Date: Thu, 12 May 1994 10:45:10 -0400 Message-Id: <199405121445.KAA03858@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org In-Reply-To: dank@alumni.caltech.edu's message of 12 May 1994 04:27:48 GMT Subject: What's with Apple and CNS ? Reply-To: simpson@math.psu.edu Status: O I wrote: let me ask you as a point of factual information, do the Apple Chinese operating systems support CNS? Apple strongly supports Big5, of course. Dan Kegel replied: > I don't know- but I do know that Apple strongly supports Unicode, > and is one of the major vendors rolling out an implementation > (MacOS 7.5 with QuickdrawGX). > > If your criterion for including a character set in your encoding is > that the character set be supported by Apple, will you now consider > adding Unicode support to your proposed HZ+S encoding? > > - Dan 'stir 'em up' Kegel Apple support is not my criterion. My principal criterion is popularity. HZ+S supports GB and Big5, because GB and Big5 are by far the most popular Chinese coding systems. -- Steve "I'm already stirred up enough!" Simpson PS. The point of my question was that a lot of Apple guys seem to be enthusiastic about CNS, so I was wondering whether any of them actually use CNS, or know anybody who uses CNS. One of the Apple guys is Ed Lai, who develops Chinese software but never even heard of CNS until a couple of weeks ago. Here on alt.chinese.computing, it took us a long time to smoke out a lone CNS enthusiast in Taiwan, Hao-yang Wang. But then it turned out that Hao-yang Wang doesn't actually use CNS himself. He uses -- surprise, surprise -- Big5. I still don't know one CNS user. Interestingly, Hao-yang Wang also seems to have an Apple connection. His e-mail is hywang@apple.com. -- Steve From simpson@math.psu.edu Thu May 12 15:36:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA06281; Thu, 12 May 94 15:36:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA11367; Thu, 12 May 1994 16:34:26 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 12 May 1994 16:34:24 -0400 Date: Thu, 12 May 1994 16:34:24 -0400 Message-Id: <199405122034.QAA04498@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Can we finally agree on HZ+S? Reply-To: simpson@math.psu.edu Status: O I'll be traveling for a few days. Probably the next time I log in will be Monday, May 16. I'm sure everybody will appreciate a temporary rest from my flood of postings about HZ+, 7-bit Big5, CNS, etc. :-) Could I ask everybody in the soft-authors group to please give some thought to HZ+? For the sake of the chinese.* newsgroup hierarchy, we need to reach a consensus on HZ+. There are more than enough proposals for HZ+. My HZ+S proposal is ready to go -- all we would have to do is change the name to HZ+. An alternative proposal is Ya-Gui's HZ-2, and I'll let Ya-Gui speak for himself on that. We are also waiting for some additional HZ+ proposals from Jing-Shin Chang, based on his B5E1 and B5E2 mapping schemes. The HZ+ discussion got bogged down on two issues which I personally think are red herrings. The two red herrings are: (1) the relative efficiency of various 7-bit Big5 mapping schemes, and (2) CNS. I believe these red herrings have now been aired adequately. I believe we know enough now to turn our gaze away from these red herrings, and standardize on HZ+S. If you agree, please speak up. If you disagree, that's OK, you can still speak up. But let's try to somehow reach a consensus. The world needs HZ+. A question for Chinese software developers: Is there any way you could possibly support HZ+ in your programs? If so, please speak up. -- Steve From mcevilly@netcom.com Fri May 13 01:36:17 1994 Return-Path: Received: from netcom.com (netcom11.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA29851; Fri, 13 May 94 01:36:17 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id XAA10043; Thu, 12 May 1994 23:34:06 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405130634.XAA10043@netcom.com> Subject: HZ+S: an elegant boomerang for 2-way conversion To: ccnet-l@uga.bitnet Date: Thu, 12 May 1994 23:34:05 -0700 (PDT) Cc: soft-authors@ifcss.org In-Reply-To: <199405130416.VAA02297@netcom.com> from "Edmund Lai" at May 12, 94 05:53:59 pm X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 3609 Status: O > Let us review how all these started. Steve is using the vendor code > as an excuse to try to bad mouth CNS. All the example about how one > user sending some vendor specific vendor code and another user got > garbage came from himself. I was just quoting him. He blame the problem > on CNS. So I quote the same passage except I show that if you > substitue CNS by HZ+S, you got exactly the same problem. And now he > is not too happy. And it turns out the problem has a better chance > to be solved by CNS. You are confusing two different problems. One is Big5's problem, and no CNS or GB or HZ+ or Unicode or anything is going to solve that one. The other problem is a conversion problem between Big5 and other schemes, including CNS. This latter problem does not exist with HZ+. The first problem exists with all conversions to and from Big5. I can understand how that point could be confusing, and appreciate this chance to clear it up. > HZ+S would work for vendor code if the sender end and receive end > uses the same vendor software. But by his own previous admittance > each vendor has his own code and it is impossible to standardised. > And if the two end don't use the same code it would be a problem. > Only that previously he claim that this is a problem for CNS so > it is a very big deal. But as soon as I point out that the problem > actually is as bad in HZ+S, then it is no big deal, it is a > "fact of life". Again, the fact of life is also a fact of life for any other scheme as soon as it maps to Big5. So the only way to solve the problem you are mentioning, is to get rid of Big5 (as Roland suggested in an earlier posting). > And now he is back to the old discredited agrument "Big5 is more > popular than CNS, we need to support Big5". As we now all know, > using CNS as exchange code would still support normal 8 bit > Big5 as internal code. The question is which is more popular, > more of a standard, 7 bit CNS or 7 bit Mule Big5. Some of us think this is not the question. What is important in my opinion is what we can do to maximise the chances of users happily and conveniently adopting the new 7-bit coding that will be used on chinese.*. The scheme we adopt should be easy to move to from the most popular coding scheme in Taiwan. The most popular scheme is Big5. If you think this is the same question, read again. 7-bit CNS and 7-bit Mule Big5 are not the coding schemes that users use. But ONE of them is very easily convertible from Big5, which is what everyone uses in Taiwan. The popularity of CNS is so low that it needs other benefits to make it worthy of consideration. So far, those benefits have not appeared. The only good argument for CNS is that it is a standard. But anyone in Taiwan will tell you that the future standard is ISO 10646 / Unicode, not CNS. And it is still early for that, unfortunately. Besides, CNS is subject to change--in fact with so few users it can afford to change more than Big5 can, and it has the user-defined codespace to ensure that it will. If the entire substance of your arguments rests on your search/ replace function, this would explain why we haven't heard any convincing arguments against HZ+S. The arguments simply do not map both ways. > /* Edmund K. Lai */ > /* Apple Computer, MS303-3A */ > /* 20525 Mariani Ave, */ > /* Cupertino, CA 95014 */ > /* (408)974-6272 */ > zW@h9cOi > From simpson@math.psu.edu Mon May 16 11:40:22 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA05764; Mon, 16 May 94 11:40:22 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA09916; Mon, 16 May 1994 12:38:21 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 12:38:19 -0400 Date: Mon, 16 May 1994 12:38:19 -0400 Message-Id: <199405161638.MAA15401@boole.math.psu.edu> To: Wei Wang Cc: soft-authors@ifcss.org Subject: mixing GB and Big5 In-Reply-To: <199405130456.AAA15728@leibniz.math.psu.edu> References: <199405122241.SAA09903@cps201.cps.cmich.edu> <199405130456.AAA15728@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Wei Wang writes: > I guess for news readers, or mail readers, the problem of going back > one page with text in HZ+S might not so obvious. Also this problem does > not exist in HZ, since ~} sequence is used only to switch from GB to > ASCII. But with BIG5 mixed in, ~} can be used to switch from either GB > mode or BIG5 mode to ASCII mode. In this case, when the text is scrolled > back from ASCII section to HZ+ section, the problem emerges. Somebody > please shed some light on it if there is a simple way to deal with this. > Or should this be considered in the standard-to-be? Thanks for raising this issue. The HZ+S standard itself need not deal with this issue, but implementation considerations could be important later. I'd like to hear the views of the authors of hztty and ZWDOS. In practice this problem may not be very serious, because not many documents will mix GB and Big5 together. For example, when people post to the chinese.* hierarchy they will probably want to post in GB or in Big5 but not both together. In designing HZ+S I took cues from ISO-2022-JP-2 as implemented in Mule, the multilingual Emacs. Mule uses completely different escape sequences but it uses them in the same way, that is an escape switches into a certain mode, be it Big5 mode or GB mode or ASCII mode. So perhaps this problem also arises in Mule. -- Steve From simpson@math.psu.edu Mon May 16 12:00:58 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA06733; Mon, 16 May 94 12:00:58 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA10019; Mon, 16 May 1994 12:58:54 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 12:58:52 -0400 Date: Mon, 16 May 1994 12:58:52 -0400 Message-Id: <199405161658.MAA15425@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: CNS and Big5 In-Reply-To: <199405131508.LAA18140@leibniz.math.psu.edu> References: <199405131508.LAA18140@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Roland Wang writes: > For the sake of argument, I'm a CNS user on Sun's Sparc, and I know many > people who use CNS on a wide variety of systems (IBM, DEC, SUN, HP, you > name it). And, at the same time, I'm a Big5 user on Mac, too. OK, fine. So there are some CNS users out there. I'm glad to hear it, and I'm not bad-mouthing CNS. In designing HZ+S, I chose to base it on Big5, because Big5 is very popular. Are you saying that this was a bad decision? If so, why? If you are only saying that HZ+S should *not* support *both* Big5 and CNS, I can understand your reasoning. But if you are saying that HZ+S should not support Big5, I must disagree. -- Steve From simpson@math.psu.edu Mon May 16 12:07:38 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA07142; Mon, 16 May 94 12:07:38 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA10072; Mon, 16 May 1994 13:05:35 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 13:05:32 -0400 Date: Mon, 16 May 1994 13:05:32 -0400 Message-Id: <199405161705.NAA15436@boole.math.psu.edu> To: HU Bo Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Code shema of various code length? No thanks. In-Reply-To: <199405132118.RAA21107@leibniz.math.psu.edu> References: <199405132118.RAA21107@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O HU Bo writes: > I personally, would not like to implement software for a system using code > of various length. HZ+S and other HZ+ proposals do not use variable length codes. In HZ+S, all of the GB and Big5 codes and all of the escape sequences are 2 bytes in length. -- Steve From rap@doc.ic.ac.uk Mon May 16 12:11:59 1994 Return-Path: Received: from santos.doc.ic.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA07407; Mon, 16 May 94 12:11:59 CDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0q36Ad-00070iC; Mon, 16 May 94 18:09 BST Message-Id: Date: Mon, 16 May 94 18:09 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Big5/CNS conversion Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Status: O I've written a pair of programs that act as Unix filters converting back and forth between Big-5 and a 7-bit ISO 2022 encoding (using CNS). They're in ftp://ifcss.org/software/unix/convert/b5cns.tar.gz The Big-5/CNS functions themselves are automatically generated from mapping data. -- Ross Paterson Department of Computing, Imperial College, London SW7 From butta1@crsa.bu.edu Mon May 16 12:25:16 1994 Return-Path: Received: from crsa.bu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA08154; Mon, 16 May 94 12:25:16 CDT Received: from ithake.bu.edu by crsa.bu.edu (8.6.4/Spike-2.1) id NAA27926; Mon, 16 May 1994 13:23:03 -0400 Received: from localhost by ithake.bu.edu (8.6.4/Spike-2.1) id NAA15453; Mon, 16 May 1994 13:22:40 -0400 Date: Mon, 16 May 1994 13:22:38 -0400 (EDT) From: C I I Project Subject: Re: mixing GB and Big5 To: Stephen G Simpson Cc: Wei Wang , soft-authors@ifcss.org In-Reply-To: <199405161638.MAA15401@boole.math.psu.edu> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O On Mon, 16 May 1994, Stephen G Simpson wrote: > > In practice this problem may not be very serious, because not many > documents will mix GB and Big5 together. For example, when people > post to the chinese.* hierarchy they will probably want to post in GB > or in Big5 but not both together. > what if following up or replying to a post that's opposite of the code that you use, then it'll be mixed, ie if the included article is BIG5, and you reply with GB, then it'll be mixed GB/BIG5 text. nelson From simpson@math.psu.edu Mon May 16 13:57:33 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA13664; Mon, 16 May 94 13:57:33 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA11044; Mon, 16 May 1994 14:55:22 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 14:55:21 -0400 Date: Mon, 16 May 1994 14:55:21 -0400 Message-Id: <199405161855.OAA15570@boole.math.psu.edu> To: Frank Tang Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Don't create another ENCODING please !!!!! In-Reply-To: <199405140612.CAA25230@leibniz.math.psu.edu> References: <199405140612.CAA25230@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Frank Tang writes: > "DON'T CREATE ANOTHER ENCODING SCHEME FOR CHINESE CHARACTER, PLEASE" The 7-bit Big5 scheme used in HZ+S is not new. It is the same as the 7-bit Big5 used in Mule. The author of Mule is a member of the ISO-2022-JP-2 committee. > Since Big5 follow the CNS standard , now the major code set in PC > is Big5. If CNS is a bad standars, why Big5 follow it. ( Don't > treat CNS as another thing. CNS is the MOTHER of Big5 ) Could you please explain this remark? When we look at the Big5 and CNS character tables side-by-side, they appear very different. The 13051 characters appear in a very different order, and there are other significant differences. Therefore, when you say that Big5 follows the CNS standard, I think you need to explain yourself. If you don't explain yourself, I conclude that you are only trying to spread confusion. My own view of this is that, whatever the history may have been, CNS and Big5 are now two different Chinese character coding schemes. We may wish that they were the same, but unfortunately, they are in fact different. > BTW, the CSN to Big5 mapping is not so complicate like what Lai > said. There are a better conversion method for it. ( FIND it ) Instead of making an assertion with no proof, could you please back this up? Where is the mapping that is simpler than what Lai has proposed? The fact that there is so much confusion about Big5 <--> CNS conversion is one reason why I don't want HZ+S to drop Big5 support. Big5 is a popular Chinese character coding scheme, so HZ+S must support it. CNS is not so popular. When I suggested that HZ+S might support CNS in addition to Big5, nobody seemed to like that suggestion. What do you think of that suggestion? -- Steve From simpson@math.psu.edu Mon May 16 14:08:18 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA14247; Mon, 16 May 94 14:08:18 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA11122; Mon, 16 May 1994 15:06:18 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 15:06:16 -0400 Date: Mon, 16 May 1994 15:06:16 -0400 Message-Id: <199405161906.PAA15584@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: chinese.* needs HZ+ now!!! In-Reply-To: <9405152112.AA14469@econ.Berkeley.EDU> References: <9405152112.AA14469@econ.Berkeley.EDU> Reply-To: simpson@math.psu.edu Status: O Pinghua Young writes: > 4. As Tuya put it a few weeks ago, the core issue is the encoding > method to be used in chinese.*. Personally I am newtral between > a new encoding method and cross-mirroring. But a new encoding > method (or interchange method) for communication of GB and Big5 > has to be devised given the popularity of Internet and the increasing > need of email communication among all the Chinese people around the > world. So I feel the effort put in now to incorporate Big5 into > HZ will be fruitful. Things will not be perfect at the beginning, > but if we take precaution and have a little bit foresight, things > can be improved. In this regard, I favor a new 7-bit encoding method > that can handle both GB and Big5. > > Let's work together and create chinese.* on 6.4. I want to retire :-) > And I mean it. I forward this comment to soft-authors. The world needs HZ+ now!!! -- Steve From simpson@math.psu.edu Mon May 16 14:32:30 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15408; Mon, 16 May 94 14:32:30 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA11259; Mon, 16 May 1994 15:30:13 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 15:30:11 -0400 Date: Mon, 16 May 1994 15:30:11 -0400 Message-Id: <199405161930.PAA15619@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: Wei Wang Subject: Re: mixing GB and Big5 In-Reply-To: References: <199405161638.MAA15401@boole.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O C I I Project writes: > what if following up or replying to a post that's opposite of the code > that you use, then it'll be mixed, ie if the included article is BIG5, > and you reply with GB, then it'll be mixed GB/BIG5 text. Yes, Nelson is quite correct. Replies may give rise to mixed GB/Big5 postings. I temporarily overlooked this point. Thanks for reminding me. As Ed Lai pointed out, there is a convention in HZ to end every line with ~}. This means that when you scroll back through a ~} you will not need to go further than one line to find out whether the preceding material is in GB or Big5. This should make it easier for terminal programs to deal with mixed GB/Big5. Perhaps HZ+ should insist on having ~} at the end of each line. This is an optional convention in HZ, but maybe we should make it mandatory in HZ+. Comments? -- Steve From wang@cps201.cps.cmich.edu Mon May 16 15:38:50 1994 Return-Path: Received: from cps201.cps.cmich.edu by ifcss.org (4.1/IFCSS-Mailer) id AA18129; Mon, 16 May 94 15:38:50 CDT Received: from gremlin (gremlin.cps.cmich.edu [141.209.20.44]) by cps201.cps.cmich.edu (8.6.8.1/8.6.6) with SMTP id QAA06474; Mon, 16 May 1994 16:36:32 -0400 Date: Mon, 16 May 1994 16:35:11 -0400 (EDT) From: Wei Wang Subject: Re: mixing GB and Big5 To: Stephen G Simpson Cc: soft-authors@ifcss.org In-Reply-To: <199405161930.PAA15619@boole.math.psu.edu> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O On Mon, 16 May 1994, Stephen G Simpson wrote: > As Ed Lai pointed out, there is a convention in HZ to end every line > with ~}. This means that when you scroll back through a ~} you will > not need to go further than one line to find out whether the preceding > material is in GB or Big5. This should make it easier for terminal > programs to deal with mixed GB/Big5. I remember to have notice this when I read a couple of HZ files. Thanks to Ed Lai for the enlightening. > Perhaps HZ+ should insist on having ~} at the end of each line. This > is an optional convention in HZ, but maybe we should make it mandatory > in HZ+. Comments? I think this is a good idea. And the scheme is suitable to the purpose of HZ/HZ+. Wang Wei [MS CS, Central Michigan University [1909B Stockman, Mt.Pleasant, MI 48858 [517.773.4934 (Home) 517.774.3987 (Office) From lee@simd.stanford.edu Mon May 16 16:29:46 1994 Return-Path: Received: from simd.stanford.edu by ifcss.org (4.1/IFCSS-Mailer) id AA20320; Mon, 16 May 94 16:29:46 CDT Received: by simd.stanford.edu (5.65/inc-1.0) id AA15904; Mon, 16 May 1994 14:27:45 -0700 Date: Mon, 16 May 1994 14:27:45 -0700 From: lee@simd.stanford.edu (Fung Fung Lee) Message-Id: <9405162127.AA15904@simd.stanford.edu> To: soft-authors@ifcss.org Subject: HZplus: 7-bit BIG5/GB standard Status: O Subject: HZplus: 7-bit BIG5/GB standard Since the recent discussion on 7-bit BIG5 standards started a few weeks ago, several persons have implicitly/explicitly in private/public asked me to express on my opinion on the subject, as I was the one who designed and wrote the HZ specification back in 1989. Being very busy at work and family matters, and being "fearful" of the endless "demand" in time to response, explain, clarify, defend again and again on the same things (as apparent in the current round of discussions on 7-bit BIG5, the previous round on the same subject with Dr. Ricky Yeung and Yagui Wei, and the rounds of discussions on 7-bit GB, i.e, the "big" HZ vs. zW debate in 1989 with me on one side, and Ed Lai and Yagui Wei on the other side), I think the only sensible way I can afford to do now is to express my viewpoint on this subject as clearly as possible in this single message, and declare that this is for reference only, so don't expect from me any prompt or detailed response. This may or may not work. Fortunately, I've found that I am almost ready to endorse a current proposal if a "minor revision" is performed. Therefore, I wishfully expect that the proposal originator who has already done most of the work will continue to guard my revision as if it were his. :-) Following are 5 questions I try to answer. As I am only on the soft-authors list and intend to remain the status quo, soft-authors@ifcss.org is the only place this message is posted. Q1. Do the Internet community need a 7-bit BIG5 standard? A1. Yes, as long as the (international) Internet is not completely 8-bit clean. However, this does NOT imply that people who have access to 8-bit clean environment need to switch from 8-bit BIG5 to 7-bit BIG5. Proper newsgroup mirroring similar to what is already happening on alt.chinese.text and alt.chinese.text.big5 would work. Q2. Should this 7-bit BIG5 standard be compatible with HZ, the de facto 7-bit GB standard, so that both GB codes and BIG5 codes can co-exist in the same file? A2. Yes, because it is desirable in the spirit of ISO-2022, and is also feasible by defining a simple extension of HZ. Q3. Let's call this extended standard HZplus, or HZ+ for short (with .hzp for file type extension?) for now. What are the desirable attributes of HZplus? A3. Desirable attributes of HZplus are a. Contain HZ as a proper subset: Pretty obvious requirement. b. Simple, fast and easily described 100% invertible conversion algorithm exists between 8-bit BIG5 and 7-bit BIG5 in HZplus c. Following HZ, choose a 7-bit encoding that preserves the relative ordering: Enable the use of many existing 7-bit utilities for the purpose of sorting, searching, etc. d. Following HZ, use two-byte escape codes, with '~' (0x7E) as the first byte: Other choices will unnecessarily complicate things. e. Following HZ, choose the second byte of an escape code such that it is NOT the first byte of any properly encoded BIG5 character: This is a feature I was aware of when I designed HZ back in 1989, but did not explicitly mention in the HZ specification document. For example, this consideration alone favors ~{, ~} over ~(, ~), since '(' or ')' may be the first byte of a valid 2-byte 7-bit GB code. This property allows simple regular string search to locate the beginning of any 2-byte escape code in HZplus (when the number of preceding ~'s is 0 or even). f. Following HZ, choose the 7-bit encoding such that it is identical to the corresponding ISO-2022 encoding, except for the choice of escape codes: Allows straightforward ASCII string substitution for conversion. Q4. Which HZplus candidiate proposal so far is the best? A4. Here are my score cards: (a) (b) (c) (d) (e) (f) Steven Simpson's HZ+S: yes mid yes yes no yes Yagui Wei's HZ-2: yes hi yes yes no no Jing-Shin Chang's B5E[12]: yes low no yes no no any scheme via CNS: ? no no ? ? yes Clearly, HZ+S and HZ-2 are better. Had BIG5's second byte code ranges been 0x21-0x7E, 0xA1-0xFE instead of 0x40-0x7E, 0xA1-0xFE, HZ+S' "in-order tight packing" scheme and HZ-2's "bit-shifting" scheme would be identical. Since there are 7652 BIG5 less-frequently used (0xC940-0xF9D5) hanzi, HZ+S requires 82 (=7652/94) code points in byte 1, while HZ-2 requires 98 (=(0xF9-0xC9+1)*2) code points in byte 1, which is larger than the maximum limit of 93, as there are only 94 ASCII graphics characters, and the last one of them is '~'. That was the main reason why HZ-2 had to break the BIG5 code range at a code point larger than 0xC940 (the less-frequently-used hanzi starting point). In view of the fact that HZ+S' tight in-order packing conversion is conceptually straightforward, and implementation speed is good enough, I think HZ+S scores higher than HZ-2. Unfortunately, no proposal scores well on (e). Actually, there are only a few choices that can satisfy (e). Q5. Can we propose something better? A5. Yes. First, let's look at the structure of BIG5 codes. (see ifcss.org:software/fonts/HBF/etm24.hbf for example) There are a total of 13973 code points. block 1 (0xA140-0xC8FE) 6280 (=40x157) code points: frequently-used hanzi + standard symbols + some vendor specific codes + some undefined code points block 2 (0xC940-0xF9FE) 7693 (=49x157) code points: less-frequently used hanzi + some vendor specific codes Let's look at the 8-bit to 7-bit code conversion process. For HZ+S, block 1 maps to 0x21-0x63 of the first 7-bit byte, while block 2 maps to 0x21-0x72 of the first 7-bit byte. According to (e), this leaves the candidate for the second byte of the escape code to be 0x73-0x7D ("stuvwxyz{|}"). But '{' and '}' have already been chosen for HZ, so we should choose any two characters from "stuvwxyz|". I'd like to avoid '|' as it may confuse some programs as a pipe operator, though not necessarily always all the time. You may argue about it. But let's choose something as "safe" as possible. I don't have a strong preference yet. But 'y' for block 1, and 'z' for block 2 seem "natural" to me, if you try to "allocate" them from the end of the list like a stack. (If the Japanese / Korean are interested, some of "stuvwx" may be used too.) For HZ-2, the "original" code space is an implicitly expanded version of BIG5: 0xA121-0xF9FE, which contains 89 * (94 + 94) = 16732 code points. To maximally "equalize" the code point distribution, each such "re-defined" block contains 89 * 94 code points (0xCDA1 is the new starting code point of the redefined second block), [[ note that this is NOT exactly the original HZ-2 proposal]] which maps to 0x21-0x79 of the first 7-bit byte. That leaves the candidate list to be 0x7A-0x7D ("z{|}"). By similar argument as for HZ+S, the ONLY choice left is 'z' for 0xA121-0xCD7E, '|' for 0xCDA1-0xF9FE. If you don't like the use of '|' as the second byte of the escape code, then there is NO solution for HZ-2 like scheme. In summary, by this rather long but simple analysis, I prefer an enhanced HZ+S over HZ-2. For the sake of reference, let's call my enhanced versio HZ+L: block 1: 0xA140-0xC8FE block 2: 0xC940-0xF9FE including all code points whose second byte is in the range 0x40-0x7E, or 0xA1-0xFE escape code to 7-bit BIG5 block 1: ~y escape code to 7-bit BIG5 block 2: ~z escape code to 7-bit GB2312-80: ~{ escape code to 7-bit ASCII: ~} HZ+L is essentially HZ+S with the changes from ~1 to ~y, and ~2 to ~z. Best regards, Fung Fung Lee, Ph.D. Consulting Assistant Professor Computer Systems Laboratory Stanford University U.S.A. lee@csl.stanford.edu From lee@simd.stanford.edu Mon May 16 16:59:06 1994 Return-Path: Received: from simd.stanford.edu by ifcss.org (4.1/IFCSS-Mailer) id AA21642; Mon, 16 May 94 16:59:06 CDT Received: by simd.stanford.edu (5.65/inc-1.0) id AA16054; Mon, 16 May 1994 14:56:59 -0700 From: lee@simd.stanford.edu (Fung Fung Lee) Message-Id: <9405162156.AA16054@simd.stanford.edu> Subject: convention vs. logical consequence (Was Re: mixed GB and BIG5) To: soft-authors@ifcss.org Date: Mon, 16 May 1994 14:56:59 -0700 (PDT) In-Reply-To: <199405161930.PAA15619@boole.math.psu.edu> from "Stephen G Simpson" at May 16, 94 03:30:11 pm X-Mailer: ELM [version 2.4 PL23] Content-Type: text Content-Length: 1834 Status: O > As Ed Lai pointed out, there is a convention in HZ to end every line > with ~}. This means that when you scroll back through a ~} you will [[ This is NOT a convention of HZ, much less an optional one. This IS a logical consequence of HZ that comes automatically. As I recalled, since the HZ vs. zW debate in 1989, and a later one shortly after the establishment of alt.chinese.text, I have at least clarified this point again and again over at least 10 times. -FFL ]] That is NOT a convention in HZ. A newline character (which is OS dependent, e.g. on UNIX, on MacOS, and on DOS) is clearly an ASCII character, NOT a GB character, and thus it is a logical consequence of HZ to have each line end with ~} if the preceding character is GB. In fact, in this case NOT ending the line with ~} is a syntax error, the recovery of which is intentionally NOT defined in HZ spec so that it will NOT become part of the standard. There are many reasons we don't want to overload the newline character. The original reason that triggers this response is just one of them. > not need to go further than one line to find out whether the preceding > material is in GB or Big5. This should make it easier for terminal > programs to deal with mixed GB/Big5. > > Perhaps HZ+ should insist on having ~} at the end of each line. This > is an optional convention in HZ, but maybe we should make it mandatory > in HZ+. Comments? > > -- Steve > In fact, what we need is to keep enforcing the convention for the USER to remember (with the help of software) to post syntatically correct HZ (HZ+) files on the net, even though the tools they use, such as zWDOS, may not complain about the non-conforming syntax. [[A quiet option and an verbose/error-checking option that can be toggled by the user will be useful.]] -FFL From simpson@math.psu.edu Mon May 16 18:39:38 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA25948; Mon, 16 May 94 18:39:38 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id TAA13866; Mon, 16 May 1994 19:37:10 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 16 May 1994 19:37:08 -0400 Date: Mon, 16 May 1994 19:37:08 -0400 Message-Id: <199405162337.TAA16184@boole.math.psu.edu> To: soft-authors@ifcss.org Cc: shann@wangwei.math.ncu.edu.tw Subject: fast Big5 <-> HZ+S conversion Reply-To: simpson@math.psu.edu Status: O My HZ+S proposal uses tight packing to map the Big5 code space into the graphical ASCII code space. Some people have criticized tight packing on grounds of "inefficiency", because the most straightforward implementation of it uses multiplication by 157 and division by 94. In order to meet the "inefficiency" objection, I have worked out a new sample implementation in which no multiplications and divisions are used. Instead, following a suggestion of Carlos McEvilly, I use tables for (a*94)/157 and (a*94)%157, a = 0,1,...,93 and (b*157)/94 and (b*157)%94, b = 0,1,...,53. If anyone would like to help me test the new sample implementation, both for correctness and for speed, please send me e-mail. -- Steve From shann@wangwei.math.ncu.edu.tw Tue May 17 09:27:21 1994 Return-Path: Received: from wangwei.math.ncu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA24286; Tue, 17 May 94 09:27:21 CDT Received: from qingzhao.math.ncu.edu.tw by wangwei.math.ncu.edu.tw (4.1/SMI-4.1) id AA06397; Tue, 17 May 94 22:24:46 CST Date: Tue, 17 May 94 22:24:46 CST From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Message-Id: <9405171424.AA06397@wangwei.math.ncu.edu.tw> To: CCNET-L@uga.cc.uga.edu, soft-authors@ifcss.org Subject: CNS_bitmap Status: O I put the bitmap files that I got together with the CNS book in anonymous ftp account on dongpo.math.ncu.edu.tw:/pub/shann/chinese/CNS_bitmap Please also get README file. -Shann From simpson@math.psu.edu Tue May 17 10:45:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA27610; Tue, 17 May 94 10:45:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA18669; Tue, 17 May 1994 11:43:35 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 17 May 1994 11:43:34 -0400 Date: Tue, 17 May 1994 11:43:34 -0400 Message-Id: <199405171543.LAA17357@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Urgent! Please speak up for HZ+ now! In-Reply-To: <9405171012.AA22162@econ.Berkeley.EDU> References: <9405171012.AA22162@econ.Berkeley.EDU> Reply-To: simpson@math.psu.edu Status: O Soft-authors: Several of us have pointed out the need for HZ+. If you agree that there is a need for HZ+, please speak up now! Here is a comment from one of the chinese.* organizers. > My position is still this: if a scheme to incorporate big5 into HZ can't > be accepted by people, and/or softwares aren't ready to support it, we > will have to resort to cross-mirroring. But I think we should give > the HZ+? proposals a chance. How are the software authors feel? And > how are the Taiwan users feel? Jeff seems to indicate it's impossible > to access Chinese news without using big5, and somehow I just don't > buy that. How are people in Taiwan accessing USENET news? I mean > what softwares do they use? How easy is it for these softwares to > be updated to support "HZ+?"? > > -- Pinghua on Tue > 5/17/94, 03:12:39 From simpson@math.psu.edu Tue May 17 12:28:46 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA01930; Tue, 17 May 94 12:28:46 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA19289; Tue, 17 May 1994 13:26:45 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 17 May 1994 13:26:44 -0400 Date: Tue, 17 May 1994 13:26:44 -0400 Message-Id: <199405171726.NAA17500@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Thank you, Fung! HZ+ now! In-Reply-To: <9405162127.AA15904@simd.stanford.edu> References: <9405162127.AA15904@simd.stanford.edu> Reply-To: simpson@math.psu.edu Status: O My sincere thanks to Fung-Fung Lee for taking the time to give us his analysis of the need and requirements for HZ+. There is an urgent need for HZ+, and I see Fung's postings as a public-spririted act. Fung's initiative and leadership is particularly valuable because of Fung's role as the author of the popular HZ standard, on which HZ+ is based. Thank you, Fung! I am also gratified that Fung saw fit to almost endorse HZ+S, except for the choice of escape sequences. As Fung said: > HZ+L is essentially HZ+S with the changes from ~1 to ~y, and > ~2 to ~z. Fung gave good technical reasons for preferring ~y and ~z, and I have no objections. I am quite willing to endorse HZ+L in order to reach a consensus. Soft-authors, can we now please try to reach a consensus? The need is urgent. Can we decide *now* that HZ+L is HZ+, and begin *now* to support HZ+? If you agree with me and Fung, please post a line to soft-authors. -- Steve From simpson@math.psu.edu Tue May 17 12:44:34 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA02622; Tue, 17 May 94 12:44:34 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA19406; Tue, 17 May 1994 13:42:18 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 17 May 1994 13:42:16 -0400 Date: Tue, 17 May 1994 13:42:16 -0400 Message-Id: <199405171742.NAA17521@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: comment on HU Bo's posting In-Reply-To: <9405162138.AA15655@sun3.lrz-muenchen.de> References: <199405161705.NAA15436@boole.math.psu.edu> <9405162138.AA15655@sun3.lrz-muenchen.de> Reply-To: simpson@math.psu.edu Status: O HU Bo writes: > ~{NRQ'AK~}ABC > The code is 13 bytes long. Now you want to say, > ~{NRQ'~}ABC~{AK~} > The code becomes 17 bytes long. This is an unavoidable feature of HZ or indeed any system that uses escape sequences to switch between character sets. If your software can't support HZ for this reason, that is your decision and I wouldn't try to change your mind. -- Steve From simpson@math.psu.edu Tue May 17 13:37:19 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA04698; Tue, 17 May 94 13:37:19 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA29585; Tue, 17 May 94 11:35:24 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA19812; Tue, 17 May 1994 14:33:41 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 17 May 1994 14:33:40 -0400 Date: Tue, 17 May 1994 14:33:40 -0400 Message-Id: <199405171833.OAA17577@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: answer to Ed's posting on CNS In-Reply-To: <199405130505.BAA15791@leibniz.math.psu.edu> References: <199405130505.BAA15791@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Edmund Lai writes: > And now he is back to the old discredited agrument "Big5 is more > popular than CNS, we need to support Big5". When did that argument become discredited? I think it is a valid argument. Big5 *is* more popular than CNS. We *do* need to support Big5. > As we now all know, using CNS as exchange code would still support > normal 8 bit Big5 as internal code. I don't know that. Does anyone other than Ed know that? Ed, Roland, and other advocates of CNS have not explained how that would be accomplished. But even if what Ed is saying about CNS turns out to be true, I still think it will be simpler and cleaner to use 7-bit Big5. The reason is that 7-bit Big5 is very close to 8-bit Big5, while CNS is *not* very close to 8-bit Big5. > The question is which is more popular, more of a standard, 7 bit > CNS or 7 bit Mule Big5. No. That is *one* question, but it is not *the* question. Another interesting question is, which is more popular, Big5 or CNS. The answer to *that* question is, Big5 is more popular, by perhaps 1000 to 1. > If Steve has so much confidence in HZ+S, why can't he play it > clean and straight? I am playing it clean and straight. If Ed has so much confidence in CNS, why does he keep trying to portray CNS as a kind of Big5? Ed insists on describing CNS as an "encoding method for Big5" or "corrected Big5" or "exchange code for Big5". These descriptions are very misleading. The simple fact is, CNS is very different from Big5, in many ways. The simple fact is that CNS and Big5 are *two* *different* *Chinese* *character* *coding* *systems*. Why doesn't Ed acknowledge this fact? Ed seems to think that I can improve HZ+S by using CNS as "an exchange code for Big5". I disagree. If I were to follow Ed's suggestion, it would amount in practice to dropping Big5 support. Big5 users would have no choice but to resort to Big5<->CNS translation. I think this is unnecessary and a very bad idea. I might be willing to add CNS support to HZ+S, but I am not willing to drop Big5 support. -- Steve From simpson@math.psu.edu Tue May 17 18:38:03 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA17195; Tue, 17 May 94 18:38:03 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id TAA22843; Tue, 17 May 1994 19:36:01 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 17 May 1994 19:35:58 -0400 Date: Tue, 17 May 1994 19:35:58 -0400 Message-Id: <199405172335.TAA17991@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu Subject: building a consensus for HZ+ Reply-To: simpson@math.psu.edu Status: O I am trying to build a consensus for HZ+, where HZ+ is defined to be HZ plus some form of 7-bit Big5. There has been a very vigorous discussion. I believe the following members of the Chinese software authors group, soft-authors@ifcss.org, have all come out in favor of the general idea of HZ+. Are there corrections or additions to this list? If you would like to add your name to this list, please send e-mail to me or to soft-authors. Hardly anybody has come out against the idea of HZ+. Shall we interpret silence to mean consent? -- Steve From lai@apple.com Wed May 18 00:39:27 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA28343; Wed, 18 May 94 00:39:27 CDT Received: from mac162.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA05807; Tue, 17 May 94 22:37:13 -0700 for soft-authors@ifcss.org Message-Id: <9405180537.AA05807@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Tue, 17 May 1994 22:42:05 -0800 To: soft-authors@ifcss.org From: lai@apple.com (Ed Lai) Subject: Re: HZplus: 7-bit BIG5/GB standard Status: O > >b. Simple, fast and easily described 100% invertible conversion= algorithm > exists between 8-bit BIG5 and 7-bit BIG5 in HZplus > >c. Following HZ, choose a 7-bit encoding that preserves the relative= ordering: > Enable the use of many existing 7-bit utilities for the purpose= of sorting, > searching, etc. > Since it is claim that CNS would fail the above two test, and this= is a misinformed statement. So I need to clarify these two points. People who have actually worked on conversion between CNS and Big5= have repeatedly confirm that conversion between the two is fast and easy. It is unavoidable that there are extra overhead when compared to= converion between real Big5 and Mule Big5, however the overhead is really minimal. Of course it can always be argued the extra overhead, no matter how small, just happen to exceed a certain threshhold and hence is not acceptable. However such overhead is much smaller than the overhead for conversion between Big5 and GB, by declaring the overhead of CNS to be acceptable, then conversion between Big5 and GB is also not acceptable. As for 100% invertable transformation between standard Big5 and CNS, it is 99.98% invertable, the two characters are not invertable because they are duplicate mistakes in Big5. They are not invertable to the same previous internal code, but they may be inverted back to its= =20 twin. As for the vendor specific code, either they should be mapped to= a commond exchange code on a per vendor basis, or they should be omitted. Is HZ meant to be merely a 7 bit encoding like uuencode, or is it meant to be a cross platform exchange code. If it meant only between machine of the same kind, then the former is OK. But if it is meant for news to be read by multiple platform, one cannot just encode code specific to your own machine and blast it out. On the Mac, "=81" are special symbols. If you read it on a Mac, you would not have any problem reading it. But on other machine, it would be garbage character. Since I know I want it to be readable on other machine, I would not normally do such a thing. Instead I would use "TM". To ignore the problem and declare it to be an advantage of Mule Big5 is like "close the ear to steal the bell". As for data processing, people from Taiwan has already say very clearly that it is doubtful if there is going to be a lot of processing with 7 bit encoded data from than 8 bit Big5. =46urthermore, I don't see why searching is a problem with CNS but not with CNS, one would not be easier than the other. As for sorting, the order would be slightly different. Sorry about that, because CNS does not count the strokes wrongly like Big5. So you find the character in the correct order. Is it wrong to do the right thing? =20 /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From lai@apple.com Wed May 18 00:39:01 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA28337; Wed, 18 May 94 00:39:01 CDT Received: from mac162.kip.apple.com by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA05749; Tue, 17 May 94 22:35:38 -0700 for soft-authors@ifcss.org Message-Id: <9405180535.AA05749@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Tue, 17 May 1994 22:40:31 -0800 To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu From: lai@apple.com (Ed Lai) Subject: Re: building a consensus for HZ+ Status: O >I am trying to build a consensus for HZ+, where HZ+ is defined to be >HZ plus some form of 7-bit Big5. > >There has been a very vigorous discussion. I believe the following >members of the Chinese software authors group, soft-authors@ifcss.org, >have all come out in favor of the general idea of HZ+. > > > > > > > > > > > > > > >Are there corrections or additions to this list? > >If you would like to add your name to this list, please send >e-mail to me or to soft-authors. > >Hardly anybody has come out against the idea of HZ+. Shall we >interpret silence to mean consent? > >-- Steve This is the strangest election I have ever seen, why don't you let people vote and speak for themselves. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From rap@doc.ic.ac.uk Wed May 18 06:15:05 1994 Return-Path: Received: from santos.doc.ic.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA05858; Wed, 18 May 94 06:15:05 CDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0q3jWv-00070hC; Wed, 18 May 94 12:11 BST Message-Id: Date: Wed, 18 May 94 12:11 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Re: building a consensus for HZ+ Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Status: O Stephen G Simpson writes: > Hardly anybody has come out against the idea of HZ+. Shall we > interpret silence to mean consent? Actually, I'm against the idea of HZ, even without the plus -- I think it would be better to use an established international standard for multilingual text. Today that means ISO 2022 (basically a standardized set of escape sequences for switching between 7-bit character sets). I hope it won't be many years before it's Unicode/ISO 10646. The benefit of this is the body of software in existence and under development that is built around this standard. The format is already supported (Mule, Mosaic-l10n, etc), and the support is improving without any effort from us. Surely this is preferable to asking all those authors to modify their programs to support a private encoding. Conversely, software developed to support Chinese text represented that way will be easy to extend for users of other languages. Moreover, a uniform treatment of existing character sets is a useful first step towards a unified character set. As for the representation of Big-5: ISO 2022 reserves the final characters 30-3F for private use, and it would be reasonable to use two of these for a 7-bit version of Big-5, as Mule does. Programs that are to support Big-5 would have to incorporate this private agreement. I happen to believe that using levels 1 and 2 of CNS (which are now in the international registry) would be better, but I think this is less important than using the international standard for the escape sequences. -- Ross Paterson Department of Computing, Imperial College, London SW7 From simpson@math.psu.edu Wed May 18 10:42:00 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12858; Wed, 18 May 94 10:42:00 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA27787; Wed, 18 May 1994 11:38:45 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 18 May 1994 11:38:43 -0400 Date: Wed, 18 May 1994 11:38:43 -0400 Message-Id: <199405181538.LAA19139@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: chinese.* newsgroups to start June 4, 1994 In-Reply-To: <9405180507.AA29374@econ.Berkeley.EDU> References: <9405180507.AA29374@econ.Berkeley.EDU> Reply-To: simpson@math.psu.edu Status: O Pinghua Young writes: > BTW, I just got word from Rob giving us green light to create > Chinese.* on June 4, 1994. And I guess each of us already know that > David is going to carry it at UUNET. > > So we are all set and ready to go. Please spread the word. -- Steve From simpson@math.psu.edu Wed May 18 12:01:15 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA15227; Wed, 18 May 94 12:01:15 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA28299; Wed, 18 May 1994 12:58:50 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 18 May 1994 12:58:49 -0400 Date: Wed, 18 May 1994 12:58:49 -0400 Message-Id: <199405181658.MAA19234@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: building a consensus for HZ+ In-Reply-To: <199405180555.BAA25458@leibniz.math.psu.edu> References: <199405180555.BAA25458@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O I wrote: > >I am trying to build a consensus for HZ+, where HZ+ is defined to be > >HZ plus some form of 7-bit Big5. > > ... Ed Lai replied: > This is the strangest election I have ever seen, why don't you > let people vote and speak for themselves. I am not opposed in principle to a vote. The biggest problem is apathy. As for "speaking for themselves", I believe that all of the 12 Chinese software authors that I mentioned have already spoken for themselves, in favor of HZ+. If anyone thinks my list needs corrections, please let me know. Ed, how about you? If there is any way that you could approve of HZ+, I would be delighted to add you to the list. (I believe Ed views CNS as a form of 7-bit Big5, so in that weak sense Ed may have already expressed approval of HZ+? But this is guesswork. I'll wait for Ed to speak for himself. The people on my list have spoken out explicitly in favor of HZ+ as defined at the beginning of this posting.) The same invitation goes out to all Chinese software developers, whether or not you belong to the soft-authors@ifcss.org mail alias. If you have written Chinese software and want to express approval or support for HZ+, please send e-mail to soft-authors@ifcss.org. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From czha@CATI.CSUFresno.EDU Wed May 18 12:08:53 1994 Return-Path: Received: from CATI.CSUFresno.EDU (caticsuf.cati.csufresno.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA15482; Wed, 18 May 94 12:08:53 CDT Received: (czha@localhost) by CATI.CSUFresno.EDU (8.6.8.1/8.6.5) id KAA13170 for soft-authors@ifcss.org; Wed, 18 May 1994 10:06:18 -0700 Date: Wed, 18 May 1994 10:06:18 -0700 From: Charlie Zha Message-Id: <199405181706.KAA13170@CATI.CSUFresno.EDU> X-Mailer: idsmail 4.0 To: soft-authors@ifcss.org Subject: Add me to your mailing list Status: O I got your address from a piece of e-mail. If this is a mailing list, please add me. I was a former IFCSS member. I am very glad to see IFCSS has its own FTP site. Thank you. Charlie Zha From simpson@math.psu.edu Wed May 18 17:14:19 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA25747; Wed, 18 May 94 17:14:19 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA08021; Wed, 18 May 94 15:10:55 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA01506; Wed, 18 May 1994 18:07:58 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 18 May 1994 18:07:56 -0400 Date: Wed, 18 May 1994 18:07:56 -0400 Message-Id: <199405182207.SAA19630@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Cc: chinesenet@uclink.berkeley.edu Subject: the high road and the low road In-Reply-To: <199405101153.HAA15739@leibniz.math.psu.edu> References: <199405101153.HAA15739@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Ross Paterson has been one of most consistently excellent participants in ccnet-l and soft-authors. His help with HBF was invaluable. His level of technical expertise and judgment in all aspects of Chinese computing is truly outstanding. Recently Ross wrote: > What about abandoning HZ in favour of a straight 7-bit ISO 2022 encoding > ... I must respectfully disagree. As everybody knows, I'm very much opposed to abandoning HZ. In fact, I'm busily campaigning to extend HZ to HZ+, for use in the chinese.* newsgroup hierarchy. Nevertheless, I want to point out that the positive part of Ross's suggestion has a lot of merit. For the long run, everybody ought to hope that Chinese computing will integrate itself with multilingual computing based on official, internationally recognized standards such as ISO-2022 and, ultimately, Unicode. This is the "high road" for Chinese computing. In contrast to the "high road", there is the "low road" based on pragmatic, popular standards such as HZ and Big5. And I would like to point out that the low road has its advantages, too. HZ and Big5 are extremely useful, today, because they are extremely popular, today. On the one hand we have CNS, ISO-2022, and Unicode -- official standards with (so far) little popular support. On the other hand we have HZ and Big5, which may never become official standards, but are very, very popular. Like star-crossed lovers (Roseanne Barr and Tom Arnold?), HZ and Big5 were made for each other! Their marriage in HZ+ represents a thoroughly pragmatic, low-road synthesis. Because of immediate practical considerations, the chinese.* newsgroup hierarchy needs a popular, pragmatic, low-road approach. That is why I advocate HZ+ for use in the chinese.* hierarchy now. Later, as high-road standards become more popular, the chinese.* hierarchy can gradually move to the high road. For now, the low road is best. Whether you agree with me or not, I would urge everyone in our debates on coding standards not to lose sight of the distinction between the "high road" and the "low road". -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From Ricky.Yeung@Eng.Sun.COM Wed May 18 18:04:54 1994 Return-Path: Received: from Sun.COM by ifcss.org (4.1/IFCSS-Mailer) id AA27557; Wed, 18 May 94 18:04:54 CDT Received: from Eng.Sun.COM (zigzag.Eng.Sun.COM) by Sun.COM (sun-barr.Sun.COM) id AA06023; Wed, 18 May 94 16:02:32 PDT Received: from hk.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA08681; Wed, 18 May 94 16:02:00 PDT Received: by hk.Eng.Sun.COM (5.0/SMI-SVR4) id AA13413; Wed, 18 May 1994 16:01:23 +0800 Date: Wed, 18 May 1994 16:01:23 +0800 From: Ricky.Yeung@Eng.Sun.COM (Ricky Yeung) Message-Id: <9405182301.AA13413@hk.Eng.Sun.COM> To: soft-authors@ifcss.org In-Reply-To: Ed Lai's message of Tue, 17 May 1994 22:42:05 -0800 <9405180537.AA05807@apple.com> Subject: Re: HZplus: 7-bit BIG5/GB standard Content-Length: 488 Status: O Just for the argument's sake, let me play the devil's advocate. > As for 100% invertable transformation between standard Big5 and CNS, > it is 99.98% invertable, the two characters are not invertable because > they are duplicate mistakes in Big5. They are not invertable to the > same previous internal code, but they may be inverted back to its= > =20 > twin. With a similar argument, maybe we should use CNS as "the" exchange code for GB as well. -Ricky From ucc02au@sunmail.lrz-muenchen.de Wed May 18 18:12:20 1994 Return-Path: Received: from cd1fddi.lrz-muenchen.de (cd1.lrz-muenchen.de) by ifcss.org (4.1/IFCSS-Mailer) id AA27761; Wed, 18 May 94 18:12:20 CDT Received: from sunserver.lrz-muenchen.de by cd1.lrz-muenchen.de; Thu, 19 May 94 01:10:19 +0200 Received: from sun3.lrz-muenchen.de by sunserver.lrz-muenchen.de (4.1/SMI-4.1) id AA25774; Thu, 19 May 94 01:10:18 +0200 Received: by sun3.lrz-muenchen.de (4.1/SMI-4.1) id AA11667; Thu, 19 May 94 01:10:17 +0200 From: ucc02au@sunmail.lrz-muenchen.de (HU Bo) Message-Id: <9405182310.AA11667@sun3.lrz-muenchen.de> Subject: Re: comment on HU Bo's posting To: soft-authors@ifcss.org Date: Thu, 19 May 1994 01:10:17 +0200 (MET DST) X-Mailer: ELM [version 2.4 PL21] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 1438 Status: O > > HU Bo writes: > > ~{NRQ'AK~}ABC > > The code is 13 bytes long. Now you want to say, > > ~{NRQ'~}ABC~{AK~} > > The code becomes 17 bytes long. > > This is an unavoidable feature of HZ or indeed any system that uses > escape sequences to switch between character sets. If your software > can't support HZ for this reason, that is your decision and I wouldn't > try to change your mind. > > -- Steve > On today's World it is not important if one or two software packages support a code system, but if all, or almost all software packages support it. Using these kind of escape sequences, it is very difficault to make standard applications useable. An example is the width table used in each word processor nowadays. I am not really interested if my software should be made for HZ+, but very worried about the new Babylonian era of Chinese Computing. Escape sequences are the first approaches to combine CJK languages with ASCII letters. But, as seen in Japan and "both parts of" China, the (!) coding systems used now are those without eascape sequences (Shift-JIS, Big5, GB). I hope that everybody going to vote for a coding system using escape sequences has also taken this fact into account. I think that there must be somme more elegant solutions for the problems around news groups and mail servers. Since code converters are used to make GB, Big5 codes to HZ+ codes, why don't we use simply the uuencode/decode? Bo From lai@apple.com Wed May 18 20:12:24 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA01995; Wed, 18 May 94 20:12:24 CDT Received: from [17.203.80.11] by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA02065; Wed, 18 May 94 18:10:03 -0700 for soft-authors@ifcss.org Date: Wed, 18 May 94 18:10:03 -0700 Message-Id: <9405190110.AA02065@apple.com> X-Sender: lai@apple.com (Unverified) Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: HZplus: 7-bit BIG5/GB standard Status: O >Just for the argument's sake, let me play the devil's advocate. > > > As for 100% invertable transformation between standard Big5 and CNS, > > it is 99.98% invertable, the two characters are not invertable because > > they are duplicate mistakes in Big5. They are not invertable to the > > same previous internal code, but they may be inverted back to its= > > =20 > > twin. > >With a similar argument, maybe we should use CNS as "the" exchange code >for GB as well. > >-Ricky Actually I think this is a prefectly valid argument. People called me a CNS advocate. This is completely off. The extension to HZ has been dicussed before, my position then was that it should be an extension of GB. I still think this is the best solution. But since nobody else is taking a position that is close to this. So I say if we are going to do something like HZ+?, we may as well use CNS there rather than do invent another coding scheme. However I never understand how HZ+? can work. I am now using Eudora to read news and write this. I am using Newswatcher to read news. Both of them only allow one font. So it is either trad font in Big5 or simplifed font in GB for me. With HZ+? am I supposed to change font on every document? What if we have news posted in traditional and reply in simplifed, am I supposed to manually switch font on every paragraph? So the software has to switch everything to either GB or Big5 for the document to be read. Then why bother to have both? Ideally there should be a single character code that can be translated to the internal code of either camp. However, using CNS only would not be upward compatiable since HZ is based on GB. Furthermore the character set of CNS plane 1 and 2 is really the same as Big5 (although the radical helps, since they are in CNS, GB but not in Big5), in theory Big5/CNS can be superset of GB, unfortunately it is not. My position has always been that I don't want to invent another character code. This is why I prefer CNS over Mule Big5. In theory there are such super set of GB, however the information is scarce. GB12345 is supposed to be a start, in the normal GB range it is supposed to be same (except trad rather than simplifed), and there are extension. The unicode table gives some of the mapping of these extension but not all of them. And I have zero information about the more extensive code table. I cannot be an advocate without any information. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From rap@doc.ic.ac.uk Thu May 19 07:06:56 1994 Return-Path: Received: from santos.doc.ic.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA21939; Thu, 19 May 94 07:06:56 CDT Received: by santos.doc.ic.ac.uk (Smail3.1.28.1 #7) id m0q46qD-00070hC; Thu, 19 May 94 13:04 BST Message-Id: Date: Thu, 19 May 94 13:04 BST From: Ross Paterson To: soft-authors@ifcss.org Subject: Re: the high road and the low road Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Status: O Stephen G Simpson writes: > On the one hand we have CNS, ISO-2022, and Unicode -- official > standards with (so far) little popular support. On the other hand we > have HZ and Big5, which may never become official standards, but are > very, very popular. Let's put CNS and Unicode to the side for a moment. I dispute the assertion that ISO 2022 is less popular or less practical than HZ. It's only true in the subset of Chinese Computing discussed on CCNET-L. Out in the wider world lots of people are using forms of ISO 2022, and a great deal of software supports it, for example the X Windows system, many X clients including Mule and exterm, many MIME user agents (mail readers), etc. Unicode may have to wait a while, but ISO 2022 is practical now. Private encoding schemes like HZ are dead ends. -- Ross Paterson Department of Computing, Imperial College, London SW7 From czha@CATI.CSUFresno.EDU Thu May 19 10:50:23 1994 Return-Path: Received: from CATI.CSUFresno.EDU (caticsuf.cati.csufresno.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA00543; Thu, 19 May 94 10:50:23 CDT Received: (czha@localhost) by CATI.CSUFresno.EDU (8.6.8.1/8.6.5) id IAA19408; Thu, 19 May 1994 08:47:48 -0700 Date: Thu, 19 May 1994 08:47:48 -0700 From: Charlie Zha Message-Id: <199405191547.IAA19408@CATI.CSUFresno.EDU> X-Mailer: idsmail 4.0 To: soft-authors@ifcss.org Subject: GB, Big5, HZ, ISO2022, Unicode, ... Cc: ccnet-l@uga.cc.uga.edu Status: O What a mess ? Were any Chinese sitting in ISO2022 or Unicode committe ? Or, does GB and Big5 has any straight forward mapping to either ISO2022 and Unicode ? It is always neat to invent your own little coding scheme. It even feels great. Unfortunately, it only cause further unnecessary confussion. Instead of arguing which code schme is better, we probably should compromise a bit and settle on one (lousy) codeing schme and be done with it. I have been working with ANSI X.12 eletronical data interchange group for over a year. Admittedly, X.12 standard is a mess. Many people can improve it to be more lean and mean and more elegant. However, people are converting to X.12 from other neat schems that they were using. My point is that it doesn't matter how neat the coding scheme is. If we can agree on one lousy scheme, we can work out something great regardless of the coding scheme. Usually, international standard is the way to go, despite the popularity of properitary coding schems. It is true at least in the eletronic data inter- change community. Let's compromise and settle this down for once and for all. Charlie Zha From jiang Thu May 19 11:36:27 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA02499; Thu, 19 May 94 11:36:27 CDT Date: Thu, 19 May 94 11:36:27 CDT From: jiang (Yuan Jiang) Message-Id: <9405191636.AA02499@ifcss.org> To: chinesenet@uclink.berkeley.edu, pinghua@econ.Berkeley.EDU Subject: Re: 7-bit Big5 and Chinese.* Cc: ccnet-l@uga.bitnet, soft-authors Status: O Pinghua, I hope chinese.* stays a clear mind on this issue. This 7-bit<->Big5 issue is to satisfy the need of chinese.*, which in turn is to satisfy Big5 users. But the majority of the big5 users are in taiwan. The issue concerns mostly in what code (HZ+ or what) Taiwan to interface with other usenet communities wh can only use a 7-bit code. So, we should let people from Taiwan to decide on this issue. If the 7-bit <-> big5 convertion is done at the usenet gateways, people in charge of these gateways, such as C. H. Huang and Shinn, have the crucial words on this matter. This is the simpliest case. If the convertion is done by each user's application, the software developers in Taiwan have the crucial word. In this case, C. H. Huang and Shinn, etc are the key persons to bridge the decision making. chinese.* despertly needs a 7-bit<-> convertion scheme. Yet, we are not here to decide on a good scheme, but to decide a workable one which can live. Such a scheme's life depends on the key persons' support (gateway operators and application programmers). Everyone says Big5 is a lossy code. Yet we let it be imposed on us because the big number of Taiwan users.us. We should let people in Taiwan to decide what 7-bit<->Big5 scheme to impose on us. CNS is a code imposed by Taiwan government. But HZ+S and HZ-2 will be impositions from North America. Yuan From jiang Thu May 19 11:55:42 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA03980; Thu, 19 May 94 11:55:42 CDT Date: Thu, 19 May 94 11:55:42 CDT From: jiang (Yuan Jiang) Message-Id: <9405191655.AA03980@ifcss.org> To: chinesenet@uclink.berkeley.edu, pinghua@econ.Berkeley.EDU Subject: Re: 7-bit Big5 and Chinese.* Cc: ccnet-l@uga.bitnet, soft-authors Status: O Sorry, in my last posting, I got the names of the Taiwan gateway operators wrong. Forgive me, I can never remember people's names. Yuan From @UTARLVM1.UTA.EDU:simpson@math.psu.edu Thu May 19 14:43:08 1994 Return-Path: <@UTARLVM1.UTA.EDU:simpson@math.psu.edu> Received: from UTARLVM1.UTA.EDU ([129.107.1.6]) by ifcss.org (4.1/IFCSS-Mailer) id AA10852; Thu, 19 May 94 14:43:08 CDT Received: from leibniz.math.psu.edu by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Thu, 19 May 94 14:41:01 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA12015; Thu, 19 May 1994 15:37:16 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 19 May 1994 15:37:15 -0400 Date: Thu, 19 May 1994 15:37:15 -0400 Message-Id: <199405191937.PAA25245@boole.math.psu.edu> To: chinesenet@uclink.berkeley.edu Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: testing HZ+ on chinese.test In-Reply-To: <9405190701.AA05943@econ.Berkeley.EDU> References: <9405190701.AA05943@econ.Berkeley.EDU> Reply-To: simpson@math.psu.edu Status: O Pinghua Young writes: > Rob has also given me the power to create chinese.* at will and at any > time. I am wondering if we should create chinese.test first so that > Steve and others can start testing HZ+? new coding method. What do > you guys think? Creating chinese.test sounds good to me, with the understanding that it is for any kind of test postings related to chinese.*, not only for HZ+. As for testing HZ+, I will have to rely on others to help me with that. If anybody would like to help, please send me e-mail. What I need most is people willing to post and read test messages in 7-bit Big5. My material related to HZ+ is available for anonymous FTP at math.psu.edu, /pub/simpson/chinese/hzps. If anyone has any questions about this stuff, please send me e-mail. > I'll write a shell script to create the rest of the newsgroups (9 if > chinese.test is already created) automatically at 0:0am of June 4, 1994. This sounds good to me. -- Steve From simpson@math.psu.edu Thu May 19 15:15:13 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12040; Thu, 19 May 94 15:15:13 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id QAA12284; Thu, 19 May 1994 16:12:34 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 19 May 1994 16:12:32 -0400 Date: Thu, 19 May 1994 16:12:32 -0400 Message-Id: <199405192012.QAA25302@boole.math.psu.edu> To: gchen@mailhost.ecn.uoknor.edu (Gongquan Chen) Cc: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: reply to Gongquan Chen In-Reply-To: References: Reply-To: simpson@math.psu.edu Status: O Gongquan Chen writes: > As a member of the soft-authors list, I'll raise my hand now to support > HZ+. BTW, have you noticed that most people on the list who have kept > silent are students? Thanks very much. I'll add you to the list of HZ+ supporters. I don't know everybody on the soft-authors list. If there are students out there, please don't be shy, speak up! I'm happy to note that several of those who have expressed support for HZ+ are old hands in Chinese software -- people like F-F Lee (author of the HZ standard and other important tools), Ya-Gui Wei (author of ZWDOS), Carlos McEvilly (author of Bamboo Helper), MC Pong (one of original authors of cxterm), Yidao Cai (author of CNprint). I'd like to have Ed Lai and Yongguang Zhang here too, but let's wait and see. > I am a little confused about the Chinese.* newsgroups. Who is Rob > and why does he have the power to nod on the creation of this > newsgroup hierarchy? Who is David and what is UUNET? Also, after > Chinese.* is set up, do we need to rename a.c.t and a.c.t.b? All > these questions seem pretty dumb. These are not dumb questions at all. I can't answer them very precisely, but as a member of the volunteer committee guiding the chinese.* newsgroup hierarchy, I have been receiving all of the relevant e-mail. I know that Rob and David are netnews administrators and they do indeed have some power with respect to creation and distribution of new newsgroups. They have been very helpful to the chinese.* organizers, and we should all be grateful to them. I'm not sure what will happen to a.c.t and a.c.t.big5 after the chinese.* hierarchy gets going. I expect there will be a fairly orderly transition. -- Steve From pinghua@econ.Berkeley.EDU Thu May 19 15:26:01 1994 Received: from econ.Berkeley.EDU by ifcss.org (4.1/IFCSS-Mailer) id AA12403; Thu, 19 May 94 15:26:01 CDT Received: from emily2.Berkeley.EDU by econ.Berkeley.EDU (4.1/SMI-4.1) id AA09380; Thu, 19 May 94 13:23:20 PDT Date: Thu, 19 May 94 13:23:20 PDT Message-Id: <9405192023.AA09380@econ.Berkeley.EDU> Received: by emily2.Berkeley.EDU (4.1/SMI-4.1) id AA14678; Thu, 19 May 94 13:23:19 PDT Return-Path: Precedence: Special-Delivery Delivered: By MPY-PMS Version 2.0 Reply-To: pinghua@econ.Berkeley.EDU Voice-Fax: 510-559-8030 (leave msg pls) From: pinghua@econ.Berkeley.EDU (Pinghua Young) To: chinesenet@uclink.berkeley.edu, jiang@ifcss.org Subject: Re: 7-bit Big5 and Chinese.* Cc: ccnet-l@uga.bitnet, soft-authors@ifcss.org Status: O |> Pinghua, I hope chinese.* stays a clear mind on this issue. |> This 7-bit<->Big5 issue is to satisfy the need of chinese.*, |> which in turn is to satisfy Big5 users. But the majority |> of the big5 users are in taiwan. The issue concerns mostly |> in what code (HZ+ or what) Taiwan to interface with other |> usenet communities wh can only use a 7-bit code. |> |> So, we should let people from Taiwan to decide on this issue. |> If the 7-bit <-> big5 convertion is done at the usenet gateways, |> people in charge of these gateways, such as C. H. Huang and Shinn, |> have the crucial words on this matter. This is the simpliest case. |> If the convertion is done by each user's application, the |> software developers in Taiwan have the crucial word. In this case, |> C. H. Huang and Shinn, etc are the key persons to bridge the |> decision making. |> |> chinese.* despertly needs a 7-bit<-> convertion scheme. Yet, |> we are not here to decide on a good scheme, but to decide a |> workable one which can live. Such a scheme's life depends on |> the key persons' support (gateway operators and application |> programmers). Yuan, I agree with you full heartedly and I have been hoping people in Taiwan are more vocal on this issue. It's still not too late for people in Taiwan to express what they think is in their best interests, and I sincerely hope and wish to see more opinions from Big5 users. There are currently 3 options for chinese.*: 1. a 7-bit code that can handle both GB and Big5; 2. cross-mirroring under chinese.*, ie, chinese.*.big5; 3. macblue's suggestion: tw.chinese.* in Taiwan, ie, whatever coding scheme we use in chinese.*, sites be set up in Taiwan to do conversion between the coding scheme in chinese.* and big5 plus cross-posting, so that users in Taiwan can still use big5 to read and post. The last thing we want to do is to exclude Big5 users, and in no way we are going to do that. Let's let people in Taiwan decide what they consider is in their best interest. -- Pinghua on Thu 5/19/94, 13:23:18 From pinghua@econ.Berkeley.EDU Thu May 19 15:59:31 1994 Received: from econ.Berkeley.EDU by ifcss.org (4.1/IFCSS-Mailer) id AA13957; Thu, 19 May 94 15:59:31 CDT Received: from emily2.Berkeley.EDU by econ.Berkeley.EDU (4.1/SMI-4.1) id AA09625; Thu, 19 May 94 13:56:19 PDT Date: Thu, 19 May 94 13:56:19 PDT Message-Id: <9405192056.AA09625@econ.Berkeley.EDU> Received: by emily2.Berkeley.EDU (4.1/SMI-4.1) id AA14755; Thu, 19 May 94 13:56:18 PDT Return-Path: Precedence: Special-Delivery Delivered: By MPY-PMS Version 2.0 Reply-To: pinghua@econ.Berkeley.EDU Voice-Fax: 510-559-8030 (leave msg pls) From: pinghua@econ.Berkeley.EDU (Pinghua Young) To: gchen@mailhost.ecn.uoknor.edu, simpson@math.psu.edu Subject: Re: reply to Gongquan Chen Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Status: O |> > I am a little confused about the Chinese.* newsgroups. Who is Rob |> > and why does he have the power to nod on the creation of this |> > newsgroup hierarchy? Who is David and what is UUNET? Also, after |> > Chinese.* is set up, do we need to rename a.c.t and a.c.t.b? All |> > these questions seem pretty dumb. Steve responded: |> These are not dumb questions at all. I can't answer them very |> precisely, but as a member of the volunteer committee guiding the |> chinese.* newsgroup hierarchy, I have been receiving all of the |> relevant e-mail. I know that Rob and David are netnews administrators |> and they do indeed have some power with respect to creation and I don't think I know all the answers either, but here they go anyway. Rob Robertson is in charge of Berkeley USENET among many other things here at Berkeley. He's done a great service to Berkeley and to us by creating alt.chinese.computing (a few months ago) and chinese.* now. David Lawrence is in charge of the standard big-7 USENET hierarchies. UUNET is a big news hub where a lot of other sites get their newsfeeds from. So UUNET's carrying of chinese.* means that chinese.* will be available to all the sites that get newsfeeds from there. We want chinese.* to be propagated as widely as possible so that everyone who wants to access it can. As Steve said, Rob and David and many others have been very helpful and supportive. |> distribution of new newsgroups. They have been very helpful to the |> chinese.* organizers, and we should all be grateful to them. I'm not Yes. We owe them a lot for the help and support that they've been giving us. I am very grateful to them. |> sure what will happen to a.c.t and a.c.t.big5 after the chinese.* |> hierarchy gets going. I expect there will be a fairly orderly |> transition. Newsgroups created under alt.* hierarchy rarely dies in the sense that they could be re-created even if one deletes them since alt.* is basically in anarchy. We hope readers of ACT & ACT.big5 will move smoothly to chinese.*, but that's up to the readers and posters. We can't impose anything on them. -- Pinghua on Thu 5/19/94, 13:56:16 From pinghua@econ.Berkeley.EDU Thu May 19 16:17:55 1994 Received: from econ.Berkeley.EDU by ifcss.org (4.1/IFCSS-Mailer) id AA14937; Thu, 19 May 94 16:17:55 CDT Received: from emily11.Berkeley.EDU by econ.Berkeley.EDU (4.1/SMI-4.1) id AA09722; Thu, 19 May 94 14:14:51 PDT Date: Thu, 19 May 94 14:14:51 PDT Message-Id: <9405192114.AA09722@econ.Berkeley.EDU> Received: by emily11.Berkeley.EDU (4.1/SMI-4.1) id AA09541; Thu, 19 May 94 14:14:51 PDT Return-Path: Precedence: Special-Delivery Delivered: By MPY-PMS Version 2.0 Reply-To: pinghua@econ.Berkeley.EDU Voice-Fax: 510-559-8030 (leave msg pls) From: pinghua@econ.Berkeley.EDU (Pinghua Young) To: gchen@mailhost.ecn.uoknor.edu Subject: Re: reply to Gongquan Chen Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Status: O I wrote: |> Rob Robertson is in charge of Berkeley USENET among many other things |> here at Berkeley. He's done a great service to Berkeley and to us |> by creating alt.chinese.computing (a few months ago) and chinese.* |> now. The last sentence could be misleading and should read as: "He's done a great service to Berkeley by accomplishing many things including the SLIP/PPP service that most of us here at Berkeley now enjoy and to us by creating alt.chinese.computing (a few months ago) and chinese.* now." Sorry for being careless. -- Pinghua on Thu 5/19/94, 14:14:50 From simpson@math.psu.edu Thu May 19 16:48:49 1994 Return-Path: Received: from leibniz.math.psu.edu ([146.186.130.2]) by ifcss.org (4.1/IFCSS-Mailer) id AA16289; Thu, 19 May 94 16:48:49 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA13065; Thu, 19 May 1994 17:46:42 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Thu, 19 May 1994 17:46:40 -0400 Date: Thu, 19 May 1994 17:46:40 -0400 Message-Id: <199405192146.RAA25464@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: Yuan Jiang's concerns about 7-bit Big5 and chinese.* In-Reply-To: <199405191636.MAA10425@leibniz.math.psu.edu> References: <199405191636.MAA10425@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > Pinghua, I hope chinese.* stays a clear mind on this issue. > This 7-bit<->Big5 issue is to satisfy the need of chinese.*, > which in turn is to satisfy Big5 users. But the majority > of the big5 users are in taiwan. The issue concerns mostly > in what code (HZ+ or what) Taiwan to interface with other > usenet communities wh can only use a 7-bit code. > > So, we should let people from Taiwan to decide on this issue. > If the 7-bit <-> big5 convertion is done at the usenet gateways, > people in charge of these gateways, such as C. H. Huang and Shinn, > have the crucial words on this matter. This is the simpliest case. > If the convertion is done by each user's application, the > software developers in Taiwan have the crucial word. In this case, > C. H. Huang and Shinn, etc are the key persons to bridge the > decision making. > > chinese.* despertly needs a 7-bit<-> convertion scheme. Yet, > we are not here to decide on a good scheme, but to decide a > workable one which can live. Such a scheme's life depends on > the key persons' support (gateway operators and application > programmers). > > Everyone says Big5 is a lossy code. Yet we let it be imposed > on us because the big number of Taiwan users.us. We should let > people in Taiwan to decide what 7-bit<->Big5 scheme to impose > on us. > CNS is a code imposed by Taiwan government. But HZ+S and HZ-2 will be > impositions from North America. I sympathize very much with what Yuan Jiang is saying here. Pinghua Young gave a very good response. I can't improve on what Pinghua said. Let me only add and reemphasize a few points. 1. The Taiwanese have done remarkable things with their tw.* newsgroup hierarchy and their BBS system. This experience could be extremely valuable in the international setting. I hope very much that Mr Huang and other Taiwanese members of the chinese.* volunteer committee will speak up and get more involved. 2. Big5 is not only popular in Taiwan. It is also very popular in places like Hong Kong and the United States. Although Big5 is not an international standard in the sense of approval by government bodies and official committees, Big5 is an international standard in terms of popularity and wide acceptance. 3. The chinese.* newsgroup organizers are trying in every possible way to reach out to Taiwanese and other Big5 users. The idea of 7-bit Big5 and HZ+ is only one part of this effort. Despite technical drawbacks, Big5 mirrors are another strong possibility. We very much hope that Taiwanese and other Big5 users will find ways to participate fully in the chinese.* newsgroups and will give us their feedback and advice about how to make such participation easier and more pleasant. -- Steve From shin@hermes.ee.nthu.edu.tw Sat May 21 06:12:31 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA02128; Sat, 21 May 94 06:12:31 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <181986(2)>; Sat, 21 May 1994 19:16:06 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA20721; Sat, 21 May 94 19:08:41 CST Date: Sun, 22 May 1994 09:08:41 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405211108.AA20721@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Subject: ONE problem with HZ+ Status: O There is so little time available such that I did not speak up for almost two weeks. Such situation will continue for a period of time. Just to mention a problem and this problem could be a serious one: When I tried a recent revision of my on line code converter Betty (BE51/2-tty, something like hztty) for viewing HZ+ documents. I found that Big5 users may find that their sentences are divided seriously by those ~1's and ~2's, NOT JUST AT THE BEGIN/END OF A LINE OR WHEN ENGLISH CHARS APPEARS. You may therefore find something like HWA~2XIA~1WEN~2JAI in your news or mail postings. When the Big5 part are mixed with HZ, the situation could be worse. The HZ users are fortunate in that, so long as they insist on using Chinese (HZ/GB) characters throughout a line, they will only see the ESCape seq.s at the BEG/END of the line. The Big5 users will not be so fortunate, since, even though they type a whole line of Chinese (B5) characters, the continuous char. streams will still be broken by the ~ stuffs. This situation, of course, will not happen for those news readers which use HZ/HZ+ style codes as exchange codes and use another internal representation as their internal codes. And it relies heavily on the multi-lingual processing capabilities of the news or mail applications. An external code converter like my Betty and hztty-like programs may not provide a satisfactory solution. Is there any such reader available for the original ACT (HZ) newsgroup, which hides the ~'s from the users? Does Mule provide a multi-lingual news readers? Any other POPULAR news readers? If a POPULAR solution is not immediately available, I guess cross-posting is probably the fastest temp. solution to meet the requirements for the 6/4 creation date of chinese.*. I also think that we should take it more seriously to move toward a multi-lingual-style processing for mixed char sets applications. This means to move toward the Mule-style encoding systems which use standard and registered encoding schemes for char code exchange and do information processing on top of such standard encoding schemes. From @UTARLVM1.UTA.EDU:simpson@math.psu.edu Mon May 23 15:03:01 1994 Return-Path: <@UTARLVM1.UTA.EDU:simpson@math.psu.edu> Received: from UTARLVM1.UTA.EDU ([129.107.1.6]) by ifcss.org (4.1/IFCSS-Mailer) id AA17198; Mon, 23 May 94 15:03:01 CDT Received: from leibniz.math.psu.edu by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Mon, 23 May 94 15:00:50 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id PAA12489; Mon, 23 May 1994 15:59:26 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Mon, 23 May 1994 15:59:24 -0400 Date: Mon, 23 May 1994 15:59:24 -0400 Message-Id: <199405231959.PAA08053@boole.math.psu.edu> To: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Subject: Jing-Shin's problem with HZ+ In-Reply-To: <9405211108.AA20721@hermes.ee.nthu.edu.tw> References: <9405211108.AA20721@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > When I tried a recent revision of my on line code > converter Betty (BE51/2-tty, something like hztty) > for viewing HZ+ documents. I found that Big5 users > may find that their sentences are divided seriously > by those ~1's and ~2's, NOT JUST AT THE BEGIN/END > OF A LINE OR WHEN ENGLISH CHARS APPEARS. I would guess that this problem is due to the fact that you are using your own private version of HZ+. Certainly this problem will not happen if you use HZ+S. In HZ+S, all of the frequently used Big5 Chinese characters are in block 1, so ~1 and ~2 rarely appear within a line of Big5 text. Is your Betty and BE51-2-3-4 material available for anonymous ftp? > Is there any such reader available for the original ACT (HZ) > newsgroup, which hides the ~'s from the users? There are several HZ-based programs which hide ~ escapes from users. I think ZWDOS works in this way. > Does Mule provide a multi-lingual news readers? GNUS, the Emacs news reading program, works under Mule. However, Mule doesn't support HZ. Mule supports several multilingual coding schemes, one of which is a 7-bit system based on ISO-2022-JP-2. This particular coding scheme is suitable for Internet e-mail and news. > I also think that we should take it more seriously to move > toward a multi-lingual-style processing for mixed char > sets applications. This means to move toward the Mule-style > encoding systems which use standard and registered encoding > schemes for char code exchange and do information processing > on top of such standard encoding schemes. I definitely agree. Officially approved, multilingual standards are what I have characterized as the "high road" for Chinese computing. However, we should also anticipate that "low road" solutions such as HZ and Big5 and HZ+ will have their place for some time to come. -- Steve From cyl Mon May 23 15:22:02 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA18108; Mon, 23 May 94 15:22:02 CDT Date: Mon, 23 May 1994 15:22:01 -0500 (CDT) From: Nelson CHIN Sender: Nelson CHIN Reply-To: Nelson CHIN Subject: Re: Jing-Shin's problem with HZ+ To: Stephen G Simpson Cc: chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org In-Reply-To: <199405231959.PAA08053@boole.math.psu.edu> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Status: O On Mon, 23 May 1994, Stephen G Simpson wrote: > > Does Mule provide a multi-lingual news readers? > > GNUS, the Emacs news reading program, works under Mule. > > However, Mule doesn't support HZ. Mule supports several multilingual > coding schemes, one of which is a 7-bit system based on ISO-2022-JP-2. > This particular coding scheme is suitable for Internet e-mail and > news. > -- Steve > actually mule can support hz in gnus using gnusutils.el, which converts HZ to and from mules internal code before reading and posting. get ifcss.org:/software/mule/lisp/gnusutils.el nelson From simpson@math.psu.edu Tue May 24 10:44:28 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA27884; Tue, 24 May 94 10:44:28 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA19508; Tue, 24 May 1994 11:41:39 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 24 May 1994 11:41:37 -0400 Date: Tue, 24 May 1994 11:41:37 -0400 Message-Id: <199405241541.LAA09821@boole.math.psu.edu> To: chinesenet@uclink.berkeley.edu, ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, mule@etl.go.jp Subject: Mule supports HZ In-Reply-To: References: <199405231959.PAA08053@boole.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O I wrote: > > However, Mule doesn't support HZ. Nelson replied: > actually mule can support hz in gnus using gnusutils.el, which converts > HZ to and from mules internal code before reading and posting. > get ifcss.org:/software/mule/lisp/gnusutils.el Sorry, I was wrong, Nelson is absolutely correct. Mule supports HZ (for instance reading and posting on alt.chinese.text) using gnusutil.el. I think it would also be possible to add support for HZ+ (for the chinese.* newsgroup hierarchy, if things go as planned). -- Steve From simpson@math.psu.edu Tue May 24 12:41:59 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA02930; Tue, 24 May 94 12:41:59 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA20188; Tue, 24 May 1994 13:39:51 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 24 May 1994 13:39:49 -0400 Date: Tue, 24 May 1994 13:39:49 -0400 Message-Id: <199405241739.NAA09963@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: Re: the high road and the low road In-Reply-To: References: Reply-To: simpson@math.psu.edu Status: O I wrote: > > On the one hand we have CNS, ISO-2022, and Unicode -- official > > standards with (so far) little popular support. On the other hand we > > have HZ and Big5, which may never become official standards, but are > > very, very popular. Ross replied: > Let's put CNS and Unicode to the side for a moment. I dispute the > assertion that ISO 2022 is less popular or less practical than HZ. > It's only true in the subset of Chinese Computing discussed on CCNET-L. > Out in the wider world lots of people are using forms of ISO 2022, > and a great deal of software supports it, for example the X Windows > system, many X clients including Mule and exterm, many MIME user agents > (mail readers), etc. If ISO-2022 is as popular as HZ and Big5, then why do Chinese-language newsgroups and BBS's always use HZ and Big5 and never use ISO-2022? I'm speaking of Chinese computing, not multilingual computing. I'm sure that ISO-2022 is very popular in the world of multilingual computing. (By the way, Mule has recently added support for the International Phonetic Alphabet.) But from what I've been able to pick up on CCNET etc., the popular standards for ordinary Chinese computer users are Big5, GB, and HZ. It's interesting that all of the examples of ISO-2022 software that Ross mentioned are for Unix only. I don't know of any popular PC or Macintosh software that supports ISO-2022. By constrast, Big5 is well supported on all platforms -- Windows, OS/2, DOS, Unix, .... > Unicode may have to wait a while, but ISO 2022 is practical now. I'm not saying that ISO-2022 isn't practical. Actually I like ISO-2022 a lot. It's a great idea and I have even contributed an add-on to Mule. But ISO-2022 simply isn't popular in the way Big5 and HZ are popular. > Private encoding schemes like HZ are dead ends. HZ isn't private. The HZ specification is available in many places, for example math.psu.edu, /pub/simpson/chinese/hzps/hz.txt. Furthermore, as Charlie Zha said, nothing lasts forever. I think all of us hope that Unicode will eventually take over and ISO-2022 and everything else will die. In that sense ISO-2022 is also a dead end. In the world of Chinese computing, HZ and Big5 are very useful, now, because they are very popular, now. The same cannot be said for ISO-2022. -- Steve From simpson@math.psu.edu Tue May 24 16:05:35 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA12162; Tue, 24 May 94 16:05:35 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA21572; Tue, 24 May 1994 17:03:23 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 24 May 1994 17:03:21 -0400 Date: Tue, 24 May 1994 17:03:21 -0400 Message-Id: <199405242103.RAA10194@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: call for a vote on whether HZ+ should support CNS In-Reply-To: <199405111304.JAA27297@boole.math.psu.edu> References: <199405111304.JAA27297@boole.math.psu.edu> Reply-To: soft-authors@ifcss.org Status: O It is time to make some decisions about HZ+. The matter is somewhat complicated, but I believe we can and should move ahead now. Earlier I wrote: > I was hoping that the soft-authors group would be able to quickly > reach a consensus on an HZ+ standard, HZ+ being defined as HZ plus > some form of 7-bit Big5. > > The discussion became bogged down on two issues: > > (1) What shall be the mapping from the Big5 code space to the > graphical ASCII range, 0x2121-0x7E7E? > > (2) What about CNS? > > There are currently two HZ+ proposals on the table, my HZ+S and and > Ya-Gui Wei's HZ-2. In addition, we are waiting for Jing-Shin Chang to > convert his B5E1 and B5E2 mappings into HZ+ proposals. That will make > four HZ+ proposals altogether. I anticipate that the four proposals > will be in essence the same, except for issue (1). > > I am thinking that we perhaps ought to take a vote now on issue (2). > I feel that we have enough information about CNS to come to a decision > on that aspect of HZ+. All of the current HZ+ proposals currently > support Big5 and do not support CNS. Thus we can phrase the vote as a > choice from the following options: > > A. Add CNS support, in addition to the existing Big5 support. > > B. Drop Big5 support. Support CNS instead. > > C. Forget about CNS for the time being. > > D. Wait for more information on CNS. > > I myself am strongly opposed to option B. If we choose option A, what > I have in mind for HZ+S is to use ~G and ~H as escape sequences for > CNS levels 1 and 2. I choose G and H because they are the registered > ISO-2022 final characters for CNS. > > Before we vote on issue (2), would anybody like to air any additional > views or information? > > -- Steve > I now think that all the necessary information and views on CNS have been aired. I would therefore now like to call for a vote on CNS. If it's all right with everybody, I'll combine options C and D into one option, "Postpone a decision." The three options with respect to HZ+ support for CNS are therefore as follows: A. Add CNS support, in addition to the existing Big5 support. B. Drop Big5 support. Support CNS instead. C. Postpone a decision on CNS for the time being. I personally remain strongly opposed to option B, but that is only my personal view. Could all members of the soft-authors group (that is, everyone who receives this announcement) please send mail to soft-authors@ifcss.org indicating your vote for A, B, or C? The deadline for votes is May 31. I appreciate everyone's cooperation. After this vote on issue (2) is finished, I plan to call for another vote on issue (1) as mentioned above. Perhaps then we will be able to present something that can be called a consensus on HZ+ -- Steve From shin@hermes.ee.nthu.edu.tw Tue May 24 16:46:28 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA13747; Tue, 24 May 94 16:46:28 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182342(2)>; Wed, 25 May 1994 05:49:56 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA24545; Wed, 25 May 94 05:42:30 CST Date: Wed, 25 May 1994 19:42:30 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405242142.AA24545@hermes.ee.nthu.edu.tw> To: shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu Subject: Re: Jing-Shin's problem with HZ+ Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Status: O |> From simpson@math.psu.edu Tue May 24 04:00:35 1994 |> |> Jing-Shin Chang writes: |> > When I tried a recent revision of my on line code |> > converter Betty (BE51/2-tty, something like hztty) |> > for viewing HZ+ documents. I found that Big5 users |> > may find that their sentences are divided seriously |> > by those ~1's and ~2's, NOT JUST AT THE BEGIN/END |> > OF A LINE OR WHEN ENGLISH CHARS APPEARS. |> |> I would guess that this problem is due to the fact that you are using |> your own private version of HZ+. Certainly this problem will not |> happen if you use HZ+S. In HZ+S, all of the frequently used Big5 |> Chinese characters are in block 1, so ~1 and ~2 rarely appear within a |> line of Big5 text. Yes, I am using my private B5E3 (Ah!! Yet Another Big5 Encoding). This version, however, has fewer switching between block 1 and 2 than HZ+S and B5E1/B5E2. The division is the same as Ya-Gui's HZ-2. But the operations are more like B5E2, simply much more simpler. (See my next posting for the detailed spec.) A rough estimation is that about 6% of the lines will have those block 2 characters interrupting the block 1 stream. If they are mixed with other ASCII (and HZ) chars. it is even uncomfortable. HZ+S should have a higher rate of such undesirable switching. Now, I begin to hate the original Big5 designer for not putting more frequently used chars together. (Why only 5401 ? Why not 94*94 ? -^^-) I guess Shih-Kun's tw.chinese.* will be much more popular by the Big5 users if such problems do not have a popular and easy solution. |> |> Is your Betty and BE51-2-3-4 material available for anonymous ftp? |> My Betty is still an ugly girl with a mess of uncleaned codes. So it will not be released now. (Plus I am busy with my projects. Sorry ...) The other B5E1/2/3 are available at hermes.ee.nthu.edu.tw:/shin/hzb5encode/*. |> > Is there any such reader available for the original ACT (HZ) |> > newsgroup, which hides the ~'s from the users? |> |> There are several HZ-based programs which hide ~ escapes from users. |> I think ZWDOS works in this way. My ZWDOS (2.20) uses some unnoticible symbols to replace the escape sequences to make the visual feeling of a user better. I am not sure whether recent versions of ZWDOS have other solutions to this since I did not use ZWDOS for a long time after the ACT.big5 mirror was set up. |> |> > Does Mule provide a multi-lingual news readers? |> |> GNUS, the Emacs news reading program, works under Mule. But I am worry that Mule is too big (4M+ compressed and 15M+ uncompressed for mule-1.1 accessed two weeks ago, excuding patches), and is only available for user with a workstation (am I right?). Many students here only have access to PC-based systems, and they rely on news readers that need not much graphics capability. |> |> However, Mule doesn't support HZ. Mule supports several multilingual |> coding schemes, one of which is a 7-bit system based on ISO-2022-JP-2. |> This particular coding scheme is suitable for Internet e-mail and |> news. |> As indicated by other persons, mule does support HZ/GB. It also support CNS and Big5. One previous posting said it doesn't support CNS but the document and a dialogue between one of the author and me indicates that it does. I will post this dialogue, which is related to 7-bit big5 implementation in Mule later. From shin@hermes.ee.nthu.edu.tw Tue May 24 17:04:18 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA14639; Tue, 24 May 94 17:04:18 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182340(2)>; Wed, 25 May 1994 06:07:51 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA24595; Wed, 25 May 94 06:00:31 CST Date: Wed, 25 May 1994 20:00:31 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405242200.AA24595@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.uga.edu, soft-authors@ifcss.org Subject: B5E3: Yet Another Big5 Encoding Status: O Dear Netter, This posting contains 3 formal proposals for 7-bit Big5 encoding and mixed HZ/ASCII/Big5 encoding scheme (HZ+). The B5E3 part is new. If you are already familiar with B5E1/B5E2, you could jump to B5E3 directly. If this posting appears to be lengthy, I am sorry for that. The characteristics of B5E3 is to use the same division as HZ-2, and use only bit-set/clear mostly except for two blocks of code space, which add constant offset to the high byte. The encoding and decoding could be implemented with only 3 C statements plus the range checking if-statements. If you are not interested in such technical stuffs, please just skip it. Otherwise, don't blame me. -^^-. (Any other comments are welcome.) Shin -^^- ========== [HZ+B5E1/HZ+B5E2/HZ+B5E3] @(#)$Id: hzb5e.spec,v 1.3 1994/05/21 21:19:21 shin Exp $ Three Proposals for Mixed ASCII/GB/Big5 Representation Jing-Shin Chang (shin@hermes.ee.nthu.edu.tw) --- ** all documents mentioned here are available from: hermes.ee.nthu.edu.tw(140.114.28.27):/shin/hzb5encode/* --- 0. Acknowledgements The original work of the HZ specification for representing mixed ASCII/GB character sets are due to Fung-Fung Lee (lee@umunhum.stanford.edu,09/04/1989) and his fellows. (See "HZ.spec" for a detailed specification of the HZ standard.) The escape sequences for switching into GB mode and switching to the default ASCII mode in the proposals are borrowed from the original HZ specification. (*) The idea to extend the current HZ specification are initiated by Professor Stephen G. Simpson (simpson@math.psu.edu). The escape sequences for entering Big5 block 1 and block 2 are borrowed from Professor Simpson's current HZ+S specification (v. 0.6), which is another proposal for mixing ASCII/GB/Big5 character sets. (See math.psu.edu:/pub/simpson/chinese/hzps/hzps.txt for details on HZ+S specifications.) (*) There are other kind persons who directly or indirectly contribute their ideas on discussing or commenting on the advantages or disadvantages of the current proposals. In particular, I had learned much from Ya-Gui Wei's HZ-2, which is yet another very valuable mixed ASCII/GB/Big5 character sets scheme. The B5E3 encoding scheme has borrowed the idea of cutting the high-byte range of 0xa1-0xfe at the middle from HZ-2. All credits concerning the related materials and efforts should be given to the respective authors or contributors. Thanks are due to all these kind persons. I love all of them. (**) --- P.S. (*) If they don't borrow me such escape sequences, I will have big trouble. I will then need to invent my own wheels. -^^- (**) What would happen if I don't love them ?? -^^- --- 1. Motivation Due to the very complex communication environment in the world-wide network systems, there is the demand for communicating with chinese safely. One possible way for safe communication with Chinese characters through the network is to use 7-bit encoding of Chinese characters. There are two major chinese user groups: one of them uses the GB code for their daily life and the other group uses the Big5 code for computer applications. To allow communication between these two groups, we need a way to integrate these two systems into a unified one. We thus need a 7-bit standard for such communication purposes. Currently, there is already a HZ standard for mixing ASCII and GB with 7-bit printable ASCII characters (0x21-0x7e). In this document, I will propose two solutions for representing the Big5 code with 7-bit printable ASCII characters. These two encoding schemes will be called B5encode-1/ B5decode-1 and B5encode-2/B5decode-2, or B5E1 and B5E2, respectively. Such encoding schemes for converting 8-bit Big5 to/from 7-bit Big5, when combined with appropriate escape sequences, will enable us to mix ASCII/ GB/Big5 with 7-bit printable ASCII characters. The mixed character set encoding schemes will be referred to as HZ+B5E1 and HZ+B5E2, respectly. (When there is no confusion, you could still call them B5E1 and B5E2.) 2. The Big5 Map The big5 code set is a two-byte code set with the following code space: /* * @(#) Jing-Shin Chang, EE, NTHU, Hsinchu, Taiwan, ROC. * ASCII: 0x00 - 0x1f - control characters 0x20 - 0x2f - space and symbols 0x30 - 0x39 - digits (0 - 9) 0x3a - 0x40 - symbols (0x20- 0x40 - symbols and digits) 0x41 - 0x5a - upper case 0x5b - 0x60 - symbols 0x61 - 0x7a - lower case 0x7b - 0x7e - symbols 0x7f (127) - DEL Big5: High Bytes: 0x81-0x8d,0x8e-0xa0,0xa1-0xfe (0x01-0x0d,0x0e-0x20,0x21-0x7e, when high bit stripped) Low Bytes: 0x40-0x7e,0xa1-0xfe a0: ' ' | 0x80, a1: '!' | 0x80, fe: '~' | 0x80 High Bytes 00 7e 80 81 8d 8e a0 a1 a3 a4 c6 c9 f9 fa fe ff 00 +----------------------------------------------------------------+ | | | | 3f +----------------------------------------------------------------+ 40 +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+ : | 40-7e: 63 chars * U * . U . * S * # S # ! A ! x U x | : | * 3 * . 2 . * P * # T # ! L ! x 1 x | : | * * . . * C * # D # ! T ! x x | 7e +--------------------*----*-.-------.-*---*-#----#-!----!-x---x--+ 80 (NULL|0x80) 81 +----------------------------------------------------------------+ a1 +--------------------*----*-.-------.-*---*-#---#--!----!-x---x--+ : | a1-fe: 94 chars * U * . U . * * # S # ! A ! x U x | : | * * . . * bf* # # ! ! x x | : | * 3 * . 2 . * c0$ # T # ! L ! x 1 x | : | * * . . * $ # # ! ! x x | : | * * . . * e0$ # D # ! Td5! x x | : | * * . . * # # ! x x | fe +--------------------*----*-.-------.-*-----#---#--!------x---x--+ ff (DEL) SPC: special symbols STD: standard (commonly used) characters ALT: alternative set of (less frequently used) characters U1: user-defined code area #1 U2: user-defined code area #2 U3: user-defined code area #3 */ There are different vender-specific extension to the original Big5 code. Therefore, I will handle the following 4 blocks of codes to cover all such possible extension: Freq-2 (F2): (0xa1-0xc8,0x40-0x7e) Freq-1 (F1): (0xa1-0xc8,0xa1-0xfe) Rare-2 : (0xc9-0xfe,0x40-0x7e) Rare-1 (R1): (0xc9-0xfe,0xa1-0xfe) The 2nd and 3rd user-defined code space (U2 and U3) will not be handled because they have very little chance to be used. In fact, we probably should not even touch such non-uniquely defined code area. Note that: Blocks Freq-1 and Freq-2 contain frequently used characters and symbols. Blocks Rare-1 and Rare-2 contain less frequently or rarely used characters. (We use the suffix '1' to designate the code space with more codes and '2' for the code space that has a smaller area.) 3. Escape Sequences for Switching between Different Character Sets The following escape sequences are used for switching from one character set to another and locked in that character set. The default character set is the ASCII character set. HZ Escape Sequences: (the same as in the original HZ standard) ~{ -- enters GB mode, each double-byte is regarded as a GB character ~} -- enters ASCII mode, each byte is interpreted as an ASCII character with the following two exceptions: ~~ -- represents the '~' character (0x7e) ~\n -- line continuation mark, the characters in the next line should be concatenated with the current line HZ+S Escape Sequences: (the same as in HZ+S, v. 0.6, proposal. However, they could be changed if desirable.) ~1 -- switches into Big5 block 1, each double-byte is regarded as a Big5 character in Block 1 ~2 -- switches into Big5 block 2, each double-byte is regarded as a Big5 character in Block 2 The division of Big5 codes into two blocks is essential since there are only 94x94=8836 double-byte printable ASCII characters (0x21-0x7e) available while there are more than 130K Big5 codes. We will sometimes refer to the Big5 block 1 simply as ~1 and block 2 as ~2 in the following sections. Different division of the original Big5 code space into two blocks and the mapping method from the codes in one block into the (0x21-0x7e,0x21-0x7e) code space determine the characteristics of the transformed 7-bit Big5 codes (hereinafter, Big5/7, in contrast to their 8-bit counterpart, Big5/8). Two proposals are given here based on two different criteria. 4. Mapping Method I: The B5E1 Mapping Method --- (* Based on B5E1/B5encode-1: 1994/05/02, B5encode/B5decode, Rev. 1.2) --- The first way to divide the original code space is shown as follows: original code space ==> transformed code space High Bytes a1 fe 21 7e L 40 ************************ 40 ************************ o * * * * w * Block 2 * ==> * Block 2 * * * * * B 7e ************************ 7e ************************ y a1 ++++++++++++++++++++++++ 21 ++++++++++++++++++++++++ t + + + + e + + + + s + Block 1 + ==> + Block 1 + + + + + + + + + fe ++++++++++++++++++++++++ 7e ++++++++++++++++++++++++ The codes in Block 1 have the 8-th bit set in both their high bytes and low bytes. On the other hand, in Block 2, the 8-th bit of the high bytes are set BUT the 8-th bit of the low bytes are cleared. For such partition, we can map the big5 codes into the printable area simply as follows: 1. clear all the 8-th bits of the high bytes when we encounter a big5 code. 2. clear the 8-th bits of the low bytes only for codes in Block 1 but keep them untouched if the codes belong to Block2 3. use the appropriate escape sequence to distinguish codes in block 1 and block 2 The encoding and decoding for each character requires at most two LOGICAL operations (ANDs or ORs) for clearing or setting the 8-th bits, instead of using multiplications and divisions for address calculation, or using table lookup for adding offsets in the mapping process. Therefore, the encoding and decoding is extremely simple. To implement the encoding and decoding of HZ+B5E1 scheme, we only need two simple finite state machines with a very small number of states. The main operations will be logical ANDs and ORs without using multiplications and divisions or table lookup. The computation load can thus be ignored. And switching between the original and the transformed code space can be done easily with two C statements. Two programs, called "b5encode.c" (rev. 1.2) and "b5decode.c" (rev. 1.2) are available in hermes.ee.nthu.edu.tw:/shin/hzb5encode/b5encode-1 to show the simple implementation. The sample decoder could parse a mixed ASCII/GB/Big5 text in the current decoder implementation, with characters in the GB modes pass-through to the output. At least 1M bytes of big5 files had been tested. Such a 8-bit to 7-bit mapping method is also used in some CNS or EUC-based chinese email implementation. For instance, the SunMicro's implementation of the Chinese/Japanese/Korean email transmission in its Asia Language Environment also used the same mapping process to change the 8-bit codes (Big5 or EUC) to 7-bit streams before the email is deliveried. (The EUC for chinese character set used in Taiwan has a very trivial mapping with CNS-11643/86). The internal codes (8 bit) and the exchange codes (7 bit) for CNS/11643-86 also has the same simple mapping. Therefore, we can easily modify the same b5encode and b5decode procedures to support existing implementations in news and email postings. (* The HZ/GB mapping is essentially using bit set/clear too.) The major criticism to such a division is: the switching between Big5 block 1 and block 2 will be frequent, and thus introduces more ~2 escape sequences, and a larger file size. However, for its extremely simple mapping and negligible computation load, people (software contributors, users) may find it a good choice as file size is no more a big deal with extremely cheap storage devices, high communication speeds and data compression protocols of the present day computing environments. Note that: the EUC/8 <-> EUC/7 and CNS/8 <-> CNS/7 does not have such a switching problem since all frequently used characters in EUC or CNS are already arranged in their 1st plane, and less frequently used characters are arranged in the 2nd plane. EUC/8 or CNS/8 here refers to their 8-bit version (e.g., internal code representation), and EUC/7 or CNS/7 refers to their 7-bit version. 5. Mapping Method II: The B5E2 Mapping Method --- (* Based on B5E2/B5encode-2: 1994/05/07, B5encode/B5decode, Rev. 1.4) --- Another way to map the Big5/8 code to Big5/7 code is to divide the code space according to the frequency of use into two blocks, then make the 8-bit bytes into 7-bit bytes. In the second encoding scheme, we therefore designate codes in Freq-1 and Freq-2 area as Block 1 codes, and codes in Rare-1 and Rare-2 areas as Block 2 codes. Under such circumstances, the switching between Block 1 and Block 2 will be reduced to the same amount as any other proposals which divide the Block boundaries in the same way. 5.1. Mapping of Frequently Used Characters It is easy to map codes in Freq-1 (F1) by bit-clear, and map codes in Freq-2 (F2) by adding a constant X-offset to the X-coordinate (high byte) without changing the Y-coordinate (low byte) of the code. The mapping table is shown as follows: ** Mapping for Frequently Used Chars: (note: all geometries are retained) ** F1 = Freq-1, F2 = Freq-2 High Bytes a1 c8 L 40 +---------+ o | F2 | w | | 7e +---------+ B a1 +---------+ +---------+ 21 y | F1 | => | F1 | t | | | | 40 +---------+ e | | | | | F2 | s | | | | | | fe +---------+ +---------+ 7e +---------+ 21 48 49 70 ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- F1 a1-c8 a1-fe CLR CLR 21-48 21-7e F2 a1-c8 40-7e -58 NOP 49-70 40-7e ---------------------------------------------------------- ================================================================ Notations: Hi: Hi byte ranges (in Big5 code) Lo: Lo byte ranges (in Big5 code) hi: Hi byte ranges (in B5E2 representation) lo: Lo byte ranges (in B5E2 representation) Hop: operations on Hi bytes (Big5->B5E2) Lop: operations on Li bytes (Big5->B5E2) NOP: no operation CLR: clear the 8th bit -YY: add a negative offset 0xYY * all numbers are HEX numbers * ================================================================ 5.2. Mapping of Less Frequently or Rarely Used Characters The mapping of codes in the rarely used area is also easy. The largest code space Rare-1 (R1) can still be changed to 7-bit simply by clearing the 8th bits of the high bytes and low bytes. Also, most of the codes in Rare-2 region (i.e., the 40-column R2 region as shown in the next figure) can be changed to 7-bit by adding a constant X-offset to the X-coordinates without changing the Y-coordinates. The mapping table is shown as follows: ** Mapping for Rarely Used Chars : ** c9 f0 f1 fe 40 +------------------+ +------+ 40 | | |R3.1 | | | | | | | +------+ 5e | R2 | +------+ 5f | | |R3.2 | | | | | | | +------+ 7d 7e +------------------+ +----+++ <- R3.3 + R3.4 ( 0xfd7e, 0xfe7e ) f1 fc fd fe a1 +---------------------------+ | | | | | | | | | | | R1 | | | | | | | | | | | fe +---------------------------+ c9 fe | | V 21 2e 2f 3c 3d 48 21 +------++------++--+ 21 +---------------------------+ 21 (+--+ : R3.3) |R3.1 ||R3.2 |++ 22 | | 22 (++ : R3.4) | || | 3e | | 3f +------++------+ 3f | | 40 +------------------+ | | | | | | | | | R1 | | | | | | R2 | | | | | | | | | | | | | | | 7e +------------------+ +---------------------------+ 21 48 49 7e ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- R1 c9-fe a1-fe CLR CLR 49-7e 21-7e R2 c9-f0 40-7e -a8 NOP 21-48 40-7e R3.1 f1-fe 40-5e -d0 -1f 21-2e 21-3f R3.2 f1-fe 5f-7d -c2 -3e 2f-3c 21-3f R3.3 f1-fc 7e -b4 -5d 3d-48 21 R3.4 fd-fe 7e -c0 -5c 3d-3e 22 ---------------------------------------------------------- Note that: the transformed X-coordinates of R1+R2 occupy all the high byte ranges of 0x21-0x7e. Therefore, we are left with 14 columns of codes in the rarely used region Rare-2 (i.e., 882 codes, BUT, including 5 columns of user defined codes). All these columns can be further divided into two small rectangular areas and one small row of codes so that they are moved to the unused part of the (0x21-0x7e,0x21-0x7e) area simply by adding a constant (X,Y) translation vector to each rectangular region. --- ** We can discard the R3.4 region, which contains only two user-defined codes in practical implementation. --- If we regard bit-clear (&0x7f or -0x80) and NOP (no operation) as two special cases for adding constant offsets, then the above mapping can be implemented with 8 (or 7) offset vectors. We can therefore map the 8-bit big5 codes into 7-bit stream without using any multiplication or division operations or resorting to any table lookup. (Note that at most 6 byte-comparisons are needed to decide which region the current code belongs to. Mostly, only about 3 comparisons are needed.) Since the major operation for the largest blocks in F1, F2, R1 and R2 is the bit-clear operation, as in the B5E1 proposal, the computation load is thus comparable with the B5E1 approach while reducing the block switching frequency to a very negligible extent. The b5encode.c and b5decode.c for this proposal is accessible from the ftp site: hermes.ee.nthu.edu.tw:/shin/hzb5encode/b5encode-2 to show the simple implementation. When choosing the 'appropriate blocks', the philosophy is to retain the geometry of the blocks of continuous codes and their sizes as closely to their BIG5 origins as possible, and the largest blocks are fit to the printable area first with the simplest possible operations. With this strategy, only those less frequently used chars, whose low bytes lie in the 0x40-0x7e range are actually divided somewhat differently from their Big5 origins. 6. Mapping Method III: The B5E3 Mapping Method (YAB5E) Another method, which retains the simplicity of encoding/decoding by bit-clear/set and removes the undesirable block switching effects, is described here as B5E3 (or YAB5E, namely, Yet Another Big5 Encoding scheme. Since the triggering idea is borrowed from Ya-Gui Wei's HZ-2, it could also be called YA-gui's value-added Big5 Encoding scheme. -^^-) The major idea is to divide the code space into two blocks so that half of the high-byte ranges (0xa1-0xcf) are designated as Block 1 and the remaining half (0xd0-0x7e) are designated as Block 2 of the Big5 code. With such an arrangement, the encoding and decoding are both easy and symmetric. The mapping tables are shown as follows. High Bytes a1 cf L 40 +---------+ o | F2+ | w | | 7e +---------+ B a1 +---------+ +---------+ 21 y | F1+ | => | F1+ | t | | | | 40 +---------+ e | | | | | F2+ | s | | | | | | fe +---------+ +---------+ 7e +---------+ 21 4f 50 7e ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- F1+ a1-cf a1-fe CLR CLR 21-4f 21-7e F2+ a1-cf 40-7e -51 NOP 50-7e 40-7e ---------------------------------------------------------- High Bytes d0 fe L 40 +---------+ o | R2+ | w | | 7e +---------+ B a1 +---------+ +---------+ 21 y | R1+ | => | R1+ | t | | +---------+ 40 | | e | | | R2+ | | | s | | | | | | fe +---------+ +---------+ 7e +---------+ 7e 21 4f 50 7e ---------------------------------------------------------- Hi Lo Hop Lop hi lo ---------------------------------------------------------- R1+ d0-fe a1-fe CLR CLR 50-7e 21-7e R2+ d0-fe 40-7e -af NOP 21-4f 40-7e ---------------------------------------------------------- For codes whose low bytes lie in the range of 0xa1-0xfe (F1+ and R1+), the 8-th bit of their high bytes and low bytes are cleared during the encoding process. For codes with low bytes in the range of 0x40-0x7e (F2+ and R2+), their high bytes are shifted by adding a constant offset so that they fall into the remaining half of the printable ASCII area. The offsets for Block 1 and Block 2 codes are -0x51 and -0xaf respectively. The low bytes need no change since they are already within 0x21-0x7e. The decoding is also easy and symmetric. For the left half of the Block 1 codes in the printable ASCII area (0x21-0x4f) and Block 2 codes in the right half (0x50-0x7e), the 8-th bit of their high bytes and low bytes are set to '1'. For Block 1 codes in the right half (0x50-0x7e) and Block 2 codes in the left half (0x21-0x4f), the high bytes are shifted by adding a constant offset (+0x51 and +0xaf, respectively) without changing the low bytes. The sample implementation of the b5encode.c and b5decode.c for this proposal is accessible from the ftp site: hermes.ee.nthu.edu.tw:/shin/hzb5encode/b5encode-3 to show the simple implementation. The encoding/decoding process is almost as easy as for B5E1, except for different numbers of range checking. (In B5E1, we check whether a code belongs to the upper half or lower half; in B5E3, we also check whether a code belongs to the right half or left half to decide the mapping operation: bit-clear or byte offset.) In the current arrangement of block boundary, all frequently used characters plus several columns of less frequently used characters (with high byte in the range of 0xc9-0xcf) are designated as Block 1. The remaining less frequently used characters are designated as Block 2. As a result, as indicated by Ya-Gui, the switching operations is even less than the case where the block boundary is defined at the start of the less frequently used characters. (This fact is also observed in a test over about 1M big5 files.) 7. Pros and Cons There are major criticisms on not preserving the sorting sequence of the original codes EXPLICITLY in B5E2. Since the current design is aimed at acting as a code exchange media, not for information processing directly over the 7-bit exchange codes (especially for the computation environments of the Big5 users), the chances to use such sorting sequence is close to zero. (In fact, it is almost equally true in the original Big5 code space for using such sorting sequence.) Even in the very rare cases, the switching between the two code spaces could be done within bit or byte changes. Therefore, it means little for designing a mapping scheme. There are also concerns for future ISO-2022 compatibility in terms of the division of the Big5 code. However, the designation of which characters as block 1 chars and which characters as block 2 chars are not yet decided for Big5 except for some private designations. So it will probably depend on what division is widely accepted after the Big5/7 standard is settled by the chinese computing societies. The major discussions on the advantages or disadvantages of the above proposals are accessible at: hermes.ee.nthu.edu.tw:/shin/hzb5encode/doc/disc.b5e Since B5E3 is new to the discussion list, further comments from all parties will be added to the file. Other discussions on 7-bit representations of Big5 code are also available in the same directory. You can read them as ordinary emails, say, by using 'mail -f disc.b5e' in the UNIX environment for all such files. ----------------------------------------------------------------------- Natural Language Processing Lab Department of Electrical Engineering Jing-Shin Chang National Tsing-Hua University shin@hermes.ee.nthu.edu.tw Hsinchu, Taiwan 30043, R.O.C. ----------------------------------------------------------------------- From shin@hermes.ee.nthu.edu.tw Tue May 24 17:37:06 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA15902; Tue, 24 May 94 17:37:06 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182344(2)>; Wed, 25 May 1994 06:40:34 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA24665; Wed, 25 May 94 06:33:16 CST Date: Wed, 25 May 1994 20:33:16 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405242233.AA24665@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.uga.edu, soft-authors@ifcss.org Subject: README.1st: the next 9 postings for 7-bit Big5 in MULE Status: O Dear Netters, There will be 9 (!!) postings concerning the 7-bit Big5 implementations in MULE (entitled: [Dialogue n/9]: 7-bit Big5 in Mule). These postings belong to a dialogue between Dr. Ken'ichi Handa and me about two weeks ago when Professor Simpson gave me some advices to know something like Mule. (Dr. Handa is one of the authors of Mule.) I have the permission from Dr. Handa for posting them in the net. Thanks. The postings are arranged in their original email form so that you can easily follow the dialogue, or delete them if you find that such a dialogue is a junk. The contents are somewhat (but not very) technical, so if you are not interested in such technical stuffs, just skip them. Some documents mentioned in the dialogue are available at hermes.ee.nthu.edu.tw:/shin/hzb5encode/doc. Personally, I think the Mule's way is a good direction for mixing chinese coding systems (though not multi-lingual -^^-). Probably we should evaluate how far is such a "high road" from what we are considering now for the HZ+ carefully. Shin -^^- From khsc@SynOptics.COM Tue May 24 18:40:33 1994 Return-Path: Received: from SynOptics.COM ([134.177.1.95]) by ifcss.org (4.1/IFCSS-Mailer) id AA18107; Tue, 24 May 94 18:40:33 CDT Received: from taichi (taichi.synoptics.com) by SynOptics.COM (4.1/SMI-4.1) id AA17911; Tue, 24 May 94 16:37:55 PDT Received: by taichi (4.1/2.0N) id AA04560; Tue, 24 May 94 16:37:54 PDT Message-Id: <9405242337.AA04560@taichi> Date: Tue, 24 May 94 16:37:54 PDT From: Simon Chow ~{V\9zP[~} To: soft-authors@ifcss.org Subject: Re: call for a vote on whether HZ+ should support CNS Status: O I vote for option C /Simon Chow From thssjyh@iitmax.acc.iit.edu Tue May 24 18:53:23 1994 Return-Path: Received: from iitmax.acc.iit.edu by ifcss.org (4.1/IFCSS-Mailer) id AA18410; Tue, 24 May 94 18:53:23 CDT Received: by iitmax.acc.iit.edu id AA03221 (5.64+/IDA/NIU-1.3.1 for soft-authors@ifcss.org); Tue, 24 May 94 18:51:00 -0500 From: Jianqing Hu Message-Id: <9405242351.AA03221@iitmax.acc.iit.edu> Subject: Re: call for a vote on whether HZ+ should support CNS To: soft-authors@ifcss.org Date: Tue, 24 May 1994 18:50:58 -17634834 (CDT) In-Reply-To: <199405242103.RAA10194@boole.math.psu.edu> from "Stephen G Simpson" at May 24, 94 05:03:21 pm X-Mailer: ELM [version 2.4 PL21] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 210 Status: O Option C is my vote. Jianqing Hu | Internet: thssjyh@iitmax.acc.iit.edu ECE Dept. Illinois Inst. of Tech. | Chicago, IL 60616 | Office phone: 312-567-6971 From mcpong@cs.ust.hk Tue May 24 20:01:02 1994 Return-Path: Received: from cssu2.cs.ust.hk by ifcss.org (4.1/IFCSS-Mailer) id AA20917; Tue, 24 May 94 20:01:02 CDT Received: by cssu2.cs.ust.hk (4.1/SMI-4.1) id AA08158; Wed, 25 May 94 08:58:57 HKT Date: Wed, 25 May 94 08:58:57 HKT From: mcpong@cs.ust.hk (Dr. Man-Chi PONG) Message-Id: <9405250058.AA08158@cs.ust.hk> In-Reply-To: Stephen G Simpson "call for a vote on whether HZ+ should support CNS" (May 25, 8:10) X-Mailer: Mail User's Shell (7.2.5 10/14/92) To: soft-authors@ifcss.org Subject: Re: call for a vote on whether HZ+ should support CNS Status: O I vote for C: forget about CNS for the time being. -- mcpong -- in GB2312-80:~{ÅÓÃñÖÎ~}, in Big5: Ãe¥Áªv From bran@monod.biol.mcgill.ca Tue May 24 20:37:14 1994 Return-Path: Received: from sifon.CC.McGill.CA by ifcss.org (4.1/IFCSS-Mailer) id AA21963; Tue, 24 May 94 20:37:14 CDT Received: from monod.biol.mcgill.ca (monod.Biol.McGill.CA [132.206.226.16]) by sifon.CC.McGill.CA (8.6.8/8.6.6) with SMTP id VAA26549 for ; Tue, 24 May 1994 21:35:12 -0400 Received: by monod.biol.mcgill.ca (4.1/SMI-4.1) id AA06136; Tue, 24 May 94 21:36:04 EDT Date: Tue, 24 May 1994 21:35:28 -0400 (EDT) From: Bing Ran/~{H=1x~} Subject: Re: call for a vote on whether HZ+ should support CNS To: soft-authors@ifcss.org In-Reply-To: <199405242103.RAA10194@boole.math.psu.edu> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O :C Bing Ran From mcevilly@netcom.com Wed May 25 01:31:05 1994 Return-Path: Received: from netcom.com (netcom8.netcom.com) by ifcss.org (4.1/IFCSS-Mailer) id AA01802; Wed, 25 May 94 01:31:05 CDT Received: by netcom.com (8.6.8.1/SMI-4.1/Netcom) id XAA23879; Tue, 24 May 1994 23:29:12 -0700 From: mcevilly@netcom.com (Carlos McEvilly) Message-Id: <199405250629.XAA23879@netcom.com> Subject: Re: call for a vote on whether HZ+ should support CNS To: soft-authors@ifcss.org Date: Tue, 24 May 1994 23:29:12 -0700 (PDT) In-Reply-To: <9405250058.AA08158@cs.ust.hk> from "Dr. Man-Chi PONG" at May 25, 94 08:58:57 am X-Mailer: ELM [version 2.4 PL23] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 65 Status: O My vote is for approach C. Carlos McEvilly mcevilly@netcom.com From lee@simd.stanford.edu Wed May 25 01:41:43 1994 Return-Path: Received: from simd.stanford.edu by ifcss.org (4.1/IFCSS-Mailer) id AA02209; Wed, 25 May 94 01:41:43 CDT Received: by simd.stanford.edu (5.65/inc-1.0) id AA09463; Tue, 24 May 1994 23:39:41 -0700 Date: Tue, 24 May 1994 23:39:41 -0700 From: lee@simd.stanford.edu (Fung Fung Lee) Message-Id: <9405250639.AA09463@simd.stanford.edu> To: soft-authors@ifcss.org Subject: call for a vote on whether HZ+ should support CNS Status: O I vote for C: forget about CNS for the time being. Fung Fung Lee From xiaokun@stats.gla.ac.uk Wed May 25 02:12:02 1994 Return-Path: Received: from hillhead.cent.gla.ac.uk by ifcss.org (4.1/IFCSS-Mailer) id AA03658; Wed, 25 May 94 02:12:02 CDT Received: from bb-gateway.stats.gla.ac.uk by jess.glasgow.ac.uk with SMTP (PP) id <22354-0@jess.glasgow.ac.uk>; Wed, 25 May 1994 08:09:48 +0100 From: Xiaokun Zhu Date: Wed, 25 May 94 08:13:14 BST Message-Id: <5080.9405250713@jupiter.stats> To: soft-authors@ifcss.org Subject: Re: call for a vote on whether HZ+ should support CNS Status: O My vote is for approach C. ---xiaokun Zhu From simpson@math.psu.edu Wed May 25 10:21:04 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA21664; Wed, 25 May 94 10:21:04 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA28391; Wed, 25 May 1994 11:18:37 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Wed, 25 May 1994 11:18:35 -0400 Date: Wed, 25 May 1994 11:18:35 -0400 Message-Id: <199405251518.LAA11490@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: Jing-Shin's problem with HZ+ In-Reply-To: <9405242142.AA24545@hermes.ee.nthu.edu.tw> References: <9405242142.AA24545@hermes.ee.nthu.edu.tw> Reply-To: simpson@math.psu.edu Status: O Jing-Shin Chang writes: > Yes, I am using my private B5E3 (Ah!! Yet Another Big5 Encoding). > This version, however, has fewer switching between block 1 and 2 > than HZ+S and B5E1/B5E2. > The division is the same as Ya-Gui's HZ-2. But the operations > are more like B5E2, simply much more simpler. > (See my next posting for the detailed spec.) > A rough estimation is that about 6% of the lines will have > those block 2 characters interrupting the block 1 stream. > If they are mixed with other ASCII (and HZ) chars. it is > even uncomfortable. How did you arrive at this 6 percent figure? I think that the 5401 more frequently used characters in Big5 block 1 are adequate for daily use and account for 99.9 percent of characters in ordinary text. I think your 6 percent figure is much too high. > HZ+S should have a higher rate of such undesirable switching. No, this hardly ever happens in HZ+S. The only exception is ancient poetry or classics, where some very unusual characters sometimes appear. > I guess Shih-Kun's tw.chinese.* will be much more popular by the Big5 > users if such problems do not have a popular and easy solution. Unfortunately, tw.chinese.* will be in 8-bit Big5. It will therefore not be available to users outside Taiwan whose news feeds are not 8-bit clean. HZ+ can provide an easy 7-bit clean solution to this problem, by means of a straightforward mapping of 8-bit Big5 into the 7-bit code space. Despite your problems with your BE51-2-3, there are no technical difficulties with HZ+. The only difficulties come about because of confusion over CNS and mapping schemes. If you think that HZ+ is a bad idea, why do you keep flooding us with alternative mapping schemes for HZ+? -- Steve From zhujun@snakemail.hut.fi Wed May 25 10:23:52 1994 Return-Path: Received: from vinkku.hut.fi by ifcss.org (4.1/IFCSS-Mailer) id AA21735; Wed, 25 May 94 10:23:52 CDT Received: from lk-hp-1.hut.fi (lk-hp-1.hut.fi [130.233.48.32]) by vinkku.hut.fi (8.6.8.1/8.6.7) with ESMTP id SAA27494 for ; Wed, 25 May 1994 18:21:51 +0300 From: Jun Zhu Received: (zhujun@localhost) by lk-hp-1.hut.fi (8.6.8.1/8.6.7) id SAA19735 for soft-authors@ifcss.org; Wed, 25 May 1994 18:21:50 +0300 Date: Wed, 25 May 1994 18:21:50 +0300 Message-Id: <199405251521.SAA19735@lk-hp-1.hut.fi> Apparently-To: soft-authors@ifcss.org Status: O SUBSCRIBE,PLS From jiang Wed May 25 12:28:08 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA28231; Wed, 25 May 94 12:28:08 CDT Date: Wed, 25 May 94 12:28:08 CDT From: jiang (Yuan Jiang) Message-Id: <9405251728.AA28231@ifcss.org> To: soft-authors Subject: Re: call for a vote on whether HZ+ should support CNS Status: O >A. Add CNS support, in addition to the existing Big5 support. > >B. Drop Big5 support. Support CNS instead. > >C. Postpone a decision on CNS for the time being. > This list is meaningless because we've never discussed whether CNS should be supported in parallel with big5. The issue has been 7-bit CNS vs. other HZ+ (the big5 representation part). From jiang Wed May 25 12:41:12 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA29068; Wed, 25 May 94 12:41:12 CDT Date: Wed, 25 May 94 12:41:12 CDT From: jiang (Yuan Jiang) Message-Id: <9405251741.AA29068@ifcss.org> Subject: 7-bit Big5 and chinese.* Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors Status: O It seems not possible to reach a consensus on what 7-bit Big5 representation code to use. Every side holds its opinion strongly. Chinese.* committee is not in the position to take side. It is to serve the end users. So, I propose let different 7-bit Big5 representation schemes-software compete on chinese.test, and let users to pick the most popular scheme-software for chinese.*. The whole problem is caused by this messy Big5. Let's use the Bit5 way to solve this problem -- let the popular software to set a de facto standard. Yuan From shin@hermes.ee.nthu.edu.tw Wed May 25 13:52:44 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA03410; Wed, 25 May 94 13:52:44 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182293(1)>; Thu, 26 May 1994 02:56:14 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA25842; Thu, 26 May 94 02:48:55 CST Date: Thu, 26 May 1994 16:48:55 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405251848.AA25842@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: Jing-Shin's problem with HZ+ Status: O |> Jing-Shin Chang writes: |> > Yes, I am using my private B5E3 (Ah!! Yet Another Big5 Encoding). |> > This version, however, has fewer switching between block 1 and 2 |> > than HZ+S and B5E1/B5E2. |> > The division is the same as Ya-Gui's HZ-2. But the operations |> > are more like B5E2, simply much more simpler. |> > (See my next posting for the detailed spec.) |> > A rough estimation is that about 6% of the lines will have |> > those block 2 characters interrupting the block 1 stream. |> > If they are mixed with other ASCII (and HZ) chars. it is |> > even uncomfortable. |> |> How did you arrive at this 6 percent figure? I think that the 5401 |> more frequently used characters in Big5 block 1 are adequate for daily |> use and account for 99.9 percent of characters in ordinary text. I |> think your 6 percent figure is much too high. I test it over my collection of HXWZ (around 1992, 1993). The file size is about 1.1M. Yes, the 5401 chars are frequently used chars in our daily life. However, I am not sure how these chars are selected. So the words "frequently used" refers to a particular statistics over a particular text corpus made several years ago. But it may not imply a 99.9 % hit rate of the block 1 chars. I think the sample is adaquate for characterizing the articles to appear in future chinese.* postings. The sample size maybe small. I would like to see testing reports based on, say, all the HXWZ+ACT postings that was archived. (Anyone interested in such a testing ?) Since my professional research is in corpus-based statistics-oriented natual language processing, I am always very careful when reporting any such figures. |> |> > HZ+S should have a higher rate of such undesirable switching. |> |> No, this hardly ever happens in HZ+S. The only exception is ancient |> poetry or classics, where some very unusual characters sometimes |> appear. Yes, HZ-S of course has a higher probability to hit the block two chars than HZ-2 and B5E3, since HZ-S include 5401 common chars (plus 441 special chars) in Block 1 and 7652 less common chars in block two while HZ-2 and B5E3 has 5401+441+8*157=7089 chars in block one and 7652-8*157=6396 chars in block. In particular, the HZ-S block 1 chars is a proper subset of the HZ-2 and B5E3 block 1 chars. I don't think this fact need any further testing. It is fair to say that the switching frequency has the following order: B5E1 > B5E2 = HZ-S > HZ-2 = B5E3 Anybody against this statement? Those interrupted chars I found are mostly from ancient articles, names food names, etc., which maybe typical in ACT and HXWZ from my experience. |> |> > I guess Shih-Kun's tw.chinese.* will be much more popular by the Big5 |> > users if such problems do not have a popular and easy solution. |> |> Unfortunately, tw.chinese.* will be in 8-bit Big5. It will therefore |> not be available to users outside Taiwan whose news feeds are not |> 8-bit clean. HZ+ can provide an easy 7-bit clean solution to this |> problem, by means of a straightforward mapping of 8-bit Big5 into the |> 7-bit code space. Yes, I never against a 7-bit channel for communication. I am just pointing out a fact of user's concern. In fact, I do think that most people here will choose the tw.chinese.* if they were setup. Today, I get several posting to chinese.test which are sent in Big5/8 from outside Taiwan (e.g., one posting was sent by cyl@ifcss.org). If the software for the 7-bit channel is not easy to update or good enough or fit a particular user's habit and he/she has access to tw.chinese.*, the choice is obvious. Isn't it? |> |> Despite your problems with your BE51-2-3, there are no technical |> difficulties with HZ+. The only difficulties come about because of |> confusion over CNS and mapping schemes. Again, what I had pointed out here (interrupted chars) is not unique to my B5E1/2/3 but also to the other proposals. Agree? I don't think it has anything to do with CNS either. |> |> If you think that HZ+ is a bad idea, why do you keep flooding us with |> alternative mapping schemes for HZ+? Did I say HZ+ is a bad idea? Does "flooding us with alternative mapping schemes" means "think that HZ+ is a bad idea"? Who are "us"? In Internet it is appropriate to skip any junk mail if you think so. I think my positions for posting such alternative mapping schemes are to provide other views for a task based on different criteria, not just for fun. If we think that we can have only one choice, why not use Mule Big5 directly. In fact, I may have different criteria for different aspects of the encoding schemes, but I did not strongly against any proposals we had. My design was directed toward "simple, fast and easily described 100% invertable conversion algorithm" (Oop!! Fung-Fung does not seem to gave a correct rank on this criteria for B5E1, and now for B5E3 -^^-). The choice is of course not based on my personal preference. The software contributors have absolute rights to support what they like. Also, I am more concern of the user's feeling and thus the user interface. The encoding issue is not my only concern. As a Big5 user, I think it is appropriate for me to tell those potential software contributors to consider those problems I found. |> |> -- Steve |> As a final word, I think your HZ-S proposal is a popular one so don't worry about that -^^-. Shin -^^- From CAI@neurophys.wisc.edu Wed May 25 14:20:20 1994 Return-Path: Received: from neurophys.wisc.edu (mve.neurophys.wisc.edu) by ifcss.org (4.1/IFCSS-Mailer) id AA05679; Wed, 25 May 94 14:20:20 CDT Received: from VMSmail by neurophys.wisc.edu; Wed, 25 May 94 14:18 CST Message-Id: <24052514181007@neurophys.wisc.edu> Date: Wed, 25 May 94 14:18 CST From: CAI@neurophys.wisc.edu Subject: Re: call for a vote on whether HZ+ should support CNS To: soft-authors@ifcss.org X-Vms-To: IN%"soft-authors@ifcss.ORG" Status: O my vote: C. Postpone a decision on CNS for the time being. Yidao CAI From Tuya@ccmail.uoregon.edu Wed May 25 15:23:22 1994 Return-Path: Received: from phloem.uoregon.edu by ifcss.org (4.1/IFCSS-Mailer) id AA09328; Wed, 25 May 94 15:23:22 CDT Received: from ccmail.uoregon.edu (ccmail-mailhost.uoregon.edu [128.223.60.105]) by phloem.uoregon.edu (8.6.9/8.6.9) with SMTP id NAA25062; Wed, 25 May 1994 13:21:18 -0700 Received: from cc:Mail by ccmail.uoregon.edu id AA769897289; Wed, 25 May 94 13:21:29 pst Date: Wed, 25 May 94 13:21:29 pst From: "Tuya" Message-Id: <9404257698.AA769897289@ccmail.uoregon.edu> To: jiang@ifcss.org (Yuan Jiang) Cc: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Subject: Re: 7-bit Big5 and chinese.* Status: O The whole problem is caused by this messy Big5. Let's use the Bit5 way to solve this problem -- let the popular software to set a de facto standard. Luan4 shi4 ying1 xiong2 qi3 4 fang1. This is very democratic and capitalistic, and hopefully not very chaotic :-). From lai@apple.com Wed May 25 18:49:44 1994 Return-Path: Received: from apple.com by ifcss.org (4.1/IFCSS-Mailer) id AA19273; Wed, 25 May 94 18:49:44 CDT Received: from [17.203.80.11] by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA13163; Wed, 25 May 94 16:47:20 -0700 for soft-authors@ifcss.org Date: Wed, 25 May 94 16:47:20 -0700 Message-Id: <9405252347.AA13163@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: call for a vote on whether HZ+ should support CNS Status: O >I now think that all the necessary information and views on CNS have >been aired. I would therefore now like to call for a vote on CNS. If >it's all right with everybody, I'll combine options C and D into one >option, "Postpone a decision." The three options with respect to HZ+ >support for CNS are therefore as follows: > >A. Add CNS support, in addition to the existing Big5 support. > >B. Drop Big5 support. Support CNS instead. > >C. Postpone a decision on CNS for the time being. > >I personally remain strongly opposed to option B, but that is only >my personal view. > I think the question is loaded. Nobody is talking about droping Big5 support, we are talking about using CNS as the exchange code for Big5 internal code. It is very interesting to look at the dialogue of 7-bit Big5 on Mule. Ken'ichi HANDA said "By the way, I'll completely change the treatment of Big5 in the next version of mule (base on emacs19). I'll adopt CNS11643 as a internal representation of Big5, which means that Big5 and CNS11643 are the same character set but the different encoding scheme. Then we can use official final character `G' and `H' for both Big5 and CNS11643." Since one of the main justification of using Mule 7 bit Big5 is that it is already been used in Mule. However, the author of Mule now favors using CNS as a representation for Big5. This is what I have been saying all along, and Mr Handa has independently come to the same conclusion. Wouldn't it be funny and a step forward if we now propose using Mule Big5 when even the author of Mule is going the way of CNS? Actually I have no problem of postponing the decision of CNS as long as it does not meant it will be pushed aside, and it is considered as just another candidate for 7 bit Big5 exchange code. Increasingly I doubt HZ+? is the way to go at all. Maybe we should discuss that first because then the CNS issue may not even show up at all. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From yawei@cs.indiana.edu Thu May 26 10:07:48 1994 Return-Path: Received: from moose.cs.indiana.edu by ifcss.org (4.1/IFCSS-Mailer) id AA19141; Thu, 26 May 94 10:07:48 CDT Message-Id: <9405261507.AA19141@ifcss.org> Received: by moose.cs.indiana.edu (5.65c/9.4jsm) id AA12147; Thu, 26 May 1994 10:05:08 -0500 Date: Thu, 26 May 1994 10:05:08 -0500 From: "Ya-Gui Wei" To: ccnet-l@uga.cc.uga.edu, shin@hermes.ee.nthu.edu.tw, simpson@math.psu.edu, soft-authors@ifcss.org Subject: Re: Jing-Shin's problem with HZ+ Status: O >|> > A rough estimation is that about 6% of the lines will have >|> > those block 2 characters interrupting the block 1 stream. >|> > If they are mixed with other ASCII (and HZ) chars. it is >|> > even uncomfortable. 6% is really low. I was expecting the number to be much higher. I thought probably a block-2 character can be seen every 2 to 3 lines (which is acceptable to me). >|> How did you arrive at this 6 percent figure? I think that the 5401 >|> more frequently used characters in Big5 block 1 are adequate for daily >|> use and account for 99.9 percent of characters in ordinary text. I >|> think your 6 percent figure is much too high. > >I test it over my collection of HXWZ (around 1992, 1993). The file size >is about 1.1M. Yes, the 5401 chars are frequently used chars in our >daily life. However, I am not sure how these chars are selected. So the HXWZ's are edited in GB and then converted to big5. Most infrequently used chars would be underrepresented in HXWZ since they don't exist in GB. From @dispatch.demon.co.uk:JD@eremita.demon.co.uk Thu May 26 14:20:03 1994 Return-Path: <@dispatch.demon.co.uk:JD@eremita.demon.co.uk> Received: from post.demon.co.uk by ifcss.org (4.1/IFCSS-Mailer) id AA29657; Thu, 26 May 94 14:20:03 CDT Received: from dispatch.demon.co.uk by post.demon.co.uk id aa08966; 26 May 94 20:01 GMT-60:00 Received: from eremita.demon.co.uk by dispatch.demon.co.uk id aa13306; 26 May 94 19:53 GMT-60:00 X-Sender: (Unverified) Mime-Version: 1.0 Content-Type: text/plain; charset="macintosh" Content-Transfer-Encoding: quoted-printable X-Mailer: Eudora 2.0.2 Date: Thu, 26 May 1994 19:54:07 +0100 To: simpson@math.psu.edu From: John Delacour Subject: HZ+S needs a compiler for Mac Cc: soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Message-Id: <9405261954.aa13306@dispatch.demon.co.uk> Status: O Steve, As you know I have made a basic compilation of your code for HZ+ and have played about with it a bit. No doubt if I had got past the first chapter of K&R or had any pretensions at all to be a programmer I could have produced a nice little application for Mac users but I'd be ashamed to propagate the minimal thing I have compiled just by adding the ANSI library and your UNIX code. Nevertheless it is good enough for testing purposes. Haven't you got anyone around you who knows the Mac and can add the very few necessary lines of code in your program that would be necessary to produce a basic application for the Mac - but not as basic as mine !! I run the program and either type or paste the Chinese into the console to get alternate lines of input and output as the output, if you see what I mean -- like:- [sorry about the QP!] =AA=F8=AC=DB=AB=E4=A1A=A6b=AA=F8=A6w=A1C ~11\4_3)!")d1\)y!$~} =B5=B8=BDn=AC=EE=B3=DA=AA=F7=A4=AB=F8=F3=A1A ~1C?P94r@A1[&k~2q%~1!"~} =B7L=C1=F7=B2Y=B2Y=F0e=A6=E2=B4H=A1C ~1EsX$=3D_=3D_~2bS~1*d@n!$~} =A9t=BFO=A4=A3=A9=FA=AB=E4=B1=FD=B5=B4=A1A ~1.wS:&c/}3)=3DDC;!"~} =A8=F7=B1c=B1=E6=A4=EB=AA=C5=AA=F8=BC=DB=A1C ~1.;<*=3D-'M1)1\OE!$~} =AC=FC=A4H=A6p=AA=E1=B9j=B6=B3=BA=DD=A1C ~15"&*)r1EISDyL'!$~} =A4W=A6=B3=ABC=AD=DF=A4=A7=B0=AA=A4=D1=A1A ~1&9*51f6D&g:n'3!"~} =A4U=A6=B3=D5=F1=A4=F4=A4=A7=AAi=C4i=A1F ~1&7*5~26V~1'V&g0M[u!'~} =A4=D1=AA=F8=B8=F4=BB=B7=BB=EE=AD=B8=ADW=A1A ~1'31\H|M@Mw5{5 Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA00623; Fri, 27 May 94 17:32:01 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA14093; Fri, 27 May 94 14:42:14 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA20649; Fri, 27 May 1994 17:40:06 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 17:40:05 -0400 Date: Fri, 27 May 1994 17:40:05 -0400 Message-Id: <199405272140.RAA19404@boole.math.psu.edu> To: soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Cc: mule@etl.go.jp Subject: Mule's support of Big5 and CNS Reply-To: simpson@math.psu.edu Status: O Ed Lai wrote: > It is very interesting to look at the dialogue of 7-bit Big5 > on Mule. The current version of Mule contains several forms of 7-bit Big5. The current version of Mule does not contain any form of CNS. The topic of Big5 and CNS support in future versions of Mule is now under discussion in the Mule mailing list, mule@etl.go.jp. Kenichi Handa appears to be leaning toward some kind of merged Big5/CNS support. The details of how this will be carried out have not been made clear. Those who would like to follow the entire discussion are encouraged to subscribe to the Mule mailing list. Send your subscription request to mule-request@etl.go.jp. There is at least one very good reason for Mule to support CNS. Namely, Mule is an ISO-2022 application, and CNS is officially registered as an ISO-2022 character set. Big5 is, unfortunately, *not* registered with ISO-2022. Big5 is a popular standard, not an official standard. > one of the main justification of using Mule 7 bit Big5 is that it > is already been used in Mule. I am not sure what Ed means by "Mule 7 bit Big5". Mule's method of support for Big5 may change over time. Ed may be referring, very indirectly, to my HZ+S proposal. That proposal uses a 7-bit representation of Big5 that is identical to one of the representations used in the current version of Mule. However, contrary to what Ed is saying, HZ+S's slight resemblance to Mule is not one of the main justifications of HZ+S. The main justifications of HZ+S are simplicity, elegance, and faithfulness to HZ and Big5. For details, please look at the material which I have made available for anonymous FTP at ftp.math.psu.edu, /pub/simpson/chinese/hzps. A sample implementation of HZ+S is included. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Fri May 27 17:32:34 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA00698; Fri, 27 May 94 17:32:34 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA12631; Fri, 27 May 94 14:05:48 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA20518; Fri, 27 May 1994 17:04:00 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 17:03:58 -0400 Date: Fri, 27 May 1994 17:03:58 -0400 Message-Id: <199405272103.RAA19360@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: reply to Ed's latest points about CNS Reply-To: simpson@math.psu.edu Status: O I wrote in soft-authors: > >The three options with respect to HZ+ support for CNS are > >therefore as follows: > > > >A. Add CNS support, in addition to the existing Big5 support. > > > >B. Drop Big5 support. Support CNS instead. > > > >C. Postpone a decision on CNS for the time being. Ed Lai replied: > I think the question is loaded. Nobody is talking about droping > Big5 support, we are talking about using CNS as the exchange > code for Big5 internal code. Once again Ed is pretending that CNS and Big5 are somehow the same thing. In this way he wants to deny Big5 its identity. The simple fact, which Ed refuses to acknowledge, is that Big5 and CNS are two different coding systems. Some software (e.g. ETen) supports both coding systems in parallel, but I don't know of any software that tries to pretend that they are the same thing. If we apply Ed's theory to HZ+, the practical consequence would be to eliminate Big5 support from HZ+. Ed has not made any specific proposal, but from the vague things he has said, we can deduce that he wants HZ+ to have escape sequences indicating CNS and *not* to have escape sequences indicating Big5. This and nothing else is what is meant by dropping Big5 support in favor of CNS support. Under such a scheme, the only way for HZ+ applications to support Big5 would be indirect, by means of Big5<->CNS translation. Although I have challenged Ed to specify his Big5<->CNS translation algorithm, he has not done so. I think he is trying to hide the fact that Big5<->CNS translation is not only unnecessary but also imperfect. Instead he resorts to vague statements such as "we can use CNS as the exchange code for Big5 internal code". If Ed would like to help us reach a consensus on HZ+, he should vote for option B, since that is what his vague proposal amounts to. As for myself, I remain very much opposed to option B. In contrast to Ed's vagueness, my detailed HZ+S proposal is available for anonymous FTP and specifies a very simple mapping of the Big5 code range 0xA140-FEFE into the 7-bit graphical ASCII range. This mapping is 100 percent well-defined and invertible. To show how HZ+S would work in practice, I have provided a sample implementation. > Actually I have no problem of postponing the decision of > CNS as long as it does not meant it will be pushed aside, > and it is considered as just another candidate for 7 bit > Big5 exchange code. This is *not* what is meant. As I stated very clearly, the purpose of this vote is to decide for the time being whether HZ+ will support CNS. The meaning of option C is to temporarily push CNS aside, i.e. not to support CNS at the present time. Later, if new argments for CNS emerge, we can always add CNS support. But if you want to continue the current debate on whether HZ+ should support CNS, you should definitely *not* vote for option C. > Increasingly I doubt HZ+? is the way to go at all. Maybe > we should discuss that first because then the CNS issue may > not even show up at all. What Ed are now suggesting is to further postpone voting on HZ+. I am opposed to that idea. I am in favor of voting now. My call for voting on HZ+ is based on the fact that there are a significant number of people who favor the creation of an HZ+ standard. If Ed is opposed to the creation of an HZ+ standard, I suggest that he vote "none of the above", i.e. abstain. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From simpson@math.psu.edu Fri May 27 17:37:28 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA01016; Fri, 27 May 94 17:37:28 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA25170; Fri, 27 May 94 08:41:45 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA19025; Fri, 27 May 1994 11:41:19 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 11:41:17 -0400 Date: Fri, 27 May 1994 11:41:17 -0400 Message-Id: <199405271541.LAA18903@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Re: call for a vote on whether HZ+ should support CNS In-Reply-To: <9405251728.AA28231@ifcss.org> References: <9405251728.AA28231@ifcss.org> Reply-To: soft-authors@ifcss.org Status: O Yuan Jiang writes: > >A. Add CNS support, in addition to the existing Big5 support. > > > >B. Drop Big5 support. Support CNS instead. > > > >C. Postpone a decision on CNS for the time being. > > > This list is meaningless because we've never discussed whether > CNS should be supported in parallel with big5. The issue has > been 7-bit CNS vs. other HZ+ (the big5 representation part). I disagree. All of the issues that you mention, and many other issues surrounding CNS, have been thoroughly aired. I several times posted an outline of my tenative ideas about how to add CNS support to HZ+, in parallel with the existing Big5 support. I repeatedly asked for comments on these ideas, and with Carlos McEvilly's help I even wrote some software to implement my ideas. The reaction of the Apple CNS guys was negative. They all asked me *not* to support CNS in parallel with Big5. At the same time, they provided no details of how they would propose to support CNS. Nobody responded to my request for volunteers to help test my CNS software. It turned out that Ed Lai, the most vocal advocate of CNS, is really in favor of GB-12345, a completely different standard about which nobody has any information, not even Ed himself. I urge everyone who is receiving this message to vote for A, B, or C if you have not already done so. We need to reach a consensus on HZ+. -- Steve From simpson@math.psu.edu Fri May 27 17:37:58 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA01092; Fri, 27 May 94 17:37:58 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA23960; Fri, 27 May 94 08:19:14 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA18887; Fri, 27 May 1994 11:18:14 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 11:18:13 -0400 Date: Fri, 27 May 1994 11:18:13 -0400 Message-Id: <199405271518.LAA18867@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Subject: 7-bit Big5 and chinese.* In-Reply-To: <9405251741.AA29068@ifcss.org> References: <9405251741.AA29068@ifcss.org> Reply-To: simpson@math.psu.edu Status: O Yuan Jiang writes: > It seems not possible to reach a consensus on what 7-bit Big5 > representation code to use. I disagree. I think it *will* be possible to reach a consensus on this issue. In the Chinese software authors group, 14 people have endorsed the general idea of HZ+ with some form of 7-bit Big5, and those people have implicitly agreed to abide by the decision of the majority on which 7-bit Big5 mapping scheme to use. I plan to call for a vote on this issue very soon. Right now we are voting on another issue, namely whether HZ+ should support CNS, but as soon as that vote is finished, I will call for another vote, on 7-bit Big5 mapping schemes. > Chinese.* committee is not in the position to take side. It is > to serve the end users. So, I propose let different 7-bit Big5 > representation schemes-software compete on chinese.test, and > let users to pick the most popular scheme-software for chinese.*. I partially agree. My understanding of chinese.test is that it is open to all sorts of tests messages related to the chinese.* newsgroup hierarchy. This means that it is open not only to 7-bit Big5 but also to test postings in 8-bit Big5 or any other coding scheme. On the other hand, the chinese.* organizers need to set some guidelines or standards for postings in the non-test newsgroups. One of these guidelines is that HZ is encouraged. Another guideline is that 8-bit Big5 is discouraged, because many news feeds are not 8-bit clean and we don't want to exclude that group of users. So there is certainly a need for some form of 7-bit Big5. Instead of spreading confusion, I think that we should all try to reach a consensus. One way to help reach a consensus would be to vote your preferences in the votes that are now being taken in the soft authors group. > The whole problem is caused by this messy Big5. No. I agree that Big5 is messy. But I don't agree that Big5 is causing a problem. Big5 is what it is. We can't change it. What we can and must do is reach a consensus on how to communicate. Big5 is a very popular Chinese communication standard, used by millions of people every day. We on the Internet can take advantage of this aspect of Big5, by adapting Big5 to our purposes. That is what I am trying to do with HZ+. -- Steve From @UTARLVM1.UTA.EDU:simpson@math.psu.edu Fri May 27 17:59:12 1994 Return-Path: <@UTARLVM1.UTA.EDU:simpson@math.psu.edu> Received: from by ifcss.org (4.1/IFCSS-Mailer) id AB02233; Fri, 27 May 94 17:59:12 CDT Received: from leibniz.math.psu.edu by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Fri, 27 May 94 11:04:03 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA19158; Fri, 27 May 1994 12:04:06 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 12:04:04 -0400 Date: Fri, 27 May 1994 12:04:04 -0400 Message-Id: <199405271604.MAA18937@boole.math.psu.edu> To: soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: HZ+S needs a compiler for Mac In-Reply-To: <9405261954.aa13306@dispatch.demon.co.uk> References: <9405261954.aa13306@dispatch.demon.co.uk> Reply-To: simpson@math.psu.edu Status: RO John Delacour writes: > Haven't you got anyone around you who knows the Mac and can add the very > few necessary lines of code in your program that would be necessary to > produce a basic application for the Mac - but not as basic as mine !! Sorry, I don't know any Mac programmers that I can coerce or cajole into doing this. But, as you suggest, it should be straightforward. HZ+S is really very simple. -- Steve From @UTARLVM1.UTA.EDU:lai@apple.com Fri May 27 18:23:03 1994 Return-Path: <@UTARLVM1.UTA.EDU:lai@apple.com> Received: from [129.107.1.6] by ifcss.org (4.1/IFCSS-Mailer) id AB03317; Fri, 27 May 94 18:23:03 CDT Received: from apple.com by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Fri, 27 May 94 11:32:42 CDT Received: from [17.203.80.11] by apple.com with SMTP (5.61/8-Oct-1993-eef) id AA20479; Fri, 27 May 94 09:30:30 -0700 for soft-authors@ifcss.org Date: Fri, 27 May 94 09:30:30 -0700 Message-Id: <9405271630.AA20479@apple.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org From: lai@apple.com (Edmund Lai) Subject: Re: 7-bit Big5 and chinese.* Status: RO Let us first look at the so called call for vote >I now think that all the necessary information and views on CNS have >been aired. I would therefore now like to call for a vote on CNS. If >it's all right with everybody, I'll combine options C and D into one >option, "Postpone a decision." The three options with respect to HZ+ >support for CNS are therefore as follows: > >A. Add CNS support, in addition to the existing Big5 support. > >B. Drop Big5 support. Support CNS instead. > >C. Postpone a decision on CNS for the time being. > First of all, this is not the kind of option one uses in an election. The question is loaded. Option B is not drop Big5 support, it is to use CNS as the exchange code to support Big5. In fact on of the main justifcation of Mule Big5 is that Mule is already using Mule Big5 to support Big5, but the author of Mule has already declared that future version will be use CNS to support Big5 instead of using Mule Big5. He is not dropping the support of Big5, he is just using CNS to support Big5. Now let us read on in another post. >Right now we are voting on >another issue, namely whether HZ+ should support CNS, but as soon as >that vote is finished, I will call for another vote, on 7-bit Big5 >mapping schemes. > So a vote to "Postpone a decision on CNS for the time being" implies HZ+ should not support CNS. So it is not sufficent to to use unfair and misleading wording in ballot, one can also change the meaning afterwards. /* Disclaimer: All statments and opinions expressed are my own */ /* Edmund K. Lai */ /* Apple Computer, MS303-3A */ /* 20525 Mariani Ave, */ /* Cupertino, CA 95014 */ /* (408)974-6272 */ zW@h9cOi From xiaofei Fri May 27 18:23:37 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA03352; Fri, 27 May 94 18:23:37 CDT Date: Fri, 27 May 94 18:23:37 CDT From: xiaofei (XiaoFei Wang) Message-Id: <9405272323.AA03352@ifcss.org> To: soft-authors@ifcss.org Subject: log files changed place Cc: ftp-admin Status: RO Since the current discussion has nothing to do with fonts, the log files are moved to new place. -- This directory ifcss.org:/software/d archieves the log files of the mail discussions among software authors. The files are in unix mbox format. Files are named as yymm.log where yy is the year and mm is the month. soft.list is the current list. The discussions on Hanzi Bitmap Fonts are archieved seperately at ifcss.org:/software/fonts/d To join the list, mail a message to xiaofei@ifcss.org to post to the list, address your message to soft-authors@ifcss.org and your message will be mailed to everyone in the list. From cyl Fri May 27 18:40:16 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA04118; Fri, 27 May 94 18:40:16 CDT Date: Fri, 27 May 1994 18:40:15 -0500 (CDT) From: Nelson CHIN Subject: FYI: font directory restructured To: Software Authors Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O per suggestion from Ross, i've restructured ifcss.org:/software/fonts so that top level subdirectories is by coding schemes, ie big5, cns, gb, unicode, misc, and under those by platform/font types, ie bdf, hbf (headers and bitmaps), vector, outline, mac, ms-win, misc, and of course will only be created subject to availability of fonts. In addition, xiaofei moved the soft-authors archive to /software/fonts/d and any future non-font related discussion maybe placed in /software/d, or a better directory name if you have a suggestion. /software/fonts/utils is kept the same, mostly font conversion utils. per suggestion from cai, /software/fonts/gb/hbf/cclib*.24 and its headers will be removed as of June 30, 1994. Please adjust your softwares to use /software/fonts/gb/hbf/cclib.*24 and its headers instead (the fonts are identical but takes less space), or may i suggest to use the new GL-encoded gb fonts released with the X11R6 distribution, which can also be found in ifcss.org. Any suggestions, objections, or comments are welcomed concerning the above. Nelson Chin From @UTARLVM1.UTA.EDU:simpson@math.psu.edu Fri May 27 18:46:57 1994 Return-Path: <@UTARLVM1.UTA.EDU:simpson@math.psu.edu> Received: from by ifcss.org (4.1/IFCSS-Mailer) id AB04307; Fri, 27 May 94 18:46:57 CDT Received: from leibniz.math.psu.edu by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Fri, 27 May 94 12:15:13 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA19584; Fri, 27 May 1994 13:15:07 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 13:15:05 -0400 Date: Fri, 27 May 1994 13:15:05 -0400 Message-Id: <199405271715.NAA19062@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: HZ vs. ISO-2022 In-Reply-To: <199405251813.OAA29579@leibniz.math.psu.edu> References: <199405251813.OAA29579@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Status: O Ross Paterson writes: > The popularity of Big-5 is not relevant to the choice of a 7-bit > encoding of Chinese+ASCII text. I strongly disagree. The popularity of Big5 is extremely relevant. I can't imagine why Ross or anybody else would say that the popularity of Big5 isn't relevant. If we try to declare that the popularity of Big5 is irrelevant, millions of Big5 users will continue to use Big5, and *we* will become irrelevant. My case for HZ+S is based squarely on the fact that Big5 is popular. HZ+S is nothing but a straightforward marriage of two popular standards, HZ and Big5. This fact should enable HZ+S to win a degree of immediate popular acceptance. Communication standards that cannot win popular acceptance are useless. -- Steve From @UTARLVM1.UTA.EDU:simpson@math.psu.edu Fri May 27 19:40:47 1994 Return-Path: <@UTARLVM1.UTA.EDU:simpson@math.psu.edu> Received: from by ifcss.org (4.1/IFCSS-Mailer) id AC06049; Fri, 27 May 94 19:40:47 CDT Received: from leibniz.math.psu.edu by UTARLVM1.UTA.EDU (IBM VM SMTP V2R2) with TCP; Fri, 27 May 94 16:31:51 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA20585; Fri, 27 May 1994 17:30:31 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Fri, 27 May 1994 17:30:29 -0400 Date: Fri, 27 May 1994 17:30:29 -0400 Message-Id: <199405272130.RAA19392@boole.math.psu.edu> To: ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: Re: Jing-Shin's problem with HZ+ Reply-To: simpson@math.psu.edu Status: O > Again, what I had pointed out here (interrupted chars) is not > unique to my B5E1/2/3 but also to the other proposals. Agree? I completely disagree. B5E1 will insert many escape sequences in the middle of almost every line of Big5 text. All of the other mapping scheme proposals are very much better than B5E1 in this respect. Tests of HZ+S carried out by John Delacour, Wei-Chang Shann, and myself show that hardly any lines of Big5 text get escape sequences in the middle. Jing-Shin's so-called problem with HZ+ was a consequence of the fact that he was looking only at his own mapping scheme, B5E1. I wrote: > |> If you think that HZ+ is a bad idea, why do you keep flooding us > |> with alternative mapping schemes for HZ+? Jing-Shin replied: > In Internet it is appropriate to skip any junk mail if you think > so. Nevertheless, I promise not to skip reading your postings. :-) > I think my positions for posting such alternative mapping > schemes are to provide other views for a task based on different > criteria, not just for fun. There are now 5 proposals for alternative mapping schemes: one from me, one from Ya-Gui, and three (3) from you. Don't you see that there is a problem here? We in soft-authors are faced with the task of choosing *one* of these 5 proposals for use in HZ+. Could you please rank-order all five HZ+ proposals, from best to worst, and explain the criteria that you used to arrive at your ranking? In addition, if you have some other idea for chinese.* that you think is better than HZ+, could you please explain your position on that too? > As a final word, I think your HZ-S proposal is a popular one so > don't worry about that -^^-. Thank you for those words of comfort. But frankly, I *am* worried about the fact that we have so many alternative mapping scheme proposals. Not everyone will have time to evaluate all of these proposals on their merits. Before you invent any more mapping schemes, I think you ought to consider the consequences in terms of increased confusion, lack of clarity about the nature and purposes of HZ+, and the need for popular acceptance of HZ+. -- Steve From shin@hermes.ee.nthu.edu.tw Sat May 28 09:53:05 1994 Return-Path: Received: from nthu.edu.tw by ifcss.org (4.1/IFCSS-Mailer) id AA00705; Sat, 28 May 94 09:53:05 CDT Received: from hermes.ee.nthu.edu.tw ([140.114.28.27]) by nthu.edu.tw with SMTP id <182710(2)>; Sat, 28 May 1994 22:56:42 +0800 Received: by hermes.ee.nthu.edu.tw (4.1/SMI-4.0/shin-1.0) id AA29180; Sat, 28 May 94 22:46:58 CST Date: Sun, 29 May 1994 12:46:58 +0800 From: shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) Message-Id: <9405281446.AA29180@hermes.ee.nthu.edu.tw> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org Subject: Don't quote selectively (was: Re: Jing-Shin's problem with HZ+) Status: O Hi, I don't understand why such clear statements turn out to introduce an illogical conclusion like: ++ Jing-Shin's so-called problem with HZ+ was a consequence of the fact ++ that he was looking only at his own mapping scheme, B5E1. **** Do we need to discuss it in Chinese?! -^^- I don't want my posting distorted by selective quotation. That will result in an unhealthy status of discussion. It is appropriate to delete irrelevant parts for follow-up, but don't quote it selectively to the extent that introduces misunderstanding. (See the original postings at the end of this posting. Sorry to burden your list in this way.) As for the testing made by Professor Shann and John, they just said that Steve's filter pass their test for some TangShi or other articles. I guess what they meant is: those Big5 articles can be converted back and forth from Big5 -> HZ-S filter -> Big5. (This is the basic requirement in everybody's implementation. Isn't that?) I didn't see their reports on anything about the block 2 char problems as they appeared in an environment like hztty and ZWDOS (2.20 or older), where the escape characters appears with the text. So I don't see what the points to relate these tests with the problem I mentioned. In fact, when I tested B5E3 (** again, B5E3 **), which has the least frequency of switching (as HZ-2) among all proposals, on Professor Shann's Chinese literature collection (about 508K, including the 56K TangShi.300 mentioned in a previous posting about testing), about 10% of the chinese text lines contains ~2, not 6%. It is quite resonable since this collection contains ancient poems and articles. I was just trying to remind those prospective software contributors to notice this fact. Ya-Gui said 6% of such lines are acceptable. I can understand and respect his viewpoints, because, I guess, he may be accustomed to this kind of HZ environment. But I am not sure whether the Big5 users here are also acceptable to such interrupting escape sequences within the character streams. (See my posting entitled "HZ+ problem & hztty-style solutions" for some related problems.) In sum, my main concern is to ask those kind persons who are familiar with POPULAR (-^^-) news readers to spend some time to chcnge the source codes so that we (Big5 Users) don't need to suffer from these esc. characters. (PingHua and Nelson had indicated that Mule has such embedded filter within the package. That's what I want in every POPULAR news reader we had.) If such news readers are not widely available, the HZ+ may even make the cross mirrowing solution more complicated. I was once asked to get familiar with the HZ user's environment. Now, can I asked people to respect Big5 Users' environments in the same manner? Again, this is not the sole problem with my own B5E1/2/3. Shin -^^- ===[ What are the logical consequences between these statements ?? ]=== I said: %% |> > Yes, I am using my private B5E3 (Ah!! Yet Another Big5 Encoding). **** %% |> > This version, however, has fewer switching between block 1 and 2 %% |> > than HZ+S and B5E1/B5E2. Steve asked: %% |> How did you arrive at this 6 percent figure? I think that the 5401 I answered: %% I test it over my collection of HXWZ (around 1992, 1993). The file size %% is about 1.1M. Yes, the 5401 chars are frequently used chars in our I further said, after explaining why B5E3/HZ-2 has less switching freq: %% ... It is fair to say that the switching %% frequency has the following order: %% %% B5E1 > B5E2 = HZ-S > HZ-2 = B5E3 **** **** **** And Steve had the following conclusion: |> (shin:) |> > Again, what I had pointed out here (interrupted chars) is not |> > unique to my B5E1/2/3 but also to the other proposals. Agree? |> |> I completely disagree. B5E1 will insert many escape sequences in the **** am I making the test with B5E1 ?? |> middle of almost every line of Big5 text. All of the other mapping |> scheme proposals are very much better than B5E1 in this respect. |> Tests of HZ+S carried out by John Delacour, Wei-Chang Shann, and |> myself show that hardly any lines of Big5 text get escape sequences in ?????????? |> the middle. |> |> Jing-Shin's so-called problem with HZ+ was a consequence of the fact |> that he was looking only at his own mapping scheme, B5E1. **** ==== My original Posting ==== |% From shin@hermes.ee.nthu.edu.tw Thu May 26 03:06:10 1994 |% Date: Thu, 26 May 1994 16:48:55 +0800 |% To: ccnet-l@uga.cc.uga.edu, simpson@math.psu.edu, soft-authors@ifcss.org |% Subject: Re: Jing-Shin's problem with HZ+ |> Jing-Shin Chang writes: |> > Yes, I am using my private B5E3 (Ah!! Yet Another Big5 Encoding). ^^^^ |> > This version, however, has fewer switching between block 1 and 2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |> > than HZ+S and B5E1/B5E2. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |> > The division is the same as Ya-Gui's HZ-2. But the operations |> > are more like B5E2, simply much more simpler. |> > (See my next posting for the detailed spec.) |> > A rough estimation is that about 6% of the lines will have |> > those block 2 characters interrupting the block 1 stream. |> > If they are mixed with other ASCII (and HZ) chars. it is |> > even uncomfortable. |> |> How did you arrive at this 6 percent figure? I think that the 5401 |> more frequently used characters in Big5 block 1 are adequate for daily |> use and account for 99.9 percent of characters in ordinary text. I |> think your 6 percent figure is much too high. I test it over my collection of HXWZ (around 1992, 1993). The file size is about 1.1M. Yes, the 5401 chars are frequently used chars in our daily life. However, I am not sure how these chars are selected. So the words "frequently used" refers to a particular statistics over a particular text corpus made several years ago. But it may not imply a 99.9 % hit rate of the block 1 chars. I think the sample is adaquate for characterizing the articles to appear in future chinese.* postings. The sample size maybe small. I would like to see testing reports based on, say, all the HXWZ+ACT postings that was archived. (Anyone interested in such a testing ?) Since my professional research is in corpus-based statistics-oriented natual language processing, I am always very careful when reporting any such figures. |> |> > HZ+S should have a higher rate of such undesirable switching. |> |> No, this hardly ever happens in HZ+S. The only exception is ancient |> poetry or classics, where some very unusual characters sometimes |> appear. Yes, HZ-S of course has a higher probability to hit the block two chars than HZ-2 and B5E3, since HZ-S include 5401 common chars (plus 441 special chars) in Block 1 and 7652 less common chars in block two while HZ-2 and B5E3 has 5401+441+8*157=7089 chars in block one +7*157=6941 (*** SORRY FOR SUCH TYPO) and 7652-8*157=6396 chars in block. In particular, the HZ-S block 1 chars -7*157=6553 (*** SORRY FOR SUCH TYPO) is a proper subset of the HZ-2 and B5E3 block 1 chars. I don't think this fact need any further testing. It is fair to say that the switching frequency has the following order: B5E1 > B5E2 = HZ-S > HZ-2 = B5E3 Anybody against this statement? Those interrupted chars I found are mostly from ancient articles, names food names, etc., which maybe typical in ACT and HXWZ from my experience. |> |> > I guess Shih-Kun's tw.chinese.* will be much more popular by the Big5 |> > users if such problems do not have a popular and easy solution. |> |> Unfortunately, tw.chinese.* will be in 8-bit Big5. It will therefore |> not be available to users outside Taiwan whose news feeds are not |> 8-bit clean. HZ+ can provide an easy 7-bit clean solution to this |> problem, by means of a straightforward mapping of 8-bit Big5 into the |> 7-bit code space. Yes, I never against a 7-bit channel for communication. I am just pointing out a fact of user's concern. In fact, I do think that most people here will choose the tw.chinese.* if they were setup. Today, I get several posting to chinese.test which are sent in Big5/8 from outside Taiwan (e.g., one posting was sent by cyl@ifcss.org). If the software for the 7-bit channel is not easy to update or good enough or fit a particular user's habit and he/she has access to tw.chinese.*, the choice is obvious. Isn't it? |> |> Despite your problems with your BE51-2-3, there are no technical |> difficulties with HZ+. The only difficulties come about because of |> confusion over CNS and mapping schemes. Again, what I had pointed out here (interrupted chars) is not unique to my B5E1/2/3 but also to the other proposals. Agree? I don't think it has anything to do with CNS either. |> |> If you think that HZ+ is a bad idea, why do you keep flooding us with |> alternative mapping schemes for HZ+? Did I say HZ+ is a bad idea? Does "flooding us with alternative mapping schemes" means "think that HZ+ is a bad idea"? Who are "us"? In Internet it is appropriate to skip any junk mail if you think so. I think my positions for posting such alternative mapping schemes are to provide other views for a task based on different criteria, not just for fun. If we think that we can have only one choice, why not use Mule Big5 directly. In fact, I may have different criteria for different aspects of the encoding schemes, but I did not strongly against any proposals we had. My design was directed toward "simple, fast and easily described 100% invertable conversion algorithm" (Oop!! Fung-Fung does not seem to gave a correct rank on this criteria for B5E1, and now for B5E3 -^^-). The choice is of course not based on my personal preference. The software contributors have absolute rights to support what they like. Also, I am more concern of the user's feeling and thus the user interface. The encoding issue is not my only concern. As a Big5 user, I think it is appropriate for me to tell those potential software contributors to consider those problems I found. |> |> -- Steve |> As a final word, I think your HZ-S proposal is a popular one so don't worry about that -^^-. Shin -^^- ==== Steve's Conclusions: ==== |> From: simpson@math.psu.edu |> |> > Again, what I had pointed out here (interrupted chars) is not |> > unique to my B5E1/2/3 but also to the other proposals. Agree? |> |> I completely disagree. B5E1 will insert many escape sequences in the |> middle of almost every line of Big5 text. All of the other mapping |> scheme proposals are very much better than B5E1 in this respect. |> Tests of HZ+S carried out by John Delacour, Wei-Chang Shann, and |> myself show that hardly any lines of Big5 text get escape sequences in |> the middle. |> |> Jing-Shin's so-called problem with HZ+ was a consequence of the fact |> that he was looking only at his own mapping scheme, B5E1. |> |> I wrote: |> > |> If you think that HZ+ is a bad idea, why do you keep flooding us |> > |> with alternative mapping schemes for HZ+? |> |> Jing-Shin replied: |> > In Internet it is appropriate to skip any junk mail if you think |> > so. |> |> Nevertheless, I promise not to skip reading your postings. :-) |> |> > I think my positions for posting such alternative mapping |> > schemes are to provide other views for a task based on different |> > criteria, not just for fun. |> |> There are now 5 proposals for alternative mapping schemes: one from |> me, one from Ya-Gui, and three (3) from you. Don't you see that there |> is a problem here? We in soft-authors are faced with the task of |> choosing *one* of these 5 proposals for use in HZ+. |> |> Could you please rank-order all five HZ+ proposals, from best to |> worst, and explain the criteria that you used to arrive at your |> ranking? In addition, if you have some other idea for chinese.* that |> you think is better than HZ+, could you please explain your position |> on that too? |> |> > As a final word, I think your HZ-S proposal is a popular one so |> > don't worry about that -^^-. |> |> Thank you for those words of comfort. |> |> But frankly, I *am* worried about the fact that we have so many |> alternative mapping scheme proposals. Not everyone will have time to |> evaluate all of these proposals on their merits. Before you invent |> any more mapping schemes, I think you ought to consider the |> consequences in terms of increased confusion, lack of clarity about |> the nature and purposes of HZ+, and the need for popular acceptance of |> HZ+. |> |> -- Steve |> ==== Testings made by Professor Shann and John ==== ==== ** sorry if this list is not complete ** === Date: Sun, 1 May 1994 17:03:39 +0800 >From: John Delacour Subject: Re: HZ+S <-> Big5 should be OK now Stephen Simpson writes:- There was a bad bug in the HZ+S <-> Big5 programs. I think I've fixed it now. Yes, no problems so far with the programme since you fixed the declarations. I have tested the conversion both ways on the Macintosh using a Li Bai poem written in Big5 (about 90 Hanzi including punctuation) and conversion was 100% accurate when I used the fonts in Mac O/S 7.1T. There were three or four wrong characters when I used MacViewHZ but this will be due, I guess, to inaccuracies in the library chinese.16 upon which it relies for Big5. As time permits I will test mixed text and long lines, but the results so far are very exciting. Congratulations! John === Date: Sun, 1 May 1994 21:53:44 +0800 >From: Stephen G Simpson Subject: HZ+S tests shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) writes: > The hzps<->big5 programs version 0.51 passed my original set of tests, > and the whole set of TangShi 300 Shou. OK, great! Thanks! But I'm not finished with you! In the meantime I made some further improvements. The new version of hzps2big is better because it preserves things like ~{....~}, that is, GB stuff. I also wrote hzps2gb which preserves things like ~1....~} and ~2....~}, that is, Big5 stuff. These are version 0.60. They are at math.psu.edu, /pub/simpson/chinese/hzps. Please test them. > When I test it with a file > with more ASCII and Big5 mixed together, I found one failure: > Original: > > > 5o+H$H: Schroedinger@totoro (Schr dinger), +H0O: NCUMA (The above quote from you is probably corrupted, due to our mailer not being 8-bit clean.) Your file here is not an ASCII/Big5 file. The character after Schr is decimal 148 or hexadecimal 0x94, which is not a legal byte in either ASCII or Big5. ASCII consists only of the range 0 to 127, that is 0x00 to 0xFF. Big5 bytes are in the range 0x40 to 0x7E and 0xA1 to 0xFE. Nevertheless, thanks for pointing this out. I'm glad you made such a thorough test! What's interesting here is that big2hzps translates the two characters after Schr as ~1 ,~}, where is ASCII 17 or 0x11, i.e. Control Q, which is not in the HZ+S preferred range 0x21 to 0x7D. We probably don't want this kind of thing to happen (it makes HZ+S less robust), so I'll fix big2hzps to avoid it. By the way, Fung F. Lee's gb2hz has a similar problem. > Maybe this is what we have overlooked: the IBM PC extended codes. I had not overlooked this. I was well aware that HZ+S as I formulated it would not be able cope with IBM PC extended ASCII, or ISO-8859-1, or any similar character set, that is any character set which uses bytes in the range 0xA1 to 0xFE in a way that is incompatible with GB and Big5. This is a limitation of HZ that carries over to HZ+S. To solve this, it would be possible (but awkward) to extend HZ+S by adding additional tilde escape sequences for ISO-8859-1, IBM PC extended ASCII, etc etc. HZ+S would start looking more and more like ISO-2022-JP-2. -- Steve ==== Date: Wed, 4 May 94 15:17:45 CST >From: shann@wangwei.math.ncu.edu.tw (Shann Wei-Chang) Subject: consensus Status: RO > Right now soft-authors has only two people with a .tw address, > and , and these > two people have been silent. I keep trying to copy everything to > shin@hermes.ee.nthu.edu.tw (Jing-Shin Chang) but I sometimes forget. > Wei-Chang Shann is another Taiwanese whom I would like to involve. I don't know why Mr Wang and Huang are silent, maybe they are busy for their final exams or defenses. They may tell us later. I don't think I can make a fair vote now. I had some experiences with HZ-S and Prof. Simpson's conversion program. I like it and it passed the tests that I thought of. But I didn't compare it with HZ-2. > In summary, I would like to have more Taiwanese input, but let's not > forget our international perspective. I think this event is mostly an international business. Since within Taiwan we don't have much problem with 8-bit stuff. The major concern is the international communications. -Shann From jiang Sat May 28 14:10:03 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA09011; Sat, 28 May 94 14:10:03 CDT Date: Sat, 28 May 94 14:10:03 CDT From: jiang (Yuan Jiang) Message-Id: <9405281910.AA09011@ifcss.org> To: ccnet-l@uga.cc.uga.edu, chinesenet@uclink.berkeley.edu, simpson@math.psu.edu, soft-authors Subject: Re: 7-bit Big5 and chinese.* Status: O > > It seems not possible to reach a consensus on what 7-bit Big5 > > representation code to use. > >I disagree. I think it *will* be possible to reach a consensus on >majority on which 7-bit Big5 mapping scheme to use. I plan to call >for a vote on this issue very soon. Right now we are voting on If a consensus can be reach, what's the need to call for a vote? The discussion has gone beyond just on technical merits, and sometimes emotions are involved. It seems that we cann't even agree on what we are arguing, and misquote and misinterpretations occurs. I see no way to reach a consensus. That's why I propose to let chinese.* users to decide what big5 representation for chinese.*. >guidelines or standards for postings in the non-test newsgroups. One >of these guidelines is that HZ is encouraged. Another guideline is >that 8-bit Big5 is discouraged, because many news feeds are not 8-bit >clean and we don't want to exclude that group of users. So there is HZ does not need testing, and encourage HZ posting to chinese.test will only overload the network. Noone needs to read the contents of chinese.test postings, so noone is excluded. The only issue is whether testing 8-bit Big5 is necessary. > > > The whole problem is caused by this messy Big5. > >No. I agree that Big5 is messy. But I don't agree that Big5 is >causing a problem. Big5 is what it is. We can't change it. What we If the Big5->7-bit conversion were as straightforward and GB->HZ and if Big5 were ISO ... registered, there would have been no such debate. Yuan From cyl Sat May 28 15:42:55 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA11983; Sat, 28 May 94 15:42:55 CDT Date: Sat, 28 May 1994 15:42:53 -0500 (CDT) From: Nelson CHIN Subject: consensus by voting To: soft-authors@ifcss.org Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O jiang says we can't reached consensus. I believe that's why we're using voting. jiang suggest let the chinese.* user decide. Since there's no consensus, maybe add jiang's suggestion to one of the options to vote on. i'm guessing those who's on this list who's not spoken up, probably couldn't come to a consensus in their own minds. as for letting chinese.* users decide, i think that it's time consuming, and a big headache for everyone, and then users will need several version of 7-bit conversion programs at hand, just in case the sender decides to use something he thinks is best for 7-bit exchange, which the receipient may not personally favor, but still wants to be able to read what he receives from the sender. as for software developers, this will just spread out the effort, and it'll be based on market demand and your sales pitch of your encoding scheme to fellow software authors, and your own motivation and conviction that what you're doing will benefits the end users... but i see that those who will be left out are those software authors who create programs based on agreed standards, who rather placed their efforts with ease of mind and focus on the usefullness of the product, than because the encoding scheme is a volatile matter. for example, would macblue telnet plug in every conceivable 7-bit format for users to decide which is the best, i think it's more efficient if only one format is provided to the end user, and we in soft-authors have the responsibility to make a wise decision so that to minimize users' rebellion as to why the software doesn't work up to expectation, and if we decide on a specific 7-bit format wisely, then the user won't draw the blame on the agreed standards of soft-authors, if the program doesn't live up to expectation. Of course, the target users of this standard is the chinese computer network users. But then there's the long-term goal of upward compatibility with merging national standards into international standards , like iso and unicode, etc...the decisions of today will determine the amount of headache tomorrow. i'll go for C, cns is still needs time to take hold on the Internet, i say the timing of cns support should coinicide with when mule takes cns into its package, maybe in a couple of months, that way the transition will be more smoother and decisions will can be in cooperation with how's mule going to handle big5/cns issue. thanx to ross, HBF headers and bitmap files are now available at ifcss.org:/software/fonts/cns/hbf, so that users and soft-authors can have time to hack with it. Nelson From jiang Sun May 29 14:11:54 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA21342; Sun, 29 May 94 14:11:54 CDT Date: Sun, 29 May 94 14:11:54 CDT From: jiang (Yuan Jiang) Message-Id: <9405291911.AA21342@ifcss.org> To: cyl@ifcss.org Subject: Re: consensus by voting Cc: soft-authors There is a consensus to support Big5, not CNS. It's not a issue. Why call for a vote on something which is not a issue? (Ross and Handa may be the only two who disagree.) The issue has always been which 7-bit Big5 representation to use for a HZ+ standard. The choices are: 1) HZ-2, 2) HZ+S, 3) BEU1,2,3, and 4) 7-bit CNS. Only a call for vote on these four makes sense. The choices of escape sequences have been agreed as a secondary issue. Yuan From simpson@math.psu.edu Tue May 31 10:15:35 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA20867; Tue, 31 May 94 10:15:35 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id LAA18244; Tue, 31 May 1994 11:13:26 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 11:13:24 -0400 Date: Tue, 31 May 1994 11:13:24 -0400 Message-Id: <199405311513.LAA29336@boole.math.psu.edu> To: Jeffrey Chen Cc: ccnet-l@uga.uga.edu, soft-authors@ifcss.org Subject: how to use escape sequences in HZ+ Newsgroups: alt.chinese.computing In-Reply-To: <9405280117.AA22061@goren1.u.washington.edu> References: <199405272210.RAA26678@news.cs.utexas.edu> <9405280117.AA22061@goren1.u.washington.edu> Reply-To: simpson@math.psu.edu Jeffrey Chen writes: > Can I ask a non-programmer, moronic question? I understand that you have > to put diff escape chars if the char I am using is in an upper plane? > Would I be required to know which is in the upper plane and which is not? > Do I need to type the code manualy in if I don't have a super duper > HZ+-enabled Terminal program or WP? For example, what is one to do if one > only have C-win and terminal? You don't need to insert escape sequences manually. Your Big5 -> HZ+ software will take care of inserting the needed escape sequences for you. The user of HZ+ will never have to worry about this. If you have only Chinese windows and a terminal program, here is how I envision it. You first use your Chinese windows Big5 editor to edit your message. Then you use a standalone program to convert your message into HZ+. Then you use your terminal program to mail your message or post it to the net. Later, if you get software that is HZ+ aware, you can skip the conversion step. -- Steve From simpson@math.psu.edu Tue May 31 11:38:28 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA24696; Tue, 31 May 94 11:38:28 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id MAA19521; Tue, 31 May 1994 12:36:13 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 12:36:11 -0400 Date: Tue, 31 May 1994 12:36:11 -0400 Message-Id: <199405311636.MAA29487@boole.math.psu.edu> To: ccnet%uga.BITNET@vm.psu.edu, soft-authors@ifcss.org, mule@etl.go.jp Subject: Big5, CNS, Mule, and HZ+ In-Reply-To: <199405272301.TAA21321@leibniz.math.psu.edu> References: <199405272301.TAA21321@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu Edmund Lai writes: > Big5 and CNS are different coding systems Thank you for finally acknowledging that fact. > Big5 and CNS are the same character set I disagree. Ed has not proved his claim that Big5 and CNS are the same character set. He can't prove it, because it isn't true. I am willing to stipulate that the 13051 standard CNS Chinese characters comprise most of the Chinese characters in Big5. However, Big5 contains a number of additional Chinese characters, also various sets of numerals, alphabets, punctuation, and special symbols, not to mention vendor-defined codes, characters, and symbols. It may be that CNS contains some of these Big5 extras, but CNS certainly does not contain all of them. > and can be represented by the same code. I disagree. If Big5 and CNS are not the same character set, there is no way they can be represented by the same code. The fact is that algorithms for converting between Big5 and CNS cannot be specified completely and are complicated and ambiguous. > Are you accusing of Mr. HANDA of dropping support of Big5 because > he is planning to support Big5 through CNS in the future version of > Mule? No. The Mule Big5 situation is somewhat complicated. The current version of Mule supports Big5 in a way that is very similar to the way that HZ+S supports it, but that may change in future versions. I have not been able to find out exactly how Mr Handa proposes to support Big5 in future versions of Mule, but he has assured me that all future versions of Mule will support Big5 in some fashion, and I believe him. One point to keep in mind is that Mule's and Mr Handa's idea of Big5 support may not be the same as HZ+'s idea of Big5 support. You have to remember that Mule is a very complicated, multilingual application. Mule has its own support for Roman, Greek, Russian, Japanese Kana, etc. etc. Therefore in a sense (at least for certain applications), Mule may have no need for any part of Big5 outside of the Big5 Chinese characters. When Mr Handa says that he plans to support Big5, he may be referring only to the 13051 standard CNS Chinese characters. This may be fine for Mule, but it would be a mistake for HZ+, in my opinion. > Numerious people have pointed out that conversion between CNS and Big5 > is not a problem. Code has been available for ftp from Taiwan and > from Ross Paterson. Obviously Mr. HANDA thinks that it is not a > problem if he is talking about using CNS to support Big5. Again Ed is refusing to specify his conversion algorithm, and he is vague on the question of whether this conversion is 100 percent invertible. The fact is that the conversion algorithms that are implicit in the code that Ed mentions do not agree and are not 100 percent invertible. By contrast, my HZ+S conversion algorithm is completely specified and is 100 percent invertible. > Indeed, I would vote for option B. Thank you for your vote. I'm sorry that you want HZ+ to drop support for Big5, but I respect your opinion. > If Mr. HANDA, who first came up > with Mule Big5, now realize CNS is a better idea and plan to replace > Mule Big5 with CNS. I should do the same. CNS may be a good idea for Mule, but it is not a good idea for HZ+. I have two reasons for saying this. (1) Mule is an ISO-2022 application, and ISO-2022 recognizes CNS. HZ+ is not suffering from this constraint. (2) HZ+ may have some need for the multilingual features of Big5. Mule can perhaps afford to ignore these features, since Mule has its own extensive multilingual support. -- Steve From simpson@math.psu.edu Tue May 31 12:33:00 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA26569; Tue, 31 May 94 12:33:00 CDT Received: from leibniz.math.psu.edu by cnd.org (4.1/4.7) id AA09981; Tue, 31 May 94 10:15:26 PDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id NAA19795; Tue, 31 May 1994 13:13:40 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 13:13:38 -0400 Date: Tue, 31 May 1994 13:13:38 -0400 Message-Id: <199405311713.NAA29542@boole.math.psu.edu> To: ccnet-l%uga.BITNET@vm.psu.edu, soft-authors@ifcss.org, chinesenet@uclink.berkeley.edu Subject: Re: Don't quote selectively In-Reply-To: <199405281646.MAA00337@leibniz.math.psu.edu> References: <199405281646.MAA00337@leibniz.math.psu.edu> Reply-To: simpson@math.psu.edu I was indeed quoting Jing-Shin selectively, but not in an improper manner. Let me remind everybody of Jing-Shin's original posting on this topic, entitled "ONE problem with HZ+". Jing-Shin wrote: > Just to mention a problem and this problem could be a serious one: > When I tried a recent revision of my on line code converter Betty > (BE51/2-tty, something like hztty) for viewing HZ+ documents. I > found that Big5 users may find that their sentences are divided > seriously by those ~1's and ~2's, NOT JUST AT THE BEGIN/END OF A > LINE OR WHEN ENGLISH CHARS APPEARS. You may therefore find > something like HWA~2XIA~1WEN~2JAI in your news or mail > postings. When the Big5 part are mixed with HZ, the situation could > be worse. I think it is important to point out that this so-called problem with HZ+ is only a problem for Jing-Shin. It will not be a problem for anybody else. The reason that Jing-Shin has a problem is that, when Jing-Shin discovered his alleged problem, he was using his own peculiar 7-bit Big5 mapping scheme, which is extremely poor, because it puts many very common Chinese characters into block 2, thus necessitating the use of ~2 in the middle of almost every line of Big5 text. Other mapping schemes do not have this property. What Jing-Shin is calling a problem with HZ+ will only be a problem if we decide to base HZ+ on Jing-Shin's mapping scheme. Jing-Shin's so-called problem with HZ+ will *not* occur if we use my HZ+S mapping scheme, or almost any other mapping scheme that anyone can think of. Recently I summarized this discussion as follows: > ++ Jing-Shin's so-called problem with HZ+ was a consequence of the fact > ++ that he was looking only at his own mapping scheme, B5E1. Jing-Shin objected to this summary, but I think this summary was fair. > I was once asked to get familiar with the HZ user's environment. Now, > can I asked people to respect Big5 Users' environments in the same > manner? Here Jing-Shin is making a good point. Discussions on HZ+ should be based on some understanding of both HZ/GB and Big5 user environments. I myself am familiar with four Big5 environments: ETen, KC, Mule, and cxterm. Based on my familiarity with these environments, I think HZ+ would be very suitable for the chinese.* newsgroups and for any other situation where 7-bit Big5 is needed, for instance international Big5 e-mail. -- Steve From simpson@math.psu.edu Tue May 31 13:43:36 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA28873; Tue, 31 May 94 13:43:36 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id OAA20932; Tue, 31 May 1994 14:41:32 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 14:41:30 -0400 Date: Tue, 31 May 1994 14:41:30 -0400 Message-Id: <199405311841.OAA29686@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: Reminder: last chance to vote on CNS In-Reply-To: <9405291911.AA21342@ifcss.org> References: <9405291911.AA21342@ifcss.org> Reply-To: simpson@math.psu.edu Yuan Jiang writes: > The issue has always been which 7-bit Big5 representation to use > for a HZ+ standard. I disagree. That is one issue, but it is not the only issue. Another issue is whether HZ+ shall support CNS. The CNS issue is the issue that we are voting on now. After that, we will take another vote on the first issue, 7-bit Big5 representations. Both of these votes are very important. They are part of the process of reaching a consensus on HZ+. If you favor the creation of a popular HZ+ standard, please vote your preferences. Reminder: Today (May 31) is the last day to vote on the CNS issue. The options are: A. Add CNS support, in addition to the existing Big5 support. B. Drop Big5 support. Support CNS instead. C. Postpone a decision on CNS for the time being. Please vote by replying to this mail now, if you have not already done so. Nelson has suggested that there may be some people who are in favor of the creation of an HZ+ standard, but are undecided as between A, B, and C. If you are in this category, please vote "present", i.e. say that you are in favor of HZ+ but cannot decide among A, B, and C. In addition, if you are against the whole idea of HZ+, you may say that too. One or two people have complained (belatedly) that options A, B, and C are not clear enough. In case any clarification is needed: Option A means that HZ+ should support Big5 and CNS as two different coding systems, with two different sets of escape sequences. Option B means that HZ+ should support CNS with its own escape sequences, but not Big5. Under option B, all Big5 support (if any) is to be accomplished by means of Big5<->CNS translation. Option C means that, for the sake of reaching a consensus on HZ+, we are dropping the CNS issue for the time being. Under option C, HZ+ will continue to support Big5 directly, as in the various HZ+ proposals (HZ+S, HZ-2, B5E1/2/3, HZ+L), and CNS will not be supported now. Option C does not prevent us from adding CNS support later, if some new reasons to support CNS emerge, but basically option C means that CNS will not be part of HZ+ in the immediate future. I remain strongly opposed to option B, but that is only my personal view. Since there have been no votes for option A and a lot of votes for option C, I will probably go along with the apparent majority and vote for option C. (I'll cast my vote at 11:59 PM tonight.) -- Steve From simpson@math.psu.edu Tue May 31 16:07:43 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA07258; Tue, 31 May 94 16:07:43 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id RAA21781; Tue, 31 May 1994 17:05:36 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 17:05:34 -0400 Date: Tue, 31 May 1994 17:05:34 -0400 Message-Id: <199405312105.RAA29938@boole.math.psu.edu> To: Ming Pei Cc: chinesenet@uclink.berkeley.edu, soft-authors@ifcss.org, ccnet-l@uga.cc.uga.edu Subject: What now? In-Reply-To: <199405311959.PAA11199@ritz.mordor.com> References: <199405311959.PAA11199@ritz.mordor.com> Reply-To: simpson@math.psu.edu Ming Pei writes: > It seems that chinese.test is on this site. Will you have hz+ > for us to test a bit My site is not getting chinese.test as yet. Also, my command of the Chinese language isn't good enough to test HZ+ properly by myself. I will need the help of others. Would you like to help? I would be very grateful. My HZ+S filter programs are available for anonymous FTP at ftp.math.psu.edu, /pub/simpson/chinese/hzps. You will need a C compiler to compile them. If anybody would like me to make DOS executables available, please let me know. The procedure to post a message in HZ+S is: 1. Prepare your message in mixed ASCII/Big5, using a Big5 editor. (I assume you have a Big5/ASCII text editor. Possibilities are Mule, Qedit under KC or ETen, celvis under cxterm in Big5 mode, etc. etc.) 2. Save your message as a mixed ASCII/Big5 file, let's call it file.big5. 2. Use my big2hzps filter to translate your file into HZ+. The command is "big2hzps < file.big5 > file.hzplus". 3. Use your favorite news-posting software to post file.hzplus as a message to chinese.test. This posting procedure is somewhat cumbersome now, but it will be easier once appropriate software becomes available. -- Steve Stephen G. Simpson Department of Mathematics, Pennsylvania State University 333 McAllister Building, University Park, State College, PA 16802 Office phone +1 814 863-0775 Home phone +1 814 238-2274 E-mail simpson@math.psu.edu FAX +1 814 865-3735 World Wide Web http://www.math.psu.edu/simpson/ From mleisher@crl.nmsu.edu Tue May 31 16:41:58 1994 Return-Path: Received: from cnd.org (cnd.cnd.org) by ifcss.org (4.1/IFCSS-Mailer) id AA08698; Tue, 31 May 94 16:41:58 CDT Received: from NMSU.Edu (dns1.NMSU.Edu) by cnd.org (4.1/4.7) id AA16870; Tue, 31 May 94 14:40:21 PDT Received: from crl.nmsu.edu by NMSU.Edu (8.6.8.1/NMSU-1.18) id PAA09871; Tue, 31 May 1994 15:34:53 -0600 Date: Tue, 31 May 1994 15:34:53 -0600 Message-Id: <199405312134.PAA09871@NMSU.Edu> Received: from pylos.nmsu.edu by crl.nmsu.edu (4.1/SMI-4.1) id AA20993; Tue, 31 May 94 15:34:50 MDT Received: by pylos.nmsu.edu (4.1/SMI-4.1) id AA17147; Tue, 31 May 94 15:34:51 MDT From: mleisher@crl.nmsu.edu (Mark Leisher) To: soft-authors@ifcss.org Subject: Vote on CNS in HZ+ I'll vote for option C (table the CNS question until later). ----------------------------------------------------------------------------- mleisher@crl.nmsu.edu Mark Leisher Computing Research Lab "Elegance is not optional!" New Mexico State University -- Attributed to Richard O'Keefe Box 3CRL, Las Cruces, NM 88003 From simpson@math.psu.edu Tue May 31 17:09:20 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA10034; Tue, 31 May 94 17:09:20 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA22395; Tue, 31 May 1994 18:07:06 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 18:07:05 -0400 Date: Tue, 31 May 1994 18:07:05 -0400 Message-Id: <199405312207.SAA00148@boole.math.psu.edu> To: D-P Deng Cc: chinesenet@uclink.berkeley.edu, ccnet-l@uga.cc.uga.edu, soft-authors@ifcss.org Subject: Re: What now? In-Reply-To: <9405312121.AA19575@bunny.rhic.bnl.gov.zoo> References: <199405312105.RAA29938@boole.math.psu.edu> <9405312121.AA19575@bunny.rhic.bnl.gov.zoo> Reply-To: simpson@math.psu.edu D-P Deng writes: > I don't use big5, but don't mind of reading it. Could you please tell me how > to read it on chinese.test? In full generality, this is a hard question. My understanding is that the chinese.test newsgroup is "anarchy" -- anybody can post anything in any format. Also, I am not getting chinese.test as yet, so I don't know what formats people are actually using. I believe John Delacour mentioned that he has posted something in HZ+S. If this is the case, then you should be able to read it. Use my hzps2big filter to translate it into Big5, then read it using your favorite Big5 software. Some other people may have posted things in 8-bit Big5. You can use these postings to test whether your news feed is 8-bit clean. Just save an article and then use some Big5 software to read it. -- Steve From simpson@math.psu.edu Tue May 31 17:13:50 1994 Return-Path: Received: from leibniz.math.psu.edu by ifcss.org (4.1/IFCSS-Mailer) id AA10241; Tue, 31 May 94 17:13:50 CDT Received: from boole.math.psu.edu (simpson@boole.math.psu.edu [146.186.131.137]) by leibniz.math.psu.edu (8.6.4/8.5) with ESMTP id SAA22421; Tue, 31 May 1994 18:11:48 -0400 From: Stephen G Simpson Received: from localhost by boole.math.psu.edu id ; Tue, 31 May 1994 18:11:46 -0400 Date: Tue, 31 May 1994 18:11:46 -0400 Message-Id: <199405312211.SAA00158@boole.math.psu.edu> To: soft-authors@ifcss.org Subject: vote on whether HZ+ should support CNS Reply-To: simpson@math.psu.edu I vote for option C, to postpone CNS support. But I am also interested in option A, support CNS in addition to the existing Big5 support. If anybody would like to discuss this with me privately, I have some sample software that needs testing. -- Steve From cyl Tue May 31 23:58:08 1994 Return-Path: Received: by ifcss.org (4.1/IFCSS-Mailer) id AA24561; Tue, 31 May 94 23:58:08 CDT Date: Tue, 31 May 1994 23:58:07 -0500 (CDT) From: Nelson CHIN Subject: tried hzp.tar.Z package from math.psu.edu To: soft-authors@ifcss.org Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII i tried out the hzp.tar.Z package from math.psu.edu i tested out mixed gb/big5 file and couldn't get it to convert successsfully, please tell me what i did wrong: % big2hzp < text.b5 > text.hzp % gb2hzp < text.gb >> text.hzp % hzp2big < text.hzp | hzp2gb > text.out text.out didn't come out with both 8-bit big5 and gb text. nelson