Subject: Statistics on usage of Big5 Infrequent characters

From: "Ya-Gui Wei" <[email-masked]>
Date: Thu, 13 May 1993 09:34:54 -0500

Fung Fung and others,

  Below is just a little statistics on usage of Big5 characters I 
did last evening. Descriptions follow the table.

      SAMPLE      TOTAL     INF    ARL    TOT/INF   F/TOT%
   ---------------------------------------------------------
    A.C.T.BIG5   107129     348   1.086    307.8    99.68%
     CLASSICS    163427    2276   1.793     71.8    98.61%
   ---------------------------------------------------------

SAMPLES - A.C.T.BIG5 is the alt.chinese.text.big5 news spool for the
  week of May 5-12, 1993, apprx. 530 articles.  CLASSICS is the
  china-studies/literature/classics directory at ifcss.org, with
  classic proses and poetry from the pre-Qin to Tang periods.
TOTAL - the total count of Big5 characters in the sample.
INF - the total count of infrequent Big5 characters in the sample.
  (The Big5 table has 5401 frequent chars, and 7652 infrequent ones.)
ARL - average run length of INF, measures how many infrequent Big5
  characters occur in a row, on average. For example, if we have
  3 characters in a row and another 1 by itself, then the ARL
  is (3+3+3+1)/4 = 2.5 (not 2.0).
TOT/INF - number of all chars for each infrequent characters.
F/TOT% - percentage of Frequent characters in the sample text.

It seems that Infrequent characters are indeed quite infrequently
used, even in pre-Qin classics. When they are used, they don't
usually occur for more than once in a row.

  Ya-Gui Wei
  May 13, 1993

p.s. This little exercise was done to justify my proposal for
     a 2-byte scheme for encoding Big5 into ASCII.  There is
     little reason to waste an extra byte when 99.7% of the times
     we stay within the 5000+ frequent characters.  I'll have
     more to say about this later (and soon).

p.p.s. There may be arguements that the infrequent char usage
     may be deflated in the a.c.t.big5 sample because most articles
     there are converted from GB. However, I can also make the
     counter arguement that the usage may be _inflated_ due to
     the fact that classic proses and poetry are over-
     represented there. So why don't we take it as accurate 
     within a few tenths of percentage points, which is good
     enough for our purpose. :-)