From pjb@imaginet.fr  Sat Jun 10 06:33:49 2000
Received: from triton.local.net (IDENT:postfix@35-MURC-X12.libre.retevision.es [62.82.105.35])
	by swi.psy.uva.nl (8.9.3/8.9.3) with ESMTP id GAA18327
	for <prolog@swi.psy.uva.nl>; Sat, 10 Jun 2000 06:33:46 +0200 (MET DST)
Received: by triton.local.net (Postfix, from userid 1000)
	id 178ED3B193; Sat, 10 Jun 2000 06:22:40 +0200 (CEST)
From: "Pascal J. Bourguignon" <pjb@imaginet.fr>
To: prolog@swi.psy.uva.nl
Subject: Suggestion regarding the classification of the characters.
X-PGP-Key-ID: 0xEF5E9966
X-PGP-fingerprint: 00 F5 7B DB CA 51 8A AD 04 5B 6C DE 32 60 16 8E EF 5E 99 66
X-PGP-Key-URL: http://www.imaginet.fr/~pjb/pgpkey
X-URL: http://www.imaginet.fr/~pjb/index
X-Accept-Language: fr, es, en
Organization: Latymer Designs Ltd.
Disposition-Notification-To: <pjb@imaginet.fr>
Return-Receipt-To: <pjb@imaginet.fr>
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
Content-Type: text/plain; charset=iso-8859-1
Reply-To: <pjb@imaginet.fr>
Message-Id: <20000610042240.178ED3B193@triton.local.net>
Date: Sat, 10 Jun 2000 06:22:40 +0200 (CEST)


Abstract:
    - The classification of characters in SWI-Prolog is not consistent.
    - I'd suggest to tune char_type to match its documentation.
    - I'd suggest to add a feature to allow the user to customize char_type
      (with effect on atom/1, var/1, ...).



$ unset LC_ALL LC_MESSAGES LC_MONETARY LC_COLLATE LC_TIME LC_NUMERIC LC_CTYPE
$ export LC_ALL=fr_FR
$ cat /usr/share/locale/fr/charset 
iso-8859-1

Let's printout what ctypes think about some of our characters:

$ char_types | egrep '^[eEéÉ¹º¼¡¢µþ]'
char_type: LC_ALL       = fr_FR
E     asci prnt grph                     alnm alph uppr           xdgt 
e     asci prnt grph                     alnm alph      lowr      xdgt 

(inverted bang)     Ok.
¡          prnt grph                pnct                               

(cent)              I would have said it more alpha than punct...
¢          prnt grph                pnct                               

(mu)                Clearly, it should be lower alpha, like thorn.
µ          prnt grph                pnct                               

(exponent 1)        I would have said it more alnum or even digit than punct...
¹          prnt grph                pnct                               

(number (as in "No."))    I would have said it more alpha than punct...
º          prnt grph                pnct                               

(1/4)               I would have said it more alnum or even digit than punct...
¼          prnt grph                pnct                               

(E acute)          Ok.
É          prnt grph                     alnm alph uppr                

(e acute)          Ok.
é          prnt grph                     alnm alph      lowr           

(thorn)            Ok.
þ          prnt grph                     alnm alph      lowr           

Ctypes  works well  enough,  even  if some  applications  may need  to
classify more finely some symbol characters.


Now, let's see what SWI-Prolog think about this:

$ pl
Welcome to SWI-Prolog (Version 3.3.6)
Copyright (c) 1990-2000 University of Amsterdam. 
Copy policy: GPL-2 (see www.gnu.org)

For help, use ?- help(Topic). or ?- apropos(Word).

1 ?- between(1,255,Code),char_code(Char,Code),catch(term_to_atom(Term,Char),Exception,fail),atom(Term),write(Char),fail.


No
!#$&)*+,-/:;<=>?@\]^`abcdefghijklmnopqrstuvwxyz|}~ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ2 ?- nl.


Yes


Anything not (alpha & upper (or some punct)) is an atom.

3 ?- char_type(C,alpha),write(C),fail.


No
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz4 ?- nl.


Yes


There's  a mismatch  here between  the behavior  of char_type  and its
documentation.  In the  documentation, it's  said that  char_type uses
ctypes:

    3.22 Classifying characters:

    These predicates are defined as built-in predicates to exploit the
    C-character classification's handling of locale (handling of local
    character-sets).

But:

20 ?- findall(_T,code_type(É,_T),_B),write(_B),nl.
[to_lower(201), to_upper(201)]

Yes
21 ?- findall(_T,code_type('E',_T),_B),write(_B),nl.
[alnum, alpha, csym, csymf, ascii, graph, upper, upper(101), to_lower(69), to_upper(101), xdigit(14)]

while the only difference reported by ctypes functions between 'E' and
'É' (E acute) is that E acute is not ascii and not xdigit.



Moreover:

5 ?- char_type(C,csymf),write(C),fail.


No
ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz6 ?- nl.


Yes

This is in  contradiction to the result of the  first question and the
two following questions.

7 ?- atom(éléphant).

Yes
8 ?- atom(Éléphant).

Yes
9 ?- 



I would  suggest to  fine tune char_type  to match  its documentation,
really using the ctypes functions.


Note that on my system:
Linux triton.local.net 2.2.13-4mdk #1 Tue Sep 7 18:23:11 CEST 1999 i586 unknown
the  locale  is  not   automatically  initialized  to  match  the  the
environment variable LC_ALL, et al. I need to do something like:

    void locale_initialize(void)
    {
        static const struct {
            char* env_name;
            int   locale;
        } variables[]={
            {"LC_ALL",          6},
            {"LC_MESSAGES",     5},
            {"LC_MONETARY",     4},
            {"LC_COLLATE",      3},
            {"LC_TIME",         2},
            {"LC_NUMERIC",      1},
            {"LC_CTYPE",        0},
            {0,-1}};
        int i;
        i=0;
        while(variables[i].env_name!=0){
            const char* value=getenv(variables[i].env_name);
            if(value!=0){
                setlocale(variables[i].locale,value);
            }
            i++;
        }
    }/*locale_initialize*/

before using the ctypes functions  to get the locale configured by the
user in the environment variables. Otherwise, it uses the C locale.



In addition, I  would suggest to add a feature  to SWI-Prolog to allow
customization   of  char_type/2   results  (and  correspondingly,   of
code_type/2, atom/1, var/1, etc).  So  we could say, for example, that
the fraction chars  (1/4, 1/2, 3/4) are actually  alnum characters but
neither upper nor lower, or that the monetary characters (cent, pound,
yen, etc) are  to be considered as lower  alpha characters. Of course,
we could  always redefine a  new locale, but  I feel it would  be more
flexible to do it within prolog.


-- 
__Pascal Bourguignon__    PGP Key ID:      0xEF5E9966
mailto:pjb@imaginet.fr    PGP fingerprint: 00 F5 7B DB CA 51 8A AD 04 5B 
http://www.imaginet.fr/~pjb/               6C DE 32 60 16 8E EF 5E 99 66

