From ok@atlas.otago.ac.nz  Mon Dec 18 00:41:29 2000
Received: from atlas.otago.ac.nz (atlas.otago.ac.nz [139.80.32.250])
	by swi.psy.uva.nl (8.9.3/8.9.3) with ESMTP id AAA09241;
	Mon, 18 Dec 2000 00:41:27 +0100 (MET)
Received: (from ok@localhost)
	by atlas.otago.ac.nz (8.9.3/8.9.3) id MAA01723;
	Mon, 18 Dec 2000 12:41:51 +1300 (NZDT)
Date: Mon, 18 Dec 2000 12:41:51 +1300 (NZDT)
From: "Richard A. O'Keefe" <ok@atlas.otago.ac.nz>
Message-Id: <200012172341.MAA01723@atlas.otago.ac.nz>
To: p.singleton@keele.ac.uk, prolog@swi.psy.uva.nl
Subject: Re: swi jpl character coding
Cc: jan@swi.psy.uva.nl

	Is it the case that, if we define atom names as UTF-8 representations
	of UCS strings, then certain byte-string atom values will strictly be
	illegal?
	
Yes.

	Is it feasible to localise the UTF-8-specific kernel code so that we
	can avoid it if we want?
	
Surely the only things that should care are the operations that convert
between atoms and character sequences?

I can think of some good reasons for a switch to UTF-8 (such as the fact
that Windows likes it, and Java likes it, except they bend the definition
of UTF-8 a bit).

I can also think of some good reasons to switch to something else.
There's a technical report from the Unicode people (I think it's number 6)
that describes a compression scheme with some rather nice properties:

(a) If you have a Latin-1 text, it is represented as itself.
(b) If you have a text using any of the ISO 8859 members, it will
    require at most one byte of overhead.
(b) If you have a CJK text, that too will need only a byte or two
    of overhead.

The basic idea of the scheme is to have a small number of 128-character
"windows"; you can switch the interpretation of the upper 128 codes to
any of the preloaded windows with a single byte, you can switch to
2-byte codes with a single byte, and you can reset any of the windows.

This is a state-dependent coding, which UTF-8 is not, but for the uses
Prolog makes of atoms, it is not clear to me that this is a disadvantage.
You can't jump around randomly in a UTF-8 string anyway.

