Subject: Re: swi jpl character coding

From: "Richard A. O'Keefe" <[email-masked]>
Date: Mon, 18 Dec 2000 12:41:51 +1300 (NZDT)

Is it the case that, if we define atom names as UTF-8 representations
	of UCS strings, then certain byte-string atom values will strictly be
	illegal?
	
Yes.

	Is it feasible to localise the UTF-8-specific kernel code so that we
	can avoid it if we want?
	
Surely the only things that should care are the operations that convert
between atoms and character sequences?

I can think of some good reasons for a switch to UTF-8 (such as the fact
that Windows likes it, and Java likes it, except they bend the definition
of UTF-8 a bit).

I can also think of some good reasons to switch to something else.
There's a technical report from the Unicode people (I think it's number 6)
that describes a compression scheme with some rather nice properties:

(a) If you have a Latin-1 text, it is represented as itself.
(b) If you have a text using any of the ISO 8859 members, it will
    require at most one byte of overhead.
(b) If you have a CJK text, that too will need only a byte or two
    of overhead.

The basic idea of the scheme is to have a small number of 128-character
"windows"; you can switch the interpretation of the upper 128 codes to
any of the preloaded windows with a single byte, you can switch to
2-byte codes with a single byte, and you can reset any of the windows.

This is a state-dependent coding, which UTF-8 is not, but for the uses
Prolog makes of atoms, it is not clear to me that this is a disadvantage.
You can't jump around randomly in a UTF-8 string anyway.