From jan@swi.psy.uva.nl  Sat Dec 16 12:58:14 2000
Received: from gollem.swi.psy.uva.nl (gollem [145.18.152.30])
	by swi.psy.uva.nl (8.9.3/8.9.3) with ESMTP id MAA16367;
	Sat, 16 Dec 2000 12:58:14 +0100 (MET)
Received: (from jan@localhost)
	by gollem.swi.psy.uva.nl (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id MAA01975;
	Sat, 16 Dec 2000 12:57:58 +0100
Date: Sat, 16 Dec 2000 12:57:58 +0100
Message-Id: <200012161157.MAA01975@gollem.swi.psy.uva.nl>
From: Jan Wielemaker <jan@swi.psy.uva.nl>
Subject: Re: swi jpl character coding
To: Paul Singleton <p.singleton@keele.ac.uk>,
        SWI Prolog <prolog@swi.psy.uva.nl>
In-Reply-To: Paul Singleton's message of Fri, 15 Dec 2000 18:11:14 +0000
Phone: +31 - 20 - 525 6121
Cc: Jan Wielemaker <jan@swi.psy.uva.nl>

> > No.  As yet it is encoded as an unsigned char * with length, so it can
> > contain 0-bytes.  So it cannot represent characters outside the range
> > 0..255, which you can use with any 8-bit encoding for which the lower
> > 128 characters match the ASCII set.  It is planned to change this to
> > use UTF-8, probably somewhere next year.
> 
> I much admire SWI-Prolog's support for atom "names" as unrestricted byte
> strings (any length, any values), since this allows me to use atoms to
> represent arbitrary external "blobs" of data, e.g. long fields from
> databases, JPEGs, OLE objects, whatever (it may not always be appropriate,
> but the liberty to do this is valuable).

This is recognised.  Whatever happens I do not want to give up that
property.

> I see no problems with the semantics of Prolog if atoms are arbitrary
> external values, not necessarily text strings (although I expect the
> theologians to come up with something unhelpful :-/
> 
> Indeed, I reckon it's neither necessary nor desirable that a Prolog
> engine is somehow committed to a particular interpretation of its
> atoms' names: as long as I have atom_bytes/2 to dynamically compose
> and decompose atoms to/from their "names" I'd be happy, I think...

Suppose in the feature an atom is taken to be the UTF-8 encoding of
an UCS string.  This means atom_chars/2 needs to be changed to convert
to/from a list of UCS characters.  Introducing encoding also means that
the system needs additional primitives to communicate.  I envision text
streams getting an encoding attribute, so the system can read/write
in multiple encodings.  The foreign interface also needs additional
primitives, telling Prolog about the encoding of external text.

You might get PL_{get,unify,put}_bytes() functions and atom_bytes/2
to allow you using atom as binary blobs.  Using any other function
(such as atom_chars/2) on these atoms yield undefined results.

> I realise that it's convenient to have a concrete source syntax for
> Prolog :-) but as long as we can compile a list of clause representations
> which represent a procedure (as suggested in another posting) we can
> implement module loaders from any concrete syntax we may devise, including
> graphical ones.
> 
> I realise also that a debugger/tracer will want to represent atoms in
> some standard, compact way, but with the hooks etc. which Jan provides,
> I assume we can implement a debugger's user interface outside the basic
> engine?

Yes, the XPCE-based GUI tracer runs entirely on these hooks.

> Is it the case that, if we define atom names as UTF-8 representations
> of UCS strings, then certain byte-string atom values will strictly be
> illegal?

Yes, but using additional primitives it should be possible to regard
atoms as arbitrary byte-arrays.  One problem might be the textual
representation of atoms containing illegal UTF-8 sequences.  It whould
be nice if write_canonical and read can deal with *any* Prolog term.

> Is it feasible to localise the UTF-8-specific kernel code so that we
> can avoid it if we want?

Normally these things start of with #ifdef O_UTF8, but after a couple
of cycles to many code starts relying on this flag that is normally
true.  We need to handle the use of binary-blobs using proper additional
interfaces.

> Is there any sense in my vision of a Prolog kernel which genuinely
> doesn't care what external values its atoms represent?
> 
> Should I resist the temptation to move to a private scheme whereby
> each atom is explicitly typed by an initial "magic number" in its
> first few bytes?  I think this could be better than wrapping them
> in constructors to denote their type, when the types are of concern
> only to I/O not to inference...

Assuming UTF-8 you could make the atom start with an illegal sequence
and add a foreign routine to return its type.  Whether that is more
elegant than using a compound term ...

Anyway, someday there will be UTF-8 (most likely), but binary blobs
are too useful to give up.

	--- Jan

