Subject: Re: [SWIPL] parsing newline bug??

From: "Richard A. O'Keefe" <[email-masked]>
Date: Fri, 16 Feb 2001 10:39:48 +1300 (NZDT)

Is there anyone around knowing the proper semantics of these predicates used
	in text-mode (or in the future with UTF-8 variable-length character encoding)?
	
The very reason that stream_position and its relatives exist is that
Quintus needed a way of talking about file positions that worked under

    - IBM MVS
    - IBM VM/CMS
    - VMS (which supports *several* stream-with-line termination modes,
           plus variable-length-with-count, plus fixed length, plus a
           couple you'd rather not know about that we didn't handle)
    - DOS (we never intended to port there ourselves, and didn't, but
           we certainly didn't want to specify anything that couldn't
           work on such a popular system)
    - MacOS
    - UNIX

So the idea was that
    - line_count counted "records", however the operating system
      defined them
    - line_position counted "characters" *within* a "record";
      the mapping of tabs to spaces varied all over the place so that we
      gave up and said a tab is one "character", this meant unfortunately
      that line_position was not useful for determining screen
      coordinates.
    - character_count is defined as "the total number of characters
      read or written", *NOT* as the number of times get0/1 or put/1
      has been called.
    - a stream position record actually contains some *extra* magic
      numbers to record any operating-system-dependent information
      that cannot easily be packed in character_count.

On IBM mainframes, the idea was to use the line count to seek to a record
(yes, CMS can actually seek to records (but not characters) in a file of
variable length records) and then skip along to the right character.
On VMS, records are located by means of a 6-byte RFA, so one would seek
to the beginning of a record using the 6-byte magic number and then skip
along to the right character.

Since we never did an MS-DOS or MacOS port, things got slightly fuzzy;
the number of *characters returned to Prolog* was the same as the number
of *bytes read from the stream*.

This is precisely the kind of questions that the ISO Prolog committee
should have settled.  But I had to scream at them for a long time before
they agreed not to require all Prolog systems to be able to seek to an
arbitrary byte (not realising that many commercially important file systems
couldn't actually *do* that).

Note that line_position is clearly a *character* count; if you read one
DBCS character, that's two bytes but one character.

Since UTF-8 can take up to 6 bytes for a Unicode 3.0 character, the
distinction between characters and bytes is now painfully clear.

The simplest solution seems to be:

    - line_count remains a count of records,
      following the Unicode recommendation that *ANY* of
      CR, LF, CRLF, NEL, LS, PS be taken to end a record.

    - line_position remains a count of CHARACTERS, not bytes.

    - character_count has the word character in its name, so it
      had darned well BETTER be a count of CHARACTERS, not bytes.

    - THEREFORE, a stream-position object must contain ADDITIONAL
      operating-system-specific information recording the current
      position (for VMS, a 6-byte RFA and a 2-byte byte offset,
      for UNIX, a 64-bit byte offset, for other systems, plug in
      whatever works).