From jan@swi.psy.uva.nl Thu Feb 15 10:27:58 2001
Received: from gollem.swi.psy.uva.nl (root@gollem [145.18.152.30])
	by swi.psy.uva.nl (8.11.2/8.11.2) with ESMTP id f1F9RwZ20134;
	Thu, 15 Feb 2001 10:27:58 +0100 (MET)
Received: (from jan@localhost)
	by gollem.swi.psy.uva.nl (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id KAA01268;
	Thu, 15 Feb 2001 10:27:57 +0100
Date: Thu, 15 Feb 2001 10:27:57 +0100
Message-Id: <200102150927.KAA01268@gollem.swi.psy.uva.nl>
From: Jan Wielemaker <jan@swi.psy.uva.nl>
Subject: Re: [SWIPL] parsing newline bug??
To: "Mike Maxwell" <mike_maxwell@sil.org>, <prolog@swi.psy.uva.nl>
In-Reply-To: Mike Maxwell's message of Wed, 14 Feb 2001 15:08:39 -0500
Phone: +31 - 20 - 525 6121

[Please wrap your lines, so we can quote properly]

> I've run into what appears to be a rather curious bug in parsing newlines.  
> 
> My editor (Visual Slickedit--nice editor!) allows me to save a file in either DOS format (with CR-L> F for newlines) or Unix format (just LFs).  I have a file which the parser I wrote in Swi-Prolog pa> rses OK if the file was saved in Unix format, but _not_ if it was saved in DOS format.
> 
> What appears to be happening when it parses the DOS format file is this: the parser chugs along unt> il it gets 4096 bytes (Hex 1000) into the file, then it suddenly drops back to a position = 4096 - > L, where L is the number of newlines it has read up to that point.
> 
> By putting a call to
>     ...
>     stream_property(Stream, position('$stream_position'(P, _, _))),
>     write(P), 
>     write(' ')
>     ...
> in the parsing code, I found that the parser thinks its byte position is L less than where it actua> lly is, until it has read in 4096 bytes.  That is, the last byte position reported by the above cod> e before the parser goes back L characters is not 4096, but position 3844, which happens to be on t> he 252nd line.  And 3844+252 = 4096.
> 
> As far as I can tell, the parser never reads a CR (ASCII 13), only LFs (ASCII 10).  That is, the [C> h] construct in a parser rule never returns any CRs.  Now since I have redefined 'C' as
> 
>     :- redefine_system_predicate('C'(_,_,_)).
>          'C'(Stream/Offset, Char, Stream/Next) :-
>          succ(Offset, Next),
>          seek(Stream, Offset, bof, _),
>          get0(Stream, Char).
> 
> --what that really means is that get0/3 never returns a CR; somehow when it reads an LF, it also re> ads the CR without returning it (and hence without advancng the counter).  get_byte has the same be> havior.
> 
> This seems like a bug, at least it sure messes up my code!
> 
>                                          Mike Maxwell
>                                          Summer Institute of Linguistics
>                                          Mike_Maxwell@sil.org

Seek and character positions are ill-defined using text-mode files.  Fetching the
position basically returns the number of characters read sofar, but that exludes
the CR characters.  Setting the position does a file-level seek and this includes
the CR characters.

The simplest solution is to open the file in binary mode and deal with the CR's
yourself.

Is there anyone around knowing the proper semantics of these predicates used
in text-mode (or in the future with UTF-8 variable-length character encoding)?

	--- Jan

