From: "Henry S. Thompson" Date: Mon, 16 Sep 96 03:49:18 BST To: w3c-sgml-wg@w3.org Subject: Summary of crypto-DTD discussion At the risk of missing something, (I have read each message on this list at least once, but there's an AWFUL lot of them :-), against the background of our experience with using an SGML-subset for inter-program communication (see NSGML in Michael's system feature summary), I'm going to attempt to summarise the emerging consensus in the general area of what kinds of declaration information is required for what tasks. In terms of the preceding discussion, this is a modest elaboration on the 'Cooked' approach. My basic take is that we are committed to saying that "[the full version] of a conforming XML document is a conforming SGML document". Furthermore, for some tasks such as validation or editing/authoring, only the full version is appropriate. HOWEVER, for another common class of tasks, roughly speaking those we might call read-only tasks, an abbreviated version is adequate. There has been considerable convergence between postings on this list, and our own implementation experience in Edinburgh, regarding what aspects of the DTD are needed in the abbreviated DTD, which together with the XML document instance would constitute the abbreviated version: 1) For each element: 1a) Whether its content model is EMPTY or not; 1b) For each of its attributes: Their value type and #FIXED or defaulted value; 2) For each SDATA and external NDATA entity, their definitions/notations respectively [following Martin's suggestion that most entity references in RCDATA are expanded already]. Our experience is that this information suffices for a simple non-validating parser to parse very fast. HOW this information is encoded for transmission to/exploitation by read-only applications such as viewers is a separate question -- we use a highly optimised relocatable hashtable which is clearly NOT what is wanted for XML, I personally like the idea of using e.g. Date: Mon, 16 Sep 96 04:01:52 BST To: w3c-sgml-wg@w3.org Subject: Re: Summary of crypto-DTD discussion Oops, missed one bit. I agree that a) continuing to allow mixed content and b) continuing to distinguish between mixed and element-only for the purposes of handling whitespace is both valuable and easy, so change the crypto-DTD content as follows: 1a) Content type: EMPTY, mixed or element-only; ht Date: Mon, 16 Sep 1996 23:53:00 -0400 To: W3C SGML Working Group From: Paul Prescod Subject: Re: Empty endtags (Was: short-tag considered unhealthy) At 03:01 AM 9/15/96 -0400, Arjun Ray wrote: In any case, as a counterargument, could anyone demonstrate -- without >resorting to the known empty element problem -- how a GI on an endtag >actually simplifies parsing? Parsing, probably not. Error reporting and recovery? Probably. GI's in end tags also make Perl hacking easier. For instance, I could do a search and replace for , and replace it with
,
to "wrap" each FOO in a section. Otherwise I have to keep a stack and write some form of tree-based thing (or work on an ESIS which does the stack building for me). Paul Prescod Date: Mon, 16 Sep 1996 00:13:34 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: A constraint on markup for EMPTY elements At 09:00 PM 9/15/96 CDT, Paul Grosso wrote: >At this point after reading all the "cute trick with delimiters" postings >that attempt to address the EMPTY element, I'm tempted to say that XML >just represent empty elements as everyone knows them in RCS 8879, and >we just have to include the "list of empty elements" information along >with the XML document instance. Did we decide that the "don't worry be smart" proposal was too difficult? This was the proposal to force parsers to keep a stack and reparent content if the next end tag doesn't match the start tag. I would much rather add a couple of extra hours of work to the tasks of the hundred or so people that will implement generic XML parsers from scratch than add a couple of minutes per document on the hundreds of thousands of documents we hope that people will create with XML. Note also that any application that either a) reads a DTD or b) uses a fixed DTD will already know which elements are empty, so we are really talking about only a subset of XML processors: primarily browsers and search tools, not really Perl hacks and editors. Consider also backwards-compatibility with HTML (and other DTDs). If we require "extra" information about which tags are empty, then all existing HTML documents are invalid XML documents because they neither include this description-list nor point to it. On the other hand, XML isn't the first time people have discussed this empty=tag-recognition problem. Is "don't worry be smart" really harder to implement than I have estimated? Paul Prescod Date: Sun, 15 Sep 1996 23:36:27 +0000 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: Re: character sets - a summary and a proposal At 12:22 PM 9/15/96 CDT, Michael Sperberg-McQueen wrote: [after an *excellent* summary-of-the-position] >Here's yet another proposal. > >6. Limited Modfied Eclecticism: compromise between Eclectic >Compromise and 100 Flowers: > - XML data streams may be in any of a number of supported encodings: > UTF-8, UTF-16, UCS-4, ISO 8859 > - XML data streams must label themselves as to which supported > encoding they are using, by means of a PI which must be the first > data in each XML entity. > - all XML systems must accept XML data in any supported encoding, > detecting the encoding in use from the internal label; > they may reject data in other encodings. > (See note on autodetection, below.) ... other good stuff Your point about "if it just reads ASCII, it's not really XML" is well-taken; but setting the bar at a point which includes 8859 *and* UTF *and* UCS for basic acceptance is I think serious infringement on our design goal #4 that says XML shall be easy to program. Also, including 8859 but not JIS is disturbingly Eurocentric. Are there grounds for compromise between minimalism and eclecticism by saying that (a) here are a list of encodings which should be supported, (b) entities have to self-label with leading PI's, and (c) all XML implementations *must* be able to read UTF8 as well as generate it? Second, might it be clever, for UTF8-encoded entities, to relax the requirement that all XML entities self-label ? Not that UTF8 is morally superior or anything, but this would have the desirable side-effect of turning a large proportion of the SGML objects in the world into XML. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Mon, 16 Sep 1996 09:37:44 +0000 From: jjc@jclark.com (James Clark) To: tbray@textuality.com Cc: w3c-sgml-wg@w3.org Subject: Re: XML character sets: the hard-minimalist manifesto > Date: Fri, 13 Sep 1996 09:46:36 -0700 > From: Tim Bray > If I'm going to write a program for processing XML, I'm going to use the > tools I have on the computer that sits in front of me. They can deal > with UTF8 today. Thus, I'm going to write a pure UTF8 program, with callouts > to converters for interchange with various other facilities. > > And at the front I'm going to have a little kludge along the lines of the > following: > > FirstByte = getc(stream); > if (FirstByte & 0xfe == 0xfe) > { > SecondByte = getc(stream); > temp = fopen(tempfile_name(), "w"); > ConvertUCS2StreamToUTF8(FirstByte, SecondByte, stream, temp); > fclose(stream); > stream = fopen(tempfile_name(), "r"); > FirstByte = getc(stream); > } I wouldn't write anything like that. If you're writing a program to handle UTF-8 data and it does any non-trivial manipulation of the data (such as displaying it), the first thing you typically do is combine the bytes representing each character into a single integer (usually a wchar_t). The program then processes this stream of wchar_t's just as a non-Unicode program would process a stream of char's. The conversion from UTF-8 to wchar_t is quite complicated and expensive. The conversion from UCS-2 to wchar_t is very simple. In fact, it is often a noop (if your wchar_t is 16 bits and bytes don't need to be swapped). So what I would end up doing would be in effect converting everything to UCS-2 or UCS-4. That's why I think UCS-2/UTF-16 has just as much right as UTF-8 to be considered the canonical way of encoding Unicode/ISO 10646. James >Received: by chuckd.allette.com.au.allette.com.au id AA00535 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Mon, 16 Sep 1996 14:58:52 +1000 Date: Mon, 16 Sep 1996 14:58:51 +1000 (EST) From: Rick Jelliffe To: bbauma1@cs.umbc.edu Cc: w3c-sgml-wg@w3.org Subject: (off-topic?)Re: XML character sets: a proposal On Sat, 14 Sep 1996 bbauma1@cs.umbc.edu wrote: > Of course one problem with the private use area is that its private. Prof. Hsieh from Academica Sinica is presenting a paper on handling variant Han ideographs and missing characters at the China Korea Japan Document Processing (CJK DOCP) meeting in a weeks time. I don't know the details yet, but I think it may build on Prof Eiji Matsuoka's comments about using standard national ideograph books in CJK to reference in SDATA entities for missing characters. (CJK DOCP has a liason with WG8 through Dr Komachi of Japanese Business Machines Association.) I am hosting this meeting this year, and I will send out details to this list. > UTF-8 is particularly offensive with its blatant western bias. As far as I know, (apart from Wired, which didn't even credit who these outraged Asians were as far as I can remember) the difficulties people have with Unicode tends to be not so much this encoding or that, but rather that for Han ideographs it follows radical order and not the order in any existing national character set order. But I suppose if it had it would be criticised for being "Japanese" or "Taiwanese" or "PRC" biased! Also, the round-trip rule and unification makes it impossible to specify other variants in the code. However, when SGML is added, who cares? All the extra information can be contained in markup: it is just a matter of setting agreed semantics of the markup. Unicode seems better than anything before, especially w.r.t. making a character catalog. (People may be away of PRC's standard GB13000 which adds all the extra Han ideographs from ISO 10646 onto the previous national standard: retaining backwards compatibility with previous character set.) And the issue of user-defined characters is irrelevent unless there is a mechanism for accessing their glyphs over the Web. Until there is, there is no point trying to allow for them in XML. Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ >Received: by chuckd.allette.com.au.allette.com.au id AA00431 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Mon, 16 Sep 1996 14:19:05 +1000 Date: Mon, 16 Sep 1996 14:19:04 +1000 (EST) From: Rick Jelliffe To: "Deborah A. Lapeyre" Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE considered confusing? On Sat, 14 Sep 1996, Deborah A. Lapeyre wrote: > On Fri, 13 Sep 1996, David Durand wrote: > >For XML we can simplify things by just treating LF and CR (well, > > actually their 10646 equivalents) as whitespace. We can simply skip > > the entire notion of non-significant and significant RS/RE entirely. > > > While not advocating this or any other mechanism, may I add my vote that > it is ESSENTIAL that RS/RE not be significant. Any system whereby they > are, in however limited a fashion, adds both complexity and dependencies > we cannot afford. Maybe someone has suggested this already, but a possible approach is to treat the whole entity as one record, i.e. to disconnect the record concept from the line concept. (One could use End Of Transmission character to signify RE, I guess: presumably this would be stripped out by the network session handler and reinserted by the entity handler.) Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ From: bbauma1@cs.umbc.edu To: w3c-sgml-wg@w3.org Date: Mon, 16 Sep 1996 07:17:19 +0000 Subject: Re: character sets - a summary and a proposal Priority: normal Michael Sperberg-McQueen writes: >Is this compromise workable? The short answer: Yes The long answer: Thanks for the summary of the discussion so far, although I would characterize my position more as "One Hundred Flowers are going to bloom, how are we going to deal with it.", and you propose a good solution. Once concern I still have is the role of UTF-8 as both the normalized form, and possibly the only form for network transmission. UTF-8 is biased toward the efficient encoding of some languages at the expense of others. Although I understand the pragmatic reasons that one would want to specify UTF-8 as the standard, I am still uncomfortable with this bias. Using UTF-16 is one solution to this problem. One other possible solution is an encoding scheme from Reuters that was presented at the UNICODE conference a couple of weeks ago. I was not actually at the conference and don't have the paper in front of me, so Gavin please correct me if I get this wrong. Basically the Reuters scheme uses a sliding window across the UNICODE character space 256 characters wide, with the window's starting point set by a shift key followed by a couple of bytes to specify the offset. By default, the window starts at the beginning of the UNICODE character space and thus ASCII is represented unchanged. This encoding also allows any other alphabetic language to be represented efficiently. It is not quite so efficient with languages that do require multiple bytes such as Chinese, Korean etc., but as the paper points out each character in these languages carries quite a bit more information then a single alphabetic character. The main problem with this encoding is that its not a standard. B. Todd Bauman Graduate Student University of Maryland, Baltimore County >Received: by chuckd.allette.com.au.allette.com.au id AA01355 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Mon, 16 Sep 1996 21:21:09 +1000 Date: Mon, 16 Sep 1996 21:21:09 +1000 (EST) From: Rick Jelliffe To: Tim Bray Cc: w3c-sgml-wg@w3.org Subject: Reads like ASCII (was Re: character sets ...) On Sun, 15 Sep 1996, Tim Bray wrote: > Your point about "if it just reads ASCII, it's not really XML" is well-taken; > but setting the bar at a point which includes 8859 *and* UTF *and* UCS for > basic acceptance is I think serious infringement on our design goal #4 that > says XML shall be easy to program. Also, including 8859 but not JIS is > disturbingly Eurocentric. Maybe all that is needed (in our blissful absense of complicating SGML declarations or non-RCS DTDs) is that 1) the first thing in the document before any non-ISO 646 characters is a PI with only ISO 646 characters that can say the encoding (if it is exotic or warranted). E.g.: 2) the encoding used for the input stream must have ISO 646 characters in the same code numbers as ISO 646. I don't know any standard national encodings (apart from EBCDIC and maybe some exoteric Asian ones that are not used for HTML anyway and so don't count) that aren't OK with this scheme (Gavin or James would know better?). It certain fits ISO 8859-n, EUC, shiftJIS, UTF8 and so on. It fits the fixed width 16-bit ones too, in that a reader can detect whether 16 bit is being used from the zero-value octets (remember, no non-1S0 646 characters before the XML PI). To put this another way: it can work with any fixed 8bit encoding that has ISO 646 in the bottom half of the character code. It works for any variable-byte encoding that uses a single octet for ISO646 characters and has ISO 646 in the bottom half of those codes. And it works for any fixed 16bit encoding (in either order) as long as the ISO 646 characters are in the appropriate place in the lowest 127 code positions. Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ From: Gavin Nicol Date: Mon, 16 Sep 1996 12:36:45 GMT To: sjd@ebt-inc.ebt.com CC: w3c-sgml-wg@w3.org Subject: Re: XML character sets: the hard-minimalist manifesto I have no problem with a canonical form, so long as it doesn't stop people using other forms. >a) Once you've parsed an XML document once, all further parses must produce >an absolutely byte-identical copy of the XML document that came back from >that first parse. I believe the right term for this is "idempotence". This >makes parser conformance testing trivial. This would necessarily take 2 passes for validation: Pass 1: Resolve entity references etc. convert to UTF-8, generate document. Pass 2: Parse document, generate document, compare results. Though things like empty elements and RE/RS complicate this no end. Date: Mon, 16 Sep 96 08:17:37 CDT From: Michael Sperberg-McQueen Subject: Re: (off-topic?)Re: XML character sets: a proposal To: Rick Jelliffe , W3C SGML Working Group On Mon, 16 Sep 1996 06:22:28 -0400 Rick Jelliffe said: >Also, the round-trip rule and unification makes it impossible to specify >other variants in the code. However, when SGML is added, who cares? All >the extra information can be contained in markup: it is just a matter of >setting agreed semantics of the markup. This seems to represent a third requirement for XML's i18n support; is that a plausible interpretation? (The first two I see are 1 handling a suitably large character repertoire, 2 allowing non-Ascii characters in names.) Is the requirement met by ensuring XML DTDs have a well-known method for indicating language (and geographic language variant?) in content, or will more information be needed than is carried by a LANG attribute of the sort we know from TEI, from DocBook, or from the HTML i18n proposals? >And the issue of user-defined characters is irrelevent unless there is a >mechanism for accessing their glyphs over the Web. Until there is, there is >no point trying to allow for them in XML. A fourth requirement? Or should this be outside the scope of XML? -C. M. Sperberg-McQueen From: Gavin Nicol Date: Mon, 16 Sep 1996 13:41:33 GMT To: papresco@calum.csclub.uwaterloo.ca CC: U35395@UICVM.CC.UIC.EDU, bbauma1@cs.umbc.edu, w3c-sgml-wg@w3.org Subject: Re: XML character sets: a proposal [In response to Paul Papresco] >>I should note that I have never said that I wish to *require* that all >>XML parsers be open-ended. I have no problem at all with seeing Latin >>1 XML systems, and SJIS XML systems, though I expect most will >>actually be UNICODE based. > >I think that a major goal of XML should be "reliable interoperability". I do >not think that "support for whatever you need to do" should be a major goal. >If people start creating standards-compliant SJIS XML documents on the Web >and standards-compliang XML client software cannot read it, then we have a >PROBLEM, in my opinion. I agree, but it is a problem which will not be solved by saying "all XML documents *must* be in either UTF-8 or UTF-16". Such a statement *will* be ignored. >One last question: Isn't it reasonable to expect that most local encodings >could be translated into XML by the HTTP server on transmission? If so, >there is no loss of convenience in requiring the on-the-wire format to be >standardized in the same way that you would expect FTP keywords to always be >ASCII and IP packets to have a certain byte-ordering. This is precisely why I don't think it's such a big deal. If you send notification to the server that you can only accept UTF-8, ASCII, or Latin 1, and the server send you a SJIS document, the server is broken. It should either convert the doducment to UTF-8 before sending it to you, *or* send an error message. NOTE 1: In the future, language like JAVA will also give applicatons the option of downloading translators. NOTE 2: One of the primary reasons for requiring a single document character set is precisely such cases: the server can *blindly* (ie. without parsing) convert the XML data, which can be made *fast*. >I don't know what we'll decide in the end, Gavin, but isn't it nice that >we're talking about Unicode _as a minimum_? "We've come a long way, Baby." I guess you can actually remember me arguing the hard minimalist path some time ago... as weel as arguing for support for the TEI subset ;-) From: Gavin Nicol Date: Mon, 16 Sep 1996 14:02:27 GMT To: Charles@sgmlsource.com CC: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Mixed content >

>Text of p. >

> >Case 2: > >

Text of p.

My understanding is that these would return *different* parse results. We could (if it is so desired) make it an application convention to strip leading and trailing whitespace. From: Gavin Nicol Date: Mon, 16 Sep 1996 14:40:16 GMT To: U35395@UICVM.CC.UIC.EDU CC: w3c-sgml-wg@w3.org Subject: Re: character sets - a summary and a proposal >performed, but everyone in the discussion seems willing to make a leap >of faith here and believe Gavin Nicol when he says it's not very hard.) At some point, I will try to dig out the wchar_t TEI parser I produced. >3. Let 100 Flowers Bloom: Gavin Nicol and Todd Bauman have argued for a >third position which I understand to have the following salient points: > - XML data streams can be in any known or documentable encoding > - XML systems may accept data streams in any format(s) they choose > to support; they are encouraged but not required to accept UTF-8 I have no objection to UTF-8 being required as a compromise position. > - all XML systems must implement and rely on external specification of > the coded character set / encoding, such as MIME or attributes on > an FSI > - each XML system must support content negotiation so clients and > servers can avoid sending or receiving XML data in unsupported > encodings Only for systems that are client/server. >This position seems, in some ways, to be even more minimalist than Tim >Bray's, since there is *no* coded character set or encoding which *all* >XML systems are required to support. ISO 10646 would need to be "supported", though there are many levels of "support". >4. The Hard Maximalist Position: this is what I originally understood >Nicol and Bauman to be arguing for; it's not wholly unlike the apparent >intent of ISO 8879, as I understand it, though there are some obvious >differences of detail. This is actually my preferred path, but it is probably too hard for most people to implement (I don't think it overly difficult though...) >5. The Eclectic Compromise (DeRose): a slight extension of the >Dual-Track approach: This is a reasonable approach. >Q1 should there be any minimal function required of all conforming XML >systems, any coded character set or character encoding they are all >required to accept as input, whether across the net or from disk? The coded character set should be ISO 10646. I am willing to accept UTF-8 as required (I argued exactly that position on HTML-WG a long time ago). >Q2 should conforming XML systems be prohibited from accepting any >input format they are not required to accept? Absolutely NOT! >Q3 if XML systems may accept different sets of input formats (whether >or not these sets overlap), can we ensure interoperability >in some way, or is that a lost cause? Interoperability is something to be greatly desired, and in fact, the primary reason I got involved in HTML I18N was precisely that. However, I do not believe that at this time, we can get to a point where all XML systems will be able to process all XML documents. At some point in the future (3-5 years), perhaps. Now, no. >Q4 if XML systems may *only* accept Unicode (whether just UTF-8 or >also UTF-16), is there anything that can be done to make life >easier for users of current systems which rely on Ascii, ISO 8859-1 >or 8859-*, JIS, Shift-JIS, EUC, etc.? No. Converters can be (perhaps even easily) written. However, as the old saying goes "You can lead a horse to water, but you can't make him drink". >Note on autodetection of character sets. Autodetection fails abysmally as soon as you get more than a few encodings. >Perhaps we should say that network transmissions (or http transmission) >should always be in UTF-8, and the other supported formats are only >for local use on disk ...) With content negotation, your application can enforce this if it wishes. >From a practical point of view, no matter what we say, I expect the following to happen: 1) People will use ISO 10646. Entity sets for ISO 10646 (like the SPREAD set), and numeric character references will use it. 2) People will tend (at least initially) to author by hand. There will be a large number of documents in ASCII, ISO 8859-?, SJIS, EUC-JP, ISO-2022-?, EUC-KR, KSC, BIG5. 3) Initial browsers will be written to support Western languages or UNICODE. 4) Localised versions of such browsers will become available (ie. pure SJIS, EUC or ISO2022 for example). 5) Client/server interaction will initially be primarily in the native encoding. 6) Over time, a transition will be made to UTF-8/UTF-16 (ie. as more and better tools become available). We should recognise, and accept this. From: Gavin Nicol Date: Mon, 16 Sep 1996 14:52:35 GMT To: ricko@allette.com.au CC: bbauma1@cs.umbc.edu, w3c-sgml-wg@w3.org Subject: Re: (off-topic?)Re: XML character sets: a proposal >And the issue of user-defined characters is irrelevent unless there is a >mechanism for accessing their glyphs over the Web. Until there is, there is >no point trying to allow for them in XML. I disagree. I would like to see the ability to use user-defined characters so that it would spur interest in font/glyph servers. If I had more time, I'd write the software myself... From: Gavin Nicol Date: Mon, 16 Sep 1996 14:54:49 GMT To: ricko@allette.com.au CC: dlapeyre@mulberrytech.com, w3c-sgml-wg@w3.org Subject: Re: RS/RE considered confusing? >Maybe someone has suggested this already, but a possible approach is to >treat the whole entity as one record, i.e. to disconnect the record >concept from the line concept. (One could use End Of Transmission >character to signify RE, I guess: presumably this would be stripped out by >the network session handler and reinserted by the entity handler.) Precisely what Dave Durand and I thought. From: Gavin Nicol Date: Mon, 16 Sep 1996 15:02:03 GMT To: ricko@allette.com.au CC: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) >1) the first thing in the document before any non-ISO 646 characters is a >PI with only ISO 646 characters that can say the encoding (if it is exotic >or warranted). E.g.: > >2) the encoding used for the input stream must have ISO 646 characters >in the same code numbers as ISO 646. This is a hack, and doesn't help with *initial* parsing of the document. Autodetection also fails very quickly when faces with a number of multibyte encodings. The only *correct* way to indicate the encoding (or BCTF) of a document is to do so external to the document. To me, this means FSI's *or* MIME labelling (the *.mim file format). So far, all of the proposal I have seen could be easily handled by the *.mim file format, in which case, no parser trickery is needed: the storage manager would always unambiguously know what the encoding is. Date: Mon, 16 Sep 1996 08:15:33 +0000 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: Re: Mixed content At 02:02 PM 16/09/96 GMT, Gavin Nicol wrote: >>

>>Text of p. >>

>> >>

Text of p.

> >My understanding is that these would return *different* parse >results. We could (if it is so desired) make it an application >convention to strip leading and trailing whitespace. I agree.

Listen to my heart beat.

would *not*, in this cenario, be the same as

Listen to my heart beat.

Is this a problem? It's certainly easy to explain. With the proper SGML declaration, they will also parse differently in SGML. The SGML setup makes it easy to set markup apart with whitespace, making it easily visually distinguishable. This is good. The price is that it becomes difficult for ordinary people to tell when some white space is actually data. This is bad. The trade-off is probably reasonable, but XML need not choose the same trade-off. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Mon, 16 Sep 96 09:44:09 CDT From: Michael Sperberg-McQueen Subject: Re: character sets - a summary and a proposal To: Gavin Nicol , W3C SGML Working Group On Mon, 16 Sep 1996 14:40:16 GMT Gavin Nicol said: >>Q1 should there be any minimal function required of all conforming XML >>systems, any coded character set or character encoding they are all >>required to accept as input, whether across the net or from disk? > >The coded character set should be ISO 10646. I am willing to accept >UTF-8 as required (I argued exactly that position on HTML-WG a long >time ago). I'm confused. Where I come from, 'coded character set' is a mapping between a set of characters and a set of bit patterns, not necessarily the same length. (I.e. I use the terms 'character set' and 'coded character set' as typically defined in ISO character set standards. I try unsuccessfully to avoid the term 'character set' precisely because it is used by SC 18 to mean what SC2 means by 'coded character set'.) Under that definition, if 'the coded character set should be ISO 10646', then we should not accept JIS 0208, Shift-JIS, EUC, ISO 8859, etc., because they are different coded character sets. Their character sets, aka character repertoires, happen to be a subset of that of ISO 10646 and of Unicode, but that does not make them the same coded character set, nor encodings of it. It just means that translation into 10646 or its encodings is not inherently lossy. So there seems to me to be an inherent contradiction in saying 'the coded character set should be ISO 10646' and 'we should allow XML documents to be in Shift-JIS or EUC or ...', which I thought you said elsewhere in your posting. I doubt that this contradiction is real, but I don't know how to resolve it. I.e., I don't understand what your position is. >>Q3 if XML systems may accept different sets of input formats (whether >>or not these sets overlap), can we ensure interoperability >>in some way, or is that a lost cause? > >Interoperability is something to be greatly desired, and in fact, the >primary reason I got involved in HTML I18N was precisely >that. However, I do not believe that at this time, we can get to a >point where all XML systems will be able to process all XML >documents. At some point in the future (3-5 years), perhaps. Now, no. This scares the pants off me. In 3-5 years, if XML is widely adopted, it will be *impossible* to impose interoperability in the form of required support for data streams in UTF-8 or UTF-16 or whatever, because by then there will be legacy systems and legacy data to be protected. The only way to achieve such uniformity is by imposing it at the outset, when there is no XML legacy data, and we have a free hand. Failing to ensure interoperability when we have a free hand is not a good sign for our ability to achieve it later when our hands will be tied by systems which have made use of whatever freedom the spec gives them now. >>Note on autodetection of character sets. > >Autodetection fails abysmally as soon as you get more than a few >encodings. Correct; that's why the proposal (a) limits itself to the cases of UCS-4, UTF-16 / UCS-2, ISO 646 and compatible encodings including UTF-8, and EBCDIC, and (b) requires an explicit label in the entity regardless. Autodetection in general fails when there are more than a few encodings. We agree. The question in my mind is, does this particular proposal for autodetection fail when we have the set of encodings described? > 5) Client/server interaction will initially be primarily in the native > encoding. > 6) Over time, a transition will be made to UTF-8/UTF-16 (ie. as more > and better tools become available). > >We should recognise, and accept this. Hmm. By analogy with this, HTML could have started by allowing any existing 7-bit national character set, as well as proprietary 8-bit character sets, and hoped that eventually, with time, all HTML users would migrate to ISO 8859-1. As it is, it started by prescribing 8859-1, which is a blessing because it allows users to protest against servers which serve data in native PC or native Mac format. Things are bad enough as they are; would they really be better if sending data in Mac's proprietary coded character set were legal HTML? On the whole, I think that HTML did the right thing in pressing for a move from 7 to 8 bits. And I think XML should do the same in pressing for a move to 16 bits. It is much easier to loosen XML restrictions in later revisions than to tighten them and break legacy data. -C. M. Sperberg-McQueen >Received: by chuckd.allette.com.au.allette.com.au id AA02100 (5.65/IDA-1.5 for tbray@textuality.com); Tue, 17 Sep 1996 02:31:24 +1000 Date: Tue, 17 Sep 1996 02:31:23 +1000 (EST) From: Rick Jelliffe To: Gavin Nicol Cc: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) On Mon, 16 Sep 1996, Gavin Nicol wrote: > Autodetection also fails very quickly when faces with a number of > multibyte encodings. This isnt autodetection except for 8 or 16 bit. When will it fail? Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ >Received: by chuckd.allette.com.au.allette.com.au id AA02063 (5.65/IDA-1.5 for tbray@textuality.com); Tue, 17 Sep 1996 02:26:35 +1000 Date: Tue, 17 Sep 1996 02:26:34 +1000 (EST) From: Rick Jelliffe To: Gavin Nicol Cc: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) On Mon, 16 Sep 1996, Gavin Nicol wrote: > >1) the first thing in the document before any non-ISO 646 characters is a > >PI with only ISO 646 characters that can say the encoding (if it is exotic > >or warranted). E.g.: > > > >2) the encoding used for the input stream must have ISO 646 characters > >in the same code numbers as ISO 646. > > This is a hack, and doesn't help with *initial* parsing of the > document. why? what is *initial* parsing? Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ >Received: by chuckd.allette.com.au.allette.com.au id AA02259 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Tue, 17 Sep 1996 03:06:39 +1000 Date: Tue, 17 Sep 1996 03:06:38 +1000 (EST) From: Rick Jelliffe To: w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) On Mon, 16 Sep 1996, Gavin Nicol wrote: > Autodetection also fails very quickly when faces with a number of > multibyte encodings. Yes, but I am not proposing that: the only thing autodetected is whether the encoding is definitely 16-bit or not, for which you only need to look at the first two octets (i.e. if one is zero valued). (This is in context of a proposal I am putting forward: "baby SGML", which is probably the most minimalising of the proposals. I think it is available on http://www.textuality.com/sgml-erb/baby/, and takes the tack "if we add constraints and rules to SGML generators, can we make produce something that is easy to parse without a DTD?" ) Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ Date: Mon, 16 Sep 1996 14:24:29 -0400 To: Michael Sperberg-McQueen , W3C SGML Working Group From: "Steven J. DeRose" Subject: Re: marked sections At 02:51 PM 09/13/96 CDT, Michael Sperberg-McQueen wrote: ... >It seems essential that we have ways of representing delimiter >strings in XML without having them parsed as delimiters. Off the top 1 CDATA marked sections for examples: ]]> 2 Entity references: &hi; <p> 3 Empty comments: <P> or <P> or <&nil;P> or <P> 4 CDATA or RCDATA elements: <P> 5 External CDATA, SDATA, or NDATA entities: &my-escaped-phrase; 6 Shortrefs: \&ht; \

>Are there other techniques people have seen? I haven't seen 6 actually used, though it has some nice features. Doing this of course does not mean XML needs to support SHORTREF; XML merely needs to define "\<" and "\&" in the XML grammar; SGML could then achieve compatibility by using SHORTREF in its domain. Another option is to set MSSCHAR in your SGML declaration, say, to '\'. It works basically like backslash in C: the following character is not recognized as starting any delimiter. Unfortunately, I have been given to understand that it differs in one crucial respect: the backslash (or other MSSCHAR character) remains in the parsed result as a data character. This is not so bad as long as you can configure your downstream processor (browser/indexer/etc) to get rid of it later. On the other hand, the fact that < and >C. Possible Approaches > >What are our options for XML? > >We could keep marked sections on the grounds that they are useful and >not too hard to parse. We could simplify the parsing a bit, e.g. by >insisting that CDATA and RCDATA be literals, not entity references, >-- or do people switch between CDATA and IGNORE?! I hope note, since if one switches between CDATA and IGNORE you invite disaster, since IGNORE marked sections can nest but CDATA can't. Thus, if you set %flag to IGNORE, this works: there AT&T ]]> but if you change %flag to CDATA, the company name will produce a syntax error (or reference the "T" entity, or the #DEFAULT entity...). If no entity reference happens to be around, then SGML recognizes the (unmatched) ]]> on the last line, but it is not considered an error that it was found with no marked section still open for it to close, so it is effectively ignored. Such behaviors also effect the "5-line Perl script writer" Jon has mentioned.... > >If we want to lose marked sections, we need to say how to get the >required function. For CDATA sections, I think either of techniques >2 or 3 above would work, though I suspect someone will want to drop >the empty comment from the language, leaving us with technique 2 (sigh). I think I vote for the space convention or the shortrefs. But I'm not sure. Steve From: Gavin Nicol Date: Mon, 16 Sep 1996 20:08:31 GMT To: ricko@allette.com.au CC: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) >> This is a hack, and doesn't help with *initial* parsing of the >> document. > >why? what is *initial* parsing? The PI is part of the entity. The parser will *parse* it, which it cannot do *blindly*, so it can't parse this correctly without some way of priming it with the knowledge. You submit that having a PI at the start of every entity will solve the problem. For a limited set of encodings that is true, in the actual infinite set, I would not guarantee that it will be true. In addition, the PI is effectively a kind of header that the storage manager will be using. If that is the case, why not define a proper header syntax instead of a hack? I would prefer Content-Type: text/xml; charset=shift-jis [data] to any day. Also, the PI idea would lead to people trying something like: [some data> which is obviously broken (or at least hard to work with). HTML has a similar problem with META. From: Gavin Nicol Date: Mon, 16 Sep 1996 20:26:45 GMT To: ricko@allette.com.au CC: w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) >> Autodetection also fails very quickly when faces with a number of >> multibyte encodings. > >Yes, but I am not proposing that: the only thing autodetected is >whether the encoding is definitely 16-bit or not, for which you only >need to look at the first two octets (i.e. if one is zero valued). OK. What do you do if you have multiple pure 16 bit encodings? How about UCS-4? What happens if the 8 bit encoding is *not* ASCII compatible? From: Gavin Nicol Date: Mon, 16 Sep 1996 20:24:03 GMT To: ricko@allette.com.au CC: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Reads like ASCII (was Re: character sets ...) >> Autodetection also fails very quickly when faces with a number of >> multibyte encodings. > >This isnt autodetection except for 8 or 16 bit. When will it fail? As soon as the number of encodings increases. Date: Mon, 16 Sep 96 14:36:42 CDT From: Michael Sperberg-McQueen Subject: Re: (off-topic?)Re: XML character sets: a proposal To: Gavin Nicol , W3C SGML Working Group On Mon, 16 Sep 1996 19:24:04 GMT Gavin Nicol said: >Both the need for a standard set of attributes/elements for I18N >and the issue of glyph servers etc. fall outside the scope of this >WG. They may, though I haven't seen that prescribed for us anywhere. The possible need for a new attribute type (declared value) with a particular meaning, and the possible need for a new defaulting mechanism, on the other hand, are clearly within the scope of this WG. It seems to me that a new declared value for language/locale flags and a new keyword for attribute-value inheritance from enclosing elements may be worthwhile, even without reference to Asian languages. If such an attribute type also makes it feasible to handle the problems Rick Jelliffe referred to, then all the more reason to provide one. -C. M. Sperberg-McQueen From: Gavin Nicol Date: Mon, 16 Sep 1996 19:52:03 GMT To: U35395@UICVM.CC.UIC.EDU CC: w3c-sgml-wg@w3.org Subject: Re: character sets - a summary and a proposal >I'm confused. Where I come from, 'coded character set' is a mapping >between a set of characters and a set of bit patterns, not necessarily >the same length. Well, this is not exactly the way I'd define it. Anyway, we can say that there is a group of characters that each can be identified by a unique integer. >Under that definition, if 'the coded character set should be ISO 10646', >then we should not accept JIS 0208, Shift-JIS, EUC, ISO 8859, etc., Well, you see, here you show a fundamental mis-understanding. JIS 0208 is a coded character set, SHIFT-JIS is an encoding, as is EUC. ISO 8859-1 is a coded character set, and an encoding. The characters that SHIFT-JIS, EUC, and ISO 8859-1 map to are also present in ISO 10646. An encoding does *not* necessarily map to the integer for a character though in practise, most do. As such, so long as an encoding identifies the *characters* in ISO 10646, all is well >from a theoretical standpoint (we can even use BCTF). In actual fact, this is all just verbiage, and hair-splitting. The important point is that you can use these encodings in an ISO 10646 based system. >>Interoperability is something to be greatly desired, and in fact, the >>primary reason I got involved in HTML I18N was precisely >>that. However, I do not believe that at this time, we can get to a >>point where all XML systems will be able to process all XML >>documents. At some point in the future (3-5 years), perhaps. Now, no. > >This scares the pants off me. In 3-5 years, if XML is widely >adopted, it will be *impossible* to impose interoperability in the >form of required support for data streams in UTF-8 or UTF-16 or >whatever, because by then there will be legacy systems and legacy >data to be protected. The only way to achieve such uniformity is by >imposing it at the outset, when there is no XML legacy data, and we >have a free hand. Failing to ensure interoperability when we have >a free hand is not a good sign for our ability to achieve it later >when our hands will be tied by systems which have made use of whatever >freedom the spec gives them now. If this is true, then we are doomed to failure. I do not think this is true however. In the 3-5 intervening years, we will see a drastic improvement in tools, and they will be the answer to the interoperability problems we will suffer initially. Anyway, as I said before, interoperability is a dream unless you state precisely the domain in which you wish interoperability to occur. For example, the statement "all XML parsers will be able to parse all XML documents" is much more reasonable than "all XML systems will be able to process all XML documents". >>Autodetection fails abysmally as soon as you get more than a few >>encodings. > >Correct; that's why the proposal (a) limits itself to the cases of >UCS-4, UTF-16 / UCS-2, ISO 646 and compatible encodings including >UTF-8, and EBCDIC, and (b) requires an explicit label in the entity >regardless. I do not think this necessary. Storage/transmission systems should provide labelling for maximum interoperability. Autodetection is an artifact of a poorly designed system. >Autodetection in general fails when there are more than >a few encodings. We agree. The question in my mind is, does this >particular proposal for autodetection fail when we have the set of >encodings described? No, but then the encodings you select are arbitrary. If we added 3 or 4 more common encodings it would fail. >> 5) Client/server interaction will initially be primarily in the native >> encoding. >> 6) Over time, a transition will be made to UTF-8/UTF-16 (ie. as more >> and better tools become available). >> >>We should recognise, and accept this. > >Hmm. By analogy with this, HTML could have started by allowing any >existing 7-bit national character set, as well as proprietary >8-bit character sets, and hoped that eventually, with time, all HTML >users would migrate to ISO 8859-1. > >As it is, it started by prescribing 8859-1, which is a blessing >because it allows users to protest against servers which serve data >in native PC or native Mac format. Things are bad enough as they are; >would they really be better if sending data in Mac's proprietary >coded character set were legal HTML? > >On the whole, I think that HTML did the right thing in pressing for >a move from 7 to 8 bits. And I think XML should do the same in >pressing for a move to 16 bits. It is much easier to loosen XML >restrictions in later revisions than to tighten them and break >legacy data. HTML met the initial needs of the WWW with the ISO 8859-1 restriction (the syntax limited it to ISO 8859-1). This will change to ISO 10646 (the HTML I18N draft has now become Proposed Standard). Before the I18N draft, people were kludging things together in a non-conformant manner. With the I18N draft, they kludge things together in a conformant manner. Initially, HTTP and client provided few I18N capabilities (poor labelling, poor forms support etc.), now at least the *standards* are moving in the right way. HTML has been driven by necessity more than by design. With XML, we can design the system to support I18N, but provide flexibility in implementation so that we can basically *repeat* what happened to HTML, but in a way that was designed for. Date: Mon, 16 Sep 1996 16:52:43 -0400 To: w3c-sgml-wg@w3.org From: "Steven J. DeRose" Subject: Re: XML and required DTDs At 12:56 PM 09/14/96 CDT, Paul Grosso wrote: >If DTDs are optional for XML editors, someone will write an XML editor >that edits a document without a DTD. I would like to explore this >avenue more. (I am not necessarily against this--I can see some >advantages--but I think it has lots of ramifications.) This is certainly true, but even if DTDs are *required* someone will write an editor that is capable of editing even without them. This has long been the case with SGML, for which vi, emacs, and xedit are quite popular editors (alas!), and they are far less capable than that: they not only *can* function without a DTD, they *can't* make use of one if they are given one (these are quite different criteria). Even if you don't have a DTD around, there's a vast different between a hex dump of a GIF file, and an SGML document that's valid under a straightforward DTD. Why not take advantage of the difference so the SGML system can just deal? Date: Mon, 16 Sep 1996 16:52:44 -0400 To: Martin Bryan , w3c-sgml-wg@w3.org From: "Steven J. DeRose" Subject: Re: XML and required DTDs At 10:14 AM 09/15/96 +0100, Martin Bryan wrote: >- a known set of elements, with a DTD that is defined but not referenced, >that can be extended in a DOCTYPE subset, e.g. > >]> > >My idea here is to adopt an HTML approach of saying that there are a certain >set of elements that are useful in a wide range of situations (the basic >text elements) which all browsers would have default presentation >characteristics for, and for which there is an agreed set of model >relationships, which the user can modify by: Standardizing tag libraries is often useful, but has nothing to do with simplifying the language. I think the point of the "to DTD or not to DTD" discussion is that there is sooooooo little in SGML that prevents parsing perfectly without a DTD, that is seems a tad silly to force *every* application to take on the (currently huge) task of implementing all of the DTD stuff (the overwhelming majority of work in building a parser, as it is the overwhelming majority of the SGML grammar, and books on SGML, and so on). The question as I see it is: Why *force* everyone into a Hobson's choice: either add a huge amount to your implementation effort, or parse some documents *just slightly* wrong? Why force a 5-line Perl script to parse a DTD (try doing that in 5 lines!), track every content model, and all that, just so it can "ignore" the occasional RE that follows an included sub-element, but not that follows a proper sub-element? I say, we shouldn't. Because if we try, many will decide the slight difference isn't worth the pain, and then we'll end up with just slightly inoperable systems -- yech! The only things blocking you from parsing one-entity minimal SGML document without a DTD are: a) Declared content (CDATA/RCDATA/EMPTY elements) b) The RE-ignoring rules. All of these constructs, interestingly, have other problems besides forcing you to have a DTD. If we address the other problems, we get DTD-less parsing capability for *free*. Sounds like a pretty nice bonus to me. Obviously if you want to use additional constructs, you must declare them (like external entities, attribute defaults, and notations). But why force people to declare more than they inherently must? From: Gavin Nicol Date: Mon, 16 Sep 1996 19:24:04 GMT To: U35395@UICVM.CC.UIC.EDU CC: ricko@allette.com.au, w3c-sgml-wg@w3.org Subject: Re: (off-topic?)Re: XML character sets: a proposal Both the need for a standard set of attributes/elements for I18N and the issue of glyph servers etc. fall outside the scope of this WG. Date: Mon, 16 Sep 1996 17:50:11 -0400 To: w3c-sgml-wg@w3.org From: "Eve L. Maler" Subject: Compatibility issues and principle #3 Cc: elm@arbortext.com Our design principle #3 currently reads, "XML shall be compatible with SGML." I'm hoping we're ready to get more specific about what this means. (I apologize for starting a thread and then bolting -- I'll be largely out of email-touch from tomorrow through Sep 26.) Here are the questions I think we need to be able to answer: o Who is the customer/audience for XML -- existing robust-SGML users, existing Web/HTML users who are not SGML-aware, or both? What "legacy information" (if any) should work with XML with no transformation? I don't think we should penalize existing users of SGML for using its awkward features before we came along with our "cleanup effort." At the same time, many undersupported features of SGML are supported *somewhere* by *someone* -- how far do we go? For example: As Paul G. mentioned, even HTML uses EMPTY elements in a totally natural way. I don't think it's reasonable to make these documents, and millions of pages of other SGML documents, change over to an EMPTY-less model. At the same time, CDATA and RCDATA elements are pretty widely supported, but little used because of the authoring complications they introduce. Would these be fair to toss? I'd rather think of XML as an effort to define a cohesive SGML "application profile" that benefits both tool creators and document creators, rather than a set of unrelated cool hacks that make it easier to write parsers. If we're trying to define the intersection of useful, understandable, and implementable characteristics, I don't want to pull the drawstring too tight. Also, I'm wary of playing around with delimiters and shortrefs when there are already widely used methods available for doing something. (E.g., what's wrong with using < instead of a backslash to escape left angle brackets? It's consistent with treatment of other special characters, and it's one thing that HTML hackers have adapted to quite easily.) o What should happen when existing SGML documents (including valid HTML) are processed by XML tools? Should a "round trip" between the two forms be possible, or is only XML->SGML or SGML->XML okay? If we provide for only XML->SGML, then I think we're setting up a situation where XML is like "SGML with macros," which gets expanded as soon as it gets into a "real" system. If we provide only for SGML->XML, then XML may just fill the role of "HTML Heavy" -- good only for static delivery, not for serious work. But round trips could be problematic, if (e.g.) we remove the ability to have EMPTY elements and users expect the ESIS of each form to be equivalent. I think a round trip should be weighted as highly desirable, with transforms relatively undesirable. Eve From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Steven J. DeRose" Cc: Martin Bryan , w3c-sgml-wg@w3.org Subject: Re: XML and required DTDs Date: Tue, 17 Sep 1996 01:02:01 GMT On Mon, 16 Sep 1996 16:52:44 -0400, "Steven J. DeRose" wrote: >The only things blocking you from parsing one-entity minimal SGML document >without a DTD are: > > a) Declared content (CDATA/RCDATA/EMPTY elements) > b) The RE-ignoring rules. > Exactly! Just eliminate declared content and mixed content and you've solved the problem. We don't need those constructs for XML, they are just forms of markup minimization parading under other names. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Gavin Nicol Cc: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: Mixed content Date: Tue, 17 Sep 1996 01:02:03 GMT On Mon, 16 Sep 1996 14:02:27 GMT, Gavin Nicol wrote: >>

>>Text of p. >>

>> >>Case 2: >> >>

Text of p.

> >My understanding is that these would return *different* parse >results. We could (if it is so desired) make it an application >convention to strip leading and trailing whitespace. > > Thanks Gavin. That's what I feared, so eliminating RE and RS characters doesn't really solve the problem. Making it an "application convention" to strip some whitespace just shifts the burden from a few parsers to all applications and increases the chance of inconsistent treatment. Better, I think, to eliminate mixed content altogether as James Clark has proposed. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Eve L. Maler" Cc: w3c-sgml-wg@w3.org, elm@arbortext.com Subject: Re: Compatibility issues and principle #3 Date: Tue, 17 Sep 1996 01:02:06 GMT On Mon, 16 Sep 1996 17:50:11 -0400, "Eve L. Maler" wrote: >Our design principle #3 currently reads, "XML shall be compatible with >SGML." I'm hoping we're ready to get more specific about what this >means. (I apologize for starting a thread and then bolting -- I'll be >largely out of email-touch from tomorrow through Sep 26.) > If we provide for only XML->SGML, then I think we're setting up a > situation where XML is like "SGML with macros," which gets expanded > as soon as it gets into a "real" system. If we provide only for > SGML->XML, then XML may just fill the role of "HTML Heavy" -- good > only for static delivery, not for serious work. But round trips could > be problematic, if (e.g.) we remove the ability to have EMPTY elements > and users expect the ESIS of each form to be equivalent. > > I think a round trip should be weighted as highly desirable, with > transforms relatively undesirable. ESIS takes pains to not distinguish empty elements from others. It signals the start and end of every element whether or not it is empty. And if it is empty, "ESIS does not indicate why it was empty; that is, whether it was declared to be empty, or whether an explicit content reference occurred, or whether it just happened to contain no data characters." That is why we can safely eliminate declared content, including declared EMPTY element types. In all cases, there is an equivalent valid SGML DTD for which there is a document instance that yields the same ESIS. This is as close to a round trip as you can hope to get when going from a rich representation to a poorer one. You'll need a trivial transformation when generating the XML and when getting it back, but there is no loss of ESIS information. Best regards, Charles -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Tim Bray Cc: w3c-sgml-wg@w3.org Subject: Re: Mixed content Date: Tue, 17 Sep 1996 01:02:04 GMT On Mon, 16 Sep 1996 08:15:33 +0000, Tim Bray wrote: >At 02:02 PM 16/09/96 GMT, Gavin Nicol wrote: >>>

>>>Text of p. >>>

>>> >>>

Text of p.

>> >>My understanding is that these would return *different* parse >>results. We could (if it is so desired) make it an application >>convention to strip leading and trailing whitespace. > >I agree.

Listen to my heart beat.

would *not*, in this cenario, be >the same as >

>Listen to my heart beat. >

> >Is this a problem? It's certainly easy to explain. I think it is a problem. It will mean that anyone creating XML with any of the popular non-XML/SGML-aware editors will have to be told why he can't put tags on a separate line from data. If XML allows PIs, you will also have to explain why a PI can't be on a line by itself. (And so on for marked sections and inclusions and comment declarations.) > >With the proper SGML declaration, they will also parse differently in SGML. > >The SGML setup makes it easy to set markup apart with whitespace, making >it easily visually distinguishable. This is good. The price is that it >becomes difficult for ordinary people to tell when some white space is >actually data. This is bad. The trade-off is probably reasonable, but >XML need not choose the same trade-off. You will still wind up trying to explain the equivalent of SGML's RE/RS handling rules, but with the additional burden of forcing the user to implement the rules himself. The _only_ way to kill this pernicious beast is to eliminate mixed content, which XML can easily live without. Best regards, Charles -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Mon, 16 Sep 96 18:46:22 CDT From: Michael Sperberg-McQueen Subject: equivalent power in SGML and XML To: W3C SGML Working Group Eve Maler's note with questions about the equivalence of SGML and XML raises serious questions, but before I try to answer them I want to pose even more questions. I apologize for the length of this posting, but I'm just working out these ideas and I don't understand them well enough to summarize them crisply. We say that XML and SGML should have 'essentially the same expressive power', and my own attempt to answer Eve's questions depends on clarifying what that means. First, let's distinguish equivalence among document instances from equivalence among DTDs. I Instance equivalence Instances can be: - byte-equivalent, if each octet of each entity in the document is the same - character-equivalent, if each 'character' in each entity is the same, but the encoding of the character may differ: a document in Ascii and the same document after translation into Ebcdic is a common-place and old-fashioned example; the same document in ISO 8859-1 and its equivalent in UCS-2 or UTF-8 is a similar and forward-looking example. - external-entity-equivalent, if each character in each *external* entity is the same (after resolving references to internal entities and character references) - ESIS-equivalent, if their document elements are E-equivalent, where elements are E-equivalent if and only if: * they have the same generic identifier * they have the same set of attributes * corresponding attributes have the same values (after expansion of entity references) * they have the same number of children, and all pairs of corresponding children are E-equivalent and character data chunks are E-equivalent if they consist of the same sequence of 'characters' (which does not mean they are in the same encoding). This may differ slightly in detail from the definition of ESIS in the Corrigendum, but not intentionally. - EE-ESIS-equivalent, if they are ESIS equivalent and the beginnings and endings of external entities occur at the same places in the ESIS. (This isn't well phrased; I hope it's reasonably clear what I mean: ESIS equivalent, plus division into external entities is preserved.) I need some examples: 1

Hello, world!

is byte equivalent to

Hello, world!

and to the equivalent in EBCDIC (or in ASCII, I should say, because as I write this, it *is* in EBCDIC. 2 If we have and then

Hello, world!

is ESIS-equivalent, but not character equivalent, to

&greet;, &someone;! 3 A document divided into one file per chapter is ESIS-equivalent, but not EE-ESIS-equivalent, with the same document in a single file (i.e. with all references to external entities resolved). If two instances have the same DTD and are byte- or character-equivalent, they are guaranteed ESIS-equivalent, but not vice versa. If they are external-entity equivalent and have the same DTD, they are ESIS-equivalent; I can't think of a way to preserve entity equivalence as defined here without preserving ESIS equivalence. II DTD equivalence Now for DTD equivalence. Normally we think of this in terms of the set of effective declarations, after resolution of entity references and marked sections, etc. Here, though, I think we should focus on the languages defined by two DTDs. We need to clarify what we mean by 'equivalent expressive power'. In general (the computer scientists among us may correct me freely) I think grammars are equivalent if they define the same languages, i.e. if they accept the same strings as members of the language and reject the same non-members. One formalism F has the same power as another formalism G if for each grammar written using F, an equivalent grammar may be written using G, and vice versa. F is more powerful than G if the set of languages definable by F's grammars is a proper superset of the set of languages definable by G's grammars. Now, if XML's DTDs are to be 'as powerful' as SGML's, they have to be able to define the same set of languages: for every SGML DTD there must be an equivalent XML DTD accepting and rejecting exactly the same set of valid and invalid document instances. This, we cannot do, if we are set on doing without CDATA and RCDATA elements. (Or even DATATAG and RANK!) On the other hand, we can say that for our purposes XML DTDs and SGML DTDs have 'essentially' the same expressive power if for any SGML DTD, there is an XML DTD such that: - for every document instance accepted by the SGML DTD, the XML DTD accepts a document which is 'equivalent' - for every document instance accepted by the XML DTD, the SGML DTD accepts a document which is 'equivalent'. We can choose, I think, to require that the documents be ESIS- or EE-ESIS-equivalent, or apply some other test, depending on how strict we want to be. We could also insist on byte-equivalence or character-equivalence, but if we do so we can't make any real changes to DTDs. Dropping DATACHAR and RANK and SHORTREF don't bother people, I think, because we're all used the notion of normalizing SGML documents, and the normalized forms of documents share ESIS with their originals, but aren't byte-equivalent. If we drop CDATA and RCDATA elements, we can meet the same standard: any CDATA element in an accepted SGML instance can be replaced, in XML, with the same thing, but with < and & escaped using any of the various possibilities discussed earlier. If we drop the & connector from the content model syntax, as I suspect some people would like to do, then we haven't changed the expressive power of the language because (in theory) it is always possible to replace a content model using & with an equivalent one not using &. It may be ugly and very very long -- and for these reasons no one is likely to do it -- but it's *possible*. Dropping EMPTY, on the other hand, would seriously affect the expressive power of XML DTDs, since we could not guarantee the second DTD-equivalence clause above. III What is needed? I think we should shoot for the following: - any existing SGML document can be translated into an EE-ESIS equivalent document (i.e. same ESIS, same gross entity structure, but there may be more, or fewer, internal entities and references) - any existing SGML DTD can be translated into an XML DTD which recognizes a set of documents EE-ESIS-equivalent to the original DTD. - a document translated from SGML into XML into SGML should be EE-ESIS-equivalent to the original; same for one that goes X-S-X. I don't think we should shoot for more than EE-ESIS equivalence after round-trip transmission. Bytes will change. References to internal entities will be resolved, or introduced. That means there may be additional entity declarations in the DTD, and thus we can't even guarantee that the document will parse with the same DTD. I think we can guarantee that it will parse with the same set of effective element, attlist, and notation declarations. -C. M. Sperberg-McQueen Date: Mon, 16 Sep 1996 21:13:26 -0400 To: Michael Sperberg-McQueen , W3C SGML Working Group From: "Eve L. Maler" Subject: Re: equivalent power in SGML and XML [Thanks, Charles, for correcting me on the ESIS/EMPTY issue...] Michael -- compared to my poor attempt, your mail seemed crystal clear to me! You've explained why EMPTY isn't just another random kind of declared content. I could easily agree to cut out *multiple* ways of expressing the same markup language without getting down to *zero* ways, which meets our principle of having only one way to do things. If we agreed to go by this rule, mixed content would have to stay in... Of course, we could still decide that some things have to go, but I think the criteria should be much more stringent. (Hmm, could RANK be simulated for all practical purposes by GIs that have the number in them?) Eve At 06:46 PM 9/16/96 CDT, Michael Sperberg-McQueen wrote: >Eve Maler's note with questions about the equivalence of SGML and XML >raises serious questions, but before I try to answer them I want to >pose even more questions. I apologize for the length of this posting, >but I'm just working out these ideas and I don't understand them >well enough to summarize them crisply. > >We say that XML and SGML should have 'essentially the same expressive >power', and my own attempt to answer Eve's questions depends on >clarifying what that means. > >First, let's distinguish equivalence among document instances from >equivalence among DTDs. > >I Instance equivalence > >Instances can be: > - byte-equivalent, if each octet of each entity in the document is > the same > - character-equivalent, if each 'character' in each entity is the same, > but the encoding of the character may differ: a document in > Ascii and the same document after translation into Ebcdic is a > common-place and old-fashioned example; the same document in > ISO 8859-1 and its equivalent in UCS-2 or UTF-8 is a similar and > forward-looking example. > - external-entity-equivalent, if each character in each *external* > entity is the same (after resolving references to internal entities > and character references) > - ESIS-equivalent, if their document elements are E-equivalent, where > elements are E-equivalent if and only if: > * they have the same generic identifier > * they have the same set of attributes > * corresponding attributes have the same values (after > expansion of entity references) > * they have the same number of children, and all pairs of > corresponding children are E-equivalent > and character data chunks are E-equivalent if they consist of > the same sequence of 'characters' (which does not mean they are > in the same encoding). This may differ slightly in detail from > the definition of ESIS in the Corrigendum, but not intentionally. > - EE-ESIS-equivalent, if they are ESIS equivalent and the beginnings > and endings of external entities occur at the same places in the > ESIS. (This isn't well phrased; I hope it's reasonably clear > what I mean: ESIS equivalent, plus division into external entities > is preserved.) > >I need some examples: > >1

Hello, world!

is byte equivalent to

Hello, world!

>and to the equivalent in EBCDIC (or in ASCII, I should say, because >as I write this, it *is* in EBCDIC. > >2 If we have and >then

Hello, world!

is ESIS-equivalent, but not character >equivalent, to

&greet;, &someone;! > >3 A document divided into one file per chapter is ESIS-equivalent, but >not EE-ESIS-equivalent, with the same document in a single file (i.e. >with all references to external entities resolved). > >If two instances have the same DTD and are byte- or >character-equivalent, they are guaranteed ESIS-equivalent, but not vice >versa. If they are external-entity equivalent and have the same DTD, >they are ESIS-equivalent; I can't think of a way to preserve entity >equivalence as defined here without preserving ESIS equivalence. > > >II DTD equivalence > >Now for DTD equivalence. Normally we think of this in terms of the set >of effective declarations, after resolution of entity references and >marked sections, etc. Here, though, I think we should focus on the >languages defined by two DTDs. We need to clarify what we mean by >'equivalent expressive power'. > >In general (the computer scientists among us may correct me freely) I >think grammars are equivalent if they define the same languages, i.e. if >they accept the same strings as members of the language and reject the >same non-members. One formalism F has the same power as another >formalism G if for each grammar written using F, an equivalent grammar >may be written using G, and vice versa. F is more powerful than G >if the set of languages definable by F's grammars is a proper superset >of the set of languages definable by G's grammars. > >Now, if XML's DTDs are to be 'as powerful' as SGML's, they have to >be able to define the same set of languages: for every SGML DTD there >must be an equivalent XML DTD accepting and rejecting exactly the same >set of valid and invalid document instances. This, we cannot do, >if we are set on doing without CDATA and RCDATA elements. (Or even >DATATAG and RANK!) > >On the other hand, we can say that for our purposes XML DTDs and SGML >DTDs have 'essentially' the same expressive power if for any SGML >DTD, there is an XML DTD such that: > > - for every document instance accepted by the SGML DTD, the XML > DTD accepts a document which is 'equivalent' > - for every document instance accepted by the XML DTD, the SGML > DTD accepts a document which is 'equivalent'. > >We can choose, I think, to require that the documents be ESIS- or >EE-ESIS-equivalent, or apply some other test, depending on how strict we >want to be. We could also insist on byte-equivalence or >character-equivalence, but if we do so we can't make any real changes to >DTDs. > >Dropping DATACHAR and RANK and SHORTREF don't bother people, I think, >because we're all used the notion of normalizing SGML documents, and the >normalized forms of documents share ESIS with their originals, but >aren't byte-equivalent. > >If we drop CDATA and RCDATA elements, we can meet the same standard: >any CDATA element in an accepted SGML instance can be replaced, in XML, >with the same thing, but with < and & escaped using any of the various >possibilities discussed earlier. If we drop the & connector from the >content model syntax, as I suspect some people would like to do, then we >haven't changed the expressive power of the language because (in theory) >it is always possible to replace a content model using & with an >equivalent one not using &. It may be ugly and very very long -- and >for these reasons no one is likely to do it -- but it's *possible*. > >Dropping EMPTY, on the other hand, would seriously affect the expressive >power of XML DTDs, since we could not guarantee the second >DTD-equivalence clause above. > > >III What is needed? > >I think we should shoot for the following: > > - any existing SGML document can be translated into an EE-ESIS >equivalent document (i.e. same ESIS, same gross entity structure, but >there may be more, or fewer, internal entities and references) > - any existing SGML DTD can be translated into an XML DTD which >recognizes a set of documents EE-ESIS-equivalent to the original DTD. > - a document translated from SGML into XML into SGML should be >EE-ESIS-equivalent to the original; same for one that goes X-S-X. > >I don't think we should shoot for more than EE-ESIS equivalence after >round-trip transmission. Bytes will change. References to internal >entities will be resolved, or introduced. That means there may be >additional entity declarations in the DTD, and thus we can't even >guarantee that the document will parse with the same DTD. I think we >can guarantee that it will parse with the same set of effective element, >attlist, and notation declarations. > > > > >-C. M. Sperberg-McQueen > > > From: "Henry S. Thompson" Date: Tue, 17 Sep 96 03:51:27 BST To: Charles@sgmlsource.com Subject: Re: XML and required DTDs <323dee3c.23027901@mail.alink.net> Cc: w3c-sgml-wg@w3.org Charles writes: > > On Mon, 16 Sep 1996 16:52:44 -0400, "Steven J. DeRose" wrote: > > >The only things blocking you from parsing one-entity minimal SGML document > >without a DTD are: > > > > a) Declared content (CDATA/RCDATA/EMPTY elements) > > b) The RE-ignoring rules. > > > > Exactly! Just eliminate declared content and mixed content and > you've solved the problem. We don't need those constructs for XML, > they are just forms of markup minimization parading under other > names. That analysis is correct, but it would in my view none-the-less be a mistake to go down that road. It would grandfather out a huge percentage of existing documents, and require huge amounts of obfuscatory markup, e.g.

TheGretellost toConstitution in1966.

[vc for 'vanilla content'] Or are you assuming we are allowing tag minimisation after all? I thought it was out the door . . . ht [apologies to Americas' Cup mavens, I know the second name and date are probably wrong] Date: Mon, 16 Sep 1996 22:17:47 -0600 (MDT) From: Bill Lindsey To: Michael Sperberg-McQueen cc: W3C SGML Working Group Subject: Re: equivalent power in SGML and XML This was an excellent summary of the types of equivalence between the documents translated from their original un-XML-fettered SGML representation to XML and back. I agree that EE-ESIS is an appropriate level. There is another type of equivalance I think I've heard some posters in this forum hint at: I don't know what to call it but "identity". There is some set of SGML documents existing today, which will need no translation to XML, XML will be defined such that these documents already will be valid XML. Is it a design goal to ensure that some percentage of today's SGML documents are "grandfathered" in, or is it enough that the EE-ESIS translation is available? If it is the former case, how do we decide which class(es) of documents shall be supported? -Bill -- William D. Lindsey blindsey@bdmtech.com +1 (303) 672-8954 From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Eve L. Maler" Cc: Michael Sperberg-McQueen , W3C SGML Working Group Subject: Re: equivalent power in SGML and XML Date: Tue, 17 Sep 1996 09:45:49 GMT On Mon, 16 Sep 1996 21:13:26 -0400, "Eve L. Maler" wrote: >[Thanks, Charles, for correcting me on the ESIS/EMPTY issue...] My pleasure ... now I'll try to show why that means that EMPTY is unnecessary. > >Michael -- compared to my poor attempt, your mail seemed crystal clear >to me! You've explained why EMPTY isn't just another random kind of >declared content. He's tried to, but I think he is wrong. >> >>Dropping EMPTY, on the other hand, would seriously affect the expressive >>power of XML DTDs, since we could not guarantee the second >>DTD-equivalence clause above. I don't think so. The effective normalized (i.e., ESIS or semantic grove form) of all empty elements, whether declared or spontaneously empty, is: "" Therefore, a content model of (#PCDATA) is equivalent to a declared value of EMPTY. (There is a difference in validatability by an SGML parser, but that is not an issue here ... and isn't much of an issue anywhere.) >> >> >>III What is needed? >> >>I think we should shoot for the following: >> >> - any existing SGML document can be translated into an EE-ESIS >>equivalent document (i.e. same ESIS, same gross entity structure, but >>there may be more, or fewer, internal entities and references) >> - any existing SGML DTD can be translated into an XML DTD which >>recognizes a set of documents EE-ESIS-equivalent to the original DTD. >> - a document translated from SGML into XML into SGML should be >>EE-ESIS-equivalent to the original; same for one that goes X-S-X. Eliminating declared content (including EMPTY) and mixed content is all that is needed to accomplish this goal. Translating from the SGML instance with declared and mixed content to the XML without those things is trivial and is reversible without information loss. The resulting XML can be parsed (but not validated, of course) without reference to a DTD. This solution does not in any way dictate the syntax of XML DTDs. What it does do is remove the need for new markup conventions or syntax in XML instances, or the need to require XML applications to enforce conventions for RE handling. This solution also has the significant advantage, described by Paul Grosso, in that existing SGML tools could easily create and receive XML documents, and that essentially all existing document types could easily be used for XML. And yes, Eve, I think you are right in thinking this technique would work for RANK, and possibly even DATATAG as well. Best regards, Charles -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Henry S. Thompson" Cc: w3c-sgml-wg@w3.org Subject: Re: XML and required DTDs Date: Tue, 17 Sep 1996 09:45:52 GMT On Tue, 17 Sep 96 03:51:27 BST, "Henry S. Thompson" wrote: >Charles writes: > >> >> On Mon, 16 Sep 1996 16:52:44 -0400, "Steven J. DeRose" wrote: >> >> >The only things blocking you from parsing one-entity minimal SGML document >> >without a DTD are: >> > >> > a) Declared content (CDATA/RCDATA/EMPTY elements) >> > b) The RE-ignoring rules. >> > >> >> Exactly! Just eliminate declared content and mixed content and >> you've solved the problem. We don't need those constructs for XML, >> they are just forms of markup minimization parading under other >> names. > >That analysis is correct, but it would in my view none-the-less be a >mistake to go down that road. It would grandfather out a huge >percentage of existing documents. Not at all. See my later posting on that point. >and require huge amounts of >obfuscatory markup, e.g. > >

TheGretellost toConstitution >in1966.

> >[vc for 'vanilla content'] > This markup is not obfuscatory, just verbose. My understanding is that for XML, verbosity is preferable to complexity. >Or are you assuming we are allowing tag minimisation after all? I >thought it was out the door . . . I think tag omission is at the door because it requires access to the DTD. On the other hand, NET is trivial to parse without a DTD and would eliminate about 50% of the added markup. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- To: Bill Lindsey , Michael Sperberg-McQueen From: Martin Bryan Subject: Re: equivalent power in SGML and XML Cc: W3C SGML Working Group Date: Tue, 17 Sep 1996 10:54:23 +0100 At 05:17 AM 17/9/96 +0100, Bill Lindsey wrote: >Is it a design goal to ensure that some percentage of today's >SGML documents are "grandfathered" in, or is it enough that the >EE-ESIS translation is available? If it is the former case, >how do we decide which class(es) of documents shall be supported? The set of documents I would like to see "grandfathered in" is that set defined in (valid) HTML. If an XML browser cannot read that set of raw HTML documents that are valid according to the 2.0 DTD (or later versions) then it will not be of much practical use, and will be ignored by the majority of potential users. (We would still have the problem of the 60% of invalid documents, but hopefully this situation will get better once standard WP tools start offering automatic conversion to HTML.) Martin Bryan ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ >Received: by chuckd.allette.com.au.allette.com.au id AA05660 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Tue, 17 Sep 1996 19:52:41 +1000 Date: Tue, 17 Sep 1996 19:52:41 +1000 (EST) From: Rick Jelliffe To: Gavin Nicol Cc: w3c-sgml-wg@w3.org Subject: Limited modified eclectism (was Re: Reads like ASCII) On Mon, 16 Sep 1996, Gavin Nicol wrote: > In addition, the PI is effectively a kind of header that the storage > manager will be using. If that is the case, why not define a proper > header syntax instead of a hack? Yes. Here is what it achieves (it = limited modified eclectisism): 1) File can be valid SGML (as external general entity) 2) File can be a canonical storage form, does not require any external labelling. 3) Encoding can be determined merely by looking at the header with any standard tool that can do an ascii dump of a file (UNIX od, DOS Xtree, etc) 4) Operates with ASCII, ISO 646, ISO 8859-n (+ Windows extended versions), ISO10646 (Unicode UCS-2/UTF16, UCS4, UTF8, reuters method(?), variable and fixed EUC for at least J & K & maybe C, shiftJIS (& maybe K version) (I don't have Lunde with me to check up about Big5), + any state-driven encoding that has ISO646 in its inital state. 5) File compatible with UNIX magic number system (not greatly important!) > I would prefer > > Content-Type: text/xml; charset=shift-jis > > [data] What if the PI was the storage form of it? There is no reason why the data stream form of it couldn't send down Content-Type: text/xml; charset=shift-jis (or even strip out the PI and reinstert it when the data is stored). Perhaps there are two different needs here that are reconcilable: MIME systems need MIME headers, SGML systems accept dumb plain text files. If the PI was transformable into the MIME header and back, both needs are met. Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ Date: Tue, 17 Sep 1996 11:41:31 +0000 From: jjc@jclark.com (James Clark) To: w3c-sgml-wg@w3.org Subject: Re: RS/RE considered confusing? The Durand/Nicol proposal on RS/RE is, if I've understood it correctly, that when an XML document is parsed by an SGML parser, then it will use an SGML declaration that includes something like: FUNCTION RE 65531 -- not a legal ISO 10646 character -- RS 65532 -- not a legal ISO 10646 character -- SPACE 32 TAB SEPCHAR 9 CR SEPCHAR 10 LF SEPCHAR 13 I have a number of concerns about this proposal. 1. Most SGML parsers won't handle such an SGML declaration. The practical benefit for XML of SGML compatibility is to enable SGML tools to be used on XML documents. If XML uses features of SGML that aren't implemented in most SGML parsers, this practical benefit is lost. 2. It's not clear to me that this in general will work with a conforming SGML system. The entity manager is supposed to transform whatever mechanism the OS uses for representing lines into RS/RE. So for example, on a Unix system it is supposed to replace the newline that the terminates each line with an RE and insert an RS at the beginning of the line. So with the SGML declaration above, the entity manager should put character number 65532 at the beginning of each line and 65531 at the end. (SP doesn't actually do this: the entity manager always uses 10 and 13 as RS and RE even if they have been changed for the parser by the SGML declaration. But I don't think I could argue that SP is correct to do so.) 3. This proposal isn't solving the RE problem, it is (at most) working around it. I would like to see XML be something that could be embraced by WG8 and incorporated into the SGML revision. I can't see this sort of hack getting into the SGML revision. 4. It is a fact of life that OSs delimit lines using different methods. One of the ideas underlying the RS/RE concept in SGML is that these different methods should canonicalized into a single form, so that applications are isolated from these differences. This is a sound idea which is used in other places (eg the C language), and I don't think we should throw it out. I certainly don't want a style sheet language to have to care about how lines are delimited. 5. In mixed content there are some newlines that must get ignored by some part of the system (whether the parser or the application). For example, if I have

This is a paragraph of text. I don't want the newline after the

to result in a space at the beginning of the paragraph. Who is going to specify these rules? Are TEI, DocBook etc each going to specify their own rules about RE significance? 6. Consider a paragraph like this:

A line Another line The SGML rules specify that the newline folling the comment is ignored. Now an XML application won't see the comment, so an XML application will see the same information for this document as for:

A line Another line But the SGML rules say that the newline that terminates the blank line is *not* ignored. This means that an XML application wouldn't be able to implement rules that process existing SGML documents (which don't redeclare RS/RE) in the same way as SGML systems. 7. Moving the responsibility of ignoring newlines to a part of the system other than the XML parser is going to mean that XML documents and SGML documents will need different processing. For example, when I write a style sheet I would have to know whether I've got an SGML document or an XML document so that I can specify that newlines are ignored in the XML document in the way I want. There may be other problems, but that's all I can think of for now. There are a few of other approaches that occur to me for dealing with the RS/RE problem: a. An approch similar to HTML. In HTML (except in PRE), line terminators are just white-space, adjacent white-space is collapsed and leading/trailing whitespace in an element is stripped. The interesting thing about this is that if you perform this process it doesn't matter whether or not you ignored the REs that SGML said you should: you'll end up with the same result. Well, almost: you'll still have the problems with REs getting moved past inclusions and PIs. The main problem with this is how to handle verbatim type elements. Can we live without these? b. Another approach is to restrict what's allowed in mixed content. For example, you could say that PIs were only allowed in element content; this avoid the problem of the interaction of REs and PIs in mixed content. c. Another possible approach is not to try to be 100% compatible in this area with SGML *as it now is*. Instead aim to be compatible with SGML as it will be revised. Devise some simplification of the SGML rules that can be incorporated in the SGML revision (maybe enabled by some flag in the SGML declaration). These aren't mutually exclusive. I suspect that some combination may be the best way forward. James Date: Tue, 17 Sep 1996 11:56:19 +0000 From: jjc@jclark.com (James Clark) To: mtbryan@sgml.u-net.com Cc: w3c-sgml-wg@w3.org Subject: Re: equivalent power in SGML and XML > From: Martin Bryan > Date: Tue, 17 Sep 1996 10:54:23 +0100 > > The set of documents I would like to see "grandfathered in" is that set > defined in (valid) HTML. I just don't think this is practical. HTML relies OMITTAG (notably for P, LI, DT, DD). I don't see how we can possibly achieve our ease of implementation goals if we support OMITTAG in XML. > If an XML browser cannot read that set of raw HTML > documents that are valid according to the 2.0 DTD (or later versions) then > it will not be of much practical use, and will be ignored by the majority of > potential users. I don't need to use an XML browser to browse HTML; I've already got plenty of HTML browsers that do that just fine. James Date: Tue, 17 Sep 96 07:02:15 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: Empty endtags (Was: short-tag considered unhealthy) At 03:05 PM 9/15/96 -0400, Eve L. Maler wrote: >I'm not sure I'm following all the intricacies of this discussion at >this point, but I do think that compact=compact is a bit of a >strawman. In this instance, the DTD was designed to optimize manual >input by taking advantage of (abusing? getting cute with?) one small >corner of SGML. A more self-explanatory and convenient design would >have been something like spacing={"compact"|"normal"}. > >And I agree it would be great to have a Boolean declared value for >attributes, but this still doesn't get around cases like >style={"compact"|"normal"|"wide"}. > >I don't think it's stupid or counterintuitive to start requiring that >all "parameter values" (which is how most people understand attribute >values) be passed, in quotes, along with the "parameter name" that >they apply to, especially since attribute specifications aren't >position-sensitive. It's straightforward and consistent to require >name="value". XML should be easy for the person who is creating as well as the person who is writing the tool that is parsing it. It's a lot easier to remember: All attribute values in XML should be written as name="value" than it is to learn/remember: If the attribute has a declared value of CDATA, or if you need to specify a whitespace delimited set of values, or ... then you must use .. You get the picture. When I'm typing SGML, like Len, I let my fingers do what they're used to (not that I have any choice in the matter anyway) and put quotes around attribute values. When I'm typing HTML, I'm irritated by the fact that I don't feel completely comfortable when specifying attribute values 'cause I'm never 100% confident as to whether or not I should use quotes. HTML parsers are just too fragile. I like the simple approach: simple to explain, simple to remember, simple to type, and simple to parse. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 From: Gavin Nicol Date: Tue, 17 Sep 1996 12:37:19 GMT To: U35395@UICVM.CC.UIC.EDU CC: w3c-sgml-wg@w3.org Subject: Re: equivalent power in SGML and XML >I think we should shoot for the following: > > - any existing SGML document can be translated into an EE-ESIS >equivalent document (i.e. same ESIS, same gross entity structure, but >there may be more, or fewer, internal entities and references) > - any existing SGML DTD can be translated into an XML DTD which >recognizes a set of documents EE-ESIS-equivalent to the original DTD. > - a document translated from SGML into XML into SGML should be >EE-ESIS-equivalent to the original; same for one that goes X-S-X. This sounds like a reasonable goal. We should also be able to build EE-ESIS based test suites for parsers then too (perhaps have a parser verification suite and testing framework). To: W3C SGML Working Group From: Martin Bryan Subject: Re: character sets Date: Tue, 17 Sep 1996 13:41:25 +0100 In case anyone else has missed the fact in the holiday period, the latest version (05) of the i18n Internationalization of the HyperText Markup Language proposal, released in August, specifies UCS-4, as opposed to the UCS-2 proposed in the previous (04) draft. ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ From: Gavin Nicol Date: Tue, 17 Sep 1996 13:06:28 GMT To: ricko@allette.com.au CC: w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) >1) File can be valid SGML (as external general entity) True for systems that understand the hack. For systems that do not understand the hack, they will still need an external way of indicating the encoding. >2) File can be a canonical storage form, does not require any external >labelling. When I say external, I mean external to the actual entity. Metadata does not belong in the objects that it refers to. >3) Encoding can be determined merely by looking at the header with any >standard tool that can do an ascii dump of a file (UNIX od, DOS Xtree, >etc) For only a limited set of encodings. Equally true for *.mim. >(or even strip out the PI and reinstert it when the data is stored). >Perhaps there are two different needs here that are reconcilable: >MIME systems need MIME headers, SGML systems accept dumb plain text >files. If the PI was transformable into the MIME header and back, >both needs are met. I'll give you one more reason to favor *.mim. Many SGML systems, and probably most XML systems, will have the ability to use the HTTP protocol to retrieve entities (SP already has this capability). This effectively means that they must have the ability to parse HTTP headers, which have exactly the same syntax as the *.mim file format headers. There is a lot of *free* code for parsing MIME bodies. FSI's also provide an equivalent functionality, and we should probably encourage deployment of them. However, this code will be more specific to XML/SGML. Summary of *.mim benefits: 1) Compatability with HTTP/MIME 2) Free code libraries (PERL, TCL, C, C++, JAVA) 3) Extensibility (you can add other parameters like Content-Encoding) 4) Doesn't confluge data and meta-data 5) Isn't a hack. From: Gavin Nicol Date: Tue, 17 Sep 1996 13:17:57 GMT To: jjc@jclark.com CC: w3c-sgml-wg@w3.org Subject: Re: RS/RE considered confusing? >The Durand/Nicol proposal on RS/RE is, if I've understood it >correctly, that when an XML document is parsed by an SGML parser, then >it will use an SGML declaration that includes something like: > > FUNCTION > RE 65531 -- not a legal ISO 10646 character -- > RS 65532 -- not a legal ISO 10646 character -- > SPACE 32 > TAB SEPCHAR 9 > CR SEPCHAR 10 > LF SEPCHAR 13 Correct. >1. Most SGML parsers won't handle such an SGML declaration. The >practical benefit for XML of SGML compatibility is to enable SGML >tools to be used on XML documents. If XML uses features of SGML that >aren't implemented in most SGML parsers, this practical benefit is >lost. Does SP allow this? >2. It's not clear to me that this in general will work with a >conforming SGML system. The entity manager is supposed to transform >whatever mechanism the OS uses for representing lines into RS/RE. I have never heard that "lines" are equivalent to "records". Many OS's do not have "records" at all. >4. It is a fact of life that OSs delimit lines using different >methods. One of the ideas underlying the RS/RE concept in SGML is >that these different methods should canonicalized into a single form, >so that applications are isolated from these differences. Again, who says that lines and records are equivalent? >5. In mixed content there are some newlines that must get ignored by >some part of the system (whether the parser or the application). For >example, if I have > >

>This is a paragraph >of text. > >I don't want the newline after the

to result in a space at the >beginning of the paragraph. What do you do if you *do* want a linefeed to occur there? It's just as easy to tell people "if you don't want a space, you must start up hard against the tag":

This is a paragraph. >6. Consider a paragraph like this: .... >The SGML rules specify that the newline folling the comment is >ignored. Does SGML actually say *newline* or record end? >7. Moving the responsibility of ignoring newlines to a part of the >system other than the XML parser is going to mean that XML documents >and SGML documents will need different processing. For example, when >I write a style sheet I would have to know whether I've got an SGML >document or an XML document so that I can specify that newlines are >ignored in the XML document in the way I want. I do not think this is true if you have a *conformant* SGML system that can accept and correctly use a declaration similar to XML. This is probably the fundamental problem: many SGML systems are not conformant in this respect. From: Gavin Nicol Date: Tue, 17 Sep 1996 13:21:19 GMT To: mtbryan@sgml.u-net.com CC: w3c-sgml-wg@w3.org Subject: Re: character sets >In case anyone else has missed the fact in the holiday period, the latest >version (05) of the i18n Internationalization of the HyperText Markup >Language proposal, released in August, specifies UCS-4, as opposed to the >UCS-2 proposed in the previous (04) draft. ISO 10646-BMP was a mistake. In my original proposal to HTML WG I proposed ISO 10646 (full 32 bit). This is a correction. Date: Tue, 17 Sep 1996 14:23:49 +0000 From: jjc@jclark.com (James Clark) To: gtn@ebt.com Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE considered confusing? > From: Gavin Nicol > Date: Tue, 17 Sep 1996 13:17:57 GMT > >1. Most SGML parsers won't handle such an SGML declaration. The > >practical benefit for XML of SGML compatibility is to enable SGML > >tools to be used on XML documents. If XML uses features of SGML that > >aren't implemented in most SGML parsers, this practical benefit is > >lost. > > Does SP allow this? Yes (at least in the multi-byte version). Do any others? > >2. It's not clear to me that this in general will work with a > >conforming SGML system. The entity manager is supposed to transform > >whatever mechanism the OS uses for representing lines into RS/RE. > > I have never heard that "lines" are equivalent to "records". My understanding is that the term "record" as used by SGML means the same as "line", in the sense of input line. It's called "record" rather than "line" to distinguish it from line in the sense of a formatted, output line. See clause 4.252 of the standard. James Date: Tue, 17 Sep 1996 09:09:46 -0500 From: Len Bullard To: Bill Lindsey CC: Michael Sperberg-McQueen , W3C SGML Working Group Subject: Re: equivalent power in SGML and XML Bill Lindsey wrote: > Is it a design goal to ensure that some percentage of today's > SGML documents are "grandfathered" in, or is it enough that the > EE-ESIS translation is available? If it is the former case, > how do we decide which class(es) of documents shall be supported? We can't support all of them, of course. A lot of use of the *exotic* features of SGML has been made. This is one issue where filtering is an acceptable solution. I think we have to do as James Clark has suggested and be very stingy with the features we include. Otherwise, we will be debating based on the size of the legacy. While I understand the realities of legacy, my experience has been that this argument has the effect of making one choose politics of investment and buttered bread. This is slippery ground and one of the places where the *use real SGML* argument applies. len bullard lockheed martin Date: Tue, 17 Sep 96 09:10:55 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: client/server XML (was Re: Marked Sections) At 08:45 AM 9/13/96 +0000, James Clark wrote: > >> From: streich@austin.sar.slb.com >> Date: Thu, 12 Sep 96 13:20:46 CDT >> >> Declared >> content is, unfortunately, essential since we use CDATA elements for our >> equations. We use TeX as the markup language since the current set of >> equation editors just makes it so easy for people to create an equation >> and literally copy via the clipboard straight into the document. > >This sounds to me like a tool issue not a language issue. Could you >not use #PCDATA if you had a tool that could take care of preventing >unwanted delimiter recognition (by inserting numeric character >references or empty comments or whatever) when copying from the >clipboard? [Sorry that this is a few days behind. I've had my head in the sand for the last few days trying to get some work out the door.] Maybe, maybe not. I think it's an issue about a particular type of data. Whether the language makes allowances for it or the tool provides a good interface to the language to allow for it is the question. Since some current tools don't provide a good interface to SGML for this type of data, why should we expect anything different from XML tools? If you take away my escape hatch, then I'm stuck. I think it would be foolish to expect any qualitative improvements in XML tools over what we have now. There are some good tools, but there are a lot of really crummy ones. Charles suggested retaining marked sections instead of declared content. This would work, but it takes away the opportunity to attach a notation to the data. Inconvenient, but it would be acceptable. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Tue, 17 Sep 96 09:09:43 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: XML and required DTDs At 09:45 AM 9/17/96 GMT, Charles F. Goldfarb wrote: >On Tue, 17 Sep 96 03:51:27 BST, "Henry S. Thompson" wrote: >>

TheGretellost toConstitution >>in1966.

>> >>[vc for 'vanilla content'] >> >This markup is not obfuscatory, just verbose. My understanding is that for XML, >verbosity is preferable to complexity. This is one of those cases where the "simple" solution only pushes the complexity into someone else's lap. Sure, you make things easier for the parser-writer, but how many users are going to tolerate this? I can't think of a single word processor/publishing system that doesn't allow mixed content. Some (like FrameMaker) don't allow it in their markup language, but they hide that very effectively behind the covers. And you can't just push this onto the application and claim that a "good application will take care of this markup for you. By trimming a day's worth of effort from the parser-writers time budget you can effectively add a month to the interface-writer's budget. There are some issues that don't have an easy solution. By focusing solely on what it means to the parser-writer, you can effectively kill the acceptance of the language by users and vendors. A simple-to-write parser doesn't do much good without an interface and some data to process. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Tue, 17 Sep 96 09:10:36 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Capitalizing on HTML (was Re: equivalent power in SGML and XML) At 11:56 AM 9/17/96 +0000, James Clark wrote: >I don't need to use an XML browser to browse HTML; I've already got >plenty of HTML browsers that do that just fine. I'm not convinced that an XML system should be able to parse "valid" HTML (there's so little of it anyway that it's probably a moot point), but there is a large community of people out there who are used to reading and writing pointy brackets. If we alienate that group by putting in too many "niceties" that result in a "That's STOOPID" comment (thank you, Len), then what have you really gained over SGML? We already have good SGML parsers (thank you, James, and others) and most of the problems that I have with current SGML applications have little to do with their parsers. If XML is to be anything more than just a stimulating mental exercise, it has to be a relatively easy jump for any HTML author who isn't afraid to use HoTMetaL, emacs, etc. to edit HTML. It's difficult enough to get most people to make the leap away from display-oriented markup. If we add a lot of restrictions or syntax oddities, we could make the jump downright undesirable. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Tue, 17 Sep 96 09:08:51 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: RS/RE considered confusing? At 11:41 AM 9/17/96 +0000, James Clark wrote: >a. An approch similar to HTML. In HTML (except in PRE), line >terminators are just white-space, adjacent white-space is collapsed >and leading/trailing whitespace in an element is stripped. The >interesting thing about this is that if you perform this process it >doesn't matter whether or not you ignored the REs that SGML said you >should: you'll end up with the same result. Well, almost: you'll >still have the problems with REs getting moved past inclusions and >PIs. The main problem with this is how to handle verbatim type >elements. Can we live without these? Only if you want to rule out most technical documents. Even the original HTML DTD had three different tags for text with significant line endings: XMP, PRE, and LISTING. Code samples, sample terminal dialogs, etc. all contain significant line endings as would some notation content. One option would be to say that all content with significant line endings be put into CDATA entities and given a notation. (We haven't yet touched on how external entities are going to be handled or what the different types will be.) The biggest drawback to this is that you force some messy management onto the author who has to cope with all the little pieces. Another option would be to force people to create "line" or "line-ending" elements, e.g., <line>main(){</line> <line>[lots of junk]</line> etc. or main() {<br></br> [yecchh, a non-empty element to identify a [lots of junk]<br></br> significant line-break!] but this forces authors to put a lot of markup around things that they really should be able to just copy into place. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Tue, 17 Sep 1996 10:11:54 -0400 To: Charles@sgmlsource.com From: "Steven J. DeRose" Subject: Re: XML and required DTDs Cc: Martin Bryan , w3c-sgml-wg@w3.org At 01:02 AM 09/17/96 GMT, Charles F. Goldfarb wrote: >On Mon, 16 Sep 1996 16:52:44 -0400, "Steven J. DeRose" wrote: > >>The only things blocking you from parsing one-entity minimal SGML document >>without a DTD are: >> >> a) Declared content (CDATA/RCDATA/EMPTY elements) >> b) The RE-ignoring rules. >> >Exactly! Just eliminate declared content and mixed content and you've solved the >problem. We don't need those constructs for XML, they are just forms of markup >minimization parading under other names. I don't see yet what mixed content has to do with it. I do see need for keeping mixed content; it's just so messy to have to tag every pseudo-element as a real element. The only difficulties with mixed content are the whitespace treatment we've already decided has problems, and a few cases of OMITTAG that will likely also go away in XML, right? Is there some problem of mixed content that we aren't going to address another way already? I should have addedd attribute normalization to the list of what you may need a DTD for (without declarations you can't tell CDATA from IDREFS). But that doesn't strike me as a problem: When in doubt a parser keeps all the whitespace; an application on top typically knows what it cares about: for example, to strip whitespace and case-fold when looking up any identifier. This is really application semantics, and many users dislike equating the *syntax* of being an 8-character [-.a-z0-9] NAME, with the *semantics* of being an ID or IDREF. Separating them is probably a benefit, in which case the parser doesn't need to know (just like a parser returns extra whitespace in PCDATA, and many applications know they can trash it except in "verbatim" mode). All we need do is state that an application may not depend on extra whitespace in attributes to determine processing semantics: it may not set an element in bold only when its FOO attribute had double spaces between its name tokens. Hardly an odious limitation. Date: Tue, 17 Sep 1996 09:36:05 -0500 From: Len Bullard To: Martin Bryan CC: Bill Lindsey , Michael Sperberg-McQueen , W3C SGML Working Group Subject: Re: equivalent power in SGML and XML Martin Bryan wrote: > > At 05:17 AM 17/9/96 +0100, Bill Lindsey wrote: > > >Is it a design goal to ensure that some percentage of today's > >SGML documents are "grandfathered" in, or is it enough that the > >EE-ESIS translation is available? If it is the former case, > >how do we decide which class(es) of documents shall be supported? > > The set of documents I would like to see "grandfathered in" is that set > defined in (valid) HTML. If an XML browser cannot read that set of raw HTML > documents that are valid according to the 2.0 DTD (or later versions) then > it will not be of much practical use, and will be ignored by the majority of > potential users. (We would still have the problem of the 60% of invalid > documents, but hopefully this situation will get better once standard WP > tools start offering automatic conversion to HTML.) Respectfully, Martin, I disagree. We would already be starting to determine XML's design by constituency. If we start that, then I have to point to the enormous amount of material in the CALS and AECMA worlds which also has a substantial investment, and might even be said to be *more valuable* information. I wish to avoid that kind of argument here but I would have to make it. Don't grandfather documents. Look strictly at features right now and the technical problems for capabilities. I too wish to field existing information without the need to do expensive conversions, but I accept the necessity of minor ones if for no other reason than a lot of the legacy is overbuilt and uses overdesigned DTDs. But to make XML contingent on grandfathering what is in many many cases, poor SGML (try to build a tag scanner for the legacy out there and keep it current) instances is not a good design criterion, IMHO. They will not accept the *valid HTML* argument and will insist that all of their invalid HTML should also work. This looks to me like a bad situation to even attempt to rectify. Let HTML and other existing DTDs take care of making the transition to XML. Make XML an attractive target because it does NOT have the foibles of these apps. len bullard lockheed martin Date: Tue, 17 Sep 1996 09:49:52 -0500 From: Robin Cover To: w3c-sgml-wg@w3.org Subject: XML and required DTDs Steve DeRose and others have made a good case for removing the burden of reading a DTD from the shoulders of XML applications, especially if reading an XML DTD is to be no easier than reading an arbitrary SGML (ISO 8879) DTD. I don't know how the equation changes if XML DTDs are "easy" to parse, by comparison, but I understand the motivation. Still, I am wondering how this strategy would fit into the larger picture: (a) how XML instances will actually be processed, and (b) what dynamics will likely be set up for XML applications development if the pronouncement is made: "Rejoice at last, programmers of the world: today we are liberated from the tyranny of the SGML DTD." To elaborate on this question, I first need to ensure that I have not made a fatal mistake in assuming that "Language" in "XML" to is meant in the same way as "Language" in "SGML," viz., elliptical for "metalanguage." I have taken "eXtensible" to be the equivalent of "meta-". Further, I have assumed that the markup languages based upon XML would set up basic lexical semantics as well as can be done using semantically perspicuous GIs, attribute names, attribute values (etc.), but that XML itself would be as innocent of processing semantics just like SGML. If both assumptions are warranted, and if I have not failed to reckon with some influence from "SGML Extended Facilities" (e.g., property sets that are assumed to apply to XML in some way), then I have the following observations and questions. Observation: if XML instances are to be processed without the DTD, then it's not possible to know with certainty from an instance what the data types (declared values) are for the attributes. This -- aside from the fact that some declarations might need to be sent with the XML instance to provide information about defaulted attributes, notations, etc. In this sense, an XML document instance is weaker semantically than an SGML document with its required DTD. See below, where the attributes 'uff' or 'ko' might encode an IDREF as (HyTime) clink, but we can't be sure. So, suppose this XML instance is acquired by my Net client: xxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx If the client assumes (or can know from a MIME type?) that the instance is a candidate for meaningful display, what does it do? (Ignore the fact that the content represented by "xx" is dummy text). The server sending the instance will also need to send a stylesheet which says that (e.g.,) "qdd is a line-breaking element" and "qkqr is to be displayed in italic typeface" and so forth. Something also has to tell my browser that QDD's attribute 'ko' encodes a HyTime clink, and that 'uff' is (therefore) for storing an ID. Etc. Thus, I still don't understand the point of enabling "DTDless" processing of XML instances, as per Tim Bray's posting #172, where "search, display, analyze" are examples of such processing, *if* it is necessary to have and process, in addition to the instance: * a rendering stylesheet * a collection of declarations to account for attribute defaulting, notations, etc) * a set of other specification of semantics like (HyTime) reftype, clink, which express fundamental relational semantics that are not expected to be in a specific stylesheet Granted that it's nice to be able to create a parse tree for an XML instance simply by having the instance in hand: what's the point of all this economy if we can't make any useful sense of the tree without also having (and processing) the stylesheet, as well as having (and processing) the other declarations that tell us how to interpret the tree? It seems to me ironic that a revised SGML could be depicted in this way: "here's the instance, here's the stylesheet, here's a small collection of declarations to let you know about defaulted attrs and notations, and here's a set of HyTime mappings based upon archForms semantics to augment the stuff in the stylesheet -- what, you want the full XML DTD too? -- naw, sorry: irrelevant and too expensive." Someone says: (1) "No need to ship a stylesheet: the XML document can just reference one, and if it's common, it should be on the receiving system's local machine." OK: why not also for the DTD? Or: (2) "Naw, we assume a common tag set, so that we know

means 'paragraph' and

  • means 'list item'." OK: how then is XML much of an improvement over HTML? I still maintain that there is a broad range of applications for which *having the DTD* (as opposed to the parse tree for the exact XML instance that just arrived), or for which *having the option of addressing the DTD* rather than the instance parse tree, allows a great deal more interesting and meaningful processing. I know I don't stand a chance of convincing anyone on the significance of this point so I won't try. But I fully expect that *most* XML applications will ignore markup declaration processing if XML says it's possible to have meaningful processing without recognizing declarations. What worries me most is that XML will lead to a fixed language with semantics (by virtue of a fixed -- yet chaotic/anarchical -- tag set, just like "HTML"), and thus, a fixed not-very-extensible "language" instead of a metalanguage. -robin Date: Tue, 17 Sep 96 10:26:46 CDT From: Michael Sperberg-McQueen Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) To: W3C SGML Working Group On Tue, 17 Sep 1996 09:08:40 -0400 Gavin Nicol said: >>1) File can be valid SGML (as external general entity) > >True for systems that understand the hack. For systems that do not All SGML systems understand how to parse PIs, and all SGML systems rely on the SGML declaration (or a hard-coded character set) to understand character encodings, so I think Rick is right; the file can be valid SGML, period. The SGML system does not need any understanding of the hack. >When I say external, I mean external to the actual entity. Metadata >does not belong in the objects that it refers to. Since tags are clearly metadata in the broad sense of the term, I think this thesis will have a tough time commanding universal assent in a group full of SGML partisans. At least, I hope so. I don't object to external metadata, but it is remarkably fragile and it is remarkable how easily it goes out of date. Internal metadata is perceptibly less fragile and goes out of date less easily, in my experience -- probably because internal metadata is there when changes are made. Humans are less prone to forget it, and programs are less likely to be unable to find it for updating. The longer this discussion goes on, the more I am tempted to migrate to the Hard Minimalist camp. -C. M. Sperberg-McQueen Date: Tue, 17 Sep 1996 08:47:53 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org Subject: Addition to the WG The following participant has been added: Ed Mooney, Sun Microsystems emooney@east.sun.com Jon From: Paul Prescod Subject: Re: XML and required DTDs To: robin@acadcomp.sil.org (Robin Cover) Date: Tue, 17 Sep 1996 12:46:55 -0400 (EDT) Cc: w3c-sgml-wg@w3.org > Observation: if XML instances are to be processed without the DTD, > then it's not possible to know with certainty from an instance what > the data types (declared values) are for the attributes. This -- > aside from the fact that some declarations might need to be sent with > the XML instance to provide information about defaulted attributes, > notations, etc. In this sense, an XML document instance is weaker > semantically than an SGML document with its required DTD. The processing application should either a) already know what it needs to know (as in the case of HTML) or b) know what external meta-data file to download to get the information (as in the case of SGML with style sheets) or c) get the DTD. But it should have the choice. Right now you CANNOT parse an SGML document correctly (in general) without the DTD. > If the client assumes (or can know from a MIME type?) that the instance > is a candidate for meaningful display, what does it do? (Ignore the > fact that the content represented by "xx" is dummy text). The server > sending the instance will also need to send a stylesheet which says > that (e.g.,) "qdd is a line-breaking element" and "qkqr is to be > displayed in italic typeface" and so forth. Something also has to > tell my browser that QDD's attribute 'ko' encodes a HyTime clink, and > that 'uff' is (therefore) for storing an ID. Etc. Only if the processing application cares about HyTime links or stylesheets. Since most Web-browsers do NOT care about HyTime links, they would NOT need the DTD (for now). At some point in the future, they may decide to add HyTime link support, and they would either have to download the DTD or some other form of "list semantics sheet". But the application designers should have the choice. > Thus, I still don't understand the point of enabling "DTDless" > processing of XML instances, as per Tim Bray's posting #172, where > "search, display, analyze" are examples of such processing, *if* it is > necessary to have and process, in addition to the instance: > > * a rendering stylesheet > * a collection of declarations to account for attribute defaulting, > notations, etc) > * a set of other specification of semantics like (HyTime) reftype, clink, > which express fundamental relational semantics that are not > expected to be in a specific stylesheet Each of these is only need for some applications. Netscape would use the first. Alta Vista would use the second and third. Panorama would use all three. It may be helpful to think of a DTD as a "validation stylesheet". You only need it if you intend to validate. > Someone says: (1) "No need to ship a stylesheet: the XML document can > just reference one, and if it's common, it should be on the receiving > system's local machine." OK: why not also for the DTD? Or: (2) > "Naw, we assume a common tag set, so that we know

    means > 'paragraph' and

  • means 'list item'." OK: how then is XML much > of an improvement over HTML? Even if this tragic situation were to come about, an extensible HTML is much better than what we have now! You probably don't remember, but we had a discussion about you using propreitary "meta-data" tags in the SGML Web Page. You and I agreed that strictly speaking, HTML doesn't allow them. I do not think that this group has decided against requiring the existance and availability of a DTD for each instance. But the goal is that processing systems should not have to fetch the DTD unless they want to. I think that an XML document that does not point to its DTD through a URL should probably be considered an invalid document. XML is being developed explicitly for the Web, after all, and we know what a mess it is to try to work with documents that do not describe their content clearly. I might advocate an explicit exception for documents with a root element with GI "HTML". Paul Prescod From: lee@sq.com Date: Tue, 17 Sep 96 13:29:21 EDT To: w3c-sgml-wg@w3.org Subject: Re: equivalent power in SGML and XML James said: > I just don't think [deleting OMITTAG] is practical. > HTML relies OMITTAG (notably for P, LI, DT, DD). > I don't see how we can possibly achieve our ease > of implementation goals if we support OMITTAG in XML. I can certainly see how we could make HoTMetaL Pro have an option to save HTML files in XML. It isn't that easy to suggest a general converter using SGML tools, since most HTML documents are not valid according to any particular HTML DTD. But I can imagine public domain conversion utilities that could be fairly painless. Would that be too much of a compromise? With no SHORTTAG and no OMITTAG and no EMPTY elements and no MIXED content, XML is becoming about as far from HTML as HTML is from RTF :-) It seems to me that limited versions of some of these things could usefully be retained: * allow an omitted end tag immediately before an end tag:

    stuff

    (is this worth it? it's easy to parse) * allow (easy to parse, and safer than NET) * allow mixed content with | but not with ",", so as to avoid the difficulties associated with whitespace in element context in a mixed content model being taken as PCDATA. It is a mistake to think that using pseudo-elements solves very much, I think. Consider:

    here is my very important document

    where in fact the newlines should be inside the elements, not between and . This is hard to explain. If comments and processing instructions make this complex, delete them and use elements instead. Heck, you can use elements instead of marked sections and attributes and entities and get a much cleaner syntax! Think of combining the wonderful work done by Tommie and Debbie on Pinnacles Reflections with the careful TEI WSD spec... entities as they could have been :-) -- as elements, using ID/IDREF to `insert' them. * consider a naming convention for EMPTY elements, if they are allowed at all -- e.g. , etc. -- if SGML is to be modified to allow end tags on EMPTY elements, this is probably irrelevant. * allow default attributes to be omitted, so that arch forms can be used (HTML IMG has 22 attributes according to HoTMetaL Pro 3.0, and almost all of them have defailt values. (a few are actually #IMPLIED I think). Having to put DIR="ltr" on eery element would be a pain, for example... Lee From: Gavin Nicol Date: Tue, 17 Sep 1996 17:36:06 GMT To: U35395@UICVM.CC.UIC.EDU CC: w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) >>>1) File can be valid SGML (as external general entity) >> >>True for systems that understand the hack. For systems that do not > >All SGML systems understand how to parse PIs, and all SGML systems >rely on the SGML declaration (or a hard-coded character set) to >understand character encodings, so I think Rick is right; the >file can be valid SGML, period. The SGML system does not need >any understanding of the hack. This is actually *not* true. SGML systems *cannot* parse the PI declaration until they know the coded character set and encoding in effect. The coded character set will be fixed, but the encoding not (even in the minimalist case). As such, only systems that can, in some way, derive the encoding, parse the PI. Rick's proposal requires some part of the system to sniff at the head of the file to figure out the encoding. >Since tags are clearly metadata in the broad sense of the term, I think >this thesis will have a tough time commanding universal assent in a >group full of SGML partisans. At least, I hope so. I don't object to >external metadata, but it is remarkably fragile and it is remarkable how >easily it goes out of date. Internal metadata is perceptibly less >fragile and goes out of date less easily, in my experience -- probably >because internal metadata is there when changes are made. Humans are >less prone to forget it, and programs are less likely to be unable to >find it for updating. One more reason why this idea is BAD (Broken As Designed): email gateways and proxy servers *may* convert the encoding of the document blindly. Most gateways and proxies understand MIME headers, but very few understand how to parse, and rewrite the entity to correct the encoding label on the PI. Date: Tue, 17 Sep 1996 11:53:09 -0600 From: "William D. Lindsey" To: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Courting Joe Homepage Apologies for the length. [Tim Bray:] > Taken at another level, more crassly sociological, XML has a greater chance > of widespread adoption if it doesn't "look weird" to the world's several > million HTML hacks. > > Someone, please, explain why this fear is unfounded. Because I would > really like to have a syntactically-obvious EMPTY element in XML. Or > (serious question) is invading HTML's turf a non-goal of XML? I'll try. Invading HTML's turf should be a goal, but cleverly disguising XML as HTML isn't necessarily the best way to do it. I am not too worried about frightening the HTML hack with weird looking delimiters. In fact, for the moment, I'm even willing to consider ideas that might make XML "look weird" to today's crop of WWW browsers. Here are some of my reasons: 1) SGML already looks weird to the world's HTML hacks. (or, If HTML looks good, I wanna look bad!) I doubt that the majority of the world's several million HTML hacks have ever seen an 8879 conformant document in their lives. I'm not sure that any flavor of SGML we come up with can have more than a passing resemblance to the bag-'o-tags they know as HTML. They're already going to have to learn a lot about XML and HTML (i.e. what elements are required within others, you can't overlap elements, etc.) anyhow. A different looking tag delimiter isn't necessarily going to hurt that process, and might even help. 2) HTML already looks weird to the world's HTML hacks. Many web page authors are using tools such as Netscape Gold, Microsoft's Internet Assistant, SoftQuad's HotMetal, etc. They really don't know, or want to know what HTML looks like. Our audience will more likely be the tool-smiths and browser makers. I think they'll accept a different looking
    tag if there's a clean, simple, precise specification of XML. 3) HTML hacks are a fickle sort. I have the feeling that these proverbial HTML hacks do not have any strong loyalty to HTML, or the RFC. They'll happily jump on the next "cool thing" to come along if it offers them the chance to enhance their website with "3-D sense-around in stereo". XML ought to provide (part of) the infrastructure to support these cool toys. (Hmmm ... maybe we should try to make XML look like Java :-)) 4) HTML is dead, long live HTxML. HTML is, IMO, increasingly being viewed as a presentation format, to be assembled on the fly. Larger web-site Internet and Intranet publishers are realizing that they have issues with document management, information reusability, and information searchability when they try to author to today's popular browser. They need SGML, but the entry cost has been too high because of its complexity. The demand for low-cost, robust, powerful SGML is there. XML can address this demand. Make it so it is super easy for our CS grad to assemble tools to process it, and the tools will proliferate. Many of them will take the form of XML->HTML CGI scripts. Put powerful tools in the hands of the WebMeister, and he'll know what to do with them. I think our initial target market is the Internet/Intranet publisher. Our champion in cracking that market is the WebMeister who finds in XML a tool that helps him solve his employer's problems. I don't believe he'll be overly concerned with how closely XML resembles HTML, as long as its rules are easily understandable with a minimum of special cases. With a sufficiently simple design, it won't take too long before browsers and search robots which can take advantage of XML's expressive power are implemented. Web page authors will move to XML because of the new things it lets them do, not because all they need to do is replace "" with "", and make sure they've explicitly opened and closed all the elements. For as long as possible, I'd like to keep the primary focus on those first four design principles identified in DD-1996-0001. When forced to choose between simplicity and elegance, or making it look like HTML, my inclination is toward doing the Right Thing(TM). -Bill -- William D. Lindsey blindsey@bdmtech.com +1 (303) 672-8954 Date: Tue, 17 Sep 96 11:36:26 CDT From: Michael Sperberg-McQueen Subject: more on equivalences and round-trip integrity To: W3C SGML Working Group Thinking about the problem of document and DTD equivalence on the train going home last night, I realized my long posting of last night didn't deal with a couple of important issues. I Further Types of Instance Equivalence To discuss them usefully, I need to propose names for two more types of document equivalence. Document instances are - isomorphic, if they can be made ESIS-equivalent by consistently renaming generic identifiers and attributes (and possibly by global changes on tokens in attribute values ...) in one document, while retaining all the distinctions made in that document -- i.e. two elements have the same GI / two attributes have the same name / two attribute values are the same after the renaming if and only if they were the same beforehand. I think this is the same as saying two documents are isomorphic if there is a one-to-one mapping between their sets of GIs and attribute names, and we can use that mapping to make the documents ESIS-equivalent. - unifiable (or structurally equivalent), if they can be made ESIS-equivalent by a consistent renaming in one or both documents that does not necessarily preserve all original distinctions. This boils down to saying the documents have the same number of elements, corresponding elements have the same number of children, and there is a mapping between their name spaces that has one:one, one:many, many:one, but no many:many relations. If two unifiable documents A and B can be made ESIS-equivalent by renaming things just in B, then A *subsumes* B; roughly, if A subsumes B then they are structurally similar and B may have more information because it may make more distinctions. A really useful definition of subsumption of attribute values will probably allow B to have some attributes that A doesn't have, and may have subtler rules about attribute values, but I think we can postpone that for a while. If each document subsumes the other, they are isomorphic. If they are isomorphic and no renaming is necessary at all, they are ESIS-equivalent. Isomorphic and unifiable document pairs will become important if we wish to consider changes to the DTD language like - elimination of inclusion exceptions (this would have an effect on our RE/RS discussions, but killing inclusions to solve RE is cracking a walnut with a steam engine) - elimination of exclusion exceptions - allowing XML DTDs to accept arbitrary regular expressions, not just deterministic ones - requiring external entities to be synchronous with the element structure (i.e. requiring start-tag and end-tag for an element to occur in the same entity, or at least requiring both to be either in the document entity or in the same external entity) If we want to hold fast to the two-way-validation test for DTD equivalence I proposed last night (for each SGML DTD there is an equivalent XML DTD that accepts a set of documents such that each member of the SGML document set maps to an equivalent document in XML and vice versa, using some measure of document-instance equivalence), then limiting the use of exceptions, or requiring synchronous entity boundaries, would require weakening the equivalence test for document instances. II Inclusion and Exclusion Exceptions Approach A: two-way validation and unifiable documents For any SGML DTD that has inclusion or exclusion exceptions, we can construct (in a potentially laborious calculation) a DTD without exceptions such that each document accepted by the original DTD subsumes some document accepted by the modified DTD. Some of the original GIs might need to be mapped into two or more GIs in the subsumed document. For example: in the TEI, an inclusion exception on TEXT allows a linebreak element LB to occur anywhere within the TEXT element. Since P (paragraph) can occur both inside TEXT and outside it (in the TEI header), this means an exception-less DTD will need two types of P (or more, if there are other possible locations of P which have different sets of effective exceptions: the number of possible effective exception states will be somewhere between 2 ** N and N!, if N elements have exceptions in their declarations; it's 2**N if no GI occurs both as an inclusion and an exclusion, I think). If we call them P (in the text) and Ph (in the header), then we can create a DTD without exceptions which will accept precisely those documents which can be mapped back into the original TEI DTD by renaming all Ph elements as P. Translation back into SGML can map the new GIs back into the original GI, of course, if we could guarantee that the XML-SGML translator knew about the original SGML-XML translation. A normal process of reading the XML document as an SGML document would generate a different DTD from the original. Approach B: two-way validation with ESIS-equivalent documents, by eliminating exceptions in the SGML In practice, of course, the advantage of normalizing an SGML document into XML and having it be valid SGML using the same DTD as the original is so great that many people, myself included, will be tempted to remove exceptions from their SGML DTDs in order to have our SGML and XML DTDs cover sets of EE-ESIS-equivalent documents, rather than just sets of unifiable documents. (This would also allow us to avoid having to explain subsumption to a lot of people, which I admit is a modest gain.) This would presumably gladden the hearts of monastic SGML-ers. Approach C: ESIS-equivalent documents, over- and under-generation Given an SGML DTD with exceptions (call it X), it should also be possible (should, meaning I haven't worked it all out yet) to generate two DTDs (W and Y) which underspecify or overspecify the language accepted by X, meaning: - every document accepted by X is accepted by W but not necessarily vice versa (W is 'wider'?) -- W underspecifies, and overgenerates, the language - every document accepted by Y is accepted by X but not necessarily vice versa -- Y overspecified, and undergenerates, the language So translating an SGML document into XML using DTD W will ensure that the XML user has all the freedom offered by the original DTD X, with exceptions, while translating it into Y (if possible) will guarantee that if I edit it with a validating XML editor, it will still be valid according to X when I give it back to you. I don't know whether being able to generate W and Y will help address concerns like Paul Grosso's or not. The key point is that if we wish to have the *option* of removing exceptions from the DTD notation of XML, we cannot insist on the EE-ESIS equivalence test in conjunction with the two-way-validation test I described last night. One or the other has got to be weakened; I'd prefer that we weaken the instance-equivalence test, but only in certain well-defined cases (such as DTDs with exceptions or instances with asynchronous entities), and not weaken the two-way-validation test. III Asynchronous entities I think I've heard it proposed that all elements begin and end in the same entity, rather than allowing entities the freedom they currently have to be asynchronous. This sometimes takes the form of requiring all elements to begin and end in the document entity itself (i.e. forbidding references to external entities, except perhaps by way of attribute values). The motive, I think, is to make it easier to parse and validate entities separated from their references. I don't want to take a firm side on this now, since I don't know all the arguments on both sides; from what I know, I think it's a useful simplification if it doesn't wreak too much havoc for other users. For me personally it's not an issue since I don't use asynchronous entities anyway; they confuse me. I believe Author/Editor users won't have a problem either, since I think A/E enforces this rule already. I don't know about other editors. Perhaps the vendors will say a word? If we choose to make XML require synchronous entities, then we won't be able to sustain a guarantee of EE-ESIS-equivalence for documents which currently have asynchronous entities. That would seem to push us back to ESIS-equivalence in all cases, and EE-ESIS equivalence for some documents. IV What's needed I think we should allow ourselves to fall short of the goal of EE-ESIS equivalence in some cases, if we think they are not too burdensome on users and help implementation. In the specific case of content-model exceptions, I think eliminating them is enough of a gain in simplicity that the cost (more complex SGML-XML translations, or loss of expressive power in DTDs) is worth it. -C. M. Sperberg-McQueen From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Steven J. DeRose" Cc: Martin Bryan , w3c-sgml-wg@w3.org Subject: Re: XML and required DTDs Date: Tue, 17 Sep 1996 22:45:33 GMT On Tue, 17 Sep 1996 10:11:54 -0400, "Steven J. DeRose" wrote: >At 01:02 AM 09/17/96 GMT, Charles F. Goldfarb wrote: >>On Mon, 16 Sep 1996 16:52:44 -0400, "Steven J. DeRose" wrote: >> >>>The only things blocking you from parsing one-entity minimal SGML document >>>without a DTD are: >>> >>> a) Declared content (CDATA/RCDATA/EMPTY elements) >>> b) The RE-ignoring rules. >>> >>Exactly! Just eliminate declared content and mixed content and you've >solved the >>problem. We don't need those constructs for XML, they are just forms of markup >>minimization parading under other names. > >I don't see yet what mixed content has to do with it. As James Clark pointed out in an earlier posting, eliminating mixed content is the only way to avoid the whitespace treatment problem. >I do see need for keeping mixed content; it's just so messy to have to tag every >pseudo-element as a real element. It doesn't have to be that way. With a few fixed shortrefs, it can be no worse than some of the proposals for enriching tag delimiters. For example, all of the following three equivalent alternatives can trivially be produced from instances of arbitrary DTDs and trivially transformed without loss between SGML and XML.

    /This is the 1st pseudo-element / /and this is another./

    "This is the 1st pseudo-element " "and this is another."

    [This is the 1st pseudo-element ] [and this is another.]

    >The only difficulties with mixed content >are the whitespace treatment we've already decided has problems, >and a few cases of OMITTAG that will likely also go away in XML, right? Is there some >problem of mixed content that we aren't going to address another way already? Unfortunately, the existing proposals for handling whitespace don't work (see earlier postings from myself and James Clark). There are only two solutions that do: 1. RE/RS handling a la SGML (or, preferably, a la the proposals for SGML97, which simplify things considerably); or 2. Explicit tagging of pseudo-elements (as you so nicely put it; this phrase conveys the idea much better since we are not actually eliminating mixed content structurally, we are just tagging the pseudo-elements). Best regards, Charles -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Robert Streich Cc: w3c-sgml-wg@w3.org Subject: Re: Capitalizing on HTML (was Re: equivalent power in SGML and XML) Date: Tue, 17 Sep 1996 22:45:35 GMT On Tue, 17 Sep 96 09:10:36 CDT, Robert Streich wrote: >At 11:56 AM 9/17/96 +0000, James Clark wrote: >>I don't need to use an XML browser to browse HTML; I've already got >>plenty of HTML browsers that do that just fine. > >If XML is to be anything more than just a stimulating mental exercise, it >has to be a relatively easy jump for any HTML author who isn't afraid to use >HoTMetaL, emacs, etc. to edit HTML. It's difficult enough to get most people >to make the leap away from display-oriented markup. If we add a lot of >restrictions or syntax oddities, we could make the jump downright undesirable. > I wasn't aware that converting HTML users was a requirement of XML. I thought the objective was a lean-and-mean dialect of SGML that would: 1. reduce the size of web transmissions of SGML documents 2. serve as a better electronic delivery target for SGML processors than HTML currently is 3. be easier to implement tools for because: a) only one (or a few) character sets are supported b) the instance can be parsed (but not validated) without reference to a DTD c) attributes can be validated, parsed, and defaulted without reference to a _full_ DTD d)tools for full SGML can easily generate and accept XML without loss of EE-ESIS information It would certainly be wonderful if XML won a host of HTML users to SGML, but that is a fringe benefit. Our real growth opportunity is winning more users to SGML because of the existence of XML. If we start confusing the requirements we run the risk of not satisfying any of them (as others have pointed out before on this list). -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: R A Milowski Subject: Re: XML and required DTDs To: w3c-sgml-wg@w3.org Date: Tue, 17 Sep 1996 22:44:57 -0500 (CDT) Well, I've let things pile up and have read hundreds of e-mails now... I was going to post a summary of my views on issues that concerned me, but this one grabbed my attention: > If I understand them correctly, it seems that Paul Grosso, Len > Bullard, and Robin Cover have all raised the same concern, by > pointing to situations in which it's necessary to have a DTD, whether > for authoring, for contractual purposes, or for other applications. > > Let's distinguish four cases: > > a DTDs Required for Parsing. > > Declarations are always required, and in practice all applications must > process them on each run, since it's impossible to parse the document > correctly without knowing which elements are EMPTY, which are CDATA, > etc. (It's also possible to cache the essential information in some > other form, but that's just a standard store-for-compute tradeoff.) > This, roughly, is the situation with 8879:1986. > > b DTDs Required for Validation > > Declarations are always required, but the language is so constructed > that the document can at least be parsed correctly* without reading the > DTD.r declarations need not always be read. I.e. it's possible to do > some kinds of useful work even without reading all the declarations. > This, roughly, is what SGML would be like if the ETAGC proposal is > adopted (at least for Minimal SGML documents without references to > external entities). > > * (within some limits -- element boundaries and content > will be correctly identified; some non-significant white > space may be preserved unnecessarily) Since there is not ETAGC delimiter, this isn't SGML. > c DTDs Optional > > Declarations are always allowed, but not always required; the system > makes certain default assumptions if no declarations are provided. This > is the approach taken by PSGML. (Or C, if a programming-language > analogy is useful. In programming, I don't find this helpful at all, > but Tim and others have suggested plausibly that it may be useful in > XML.) This is certainly acceptable. There are some rules that could be applied to this... Essentially, you could still deliver an SGML document with having to change the reference concrete syntax or convert to some "SGML-like" language. > d DTDs Forbidden > > Declarations are never allowed; the system makes certain assumptions > about things, and your usage had better agree. I don't know of any > serious markup languages that do this; the only analogy I can think of > is Basic, in the form that requires names of string variables to end in > $ and so on. Ick. This is like saying: "You can't define you classes to your complier, it just *knows*!" > Cover, Grosso, and Bullard seem to be arguing against (d), but I'm not > sure whom they are arguing against. My own reading of the goals > document is that (b) or (c) should apply -- or at least, that (a) is not > what's wanted here. I don't think anyone is actually in favor of (d), > and if the current phrasing of the goals statement gives readers that > impression, then it needs to change. Can someone who finds the current > phrasing confusing suggest a less confusing alternative? What I don't understand is why (a) is not wanted? What is the big deal about DTDs? A document type is a contract between application--including a browser--and the document. All bets are off if you don't conform to the DTD. It might be ok, and then again, it might not. Having worked a great deal in financial and legal printing, the issue of validation is quite important. Without a DTD, you can't even assume the document was authored correctly, let alone, be legally compliant. For legal-related publishing systems, DTD are *necessary*. I would like to think that XML is going to be able to be used for legal documents. Since a browser need not be a validating parser, the DTD can serve as a rule for whether or not a particular element is empty and what attributes are implied, etc. I just don't see the need to say that we don't want definitions--or even further, to say that we don't need validation. It is very interesting to see HTML authors realize the benefits of validation. Once they know that their document is conformant, they can proceed to verify that it looks correctly. If a valid document is not interpreted correctly by tool X, tool X is broken or misconfigured. Obviously, I'm in favor of (a). > A validating editor just doesn't belong to the class of applications for > which declarations are inessential. When presented with an XML document > which has no DTD, it might (a) warn about missing declarations, (b) > silently assume , or (c) something else entirely. I'd be happy with (a) if at all possible, and some default mode if declarations are not possible. The problem is that this allows applications to never check for validation. Then we are back to square one in that someone can just toss in some element, however they please, and expect it to work. We all have to have rules to play by. The question is who is in control of those rules. The point of SGML was to put that control back into the hands of the user and producer of the information, not in the hands of the software vendor. XML should have that goal as well. DTDs are a necessary component. BTW, why call XML XML? Why not "conventions for SGML on the Web" etc. Do we really want to create "Yet Another Markup Language" (YAML) and go against ISO standards? I realize that there are problems *some* parts of SGML. We should really try to define a set of conventions that work within ISO standard SGML and work with SGML Open and the ISO working group to allow SGML to adjust to fulfill needs that fall outside of ISO 8879. Lets get onto the more important issues: * Use of HyTime for hyper linking. * Formal System Identifiers and Entity Management * Use of DSSSL for active applications (e.g. DSSSL transformations can specify active transformation of documents in the clients browser. Imagine CGIs becoming self-transforming documents). * Transportation of SGML, entities, catalogs, DSSSL in a sane fashion. And if you really want something to think about: * DSSSL defines a standard post-parsed form for a document--a grove. Since this is standard, it should be possible to define a linearization of a grove into a content type such that an application has to make no decisions upon reading the data. It just re-formulates the grove into a tree-like construct (or whatever). No DTD here! No Parsing Here! ...hmmmm, sounds like BSGML! A wise man once said: A hard problem is always a hard problem. Generic and extendible markup for use in distributed systems is a hard problem. ============================================================================== R. Alexander Milowski http://www.copsol.com/ alex@copsol.com Copernican Solutions Incorporated (612) 379 - 3608 From: R A Milowski Subject: Re: marked sections To: w3c-sgml-wg@w3.org Date: Tue, 17 Sep 1996 23:03:34 -0500 (CDT) > I'm trying to get a grasp of the group's sentiment on marked sections, > and am not sure what all the options and positions are. This morning, > Charles Goldfarb suggested that there are really two cases: > > a) conditional marked sections are needed in DTDs, for version > control and tweaking > b) (R)CDATA marked sections are needed in instances, for non-SGML > notations (and, for that matter, for SGML notation -- CDATA > marked sections are the simplest way I know to do SGML examples > in documentation of a DTD > > But there appears to be another position: > > c) some argue that marked sections just complicate life too much > for the parser, and can *always* be done away with, possibly by > improving the tools we use > > Have I left out any major positions? Yes, we have a client who is successfully using marked sections to control the all-possible legal case. They encode SGML document that model via marked section language that could appear in legal banking documents (variants for each state and the federal government). Each marked section is controlled by a parameter entity. These parameter entities are hooked into an expert system using storage managers (available in James Clark's SP). Thus, when they parse a document, they get the appropriate language for the banking transaction at hand. In short, they using marked sections to control variants. There is another solution to this. The solution involves DSSSL and HyTime and is what my talk an SGML '96 will be about. Its basically a transformation/merge model. So, if you have DSSSL, the above goes away. The reason I bring this up is there is quite a few industries that have to solve the documentation problem I outline above. They want to be able to have a publishing system that allows them to say: "here's my situation, gimmie my documents!" ...and out poofs the correct legal language. > It seems to me we are close to consensus on some things, but not all. > > - We seem to agree that instances don't need conditional sections. > - There is some sentiment, not unanimous, that instances don't > *have* to have CDATA or RCDATA marked sections, either; there > are other ways to get the job done. > - Some of us think DTDs don't need marked sections at all. > - Some of us think DTDs need marked sections at all. > - Some of us think XML must provide support for versioning in DTDs, > through marked sections or by some other means. My opinion of marked section in instances is that they are nasty. You get into this whole issue of computability. I can never gauranty that my documents will always parse under all possible variants. The combinatorics of the test cases becomes in-practical very fast. > C. Possible Approaches > > What are our options for XML? > > We could keep marked sections on the grounds that they are useful and > not too hard to parse. We could simplify the parsing a bit, e.g. by > insisting that CDATA and RCDATA be literals, not entity references, > -- or do people switch between CDATA and IGNORE?! > > If we want to lose marked sections, we need to say how to get the > required function. For CDATA sections, I think either of techniques > 2 or 3 above would work, though I suspect someone will want to drop > the empty comment from the language, leaving us with technique 2 (sigh). > For conditional sections, it's harder: > > - we can revise the declaration rules to provide for customization > by other means. For example "Multiple element declarations are legal; > the first one wins" etc. > - we can reinvent the C preprocessor, and say that conditional > sections (and perhaps selective expansion of entity references) > is handled by a simple pre-processor. Dividing up the functionality > this way helps keep it manageable; it also allows us to imagine a > situation in which the server (normally a Big machine) does the > preprocessing, to keep parsing simple for the client (often a Little > machine). If our motive for keeping XML as light-weight as possible > is to ensure that we can write light-weight processors for low-end > machines, then this would allow us to have an XML Extra-Lite for those > of us with 8086s still on our desks. The pre-processor can be > pretty light-weight, too, if we make it easy to detect the relevant > parameter-entity declarations and ensure that the DTD can otherwise > be skipped. > > Drawback: separating some function into a pre-processor could be > the thin end of a wedge leading to an XML with optional features, > which would lead to interoperability problems. We do *not* want > optional features: any XML processor handling any XML document is > a much more attractive model. > Preprocessors have all kinds of hack-like advantages. My question would be if we need a preprocessor, what are we doing wrong? In the case of the client I described above, it turns out that a transformation model is much more elagant and proveable. You can parse every document and be assured that it is valid. Though, it needs a leap in technology OTOH, it is very nice to be able to: I don't have a clue what I should be doing at this point in the document. So I left it to you. ]]> There are all kinds of times when you need to just "shut off" a portion of a document in a production system because it make things explode. ============================================================================== R. Alexander Milowski http://www.copsol.com/ alex@copsol.com Copernican Solutions Incorporated (612) 379 - 3608 Date: Tue, 17 Sep 1996 22:29:06 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: XML and required DTDs [Alex Milowski:] | BTW, why call XML XML? Why not "conventions for SGML on the Web" etc. Do | we really want to create "Yet Another Markup Language" (YAML) and go against | ISO standards? I realize that there are problems *some* parts of SGML. It's called XML because we had to call it something. It's not full SGML. | We should really try to define a set of conventions that work within | ISO standard SGML and work with SGML Open and the ISO working group to | allow SGML to adjust to fulfill needs that fall outside of ISO 8879. That's essentially what we are doing. Charles Goldfarb's posting earlier today summed up the XML agenda very well, I thought. | Lets get onto the more important issues: | | * Use of HyTime for hyper linking. | * Formal System Identifiers and Entity Management | * Use of DSSSL for active applications (e.g. DSSSL transformations | can specify active transformation of documents in the clients | browser. Imagine CGIs becoming self-transforming documents). | * Transportation of SGML, entities, catalogs, DSSSL in a sane fashion. One thing at a time. Right now we are talking about an application of SGML called XML that is designed for the transmission of extensible, structured data over the Web in a way that is easy enough to implement that we can get past the objections leveled at full SGML by the Web browser vendors. Next comes hypermedia semantics and linking. Catalogs and fragment wrapping (an essential technology not on your list) are coming along nicely in SGML Open and don't need our attention at the moment. DSSSL is already a standard and dsssl-o is shaping up solidly as the reference implementation approaches beta status. We will need to visit DSSSL before this is all over, but that should probably come at the end of this effort. | Generic and extendible markup for use in distributed systems is a hard | problem. Right. Which is why we have to keep the discussion focused on one task at a time. This group has been given until October 6 to air all the (non-hypermedia) issues related to XML; I think the discussion so far has demonstrated that we will be doing very well to accomplish that. Jon Bosak ERB/WG Chair >Received: by chuckd.allette.com.au.allette.com.au id AA08440 (5.65/IDA-1.5 for U35395@UICVM.CC.UIC.EDU); Wed, 18 Sep 1996 16:26:04 +1000 Date: Wed, 18 Sep 1996 16:26:03 +1000 (EST) From: Rick Jelliffe To: Gavin Nicol Cc: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) On Tue, 17 Sep 1996, Gavin Nicol wrote: > One more reason why this idea is BAD (Broken As Designed): email > gateways and proxy servers *may* convert the encoding of the document > blindly. Most gateways and proxies understand MIME headers, but very > few understand how to parse, and rewrite the entity to correct the > encoding label on the PI. I thought: Me: it is better to have webserver smart enough to add MIME headers from looking at PI, then the files are nice SGML entities without processing You: it is better to have the MIME headers attached to the file, even if it means an SGML entity manager must be smart enough to strip them out. However, your comments about gateways changing encoding makes me think I have missed your point. I had being taking (explicitly in baby SGML at least) the file as an array of octets self-labeled with encoding/charset info (e.g. binary not text): so is the problem that type "text/xml" is inappropriate? Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ To: jjc@jclark.com (James Clark) From: Martin Bryan Subject: Re: equivalent power in SGML and XML Cc: w3c-sgml-wg@w3.org Date: Wed, 18 Sep 1996 08:39:58 +0100 At 12:56 PM 17/9/96 +0100, James Clark wrote: >> The set of documents I would like to see "grandfathered in" is that set >> defined in (valid) HTML. > >I just don't think this is practical. HTML relies OMITTAG (notably >for P, LI, DT, DD). I don't see how we can possibly achieve our ease >of implementation goals if we support OMITTAG in XML. As currently discussed it would not be possible with XML - this gives me a problem. >> If an XML browser cannot read that set of raw HTML >> documents that are valid according to the 2.0 DTD (or later versions) then >> it will not be of much practical use, and will be ignored by the majority of >> potential users. > >I don't need to use an XML browser to browse HTML; I've already got >plenty of HTML browsers that do that just fine. Yes, but if its a choice between viewing a few documents on XML, a huge number on HTML and even more on SGML I'm going to stick with HTML/SGML and forget all about XML. Unless XML browsers can reference and display HTML documents they are not going to form worthwhile information management tools! What I want XML to do is to provide me with all the good stuff (and none of the sloppy stuff) that HTML provides while allowing me to extend the basic concepts beyond the limits currently set by W3C. ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: "William D. Lindsey" , w3c-sgml-wg@w3.org From: Martin Bryan Subject: Re: Courting Joe Homepage Date: Wed, 18 Sep 1996 09:51:47 +0100 At 06:53 PM 17/9/96 +0100, William D. Lindsey wrote: > Web page authors will move to XML >because of the new things it lets them do, not because all they need >to do is replace "" with "", and make sure they've >explicitly opened and closed all the elements. > If Web page authors can't reference other web pages, or be referenced by them, without a seamless interface in their XML tools they will not care a fig for the extra things they can do with XML. At present XML is only likely to be of interest to corporations who specifically do not want their information made generally available on the Web. Joe Homepage will have to go elsewhere! Martin Bryan ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ >Received: by chuckd.allette.com.au.allette.com.au id AA09287 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Wed, 18 Sep 1996 21:38:27 +1000 Date: Wed, 18 Sep 1996 21:38:27 +1000 (EST) From: Rick Jelliffe To: Paul Prescod Cc: w3c-sgml-wg@w3.org Subject: Reparenting (was Re: A constraint on markup for EMPTY elements) (Not important point) On Mon, 16 Sep 1996, Paul Prescod wrote: > At 09:00 PM 9/15/96 CDT, Paul Grosso wrote: > >At this point after reading all the "cute trick with delimiters" postings > >that attempt to address the EMPTY element, I'm tempted to say that XML > >just represent empty elements as everyone knows them in RCS 8879, and > >we just have to include the "list of empty elements" information along > >with the XML document instance. > > Did we decide that the "don't worry be smart" proposal was too difficult? > This was the proposal to force parsers to keep a stack and reparent content > if the next end tag doesn't match the start tag. A simpler way is (if scanning can be done by a single DFA, and if end-tag ommision & #CONREF are not allowed, etc.) to simply scan through for all end-tags: if an element has an end-tag it can be added to the list of non-empty elements. Then, the document can be parsed by a second pass with this knowledge: no trees and reparenting, or "list of empty elements" required. (Whether this is a good method for the kind of tasks XML is intended is another matter.) Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ From: Gavin Nicol Date: Wed, 18 Sep 1996 13:06:49 GMT To: milor001@gold.tc.umn.edu CC: w3c-sgml-wg@w3.org Subject: Re: XML and required DTDs > * Transportation of SGML, entities, catalogs, DSSSL in a sane fashion. MIME-SGML provides a good basis for discussion, I think. > * DSSSL defines a standard post-parsed form for a document--a grove. > Since this is standard, it should be possible to define a linearization > of a grove into a content type such that an application has to make > no decisions upon reading the data. It just re-formulates the > grove into a tree-like construct (or whatever). No DTD here! > No Parsing Here! ...hmmmm, sounds like BSGML! Sounds like LISP to me ;-) From: Gavin Nicol Date: Wed, 18 Sep 1996 13:14:40 GMT To: ricko@allette.com.au CC: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) One more point: even for gateways smart enough to rewrite the entity, you then lose security because checksums etc. will fail. >Me: it is better to have webserver smart enough to add MIME headers from >looking at PI, then the files are nice SGML entities without processing >You: it is better to have the MIME headers attached to the file, even >if it means an SGML entity manager must be smart enough to strip them out. The point is that in your scenario, the entity manager must be smart enough to sniff at the data to figure out how to parse it. With MIME-SGML, SGML across HTTP, and other such things, it seems to me that most XML entity managers will be supporting MIME anyway... >However, your comments about gateways changing encoding makes me think I >have missed your point. I had being taking (explicitly in baby SGML at >least) the file as an array of octets self-labeled with >encoding/charset info (e.g. binary not text): so is the problem that >type "text/xml" is inappropriate? No. From: R A Milowski Subject: Re: XML and required DTDs To: w3c-sgml-wg@w3.org Date: Wed, 18 Sep 1996 07:57:04 -0500 (CDT) >From Jon Bosak: > One thing at a time. Right now we are talking about an application of > SGML called XML that is designed for the transmission of extensible, > structured data over the Web in a way that is easy enough to implement > that we can get past the objections leveled at full SGML by the Web > browser vendors. Next comes hypermedia semantics and linking. > Catalogs and fragment wrapping (an essential technology not on your > list) are coming along nicely in SGML Open and don't need our > attention at the moment. DSSSL is already a standard and dsssl-o is > shaping up solidly as the reference implementation approaches beta > status. We will need to visit DSSSL before this is all over, but that > should probably come at the end of this effort. > > | Generic and extendible markup for use in distributed systems is a hard > | problem. > > Right. Which is why we have to keep the discussion focused on one > task at a time. This group has been given until October 6 to air all > the (non-hypermedia) issues related to XML; I think the discussion so > far has demonstrated that we will be doing very well to accomplish > that. > I completely understand focusing the discussion on on thing at a time. The problem with this is that some decisions will have larger ramifications when we go to discuss the use of HyTime or DSSSL. I would rather not be in the position of saying: "Well, since we don't require a DTD, we can't use HyTime" or whatever. We have only talked (e-mailed?) about parsing documents without DTDs. What are the ramifications of using a document without having parsed against a DTD? What happens with HyTime? DSSSL? etc. We want useable information to come in the form of XML. Is structured information useful without knowing the structure? BTW, if we have a date to end the current discussion, that means we have some kind of schedule. Where might I find that? ============================================================================== R. Alexander Milowski http://www.copsol.com/ alex@copsol.com Copernican Solutions Incorporated (612) 379 - 3608 Date: Wed, 18 Sep 1996 11:07:31 -0500 From: Len Bullard To: Dave Hollander CC: w3c-sgml-wg@w3.org Subject: Re: Courting Joe Homepage Dave Hollander wrote: > > > At 06:53 PM 17/9/96 +0100, William D. Lindsey wrote: > > > > > Web page authors will move to XML > > >because of the new things it lets them do, not because all they need > > >to do is replace "" with "", and make sure they've > > >explicitly opened and closed all the elements. > > > > > If Web page authors can't reference other web pages, or be referenced by > > them, without a seamless interface in their XML tools they will not care a > > fig for the extra things they can do with XML. At present XML is only likely > > to be of interest to corporations who specifically do not want their > > information made generally available on the Web. Joe Homepage will have to > > go elsewhere! > > > > I have not had time to study all the issues that this argument impacts, > however, I believe that if the results of our efforts are only used by > large publishers/corporations we will have missed an opportunity to > serve the greatest need. Are the needs of the homepage maker more urgent than the needs of the corporation? HTML gets a lot of support. The adopters of SGML who have invested a lot of resources in sticking with the standards, creating well-modeled data, in other words, playing to the rules have taken the shaft from the WWW. I am much more interested in enabling Internet access to SGML as practiced than I am in fixing the problems of bad practice. XML is to simplify SGML. I currently use a browser that already does this. In no way does its features make it something only a large publisher or corporation can use. All a user has to do is define elements and create a stylesheet for them. That's all point and click. If they want to use a DTD, they can write one and batch validate. That works. If they want to do configured editing, take the DTD and give it to a validation editor like SoftQuad's Author/Editor and tell the browser that is the editor to call when asked. Easy. A bit cheaper editor is needed, but that would be an immediate benefit of XML. > When I look around at of the information trapped on the web in > presentational markup, aka HTML, I get depressed. When written, most > authors are only responding to the pressure to get a page on the web > and viewable with the browser de-jour. However, later I need to make > better use of that information and am unable to because of the complete > lack of non-presentational semantics. Yes. That is a situation created by vendors and willing authors. It will not change because we tell them to. It is a manufactured crisis. Only the availability of alternatives and the need and will to use them will alter that course. > Please, XML must be able to improve the value of the millions of web pages. > > Do we agree with this goal? The goal is too ambiguous to agree to as stated. We can't, we musn't weaken XML to fix the problems that popular bad practice creates. We can certainly state that the HTML 3.2 DTD is one which *should* be expressible as an XML DTD. But there are many other sources of legacy and their requirements are also important. It may be that HTML 3.2 has to change to meet XMLs requirements. That is not unthinkable. This is, after all, a W3C working group. If the WG8 members present are helping to ensure that the revision of SGML and the creation of XML are cooperative, isn't it reasonable to expect the same of the owners of the HTML DTD? So, yes, I agree XML must improve the value of the work of those who create web pages, because where HTML is an XML application, it falls within the goals to provide simpler SGML. Where it is not and can't be, then any overlap is coincidental. len bullard lockheed-martin Date: Wed, 18 Sep 1996 09:02:30 -0700 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: Re: Courting Joe Homepage In my former life I probably worked with more varieties of broken HTML than anyone should ever have to. Perhaps as a result of this, I believe there is little value in structuring XML to catch the output of Joe Homepage. That said, it is nonetheless clear that it would be good if XML could be read by HTML browsers. Basically, I think that anything on the Web that is worth saving was created by somebody with a modicum of care, such that it could easily be turned into a valid form or (even more likely) the HTML was generated from something else anyhow, thus would be easily replaced by XML. The design compromises involved in attempting to grandfather the population of bogus pseudo-HTML documents seem way too expensive at this point in time. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Wed, 18 Sep 1996 09:35:50 -0700 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: Re: XML and required DTDs At 10:45 PM 9/17/96 GMT, Charles F. Goldfarb wrote: >Unfortunately, the existing proposals for handling whitespace don't work (see >earlier postings from myself and James Clark). There are only two solutions that >do: >1. RE/RS handling a la SGML (or, preferably, a la the proposals for SGML97, >which simplify things considerably); or Could you post a reference for (or even better, summary of) the SGML97 proposals?; I would love nothing better than to dodge the RS/RE issue in XML in favor of a WG8-invented solution. Clearly, there is a near-consensus here that the SGML RS/RE rules are "too complicated" (the quotes signal my lack of interest in arguing about the correctness of this perception) for use in XML. I at least am very very VERY reluctant to give up mixed content and PI's to work around the RS/RE problem. In fact, this is the one area where, in order to produce something truly usable, some of us, I think, might be willing to live with being non-ISO-8879-compliant (ideally only for a while). If it's clear which way WG8 is going to go on this, it would be very helpful to know that. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Wed, 18 Sep 1996 10:46:00 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: Courting Joe Homepage The ERB wishes to clarify the discussion arising from the suggestion that we make it possible to grandfather existing valid HTML documents as valid XML documents without changing them in any way. We certainly do wish to make Joe Homepage comfortable, but we wish to accomplish this by moving him into a new home. The way to do this is to ensure that existing valid HTML documents can straightforwardly be converted to XML documents without any loss of information using a generally available normalization tool. In particular, it should be easy for the developers of XML applications such as XML Web browsers to build in converters that will map HTML to XML on the fly. But a requirement that existing valid HTML documents be valid XML documents with no changes whatsoever would impose unacceptable limitations on our ability to design the best possible SGML application profile for documents on the Web. It is a design goal of XML to function as a metalanguage for document markup on the Web. It is not a design goal of XML to automatically grandfather in existing HTML documents or documents conformant to any other specific SGML markup language. Jon Bosak ERB/WG Chair Date: Wed, 18 Sep 1996 10:52:34 -0700 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: Unsolicited fan mail I'm off to Japan for a week, so if I'm silent around here, it doesn't indicate a lack of interest - depends on how good the Net connectivity is. Just wanted to take the opportunity to say that I have been been dazzled by the quality of discussion around here - I have seen long-held positions, including some of mine, change 180 degrees - I think we've all had the unusual experience of continuing to disagree with people while keenly perceiving the force and reason of their arguments. I think there's a solid chance here to make something that will outlive us all. Which is damn rare in the technology professions. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Wed, 18 Sep 1996 11:14:20 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: XML and required DTDs [Alex Milowski:] | We have only talked (e-mailed?) about parsing documents without DTDs. What | are the ramifications of using a document without having parsed against a | DTD? What happens with HyTime? DSSSL? etc. | | We want useable information to come in the form of XML. Is structured | information useful without knowing the structure? These questions are certainly within the scope of the present discussion. | BTW, if we have a date to end the current discussion, that means we have | some kind of schedule. Where might I find that? Our kickoff letter set a date of October 6 for the point at which the "voting booth" shuts down and ERB work on editing a draft begins. Our target date for completing the XML draft (not including specifications relating to linking, entity management, hypermedia semantics, etc.) is the first WG meeting on the evening of November 17 in Boston. Beyond that, we have tentatively targeted the Sixth World Wide Web Conference in April, 1997 as the occasion for releasing a draft that addresses the linking, entity management, and hypermedia issues, though that date seems very ambitious for covering all three of those subjects. We hope to have the entire set of specifications needed to implement SGML on the Web completed by the SGML 97 conference. Jon Date: Wed, 18 Sep 1996 10:55:23 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: XML and required DTDs [Gavin Nicol:] | > * Transportation of SGML, entities, catalogs, DSSSL in a sane fashion. | | MIME-SGML provides a good basis for discussion, I think. Maybe so, *but* *not* *now*. If people feel the overwhelming desire to get into the entity and linking issues early (not enough email to read?), please take the discussion offline and be prepared to present coherent proposals when the time comes. We have a specific milestone to reach here. Jon Bosak ERB/WG Chair and resident traffic cop Date: Wed, 18 Sep 1996 15:21:42 -0400 To: Charles@sgmlsource.com From: "Steven J. DeRose" Subject: Re: XML and required DTDs Cc: Martin Bryan , w3c-sgml-wg@w3.org At 10:45 PM 09/17/96 GMT, Charles F. Goldfarb wrote: >>I do see need for keeping mixed content; it's just so messy to have to tag every >>pseudo-element as a real element. >It doesn't have to be that way. With a few fixed shortrefs, it can be no worse >than some of the proposals for enriching tag delimiters. For example, all of >the following three equivalent alternatives can trivially be produced from >instances of arbitrary DTDs and trivially transformed without loss between SGML >and XML. That pushes up implementation complexity, because there are a bunch of extra rules. It also forces anyone typing markup to remember to put *something*, however small, around every pseudo-element, which I find prohibitively burdensome. >Unfortunately, the existing proposals for handling whitespace don't work (see >earlier postings from myself and James Clark). There are only two solutions that >do: They only "don't work" if we choose certain decision regarding the ultra-fine details of what constitutes compatibility. As a proof-of-concept, we could state a rule of XML that whitespace is normalized for data content (that is, removed entirely at the beginning and end of elements and pseudo-elements, and normalize to a single space elsewhere). One can certainly state the same rule as an application convention applicable to certain SGML documents that choose to use it. Yes, the ESIS is a tiny bit different, but it has no processing consequences because we state that the semantics of XML documents prohibit doing any different processing based on that tiny different. Also, ESIS is not carved in stone here; we've said from the beginning that we may have to add or subtract from ESIS slightly to get the level of abstractness we desire, and on which we will define equivalence. Likewise, we can keep mixed content but constrain it in some way so the problems don't arise (the most obvious being to prohibit things like non-|* models, etc). S Date: Wed, 18 Sep 1996 15:21:43 -0400 To: R A Milowski , w3c-sgml-wg@w3.org From: "Steven J. DeRose" Subject: Re: XML and required DTDs At 07:57 AM 09/18/96 -0500, R A Milowski wrote: >We have only talked (e-mailed?) about parsing documents without DTDs. What >are the ramifications of using a document without having parsed against a >DTD? What happens with HyTime? DSSSL? etc. Nothing different happens, because the point is that the ESIS must be the same either way, and all these things operate off the ESIS in practice. That what the default DSSSL grove really is (yes, the existing ESIS spec forgot a few tiny details. They're known and that's what TCs are for). >Received: by chuckd.allette.com.au.allette.com.au id AA09627 (5.65/IDA-1.5 for U35395@UICVM.CC.UIC.EDU); Thu, 19 Sep 1996 01:03:56 +1000 Date: Thu, 19 Sep 1996 01:03:56 +1000 (EST) From: Rick Jelliffe To: Gavin Nicol Cc: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) On Wed, 18 Sep 1996, Gavin Nicol wrote: > The point is that in your scenario, the entity manager must be smart > enough to sniff at the data to figure out how to parse it. With > MIME-SGML, SGML across HTTP, and other such things, it seems to me > that most XML entity managers will be supporting MIME anyway... Not the SGML entity manager when the XML file is read as SGML, since it won't use the PI. It can just use the plain file as is, or be aided by FSIs or whatever if needed. (If there is no requirement that XML files can be read directly as a (head-less) SGML instance, then I completely defer to you, by the way.) The header that looks like a PI is trivial to convert into one that looks like the MIME header. So a XML entity manager could implement reading local XML files just by doing that header translation and submitting the file to itself as a MIME type. So their needn't be a duplication between MIME decoders and ones selectable by the PI. (Or, conversely, an XML entity manager could convert an text/xml header into a PI if it that was more convenient for the implementor.) Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ From: Gavin Nicol Date: Wed, 18 Sep 1996 20:08:14 GMT To: bosak@atlantic-83.eng.sun.com CC: w3c-sgml-wg@w3.org, bosak@atlantic-83.eng.sun.com Subject: Re: XML and required DTDs >| > * Transportation of SGML, entities, catalogs, DSSSL in a sane fashion. >| >| MIME-SGML provides a good basis for discussion, I think. > >Maybe so, *but* *not* *now*. Understood. From: lee@sq.com Date: Wed, 18 Sep 96 00:16:45 EDT To: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: more on equivalences and round-trip integrity > III Asynchronous entities > > I think I've heard it proposed that all elements begin and end in the > same entity, rather than allowing entities the freedom they currently > have to be asynchronous. [...] > I believe Author/Editor users won't have a > problem either, since I think A/E enforces this rule already. I don't > know about other editors. Perhaps the vendors will say a word? There are two cases for Author/Editor... editing and validating. For editing, A/E requires marked sections and entities to be synchronous: a MS must begin and end in the same directly-containing element. A/E does not support file entities for editing at all, so the entire document must be in a single entity. For editing, A/E does not support entities containing markup, whether or not they are synchronous. For validating, markup contained within entities is checked, and an MS can probably end in a context different to that in which it began. I don't believe file entities are supported, though. I'll leave it as an exercise to work out how to get a document with a marked section positioned so as to let you test this :-) (hint: use open, not import) We have no customers (of course) who use asynchronous entities or MSs. I've once encountered someone who put a number of `beginnings' in an entity to save typing, e.g. "> <!Entity t2 "

    "> . . . &t;this is a title&t2;document start although I thing this probably counts as an `obfuscatory' use :-), and the individual expressed a willingness to conform if tools became available to him so he didn't use windows NOTEPAD (or whatever) any more. I'd certainly vote for requiring synchronous things in general. Lee From: streich@austin.sar.slb.com Date: Wed, 18 Sep 96 18:46:23 CDT To: w3c-sgml-wg@w3.org Subject: Re: Capitalizing on HTML (was Re: equivalent power in SGML and XML) > From Charles@SGMLsource.com Tue Sep 17 17:43:18 1996 > On Tue, 17 Sep 96 09:10:36 CDT, Robert Streich wrote: > >If XML is to be anything more than just a stimulating mental exercise, it > >has to be a relatively easy jump for any HTML author who isn't afraid to use > >HoTMetaL, emacs, etc. to edit HTML. It's difficult enough to get most people > >to make the leap away from display-oriented markup. If we add a lot of > >restrictions or syntax oddities, we could make the jump downright undesirable. > > > I wasn't aware that converting HTML users was a requirement of XML. I thought > the objective was a lean-and-mean dialect of SGML that would: > > 1. reduce the size of web transmissions of SGML documents My apologies on this one. I didn't see it on the list of "design principles" so I thought it was a non-issue especially with the trend in many of the discussions towards increasing the amount of markup to make it easier to write the parser. Since this is a requirement that Jon validated in a later mail (I think he was referring to this message), I'll have to change my opinion on a few issues. > 3. be easier to implement tools for because: > a) only one (or a few) character sets are supported > b) the instance can be parsed (but not validated) without reference to a DTD > c) attributes can be validated, parsed, and defaulted without reference to a > _full_ DTD > d)tools for full SGML can easily generate and accept XML without loss of EE-ESIS > information And I agree that these are all good things and worth pursuing. However, I remain very skeptical that there will be a big surge of new tools. I hope that one of the vendor-types will correct me if I'm wrong, but I'd bet that the changes that we are discussing affect less than 1% of the code base of the typical SGML system on the market today. If the changes to the language have a significant effect on the authors, then it really doesn't matter how simple it is to write a parser. Without authors, there won't be anything for the parsers to parse. Resistance to change has been the biggest impediment to SGML acceptance that I've faced. If we make XML more difficult to author than SGML, it doesn't stand a chance. > It would certainly be wonderful if XML won a host of HTML users to SGML, but > that is a fringe benefit. Our real growth opportunity is winning more users to > SGML because of the existence of XML. If we start confusing the requirements we > run the risk of not satisfying any of them (as others have pointed out before on > this list). I think you might be missing my intent. I'm not concerned about winning over generic HTML users or "Joe Homepage" as others have referred to them. I'm concerned about *providing an easy path* for the thousands of HTML users we have in my company. We have several hundred WWW servers on our net (nobody really knows for sure how many). That's a lot of people typing HTML. It'd be really great if I could offer them something that was easier to get started with, i.e., didn't introduce too many new concepts, cheaper, and got them on the track towards SGML. I was hoping that XML could fill that need. Robert Streich streich@slb.com Schlumberger 512-331-3318 (voice) Austin Research 512-331-3760 (fax) From: Gavin Nicol Date: Wed, 18 Sep 1996 21:05:19 GMT To: ricko@allette.com.au CC: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: Limited modified eclectism (was Re: Reads like ASCII) >> The point is that in your scenario, the entity manager must be smart >> enough to sniff at the data to figure out how to parse it. With >> MIME-SGML, SGML across HTTP, and other such things, it seems to me >> that most XML entity managers will be supporting MIME anyway... > >Not the SGML entity manager when the XML file is read as SGML, since it >won't use the PI. It can just use the plain file as is, or be aided by >FSIs or whatever if needed. (If there is no requirement that XML files >can be read directly as a (head-less) SGML instance, then I completely >defer to you, by the way.) This is my point. You *cannot* read the entity in unless you know the coded character set and encoding. The entity manager therefore must either understand the PI hack, or some other external mechanism (FSI, MIME, etc). If the entity manager doesn't understand the PI hack, then what's the point of having it? >The header that looks like a PI is trivial to convert into one that looks >like the MIME header. So a XML entity manager could implement reading >local XML files just by doing that header translation and submitting the >file to itself as a MIME type. So their needn't be a duplication between >MIME decoders and ones selectable by the PI. (Or, conversely, an XML entity >manager could convert an text/xml header into a PI if it that was more >convenient for the implementor.) I don't see the point in duplication. One way or another, you either use the PI hack alone, or together with some other external labelling method. In the latter case, the external labelling method alone suffices. To: w3c-sgml-wg@w3.org Subject: Re: equivalent power in SGML and XML Date: Wed, 18 Sep 1996 21:33:45 -0700 From: Joe English Martin Bryan wrote: > The set of documents I would like to see "grandfathered in" is that set > defined in (valid) HTML. If an XML browser cannot read that set of raw HTML > documents that are valid according to the 2.0 DTD (or later versions) then > it will not be of much practical use, and will be ignored by the majority of > potential users. That may be biting off more than we wish to chew... HTML 2.0 makes liberal use of end-tag omission, a couple cases of start-tag omission, contains elements with CDATA, RCDATA, and EMPTY declared content, uses inclusion exceptions, and encourages omitted attribute name minimization. It does disallow most other forms of SHORTTAG minimization by application convention, and in "Strict" or "Recommended" mode HTML 2.0 does not use CDATA or RCDATA declared content (though HTML 3.2 does), but still... (FWIW, I agree with you that XML should be able to handle HTML -- not necessarily HTML 2.0, but possibly a similar, slightly stricter DTD -- but the general consensus seems to be that the cost of supporting the necessary features is too high.) > (We would still have the problem of the 60% of invalid documents More like 96% invalid from what I've seen... > , but hopefully this situation will get better once standard WP > tools start offering automatic conversion to HTML.) --Joe English jenglish@crl.com Date: Wed, 18 Sep 1996 20:07:42 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: Capitalizing on HTML (was Re: equivalent power in SGML and XML) [Bob Streich:] | > From Charles@SGMLsource.com Tue Sep 17 17:43:18 1996 | > On Tue, 17 Sep 96 09:10:36 CDT, Robert Streich wrote: | > >If XML is to be anything more than just a stimulating mental exercise, it | > >has to be a relatively easy jump for any HTML author who isn't afraid to use | > >HoTMetaL, emacs, etc. to edit HTML. It's difficult enough to get most people | > >to make the leap away from display-oriented markup. If we add a lot of | > >restrictions or syntax oddities, we could make the jump downright undesirable. | > > | > I wasn't aware that converting HTML users was a requirement of XML. I thought | > the objective was a lean-and-mean dialect of SGML that would: | > | > 1. reduce the size of web transmissions of SGML documents | | My apologies on this one. I didn't see it on the list of "design principles" | so I thought it was a non-issue especially with the trend in many of the | discussions towards increasing the amount of markup to make it easier to | write the parser. | | Since this is a requirement that Jon validated in a later mail (I think | he was referring to this message), I'll have to change my opinion on a few | issues. Oops, I screwed up on my blanket endorsement of Charles's statement. First of all, I was speaking for myself that time, not ex cathedra. I will try to make this distinction clearer in future postings. Second, I was flat wrong in seeming to endorse the position that XML documents should be optimized for size. When I think of SGML on the Web, I think in terms of fragments served out according to SGML Open TR9601, and I assume that the server is smart enough to deliver appropriately sized chunks. Bob's initial impression was the correct one. As our discussion of end-tag omission makes clear, size optimization is *not* a priority for XML. Unless I speak directly for the ERB or am acting in an administrative role with regard to the list, please don't take any statements of mine as authoritative. For our current position, refer to the statement of principles. Jon To: w3c-sgml-wg@w3.org Subject: Re: Courting Joe Homepage <96Sep18.095150bst.61015(1)@u-net.net> Date: Wed, 18 Sep 1996 08:44:08 -0600 From: Dave Hollander > At 06:53 PM 17/9/96 +0100, William D. Lindsey wrote: > > > Web page authors will move to XML > >because of the new things it lets them do, not because all they need > >to do is replace "" with "", and make sure they've > >explicitly opened and closed all the elements. > > > If Web page authors can't reference other web pages, or be referenced by > them, without a seamless interface in their XML tools they will not care a > fig for the extra things they can do with XML. At present XML is only likely > to be of interest to corporations who specifically do not want their > information made generally available on the Web. Joe Homepage will have to > go elsewhere! I have not had time to study all the issues that this argument impacts, however, I believe that if the results of our efforts are only used by large publishers/corporations we will have missed an opportunity to serve the greatest need. When I look around at of the information trapped on the web in presentational markup, aka HTML, I get depressed. When written, most authors are only responding to the pressure to get a page on the web and viewable with the browser de-jour. However, later I need to make better use of that information and am unable to because of the complete lack of non-presentational semantics. Please, XML must be able to improve the value of the millions of web pages. Do we agree with this goal? Dave Hollander From: lee@sq.com Date: Thu, 19 Sep 96 09:16:51 EDT To: jenglish@crl.com, w3c-sgml-wg@w3.org Subject: Re: equivalent power in SGML and XML >> (We would still have the problem of the 60% of invalid documents > > More like 96% invalid from what I've seen... A while ago I asked Tim Bray to check for me with the Open Text index... and a little under 5% of HTML documents had a DOCTYPE line. That doesn't mean that they were valid, and there may be documents that are otherwise valid but don't have DOCTYPE, but it's unlikely that the overall figure is much higher. So I'd say that 90% to 95% invalid is a good guess. I'd be interested to run the same test again now a few million more copies of HoTMetaL have shipped... >> , but hopefully this situation will get better once standard WP >> tools start offering automatic conversion to HTML.) The conversions we've seen do not generally attempt to create valid SGML. I think there isn't enough encuragement. This is an aspect of the demise of the IETF HTML WG that is unfortunate, I think -- there are things that are easier for that sort of very approachable standards body. However that may be, I think it's reasonable to expect to have to run sme sort of transformation from arbitrary/normal/typical HTML into XML. Most HTML documents won't go automatically. In addition, it's very common for HTML documents to be different on the server than when they are delivered-- * processing instructions and significant comments are used by some servers: to replace the PI with a DVI image on the fly "and beat and beat."

    Advantages: [a] & [b] exactly as above, and [c] it's 8879-compliant Costs [a] people have to put in a little more markup [b] it is unlikely that XML can ever be fed to an HTML reader. Not really an easy choice; but I think we've succeeded in shaking out what the issues are. Date: Sat, 21 Sep 1996 06:39:24 +0000 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: XML Entity Manager I am surprised that nobody has remarked on the woeful absence of discussion of the entity management problem both in our goals document and here on the list. So... let me as usual advance the minimalist position: XML offers only 4 kinds of external entities: the cross-product of (PUBLIC or SYSTEM) x (PCDATA or NDATA) PCDATA/NDATA offer little for discussion, you either parse it or you don't, right? SYSTEM entities, somebody at some point suggested outlawing them, and there are certainly good reasons for that, but in XML I think we'd like easy stuff to be easy, which SYSTEM does; it should be fenced around with stern warnings about nonportability. Which leaves PUBLIC. Any chance of getting away with saying "all PUBLIC entities are to be resolved by recourse to an SGML/Open Catalog, ref. 9401:1995"? It is now appropriate for James or someone who has built these things to step in and point out the woeful error of my ways. But someone had to kick it off. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 From: R A Milowski Subject: Re: XML vrs SGML tools [was Re: Capitalizing on HTML (was ...)]]] To: w3c-sgml-wg@w3.org Date: Sat, 21 Sep 1996 10:44:25 -0500 (CDT) > XML on its own does not solve the browser problem. The browser > writers are still faced with the problem of style sheets. The reason > there are so many cheap HTML browsers is because it is _very_ easy to > figure out how to display HTML (well, for the most part). DSSSL may > solve the problem, but it solves it in a rather difficult to > understand and implement way. XML will be an easy sell to browser > writers if it comes packaged with a useful but simple style-sheet > mechanism. DSSSL might work, if someone would write a good book about > it, or at least about DSSSL online. Maybe Cascading Style Sheets > (CSS) is an easy (quick and dirty) answer; I have not had a chance to > look into it properly. There was a great deal of discussion about why > CSS is too simplistic a month back on comp.text.sgml, but it may be > "good-enough" for a first run. Every one here seems to want XML tools > today, but we don't even have a DSSSL capable browser now, how many > months after the standard's publication? Hey, DSSSL isn't trivial--neither is the problem of "presentation." There are three DSSSL engines in the works. James Clark's engine is already in alpha testing. > I do not mean to cut down DSSSL. DSSSL looks to me to be a great > possiblity, for the future. A XML/DSSSL browser will not be able to > compete with a HTML browser in today's market though. It will look too > slow. It will definitely have a market, but it will lack a real > mass-market appeal. (But boy would I kill for a good, extensible > DSSSL browser to play with... with some work on on-line display > extensions to the standard, you could use SGML/DSSSL to do everything that > HTML, CSS, Java, Javascript, etc are _trying_ to do now. Now that > would be fun.) Too slow? I don't really think that is necessarily true. We are seeing very good response times on DSSSL formatting without *any* optimization or use of a JIT compiler. (Our DSSSL engine is written in Java). > XML is one important part of the picture, and although DSSSL is one > way to help fill in the whole picture, I don't see it as being an > immediate answer. No, I don't have any better ideas, and I would love > to be proved wrong. (This is one of those rare cases where I _want_ to > be wrong.) The only way to ensure that XML and DSSSL will be a solution is to require some level of DSSSL compliance. The question is (James, correct me on this one) is there a minimal set of features that could be defined that would allow a DSSSL engine to be "easy" to implement but still be compliant with the DSSSL standard? Is this DSSSL-online? DSSSL is a "today" solution if we want to make it so. ============================================================================== R. Alexander Milowski http://www.copsol.com/ alex@copsol.com Copernican Solutions Incorporated (612) 379 - 3608 From: R A Milowski Subject: Re: XML Entity Manager To: w3c-sgml-wg@w3.org Date: Sat, 21 Sep 1996 11:10:56 -0500 (CDT) > I am surprised that nobody has remarked on the woeful absence of discussion > of the entity management problem both in our goals document and here on > the list. So... let me as usual advance the minimalist position: > > XML offers only 4 kinds of external entities: the cross-product of > > (PUBLIC or SYSTEM) x (PCDATA or NDATA) > > PCDATA/NDATA offer little for discussion, you either parse it or > you don't, right? > > SYSTEM entities, somebody at some point suggested outlawing them, and there > are certainly good reasons for that, but in XML I think we'd like easy stuff > to be easy, which SYSTEM does; it should be fenced around with stern warnings > about nonportability. In every SGML system I have ever built, setup, or used, I have *never* used system entities. It will always lead to trouble. I would support banning them, but would concede to allowing them. There is a third type of entity to look at: system default. System default entities are a lookup by name. They are not very good for interchange but are excellent means for achieving system independence in a document without having to declare the entities explicitly. The reason why this is good is that in a internal publishing system, you could have thousands of entities (I have a client who has over 10000 separate language entities). In their internal use of entities, to declare all the possible entities that a document *could* use is practically speaking impossible--the parse of the document would take a *very* long time. In my opinion, there are three to decide on: 1. System entities (Explicit system ids--non-transportable). 2. System defaulted (Lookup by name). 3. Public entities (Lookup by public identifier). 2 & 3 are covered by the SGML Open catalog format. In SGML open, 2 is: ENTITY foo "blah" 3 is: PUBLIC "-//Someone//DTD Something//EN" "somewhere" We should also embrace the concept of Formal System Identifiers. This would allow a document to push transport issues into the environment (catalogs) instead of being embedded in the document. For example, an HTML HREF could be: Foo and the catalog entry could be: ENTITY DocumentFoo "http://mydomain.com/documentfoo.html" Of course, there's a little more to location-independent linking than the above. (See the HyTime Clink and Nameloc.) Comments? ============================================================================== R. Alexander Milowski http://www.copsol.com/ alex@copsol.com Copernican Solutions Incorporated (612) 379 - 3608 Date: Sat, 21 Sep 1996 12:19:37 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: XML Entity Manager At 06:39 AM 9/21/96 +0000, Tim Bray wrote: >I am surprised that nobody has remarked on the woeful absence of discussion >of the entity management problem both in our goals document and here on >the list. So... let me as usual advance the minimalist position: > >XML offers only 4 kinds of external entities: the cross-product of > >(PUBLIC or SYSTEM) x (PCDATA or NDATA) What about text entities vs. external text entities, vs. subdocument entities vs. some other fragment technology? Eliot Kimber has a vote in favour of subdocument entities at http://www.sil.org/sgml/kimber-subdoc.html . Paul Prescod Date: Sat, 21 Sep 1996 09:24:14 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: XML Entity Manager | I am surprised that nobody has remarked on the woeful absence of | discussion of the entity management problem both in our goals document | and here on the list. So... let me as usual advance the minimalist | position: Entity management hasn't been mentioned because that discussion has been scheduled for a separate phase of design immediately following the November draft. The discussion, when it occurs, is expected to be complex and probably rather heated. Please defer all further consideration of this question until after the November WG meeting in Boston. Jon (as Chair) From: lee@sq.com Date: Sat, 21 Sep 96 14:12:06 EDT To: bosak@atlantic-83.eng.sun.com, w3c-sgml-wg@w3.org Subject: Re: Back to basics Jon wrote: > [...] the goal of the W3C SGML activity is to enable Web browsers to > receive and process generic SGML in the way that they are now > able to receive and process HTML. Agreed.... however, if that were the only issue, SoftQuad Panorama would satisfiy it, no? We can all go home :-) Well, OK, there are still issues -- * Panorama is only shipping on Windows (Unix and Mac are in early beta) * Panorama deals with documents, not fragments. I don't want to subvert the discussion towards SGML OPEN fragments, but only to note that Panorama can't deal with 100 MByte documents * Panorama works with Netscape and MSIE and Mosaic, but other browser writers need to work with us & write a DLL (for Windows) to communiate * Panorama can't (as far as I can tell) work on Unix as a helper-app unless you edit your SGML to add a processing instruction giving the document's URL. This is because under Unix you can't ask Netscape for the URL of the document you've been asked to fetch -- you just get handed a tmp file. We might be able to use the Netscape plugin API to deal with this. The CCI spec (asking the user to specify an IP port number) is unworkable. However, Panorama under Windows with documents no larger than 4 MBytes works pretty much like a standard web browser. I therefore assert that the goal stated above is already satisfied. That doesn't mean I don't think XML is not a bad idea. I fully support the idea of a regular, simplified version of SGML that can be parsed easily. Let's make SGML uniquitous, as Michael SMcQ put it in his fabulous closing speech at SGML 95 last December (is that online anywhere??). So perhaps I am unclear on how to interpret the Goals. > One factor that we know has prevented the wide deployment of generic SGML > browsers is the difficulty of implementing the complete 8879 specification. Well, it has prevented devlopment of multiple browsers. And even Panorama implements only the required features of SGML. No OMITTAG, no SHORTTAG (actually some SHORTTAG features are there, but I don't think net enabling tags are supported, for example) no DATATAG or SHORTREF (I've hardly ever even seen those used, but maybe that's self-selection), no LINK, no CONCUR, no RANK (I think), no SUBDOC... don't know about CURRENT. You can always run James' spam on the server, right out of the can, and remove the minimisation features that Panorama doesn't understand. In fact, use of OMITTAG is a frequent source of error, so it should probably be added. I have listed lots of things Panorama can't do, not because I don't like it (it's a fabulous piece of software) but because I want to show that you can go a long way without supporting those features. They are not needed. Perhaps LINK and SUBDOC are hardest to do without; I have recently seen a way (I think) to do subdoc, as long as it is used as an entity-valued attribute and not just &otherdoc;... which is somewhat harder. LINK can be done in an odd sort of way with HyTime. Using external markup and connecting it to a document frame with HyTime is very powerful. So for deploying SGML on the web, I think we are there today, and the right question may be `Is this where we want to be? Why are we not satisfied?'. I don't want to start a flame war about this. Clearly Panorama does _not_ satisfy the goal in everyone's minds, or this list wouldn't exist. Or is the stated goal merely a political front for a Higher Goal? (better reply privately on that one!) Lee From: lee@sq.com Date: Sat, 21 Sep 96 14:31:08 EDT To: tbray@textuality.com, w3c-sgml-wg@w3.org Subject: Re: XML Entity Manager > Which leaves PUBLIC. Any chance of getting away with saying "all PUBLIC > entities are to be resolved by recourse to an SGML/Open Catalog, ref. > 9401:1995"? Seems reasonable. Note that the draft is to be updated, partly to allow (from my awful memory) INSTANCE, and partly to change the fiename to catalog.soc and reword the case insensitivity requirement so that it is actually implementable on filesystems where a file probe or directory list are either expensive or prohibited (e.g. common on the web). So check on the draft status and refer to the latest version if it's passed in time. Lee From: Charles@sgmlsource.com (Charles F. Goldfarb) To: lee@sq.com Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Sat, 21 Sep 1996 20:21:13 GMT On Thu, 19 Sep 96 23:21:36 EDT, lee@sq.com wrote: > <p>Listen to my heart beat. > <?DIRECTOR: audio on> > And beat and beat and beat.</p> > >If the required form of this in XML were either > ><p>Listen to my heart beat.<?DIRECTOR: audio on> >And beat and beat and beat.</p> > >or > ><p>Listen to my heart beat. ><?DIRECTOR: audio on>And beat and beat and beat.</p> > >then both SGML and XML would parse the document in the same way, no? > >Lee > Yes, but then you are requiring the author to enforce the rules as well as remember them. With smart record handling in the parser, the author only has to remember the rules; the parser enforces them. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Tim Bray <tbray@textuality.com> Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Sat, 21 Sep 1996 20:21:14 GMT On Sat, 21 Sep 1996 06:08:19 +0000, Tim Bray <tbray@textuality.com> wrote: > >Charles' proposal in this area is to require explicit markup around >pseudo-elements in mixed content, probably minimized into something like '"' >via shortref, e.g. ><p>"Listen to my heart beat " <!-- trailing space required, right? --> ><?Director Audio On> >"and beat and beat." ></p> > >Advantages: >[a] & [b] exactly as above, and >[c] it's 8879-compliant > >Costs >[a] people have to put in a little more markup >[b] it is unlikely that XML can ever be fed to an HTML reader. > >Not really an easy choice; but I think we've succeeded in shaking out what >the issues are. > [a] According to the Design Principles: " 10. Terseness is of minimal importance." [b] Native processing of XML by unaware HTML readers is not an XML objective. (judging by earlier postings, it isn't even a credible fantasy) See, it really is an easy choice (although perhaps not a pleasant one). -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Sat, 21 Sep 1996 16:13:47 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: SGML Entities or Web-style objects in XML? Before we get too far into a discussion of entities, I'd like to take a second to reflect on XML's dual parentage. (risking Len's ire =) ) The SGML Way: In the SGML world, entity referncing and document parsing are intertwined, though separate. A document cannot be considered "valid" until its entities have been resolved (especially its external text entities!. Entities are declared (and theoretically resolved) in the DTD or DTD subset by the entity manager and used by the parser to create the ESIS. In all but a few SGML systems, all document content reuse is done through this mechanism. The Web Way: In the Web world (especially in the HTML world), entities (or "objects") are, conversely, resolved by the application AFTER parsing. So, the parser parses the document and returns the ESIS to the application. The application starts to process it and fetches (perhaps through an entity manager) any objects it needs to complete that task. In reality, "transclusion" or "inclusion" or "fragment inclusion" is just a special case of linking. The standardized mechanism for including HTML content is the same as for including JAVA or Active-X content: <OBJECT>. In other words, in HTML, document transcusion is just a special case of linking. The parser doesn't know or care about it. The Heresy: Do we really need parse-time entities in XML? What do they "buy?" In a networked environment, the decision to resolve entities or not should be entirely left up to the application (not the parser) because only the application knows which entities it "needs", and the cost of resolving entities it does not need is quite high. Furthermore, since the number of entity-resolution failures will be quite high (relatively speaking) going over the Internet, the application should be able to choose which entities are strictly need and which may be absent. In other words, I am proposing that we should not let entities affect the parsing of their containing documents. Every XML document would be validated without regard to the content or existance of its sub-documents. Further, the content of sub-documents should not affect the parse-tree of the parent document in any way. They would all be "opaque" to the containing document at the parser level. At the application level, of course, it might be invalid to transclude a footnote as if it were a chapter, but that's the same as if you hyperlinked to a chapter as if it were a footnote. But HyTime (and XTime =) ) has/will have facilities for specifying those constraints at the "link manager" level. If we can agree that the parser doesn't need to care about fragments/entities/objects, then we can wait until the spring to talk about them. Paul Prescod From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Cc: w3c-sgml-wg@w3.org Subject: Re: XML vrs SGML tools [was Re: Capitalizing on HTML (was ...)] Date: Sat, 21 Sep 1996 20:21:17 GMT On Fri, 20 Sep 1996 09:16:50 -0400, Paul Prescod <papresco@calum.csclub.uwaterloo.ca> wrote: >At 02:17 AM 9/20/96 GMT, Charles F. Goldfarb wrote: >>It seems like the two Pauls are agreeing that > >I can only speak for me. =) > >>1. XML has to be easy for SGML editors (and word-processor add-ons) to generate >>and for arbitrary browsers to read. > >Right. That's why I'm worried about this RS/RE correction proposal. I don't >think today's SGML editors will support it. Otherwise, the proposal is quite >elegant and simple. They don't actually have to. Any SP-based hacking tool (and probably Omnimark as well) could perform the SGML->XML transform, and the reverse transform as well. > >>We needn't worry about XML-only editors or >>dedicated XML browsers because there won't be any. > >I would word this a little differently: "We can't _depend_ on XML-only >editors or dedicated XML browsers because there MAY NOT be any." (especially >in the short term) I stand corrected. But I think the implication for the XML design is the same. For myself (not paraphrasing two Pauls, as before), I am troubled by the idea of no XML browsers because I thought that the purpose of XML was to encourage tool development. If we don't expect any XML tools, why design XML? My personal belief is that if we do XML well, there will be XML browsers. I don't believe we'll see any dedicated XML editors any time soon, however; just XML modes in SGML editors and trivial transform tools for SGML<->XML. But that is not necessarily a bad thing, given the objectives of XML. > >>2. The browsers need to be able to work without accessing a DTD. > >Exactly! To summarize my proposal, then, insofar as it affects the document instance. 1. Delimit pseudo-elements. 2. Supply end-tags for empty elements. 3. Allow CDATA and RCDATA marked sections. An XML browser can handle all this with just a simple stack to match ends of elements and marked sections with their starts. We can decide whether to allow comment declarations, PIs, SHORTTAG, ignored marked sections, inclusions, etc. on their own merits, as they won't affect the basic parse. And this solution is valid 8879 because an arbitrary DTD (including one with mixed content and declared content) can trivially (and reversibly) be transformed to an equivalent DTD that supports the above XML instance conventions. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Sat, 21 Sep 1996 16:46:35 -0500 From: Len Bullard <cbullard@HiWAAY.net> To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> CC: w3c-sgml-wg@w3.org Subject: Re: SGML Entities or Web-style objects in XML? Paul Prescod wrote: > > Before we get too far into a discussion of entities, I'd like to take a > second to reflect on XML's dual parentage. (risking Len's ire =) ) Doesn't even a little bit. This is what we came to do. Shake out issues, look at alternatives, try to get the work done. so far, so good. len From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Cc: w3c-sgml-wg@w3.org Subject: Re: SGML Entities or Web-style objects in XML? Date: Sun, 22 Sep 1996 02:52:04 GMT On Sat, 21 Sep 1996 16:13:47 -0400, Paul Prescod <papresco@calum.csclub.uwaterloo.ca> wrote: Jon says we're not to discuss this subject now, but so many erroneous impressions of SGML have been created that I feel a need to get the facts straight. >Before we get too far into a discussion of entities, I'd like to take a >second to reflect on XML's dual parentage. (risking Len's ire =) ) > > The SGML Way: > >In the SGML world, entity referncing and document parsing are intertwined, >though separate. (snip) > The Web Way: > >In the Web world (especially in the HTML world), entities (or "objects") >are, conversely, resolved by the application AFTER parsing. SGML has entity types for both of these "ways". The SGML Way: SGML text entities, internal and external, are accessed and parsed in context, as though their replacement text was physically in the entity from which they are referenced. The Other SGML Way (aka the Web Way, but we did it first): External data entities, whether CDATA, SDATA, or NDATA, are *never* parsed in context. The application is given a pointer to them and it can request access or not, as it sees fit. When accessed, parsing is based on the data content notation, which may or may not be SGML. SUBDOC entities are treated like CDATA external data entities with SGML as the notation. They are not automatically parsed in context. And just for the record, whether an external entity is located via a public identifier or a system identifier is totally independent of what kind of text is in the entity. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Sun, 22 Sep 1996 00:29:07 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: Back to basics [Liam Quin:] | So for deploying SGML on the web, I think we are there today, and the | right question may be `Is this where we want to be? Why are we not | satisfied?'. | | I don't want to start a flame war about this. Clearly Panorama does | _not_ satisfy the goal in everyone's minds, or this list wouldn't | exist. Or is the stated goal merely a political front for a Higher | Goal? (better reply privately on that one!) I don't mind responding publicly as long as it's clearly understood that this is just my own opinion. Panorama is a magnificent application. It's so good, in fact, that it serves as a proof that the lack of an implementation isn't the problem here. To my mind, it's the lack of ubiquity that's the problem. I believe that SGML functionality needs to be ubiquitous before content providers will be encouraged to make SGML data available as an optional high-level alternative to the HTML that they will be serving >from their SGML databases (which is how I visualize this happening). In my vision of a transition to SGML on the Web, two things occur to make SGML browser capability ubiquitous: 1. Entrepreneurial ISVs are enabled to try their luck in a new market by developing XML applications, i.e., applications that can consume an SGML that has been made much easier to implement. These applications could be browsers or they could be special-purpose applets. I think that this is what Len means when he speaks of another house to burgle. 2. One or more of the big and soon-to-be-big browser vendors (Netscape, Microsoft, Spyglass, JavaSoft) are encouraged to build XML support into their standard HTML offerings. The introduction of XML applications from multiple vendors is enabled simply by the existence of a properly designed XML. Whether the transition to XML as a standard browser format can be made to occur without participation of one of the larger players is an open question. My own opinion on this changes daily, but I do believe that our job will be much easier if we can persuade one of the HTML browser vendors to choose XML for their next forward leap in functionality rather than choosing something that they invent on their own. I suspect that people will have different opinions on this last point, but I don't think that it really matters for purposes of the current effort; as far as I can tell, the simplification that makes XML attainable for the independent developer is exactly the simplification that allows us to make a reasonable case for its adoption by the big guys. Jon Date: Sun, 22 Sep 96 03:31:16 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: some statistics on the impact of various proposals I put these statistics together a few days ago in the momentary confusion over document sizes. I was curious about the implications of various proposals on the size of a document. When the confusion was cleared up, I began wondering why we were concerned about being able to parse without the DTD. So, I decided to send this out anyway just to provide a few data points. The size of the sample of documents that I used to get these statistics was a little over 44 Mbytes of SGML in 138 documents. The average size of a document is 320 Kbytes. The range was from 2 Kbytes to 2.7 Mbytes. Empty end tags: The names of end tags comprise 15% of the total size of the document. I didn't try to differentiate, but this is probably due in large part to the number of tables in these docs (just over 5000 of them). I didn't try to determine the mode which would have been interesting but more work than I was willing to invest. The range was from 7.5% to 20.6%. "Pseudo-element" delimiters Eliminating mixed content as James and Charles have proposed, would increase the average document size in this set by 0.98% for every character used to bound the "pseudo-element." So, for example, if you had a single character name for the element, file size would increase by 6.86% (three characters in the start tag, four in the end tag. The range was from 0.5% to 1.9%. End tags on EMPTY elements Adding end-tags to EMPTY elements increased file size 0.9%. The range was from 0.3% to 3.3%. Attribute value literals Currently, all of the attribute values in these docs are quoted, i.e., attribute value literals. If I were to strip out all unnecessary quotes, I could reduce the file size by 2%. The range was from 0.3% to 4.2%. DTD size As is, our DTD, without comments, is 25 Kbytes. If I were to trim out all of the fat, and rebuild it with an emphasis on keeping size to a minimum, I'm sure I could get it to around 20 Kbytes. There are lots of declarations that could be paired up. At 20 Kbytes, the DTD is 6.3% the size of an average document. Stylesheets The DynaText stylesheets associated with these documents are 94 Kbytes, 29% of the size of an average document and 4.7 times the size of the DTD. There are some oddities in the DynaText stylesheet language that might make it more verbose than a corresponding DSSSL stylesheet, but I kind'a doubt it since there are also a lot of pieces that encapsulate a lot of behavior into single functions. Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Sun, 22 Sep 1996 09:47:09 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: Re: SGML Entities or Web-style objects in XML? At 02:52 AM 9/22/96 GMT, Charles F. Goldfarb wrote: >On Sat, 21 Sep 1996 16:13:47 -0400, Paul Prescod ><papresco@calum.csclub.uwaterloo.ca> wrote: > >Jon says we're not to discuss this subject now, but so many erroneous >impressions of SGML have been created that I feel a need to get the facts >straight. Thanks for using the agentless passive voice, Charles. =) You are right, I should have mentioned those other forms of entities. I do still have concerns about their applicability that I will bring up again in November. There still is a distinct "Web Way" although the second "SGML Way" is closer to it. More in November. Paul Prescod Date: Sun, 22 Sep 1996 11:11:37 -0500 From: Len Bullard <cbullard@HiWAAY.net> To: Jon Bosak <bosak@atlantic-83.eng.sun.com> CC: w3c-sgml-wg@w3.org Subject: Re: Back to basics Jon Bosak wrote: > > In my vision of a transition to SGML on the Web, two things occur to > make SGML browser capability ubiquitous: > > 1. Entrepreneurial ISVs are enabled to try their luck in a new market > by developing XML applications, i.e., applications that can consume an > SGML that has been made much easier to implement. These applications > could be browsers or they could be special-purpose applets. I think > that this is what Len means when he speaks of another house to burgle. Yes. It just doesn't work in a 5 second sound byte. > 2. One or more of the big and soon-to-be-big browser vendors > (Netscape, Microsoft, Spyglass, JavaSoft) are encouraged to build XML > support into their standard HTML offerings. Yes. But the market has to make it necessary. The market won't unless the content developers do. They won't unless they have tools or see it as "the next cool thang". In any event, it's a chicken and the egg. To break the circle, you need either vendors or college students. College students with encouragement from the federal government or enlightened professors have been the traditional engine of the internet. OTH, the guys in DoD are sitting on massive lumps of SGML data that they would like to move onto the net in some cases, or into interoperable IETMs in others. Because the creme de la creme of SGML is here, we all know of a few projects that could use cheaper tools. The barrier to SGML is not how hard it is to implement, it is the cost of obtaining the implemented tools and that everytime someone tried to do this in the past, members of our own community stifled the attempts. > I do believe that > our job will be much easier if we can persuade one of the HTML browser > vendors to choose XML for their next forward leap in functionality > rather than choosing something that they invent on their own. True. But go forward with or without them. Otherwise we repeat the same mistake we have always made. We assume the market must exist before the work is done and the market assumes the work must exist. All we are investing right now is our time. > I suspect that people will have different opinions on this last point, > but I don't think that it really matters for purposes of the current > effort; as far as I can tell, the simplification that makes XML > attainable for the independent developer is exactly the simplification > that allows us to make a reasonable case for its adoption by the big > guys. Exactly. Just remember, the big guys are the former little guys. They are building up legacy code just as some of our companies did. They become dinosaurs and we become little big eyed mammals coming out at night to eat eggs. This is like gigging in an indy band: they don't come to you. You make the opportunity. It takes brains and bravery, and just a bit of chutzpah. len From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Cc: Liam Quin <lee@sq.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> Subject: Re: RS/RE: basic questions Date: Sun, 22 Sep 1996 20:27:10 GMT On Fri, 20 Sep 96 10:53:46 CDT, Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> wrote: >On Thu, 19 Sep 1996 23:23:03 -0400 Liam Quin said: >> <p>Listen to my heart beat. >> <?DIRECTOR: audio on> >> And beat and beat and beat.</p> >> >>If the required form of this in XML were either >> >><p>Listen to my heart beat.<?DIRECTOR: audio on> >>And beat and beat and beat.</p> >> >>or >> >><p>Listen to my heart beat. >><?DIRECTOR: audio on>And beat and beat and beat.</p> >> >>then both SGML and XML would parse the document in the same way, no? > >Or even > > <p>Listen to my heart beat. > <?DIRECTOR: audio on > >And beat and beat and beat.</p> > >I find it just as hard to explain to users why some newlines disappear >as some people seem to find it hard to explain that a newline in >character data is character data. Actually, a lot harder, since to >this day I wouldn't claim to be able to tell accurately whether a >newline is actually ignored or not, in any case except the simple one >(newline immediately after start-tag or immediately before end-tag), >without running a test through SP or sgmls first. Which means I *can't* >explain it, 'cause I don't understand it. > >Can someone actually explain the RE rules here, in terms *not* those >used in the relevant clause of 8879? (If I understood it in those >terms I wouldn't have this problem.) > >Michael Sperberg-McQueen > There's only ever been one rule: An RE that exists only because of markup is ignored. (It's the application of the rule that causes confusion.) If we delimit pseudo-elements, then the rule is: undelimited REs are ignored, delimited ones are data. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Sun, 22 Sep 96 15:29:08 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: XML and required DTDs At 07:48 PM 9/13/96 -0700, Tim Bray wrote: > o I believe that there are many things you can do with structured > text that do not require Full Markup Declaration (search, display, > analyze). We want you to be able to do those things in XML without > having a Full Markup Declaration at hand. I agree wholeheartedly with #4 of the design principles especially with the examples that Tim provided above. The problem seems to arise at the definition of "useful" in the design principles: In particular, it shall be straightforwardly possible to construct useful XML applications which do not read, or need to read, the DTD of the XML document. Search engines may never be able to find a DTD if they're indexing docs retrieved from the net, some browsers can already do a pretty good job of presenting a doc without a DTD (DynaText, for example), and there are any number of applications that can perform a lot of "useful" processing without a DTD. There are also a lot of applications where given a relatively simple doctype, a DTD may never be necessary. A specialty doctype and a chunk of java client-side can take a lot of pressure off of a gateway (to a DBMS) server. If you've ever had a key gateway crash because its process table filled up you understand how valuable this can be. I also agree that it might be desirable to express the DTD using the same syntax as the doc. This would eliminate the need for a completely separate parsing module. Kimber posted a good start to c.t.s a year or two ago, Wayne's encapsulated the productions making it pretty verbose but another option, ours is built to encapsulate the documentation with the declarations so wouldn't really apply, Michael? Yours? However, when we start talking about partial DTDs or XML editors that don't use a DTD, my hackles go up. First, I suggest killing the idea of a partial DTD. If any of the declarations are required, just send the whole DTD. I don't see any benefit in trying to extract a partial DTD (who's going to do that anyway?). If it's an issue of syntax, then let's change the syntax of the DTD. I care about the syntax of the instances, I don't care if we use COBOL to express the DTD. Unless we made it some incredibly verbose syntax, I don't think the size of the DTD is an issue. I would bet that it will always be significantly smaller than the stylesheet. I also believe that the vast majority of XML applications will be inside corporate networks where the number of DTDs will be relatively small and caching systems will eliminate most DTD fetches anyway. If it's syntax, let's address it. I think it would be time better spent than trying to come up with partial DTD delivery schemes. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Sun, 22 Sep 96 15:29:03 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 10:53 AM 9/20/96 CDT, Michael Sperberg-McQueen wrote: >Or even > > <p>Listen to my heart beat. > <?DIRECTOR: audio on > >And beat and beat and beat.</p> This is an excellent technique. DynaTag uses it to great success to get around mixed content problems in the really screwy DTDs that it creates. At 06:08 AM 9/21/96 +0000, Tim Bray wrote: >And I agree 100%. I am proposing (perhaps only as a strawman) that in XML we >make it 100% crystal clear that in in case [2], the "true information" is, in >C notation, > "\nListen to my heart beat\n\nand beat and beat\n" >or if you typed it in on a windows box > "\r\nListen to my heart beat\r\n\r\nand beat and beat\r\n" I agree, sort of. If possible, I'd like to have the first and last discarded. These should be easy to differentiate and discard and are "logically" insignificant. >No ambiguity whatsoever. The costs are: >[a] difficulty in figuring out how to make this 8879-compliant, and This is tricky (if not impossible) and I could accept a divergence here as it's impact is very, very small. >[b] the fact that you can no longer use whitespace around markup if you are > worried about the application's handling of line breaks in the data, so you > might in fact have to use > > <p>Listen to my heart beat<? Director Audio on> and beat and beat.</p> > > which is harder to read, and hard to type in vi, for long paragraphs. I can easily live with this as this is the current situation. Applications decide what a newline sequence means based on the stylesheet. In Author/Editor and DynaText they look like spaces unless I specify "verbatim" formatting in the stylesheet. >The advantages are: >[a] you can explain quickly, clearly, and precisely exactly what the "true > information" is >[b] Implementation is very easy I think the advantages far outweigh the disadvantages and it is easy to explain to the author. The only place it becomes an issue is when you have markup in data content that is not a proper subelement and it's the only thing on the line. Worst case: if line breaks are significant (to the presentation), you get an empty new line; if line breaks are not significant, you get an extra space. These things are easily picked up during proofing. Good spellcheckers even pick up the two or more spaces for you. If I'm in line-break-significant content, I'm also very unlikely to put in any extraneous markup anyway. If I need to, I can easily remember to "hide" the line break in the added markup using either Lee's or Michael's suggestions. Another advantage is that it makes it very easy for an SGML editor to "save as XML," at least in this case. The biggest disadvantage is that in the probably very, very few cases where someone wanted the extraneous line-break after some markup, an SGML parser would discard it. I can live with this risk. At 08:21 PM 9/21/96 GMT, Charles F. Goldfarb wrote: >Yes, but then you are requiring the author to enforce the rules as well as >remember them. With smart record handling in the parser, the author only has to >remember the rules; the parser enforces them. But I think the rules are much simpler to remember and a lot easier to digest than having to sometimes "quote" data content. This requires that the author know what mixed content is and which elements are mixed. This is a lot more to bite off than the alternative. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Sun, 22 Sep 1996 17:09:41 -0900 To: w3c-sgml-wg@w3.org From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: Element Structure for XML (Clause 7) At 05:05 PM 9/12/96 -0400, Steven J. DeRose wrote: I've been fortunate enough never to encounter a conflict in those >self-assigned PI keywords -- anybody else actually hit one yet? (I'm not >saying that would prove reliability, just good fortune or remarkable >implementor restraint...) The processing instructions used by Panorama to include style sheets and navigators are generic enough that they could easily conflict: STYLESPEC, NAVIGATOR. I agree with Steve and Charles that the initial "PI keyword" is, semantically, a notation name. We could even propose a new public text keyword for use in notation declarations: PROCINST, e.g.: <!NOTATION Panorama PUBLIC "-//Synex AB//PROCINST Panorama control instructions//NE" > While I think PIs are a poor (if easy) way to do what Arbortext does, I think Paul's argument that PIs exist and thus excluding them would cause a lot of pain is a good one. In the abstract, I agree with Martin that PIs should be restricted to the prolog. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.squirrel.com/squirrel/drmacro (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Sun, 22 Sep 96 15:02:56 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: what *are* the RE rules? To: W3C SGML Working Group <w3c-sgml-wg@w3.org> The other day, I asked for a summary of SGML's RE rules, in terms other than those of clause 7.6.1. Paul Prescod pointed us all to a 1993 posting from Erik Naggum on comp.text.sgml, which does in fact seem to me very useful. Some people may quibble over his interpretation of the word 'ignore', but his interpretation has the advantage of being explicit, plausible, and on the record. On the other hand, Erik's paraphrase, like clause 7.6.1, refers rather obscurely to situations in which "no data or proper subelement" has occurred within the element content, which passes over in silence the question what *can* occur in these situations. Inspired by the recent discussion and by Erik Naggum's old posting, I've made my own attempt at summarizing the rules in clause 7.6.1 in a less indirect way. Is this a full and accurate summary? ------- RS is significant only if it's markup -- since it can be markup only in a shortref, it's of no interest to XML. For our purposes, RS is always ignored, period. SGML documents consist of markup and content. If an RE occurs in an SGML document, it's in markup or in content; there's no place else for it to be. In markup, RE is easy; an RE can occur only: - as a separator (in declarations or tags), in which case it's ignored - within a PI, in which case it's passed through to the application - within a literal in an attribute value specification (in a tag or entity declaration or attlist declaration), in which case it's replaced by a SPACE before the attribute value is passed to the application - within a literal in an entity declaration, in which case its treatment is determined when the entity is expanded - as the refc delimiter on a reference, in which case it's eaten by the parser In content, RE can occur: - in element content, in which case it's ignored - within mixed content or (replaceable) character data; it is here that the rules for ignoring it or treating it as data seem, to some people, to become complex. (Or, as Charles Goldfarb has just reminded me, the rule is simple: RE is ignored when caused by markup. The determination of causality is what seems complex. Fair point.) Clause 7.6.1 a says "the first RE in an element is ignored if no RS, data, or proper subelement preceded it." Phrased the other way around, and working from the list in clause 7.6 of all the things that can occur in mixed or character data content, this means that the pattern starttag negligible* RE reduces to starttag negligible* where 'negligible' is defined as negligible ::= comment declaration | shortref use declaration | link set use declaration | processing instruction | character reference | entity reference | marked section declaration | included subelement | short reference | entity-end Rule (b) in the same clause says, in effect, that the same applies at the end of an element: so if the end of an element matches the pattern RE negligible* end-tag the RE is ignored. Rule (c) in the same clause says, I think, that if a record (i.e. the space between an &#RS; and the next &#RE;) is not empty, but contains no data, then the RE is ignored. I think this means that if any record containing nothing other than comment declaration(s), PI(s), the beginning of an entity, a marked section declaration, or an included subelement, then its RE is ignored. So the element Q contains no REs in any of the following cases: <q> Listen to my heart beat. </q> This is the simple case: RE adjacent to a start-tag or end-tag. All the most persuasive examples I've seen involve REs adjacent to the tag, which has often led me to wonder why the rule couldn't be just that that the TAGC of a start-tag is defined as ">" followed by an optional newline, and ETAGO as an optional newline followed by "</". <q> <!-- sound track is silent --> Listen to my heart beat <!-- -- ><?DIRECTOR begin: audio> and beat and beat and beat. </q> Here rule (a) takes care of line 1, rule (c) of line 2, the comment of line 3, rule (c) again of line 4, and rule (b) of line 5. The only case I can think of where rule (a) applies without the RE being adjacent to the start-tag is something like <q><!-- sound track is silent --> Listen to my heart beat. </q> Perhaps most of the obscurity lies in 8879's omitting any explanation of what 'ignoring' data means. (For example, RS is ignored unconditionally. But it can't be ignored in the sense that the software takes no notice of it, since the RE rules involve knowing precisely where the last RS occurred.) Well, no. Not *most* of the obscurity. Clause 7.6.1 really begins to challenge my hermeneutical skill when it mentions that REs are 'deemed' to occur immediately preceding the next following data or proper subelement. So the element <q> Listen. <!-- silence. --> <!-- The clock ticks. --> <!-- The wind sighs. --> <!-- The clock chimes. --> <?DIRECTOR: start audio-track 1 > Listen to my heart beat. </q> is, if I am reading the standard right, to be 'deemed' identical to <q> Listen. <!-- silence. --> <!-- The clock ticks. --> <!-- The wind sighs. --> <!-- The clock chimes. --> <?DIRECTOR: start audio-track 1 > Listen to my heart beat. </q> with the RE originally situated after "Listen." having migrated five lines down, past four comments and a processing instruction. Rule (c) also appears to mean that <q> Listen. <!-- silence. --> <!-- The clock ticks. --> Listen to my heart beat. </q> has a record end between the two sentences, since the RE after the first comment cannot be ignored by rule (c), but that <q> Listen. <!-- silence. The clock ticks. --> Listen to my heart beat. </q> has no record end, since the record end after the second line of the comment ends a non-empty record which contains no data. ------- Is this an accurate summary? If so, I suggest that either of the following alternative rules are simpler for both implementor and user; the first one handles all the cases I think people are likely to agree aren't confusing. - RE is ignored when immediately adjacent to the TAGC of a start-tag, the ETAGO of an end-tag, the PIC of a processing instruction that began at or before the beginning of the current line, or the MDC of a comment that began at or before the beginning of the current line or - RE is ignored when it occurs outside literals within markup - RE is ignored in element content - in character data, RE is a data character -C. M. Sperberg-McQueen From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Robert Streich <streich@slb.com> Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Sun, 22 Sep 1996 22:39:12 GMT On Sun, 22 Sep 96 15:29:03 CDT, Robert Streich <streich@slb.com> wrote: >The only place it becomes an issue is when you have markup in data content >that is not a proper subelement and it's the only thing on the line. Worst >case: if line breaks are significant (to the presentation), you get an >empty new line; if line breaks are not significant, you get an extra space. > >These things are easily picked up during proofing. Good spellcheckers even >pick up the two or more spaces for you. If I'm in line-break-significant >content, I'm also very unlikely to put in any extraneous markup anyway. >If I need to, I can easily remember to "hide" the line break in the added >markup using either Lee's or Michael's suggestions. As you just demonstrated, you are forcing the author to both remember the rules and implement them himself. They are not simple rules; if they were, we wouldn't be having this discussion. > >But I think the rules are much simpler to remember and a lot easier to >digest than having to sometimes "quote" data content. This requires that >the author know what mixed content is and which elements are mixed. This >is a lot more to bite off than the alternative. That isn't my proposal, Bob. My proposal is that data always be "quoted", just as in programs. The author does *not* have to know about mixed content (in fact, mixed content won't actually exist). White space within quotes is data; other white space is ignored. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Sun, 22 Sep 1996 22:18:47 -0900 To: Charles@sgmlsource.com, w3c-sgml-wg@w3.org From: "W. Eliot Kimber" <kimber@passage.com> Subject: How you respond to notes Charles, Small point, but your responses would be much easier to read if you put a line before your responding text. I find it very difficult to distinquish your responses from the quoted text. Cheers, E. >>If PIs are retained then PIC should be changed so that it is not > as this >>is required for many processing instructions. (An alternative may be to >>allow a character reference to be entered within a PI, but this would make >>XML incompatible with SGML.) >This is backwards. Changing PIC breaks SGML. A character reference would not be >recognized by the SGML parser, but could be recognized by the XML application. >XML would strip out the char refs before passing the PI to the system (or to the >designated notation processor if my previous suggestion is adopted). -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.squirrel.com/squirrel/drmacro (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" To: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Sun, 22 Sep 1996 21:30:55 -0700 From: Joe English <jenglish@crl.com> Charles@sgmlsource.com (Charles F. Goldfarb) wrote: > > There's only ever been one rule: An RE that exists only because of markup is > ignored. (It's the application of the rule that causes confusion.) > > If we delimit pseudo-elements, then the rule is: undelimited REs are ignored, > delimited ones are data. ... but they can still be moved around in the presence of processing instructions, markup declarations, and included subelements. (Personally I wouldn't mind disallowing inclusion exceptions, and restricting PIs and comment declarations to element content or even to the prolog; but I do think that disallowing mixed content is too restrictive.) --Joe English jenglish@crl.com Cc: Charles@sgmlsource.com To: w3c-sgml-wg@w3.org To: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: How you respond to notes Date: Sun, 22 Sep 1996 21:47:23 -0700 From: Joe English <jenglish@crl.com> "W. Eliot Kimber" <kimber@passage.com> wrote: > Small point, but your responses would be much easier to read if you put a > line before your responding text. I find it very difficult to distinquish > your responses from the quoted text. That would just introduce record-ends whose presence can be attributed solely to markup and hence should be ignored anyway :-) --jenglish@crl.com Date: Mon, 23 Sep 96 01:50:54 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 10:39 PM 9/22/96 GMT, Charles F. Goldfarb wrote: >As you just demonstrated, you are forcing the author to both remember the rules >and implement them himself. They are not simple rules; if they were, we wouldn't >be having this discussion. I think they are pretty straightforward: In data content: 1. If an element begins or ends with a newline [not entirely accurate, but this is what people see], the newline is ignored. 2. Newlines inside markup are ignored. 3. All other newlines are passed on. I could be convinced to drop #1 to make it even simpler, but I'd like to keep it if possible just to reduce the amount of inconsistency with SGML. >That isn't my proposal, Bob. My proposal is that data always be "quoted", just >as in programs. The author does *not* have to know about mixed content (in >fact, mixed content won't actually exist). White space within quotes is data; >other white space is ignored. My fault. I missed that point. That does diminish my disagreement with this proposal, not enough to agree with it though. I'd prefer to leave out the "quoting." bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 To: bosak@atlantic-83.eng.sun.com (Jon Bosak), w3c-sgml-wg@w3.org From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: Back to basics Cc: bosak@atlantic-83.eng.sun.com Date: Mon, 23 Sep 1996 09:32:03 +0100 Jon > I do believe that >our job will be much easier if we can persuade one of the HTML browser >vendors to choose XML for their next forward leap in functionality >rather than choosing something that they invent on their own. I could not agree more, but I fail to see how this can be possible when we say that XML cannot read even valid HTML documents. A leap into the dark without taking the light you already have takes a vast amount of faith. ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ Date: Mon, 23 Sep 1996 11:07:38 +0000 To: Robert Streich <streich@slb.com> From: James Clark <jjc@jclark.com> Subject: Re: RS/RE: basic questions Cc: w3c-sgml-wg@w3.org At 01:50 23/09/96 CDT, Robert Streich wrote: >At 10:39 PM 9/22/96 GMT, Charles F. Goldfarb wrote: >>As you just demonstrated, you are forcing the author to both remember the rules >>and implement them himself. They are not simple rules; if they were, we >wouldn't >>be having this discussion. > >I think they are pretty straightforward: > > In data content: > 1. If an element begins or ends with a newline [not entirely > accurate, but this is what people see], the newline is ignored. > 2. Newlines inside markup are ignored. > 3. All other newlines are passed on. According to your statement of the rules all of the three newlines in the following would be passed on: <doc><?> <?> data <?></doc> In fact none of them are. James From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "W. Eliot Kimber" <kimber@passage.com> Cc: w3c-sgml-wg@w3.org Subject: Re: How you respond to notes Date: Mon, 23 Sep 1996 10:12:11 GMT On Sun, 22 Sep 1996 22:18:47 -0900, "W. Eliot Kimber" <kimber@passage.com> wrote: >Charles, > >Small point, but your responses would be much easier to read if you put a >line before your responding text. I find it very difficult to distinquish >your responses from the quoted text. Thanks. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Joe English <jenglish@crl.com> Cc: w3c-sgml-wg@w3.org Subject: Re: How you respond to notes Date: Mon, 23 Sep 1996 10:20:13 GMT On Sun, 22 Sep 1996 21:47:23 -0700, Joe English <jenglish@crl.com> wrote: > >"W. Eliot Kimber" <kimber@passage.com> wrote: > >> Small point, but your responses would be much easier to read if you put a >> line before your responding text. I find it very difficult to distinquish >> your responses from the quoted text. > >That would just introduce record-ends whose presence >can be attributed solely to markup and hence should >be ignored anyway :-) Actually, it wouldn't. The SGML record handling rules recognize that data could contain intentionally empty records. > >--jenglish@crl.com > -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Joe English <jenglish@crl.com> Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Mon, 23 Sep 1996 10:20:10 GMT On Sun, 22 Sep 1996 21:30:55 -0700, Joe English <jenglish@crl.com> wrote: > >Charles@sgmlsource.com (Charles F. Goldfarb) wrote: >> >> There's only ever been one rule: An RE that exists only because of markup is >> ignored. (It's the application of the rule that causes confusion.) >> >> If we delimit pseudo-elements, then the rule is: undelimited REs are ignored, >> delimited ones are data. > >... but they can still be moved around in the presence >of processing instructions, markup declarations, and >included subelements. Not at all. Nothing is allowed within a delimited pseudo-element except data. >(Personally I wouldn't mind disallowing inclusion >exceptions, and restricting PIs and comment declarations >to element content or even to the prolog; but I do think >that disallowing mixed content is too restrictive.) What you've just said you "wouldn't mind" has the effect of disallowing mixed content. My proposal is *less* restrictive, because by delimiting the pseudo-elements you don't need to restrict PIs, etc. <p> "Here is data on multiple lines " <?XMLPI with a PI> "and more data after the PI." </p> -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Mon, 23 Sep 1996 08:25:32 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: Re: RS/RE: basic questions At 09:30 PM 9/22/96 -0700, Joe English wrote: >(Personally I wouldn't mind disallowing inclusion >exceptions, and restricting PIs and comment declarations >to element content or even to the prolog; but I do think >that disallowing mixed content is too restrictive.) Can somebody help me? Joe seems to be proposing that if we * restrict PIs and comments to element content * restrict mixed-content-models to "|" * disallow inclusion exceptions then we can reduce the RS/RE handling rules to "Robert's Rules" ( =) ) of In data content: 1. If an element begins or ends with a newline [not entirely accurate, but this is what people see], the newline is ignored. 2. Newlines inside markup are ignored. 3. All other newlines are passed on. Is this really the case? Is this simplification an option? Paul Prescod Date: Mon, 23 Sep 1996 14:04:36 +0000 To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> From: James Clark <jjc@jclark.com> Subject: Re: RS/RE: basic questions Cc: w3c-sgml-wg@w3.org At 08:25 23/09/96 -0400, Paul Prescod wrote: >At 09:30 PM 9/22/96 -0700, Joe English wrote: >>(Personally I wouldn't mind disallowing inclusion >>exceptions, and restricting PIs and comment declarations >>to element content or even to the prolog; but I do think >>that disallowing mixed content is too restrictive.) > >Can somebody help me? > >Joe seems to be proposing that if we > >* restrict PIs and comments to element content >* restrict mixed-content-models to "|" >* disallow inclusion exceptions > >then we can reduce the RS/RE handling rules to "Robert's Rules" ( =) ) of > >In data content: > 1. If an element begins or ends with a newline [not entirely > accurate, but this is what people see], the newline is ignored. > 2. Newlines inside markup are ignored. > 3. All other newlines are passed on. > >Is this really the case? For that to be the case, you would also need to disallow marked section declarations in mixed content. Date: Mon, 23 Sep 1996 09:22:46 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: Re: RS/RE: basic questions [Incorporating James' feedback] If we * restrict PIs and comments to element content * restrict marked sections similarly * restrict inclusion exceptions to element content or, (more likely) disallow them * restrict mixed-content-models to "|" then we can reduce the RS/RE handling rules to "Robert's Rules" ( =) ) of In data content: 1. If an element begins or ends with a newline [not entirely accurate, but this is what people see], the newline is ignored. 2. Newlines inside markup are ignored. 3. All other newlines are passed on. First, does anyone else have revisions to these rules? Restrictions to add, or complications I haven't forseen? Second, are these restrictions acceptable? Are "Robert's Rules" simple enough? Finally, ar there any hidden costs to this approach? Paul Prescod From: Gavin Nicol <gtn@ebt.com> Date: Mon, 23 Sep 1996 13:51:05 GMT To: milor001@gold.tc.umn.edu CC: w3c-sgml-wg@w3.org Subject: Re: XML Entity Manager >There is a third type of entity to look at: system default. I would like to keep these as they can simplify the automatic generation of catalogs. Date: Mon, 23 Sep 96 09:26:37 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: Re: RS/RE: basic questions To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca>, W3C SGML Working Group <w3c-sgml-wg@w3.org> On Mon, 23 Sep 1996 08:35:04 -0400 Paul Prescod said: >Joe seems to be proposing that if we > >* restrict PIs and comments to element content >* restrict mixed-content-models to "|" >* disallow inclusion exceptions > > ... > ... Is this simplification an option? Perhaps I undervalue the SGML record-end rules, but it seems to me that prohibiting PIs and comments in mixed content solely in order to preserve the RE rules is a case of throwing out baby in order to preserve bathwater. If we think it's hard to explain to users where REs are preserved and where they are suppressed, wait til we try explaining why comments are allowed between paragraphs but not within paragraphs. So no, I hope it's not considered a realistic option. The other two restrictions have independent motivations, and I think I favor each of them, myself. To come back to our core test: Are the RE rules *essential* to good reprsentation of textual data? I haven't heard anyone argue this position, only other positions: - the RE rules are part of 8879, and our goal of alignment with 8879 requires that we follow them somehow - the RE rules are a convenience for data entry - the RE rules are simpler to explain than the alternatives The only answer I know to the first argument is that in this particular case strict conformance may exact too high a price in simplicity and comprehensibility. I'd like to minimize the difference between SGML and XML in this area, but it may be that this is one place where the conformance goal has to give. (I suppose this boils down to my saying that for me, the RE rules don't even pass the weaker test James was arguing against: I *don't* like them that much, and they *don't* seem reasonably easy to implement.) About the other two positions there is nothing to say except that they do not seem to me to be true; others seem to find them more persuasive. -C. M. Sperberg-McQueen Date: Mon, 23 Sep 1996 10:12:28 -0600 From: "William D. Lindsey" <blindsey@bdmtech.com> To: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions All of the alternatives to the quoted pseudo element proposal for dealing with the problem of REs in mixed content seem to have one or more of the following problems: a) Restrictive of what can go in mixed-content elements. b) Subtly different from 8879. c) Places a burden on a typist to learn and enforce rules. If any of the proposals have none of these problems, I apologize for not having paid close enough attention. I'm wary of breaking 8879 conformance, however subtly, without a compelling reason. It says that SGML isn't up to the job. So, how compelling are the disadvantages of the quoted pseudo element proposal? I tried, once again, to list them. What have I missed? Disadvantages: 1) Typists must quote data. 2) Many of today's SGML editing tools have difficulties with shortrefs. Advantages: 1) Extremely simple, regular rules for RE/RS handling. 2) Completely 8879 conformant. 3) Inclusion exceptions less problematic. 4) PIs less problematic. I'll add another of my own -- 5) Precisely what the tool builders, whose support is fundamental to XML's success, are used to seeing in computer languages. With this proposal, all we're doing is using SGML to create some syntactical sugar which makes more apparent what SGML really does: Provides for the linearized representation of hierarchically structured data through a fully bracketed form and a prefix syntax. In other words, we use SGML to make an SGML that looks more like SGML. -Bill -- William D. Lindsey blindsey@bdmtech.com +1 (303) 672-8954 Date: Mon, 23 Sep 1996 12:21:36 -0400 To: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU>, Paul Prescod <papresco@calum.csclub.uwaterloo.ca>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "David G. Durand" <dgd@cs.bu.edu> (David G. Durand) Subject: Re: RS/RE: basic questions At 9:26 AM 9/23/96, Michael Sperberg-McQueen wrote: >To come back to our core test: > >Are the RE rules *essential* to good reprsentation of textual data? > >I haven't heard anyone argue this position, only other positions: > - the RE rules are part of 8879, and our goal of alignment with > 8879 requires that we follow them somehow > - the RE rules are a convenience for data entry > - the RE rules are simpler to explain than the alternatives > >The only answer I know to the first argument is that in this particular >case strict conformance may exact too high a price in simplicity and >comprehensibility. I'd like to minimize the difference between SGML and >XML in this area, but it may be that this is one place where the >conformance goal has to give. (I suppose this boils down to my saying >that for me, the RE rules don't even pass the weaker test James was >arguing against: I *don't* like them that much, and they *don't* >seem reasonably easy to implement.) Hear! Hear! >About the other two positions there is nothing to say except that they >do not seem to me to be true; others seem to find them more persuasive. As a coutnerexample to their necessity, I'll note that even "Joe HTML" has learned that whitespace needs to be used carefully (to get images and tables to format correctly). It's a pain to be forced not to use a line-break, but the rule "line-breaks always count as whitespace" is certainly very easy to remember. Like Michael, I don't vaslue compatibility with 8879 so much that I would preserve the current rules. There are still a some hard questions about how, and whether to try to normalize the marking of input lines (ie. should we preserve the input character stream intact, convert to an XML-standard linend convention before we send data to the application, require a standard linend convention on XML input, or simply ignore line endings entirely and require markup for significant linends). But I think the current rules for RS/RE just don't make the grade. We should not let our decisions about other SGML features (like includions, comments, and markjed sections) depend on the handling of this one issue -- we should resolve markup features on their value as markup, not their ability to preserve line endign comaptibility with 8879. I think treating the entire SGML input as a single record is still the best way to go, and it can be implemented by any entity manager. If we are not compatible with current parsers on this feature, that can be regarded as _strong_ input to the 8879 revision. RS/RE Delenda Est! >-C. M. Sperberg-McQueen Never mind that if Michael can't understand the rules easily, that most of the rest of us who think we do understand them are probably wrong... -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ To: Paul Prescod <papresco@calum.csclub.uwaterloo.ca>, w3c-sgml-wg@w3.org From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: RS/RE: basic questions Date: Mon, 23 Sep 1996 17:35:45 +0100 At 02:22 PM 23/9/96 +0100, Paul Prescod wrote: >* restrict marked sections similarly >Second, are these restrictions acceptable? Are "Robert's Rules" simple enough? > >Finally, ar there any hidden costs to this approach? What does it mean to restrict marked sections to element content? How would an author know when they were permitted without detailed knowledge of the DTD? Are we presuming only CDATA and RCDATA within an element, or are you thinking in terms of ignored and included marked sections containing whole elements? If my model is <!ELEMENT x (a,b,c)>, where the subelements are defined as <!ELEMENT (a,b,c) (#PCDATA|em)> and <!ELEMENT em (#PCDATA)>, which of the following are valid? <x> <![INCLUDE[<a>This is a <em>bastardized</em> example</a>]]> <b>This is an <em><![CDATA[<SGML>]]></em> example. <![INCLUDE[This is a French markup tag: <em><![RCDATA[<example lang=français>]]>.]]> </x> ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Mon, 23 Sep 1996 10:52:55 -0700 From: Joe English <jenglish@crl.com> (The discussion so far has focussed on RS/RE handling, but there is also a problem with separator characters _other_ than record-ends in mixed vs. element content.) Paul Prescod <papresco@calum.csclub.uwaterloo.ca> wrote: > Joe seems to be proposing that if we > > * restrict PIs and comments to element content > * restrict mixed-content-models to "|" [ actually, "restrict mixed content to OR groups with a REP occurrence indicator and which only contain primitive content tokens", or something equivalent; IOW, "no pernicious mixed content" ] > * disallow inclusion exceptions There's one more rule (which is the important one): * disallow separator characters in element content This is because things like: <a> <b>blah</b> <b>blah</b> </a> have different meanings depending on whether A has mixed content or element content. The record-end after the first </b> end-tag is significant in the former case, and is ignored in the latter. If A has element content, the above would have to be written like: <a><b>blah</b><b>blah</b></a> or <a ><b>blah</b ><b>blah</b ></a> instead. > then we can reduce the RS/RE handling rules to "Robert's Rules" ( =) ) of > > In data content: > 1. If an element begins or ends with a newline [not entirely > accurate, but this is what people see], the newline is ignored. > 2. Newlines inside markup are ignored. > 3. All other newlines are passed on. Yes, as far as I can tell. Charles' proposal is similar: * restrict PIs and comment declarations to element content * disallow mixed content * disallow inclusion exceptions (or perhaps, disallow included subelements in pseudoelement content). * require data content to be delimited The chief difference is in the second and fourth rules. With these restrictions the RS/RE/separator character rules are even simpler: 1. Delimited separator characters are data. 2. Undelimited separator characters are ignored. --Joe English jenglish@crl.com Date: Mon, 23 Sep 1996 15:14:33 -0400 To: <w3c-sgml-wg@w3.org> From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: Re: RS/RE: basic questions At 12:21 PM 9/23/96 -0400, David G. Durand" (David G. Durand wrote: > I think treating the entire SGML input as a single record is still the >best way to go, and it can be implemented by any entity manager. If we are >not compatible with current parsers on this feature, that can be regarded >as _strong_ input to the 8879 revision. One concern to me is that this proposal will be _subtly_ incompatible with existing tools. The user will want <P>Foo</P> The editing tool will output <p> Foo </P> SGML/XML parsers will interptet that as (P " Foo " P) We will have to make it VERY clear that although XML _looks_ like SGML, it is actually subtly different. If we are going to do something incompatible with SGML, I prefer the eliminate-mixed-content proposal. It is sufficiently different that nobody will think that XML is SGML and it allows authors to use whitespace to organize their text (which was the original goal of the RS/RE rules). Paul Prescod >Received: by chuckd.allette.com.au.allette.com.au id AA12078 (5.65/IDA-1.5 for w3c-sgml-wg@w3.org); Tue, 24 Sep 1996 02:08:40 +1000 Date: Tue, 24 Sep 1996 02:08:39 +1000 (EST) From: Rick Jelliffe <ricko@allette.com.au> To: Tim Bray <tbray@textuality.com> Cc: w3c-sgml-wg@w3.org Subject: Re: XML Entity Manager On Sat, 21 Sep 1996, Tim Bray wrote: > I am surprised that nobody has remarked on the woeful absence of discussion > of the entity management problem both in our goals document and here on > the list. So... let me as usual advance the minimalist position: The minimalist position would be to have no user-defined entities at all. Is this out of the question? Rick Jelliffe http://www.allette.com.au/allette/ricko email: ricko@allette.com.au ================================================================ Allette Systems http://www.allette.com.au email: info@allette.com.au 10/91 York St, 2000, phone: +61 2 9262 4777 Sydney, Australia fax: +61 2 9262 4774 ================================================================ Date: Mon, 23 Sep 1996 17:43:14 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: Re: RS/RE: basic questions I want to be careful to make sure that we aren't mixing up proposals. I was not proposing that we treat the document as a single record. I was proposing that we restrict all of the confusing RS/RE cases in mixed content so that XML is SGML compatible, but not as confusing. At 10:52 AM 9/23/96 -0700, Joe English wrote: >There's one more rule (which is the important one): > > * disallow separator characters in element content I don't see how that is required. As far as I know, the SGML separator character rules are not complicated in element content and would not have to be changed. >This is because things like: > > <a> > <b>blah</b> > <b>blah</b> > </a> > >have different meanings depending on whether A has mixed content or >element content. The record-end after the first </b> end-tag is >significant in the former case, and is ignored in the latter. > >If A has element content, the above would have to be written like: > > <a><b>blah</b><b>blah</b></a> > >or > > <a > ><b>blah</b > ><b>blah</b > ></a> > >instead. As long as all of our restrictions are on mixed content, which is where the problem is, then I don't wee why we would need any restrictions on element content. If I'm wrong on that, then this proposal is completely unworkable. People need to be able to organize their element content with whitespace. Paul Prescod To: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Mon, 23 Sep 1996 16:10:52 -0700 From: Joe English <jenglish@crl.com> Paul Prescod <papresco@calum.csclub.uwaterloo.ca> wrote: > Joe English wrote: > >There's one more rule (which is the important one): > > > > * disallow separator characters in element content > > I don't see how that is required. As far as I know, the SGML separator > character rules are not complicated in element content and would not have to > be changed. The problem is that the parser has to examine (at least part of) the DTD in order to determine which elements have mixed content and which have element content. If the goal is to enable DTD-less parsers to construct the same grove as DTD-aware ones, then SEPCHARs in element content have to be distinguished from those in mixed content through some other mechanism. One way is to explicitly delimit the latter; another is to disallow the former. This is independant of dealing with the RS/RE rules. For that problem, David Durand's suggestion that XML entity managers treat documents as a single record sounds like a good one since it places the fewest restrictions on DTDs and instances of all the proposals so far. > As long as all of our restrictions are on mixed content, which is where the > problem is, then I don't wee why we would need any restrictions on element > content. If I'm wrong on that, then this proposal is completely unworkable. > People need to be able to organize their element content with whitespace. You're probably right... --Joe English jenglish@crl.com From: lee@sq.com Date: Mon, 23 Sep 96 19:51:19 EDT To: w3c-sgml-wg@w3.org Subject: Re: RS/RE quoted data proposal > An SGML-editor friendly one like this: > > <P><C>abcdefgh</C><EM><C>ijklmnopq</C></EM></P> > > and a a typist-friendly alias like this: > > <P>"abcdefgh"<EM>"ijklmnopq"</EM></P> The latter syntax is certainly not typist friendly. What happens if you omit a quote? If you're going to do that, use @p{abcdefgh@em{ijklmopq}} instead -- can also be done with SGML. But I think this is all besides the point. If you allow mixed content, and say (SGML notwithstanding) that any amount of whitespace is equivalent to a single space, in all circumstances, you get something that can be handled with no DTD, with no need to distinguish element & text contexts, and that is easy to explain. Although the "..." proposal is a very clever use of relatively obscure SGML, I don't fancy explaining it, and when you can include newlines in quoted `strings'', and why the emphasised text in the example is actually embedded in the text it surrounds, conceptually, even though the syntax makes this impossibly obscure. A language needs to be * consistent & regular * easy to explain * clearly and unambiguously defined. It is a fact of history that SGML does not meet these goals, but any new system for which wide deployment is desired must do so. Change the whitespace rules. Accept the idea of a mechanical translation between HTML and XML, and between SGML and XML, and that if you go from a richer format to a weaker one you may lose information. Then you'll get something that can actually be used. If you have to accept all currently legal HTML syntax (as some have opined), or if you are encumbered by needless complexities that are in the more powerful language because of features not used in XML, you'll end up with something irregular and complex. Lee <P>this is a <em>simpler</em> approach.</P> From: lee@sq.com Date: Mon, 23 Sep 96 19:59:55 EDT To: U35395@UICVM.CC.UIC.EDU Subject: Re: RS/RE: basic questions Cc: papresco@calum.csclub.uwaterloo.ca, w3c-sgml-wg@w3.org > Intelligent RE handling is *essential* to good (=accurate) document > representation I would say rather intelligently _designed_ whitespace handling... For example, TeX has a uniform and regular rule, but it does not involve the concept of records. You either have white space or you don't, just as in most modern programming languages. I don't mean to comment on the design of the SGML white space mechanism here, by the way. But I think it necessary to consider things as a whole: SGML is doing a lot more than XML, and many of its rules wil be more complex than those of XML -- or XML wouldn't be simpler. Lee -- Liam Quin, SoftQuad Inc | lq-text freely available Unix text retrieval lee@sq.com +1 416 544-9000 | FAQs: Metafont fonts, OPEN LOOK UI, OpenWindows SGML: http://www.sq.com/ | We've moved; new 'phone number & postal address! The barefoot programmer | `who is my neighbour?' Date: Mon, 23 Sep 96 19:15:46 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: some topics we haven't yet discussed To: W3C SGML Working Group <w3c-sgml-wg@w3.org> We have two weeks left in our month of open discussion; I've been trying to see what topics need to be (further) discussed, and which probably don't need (further) discussion. I have arrived at a satisfyingly long list of things I think we don't need to discuss, and a mildly alarming list of things that haven't yet been discussed enough, or in some cases at all. I'm not going to post the first list, of things I think we don't need to discuss, since that could backfire badly. The topics I think we need to discuss before we close the voting booth, so people can make up their minds intelligently, fall into several classes: those relating to entities and their declarations, those relating to element declarations, those relating to attribute declarations, and a fourth group of general and miscellaneous issues. I'll separate out the other three groups into distinct postings, for the sake of mail archives with threading mechanisms. Here's my list of general and miscellaneous issues raised by the various specific proposals on the table, which I think the WG needs to address before the ERB actually starts drafting. Clause numbers are attached, for those who want to look at the voting booth to see how the proposals now deal with these issues, or at 8879 to see what it says. I've also added a few parenthetical comments; apologies to those who think I'm flogging any dead horses.) Participants may also wish to read the relevant proposals, if they haven't already. * Should XML use the markup-declaration syntax described by ISO 8879 clauses 10-11, or should XML define a specialized document type and let its markup declarations use the document-instance syntax, as proposed by MGML? (Using the same syntax for declarations and instances cuts the size of the grammar approximately in half. It also reflects a firm belief that structured information belongs in SGML documents.) * How should XML deal with the need for conditional inclusion of markup declarations, if XML has no marked sections (10.4.1)? (In instances, people seem happy to use elements to contain the conditional text. If XML uses instance syntax for markup declarations, such conditional-inclusion elements can be used for the purpose.) * Should XML change the delimiter-in-context rules to require the STAGO and ERO strings to be escaped whenever they are not to be recognized as delimiters (9.6)? (Some people have proposed this; certainly many user manuals I've seen prescribe this behavior for users, rather than explaining the d-i-c rules.) * Should XML use MSOCHAR, MSSCHAR, and MSICHAR strings (9.7)? (If MSSCHAR is '\', then \< and \& are escaped in the expected way. Unlike most Unix-style backslash escapes, however, the MSSCHAR is not removed from the data stream; this means \ characters in existing docs are safe.) * Should XML require system and public identifiers to be FORMAL (13.5)? * Should XML restrict comment declarations to a single comment (10.3)? (Motivation: the hash made of SGML comments by some current HTML browsers; simplifying the syntax might make it harder to get wrong.) -C. M. Sperberg-McQueen Date: Mon, 23 Sep 96 19:33:33 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: questions about attribute declarations To: W3C SGML Working Group <w3c-sgml-wg@w3.org> Questions about attribute declarations I'd like to see discussed while the voting booth is still open: * Should XML retain SGML's prohibition on multiple attribute-list declarations for the same element (11.3.1) or on multiple declarations for the same attribute (11.3.2)? * Should XML change the set of types available for attributes? E.g. by suppressing NAME(S), NUMBER(S), NMTOKEN(S), NUTOKEN(S) and adding constraints in the form of regular expressions, ISO dates, language-code, external-id, typed IDREF, ... (7.9.4, 11.3.3) * Should XML remove SGML's prohibition on multiple ID or NOTATION attributes on the same element (11.3.3)? * Should XML allow more than one enumerated type (name-group declared value) to contain the same possible value (11.3.3)? (Since XML seems unlikely to support SHORTTAG, this restriction seems pointless in XML. It does affect the translation of XML DTDs into SGML.) * Should XML define new specific methods of inferring values for attributes with no attribute-value specifications (11.3.4)? E.g. INHERITED, to signify that the value is taken from the attribute of the same name (and type) on the smallest enclosing element with such an attribute. (This is the method normally used for setting the default of language attributes, among others; currently it has no SGML keyword.) * When given an incomplete DTD, should XML parsers assume all attributes are implicity declared CDATA, or should some attribute names cause other assumptions (e.g. attributes with the same name as some type of declared value might be assumed to have that declared value)? Date: Mon, 23 Sep 96 19:36:44 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: questions about entities and entity declarations To: W3C SGML Working Group <w3c-sgml-wg@w3.org> Questions about entity structure and entity declarations. * should XML require all entities to be synchronous with the document's logical structure? * should XML restrict the types of entities? We seem to agree to keep document entities, text entities, and non-SGML data entities, and to drop subdocument entities, so this boils down to * whether XML should keep or discard the concepts of CDATA and SDATA entities (6.1). * whether XML should retain 'data text' entities (CDATA, SDATA, PI) (10.5.3, 8.1) * whether XML should retain 'bracketed text' entities (STARTTAG, ENDTAG, MS, MD) (10.5.4) * should XML prescribe the use of an ENTITY-END character as the canonical method of handling entity boundaries, as a way of simplifying exposition and implementation (6.2.2)? * should XML retain or relax SGML's prohibition on ENTITY attributes referring to SGML text entities (7.9.4.3)? * if XML makes DTDs optional and allows partial DTDs, what must or may a parser do when it encounters references to undeclared entities (9.4)? Should XML declare any set of entities automatically? * if XML uses ISO 10646, should there be a special form of character reference using hexadecimal, not decimal, numbers, since most references to ISO 10646 and Unicode use hex, not decimal (9.5)? (So references to schwa could take a form like &u0259; or &x0259;, not *#601;, which is rather error-prone, given that nothing in the Unicode documentation gives decimal numbers for the character positions.) * Should XML remove SGML's prohibition on ENTITY attributes for notations (11.4.1)? Date: Mon, 23 Sep 96 19:31:35 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: questions about element declarations To: W3C SGML Working Group <w3c-sgml-wg@w3.org> The proposals on the table seem to me to raise several questions about element declarations which we haven't yet discussed here: * Should XML retain SGML's prohibition on multiple declarations for the same element (11.2.1)? * Should XML restrict the use of #PCDATA to content models of the form (#PCDATA), or of the form (#PCDATA | A | B | ...)*, as a way of avoiding the mixed-content trap (11.2.4) and/or of simplifying RE handling (7.6.1)? * Should XML prohibit the use of inclusion and exclusion exceptions in element declarations? (11.2.4, 11.2.5)? * Should XML prohibit content-model references to undeclared elements (11.2.4)? * Should XML forbid use of the '&' connector in content models (11.2.4.1)? * Should XML allow nondeterministic content models (11.2.4.3)? Date: Mon, 23 Sep 96 20:08:46 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 11:07 AM 9/23/96 +0000, James Clark wrote: >At 01:50 23/09/96 CDT, Robert Streich wrote: >>I think they are pretty straightforward: >> >> In data content: >> 1. If an element begins or ends with a newline [not entirely >> accurate, but this is what people see], the newline is ignored. >> 2. Newlines inside markup are ignored. >> 3. All other newlines are passed on. > >According to your statement of the rules all of the three newlines in the >following would be passed on: > ><doc><?> ><?> >data ><?></doc> > >In fact none of them are. Exactly right on both counts. I'm willing to accept this inconsistency with SGML. An XML/SGML editor could write it out such that it would parse the same in SGML, i.e., <doc><?><?>data<?></doc> or <doc ><? ><? >data<? ></doc> Someone who was hand coding the doc would have to know how to conceal the newlines if they were in an element where newlines were considered "significant." But then, I'm inclined to believe that they would expect them in a case like this. If the line endings weren't considered significant, I'd expect a browser to discard them just as it would any other leading or trailing whitespace. With this approach, the only place you run into a problem is when you have significant line breaks. I'd rather have this then get rid of them altogether. Leave it up to the application to deal with the whitespace. If we don't have shortrefs then it's no longer important for the parser to care about what whitespace character it is. Tabs, newlines, carriage-returns, spaces, all get treated the same in data content--passed through. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Mon, 23 Sep 1996 18:19:16 -0700 (PDT) From: Bill Smith <bill.smith@Eng> Subject: Re: RS/RE quoted data proposal To: w3c-sgml-wg@w3.org Liam Quin wrote: > A language needs to be > * consistent & regular > * easy to explain > * clearly and unambiguously defined. > > It is a fact of history that SGML does not meet these goals, but any new > system for which wide deployment is desired must do so. Just so. The fact that this group does not universally understand SGML's RS/RE rules and/or their application indicates that SGML fails on at least two of the three points above. XML should not follow SGML's lead here. Neither implementors nor authors can understand the rules. David Durand's "treating the entire SGML input as a single record" has immediate, minimalist appeal to me. Line breaks would be whitespace and would always be treated as such. There would be no need to describe RS/RE (pseudo) characters or their associated rules - there wouldn't be any. Measuring David's proposal against Lee's list show's that whitespace handling would be: * consistent & regular, * easy to explain, and * clearly and unambiguously defined. What remains is sufficiency. Is this proposal sufficient to meet the needs of authors and implementors? If so than I believe we have an excellent solution to a rather nasty problem. Relax the conformace requirement and declare victory. Bill Date: Mon, 23 Sep 96 20:29:37 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 11:17 PM 9/23/96 GMT, Charles F. Goldfarb wrote: >Making it an "application convention" to strip what appears to be extraneous >whitespace (i.e., to figure out what is the "true information", just shifts the >burden from a few parsers to all applications and increases the chance of >inconsistent treatment). Applications already have this burden though, Charles. I'd expect any browser to collapse multiple whitespace characters into a single space unless I directed that I want "verbatim" formatting. Even with the quoted- content model, I'd expect this behavior. > Alternatively, telling the user that he can't put >markup or an included element on a line by itself just shifts the burden to him, >with even more chance of error if he doesn't have a validating editor. I'm sure you'd see a lot more errors with quoting. Even when programming, forgetting to put a closing quote on a string is probably the most common error that I make. Unless you treat the content of the quoted text as RCDATA, I think you'd also get a lot of strange looks from new-to-XML typists when you tell them that you can't put elements inside quoted text. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Cc: Paul Prescod <papresco@calum.csclub.uwaterloo.ca>, W3C SGML Working Group <w3c-sgml-wg@w3.org> Subject: Re: RS/RE: basic questions Date: Mon, 23 Sep 1996 23:17:31 GMT On Mon, 23 Sep 96 09:26:37 CDT, Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> wrote: >Are the RE rules *essential* to good reprsentation of textual data? > >I haven't heard anyone argue this position, only other positions: Intelligent RE handling is *essential* to good (=accurate) document representation because it is the only way to distinguish the true information content from the "source document formatting"; that is, from the rendition of the document that is created in the text editor when you enter the document. A principal objective of SGML is that all applications should receive the same "true information" about the document. When an SGML document is created with an editor that preserves line breaks (which SGML calls "record" breaks to avoid confusion with formatted output lines), the possibility exists that some record breaks are not part of the "true information". For example, in <p>Listen to my heart beat. <?DIRECTOR: audio on> And beat and beat and beat.</p> the true information is: "Listen to my heart beat. And beat and beat and beat." because the record end after the PI is not part of the data Similarly, if the user chose to set the tags off clearly by putting them in their own records, as in <p> Listen to my heart beat. <?DIRECTOR: audio on> And beat and beat and beat. </p> the true information still would be "Listen to my heart beat. And beat and beat and beat." With a mechanism like SGML's RS/RE handling (properly implemented), the parser always gives the identical "true information" to the application, regardless of the user's input style. Without intelligent record handling, in the last example the application instead sees: " Listen to my heart beat. And beat and beat and beat. " These two are very different character strings, so there is no guarantee that two different products, asked to do the identical processing will produce anything close to the same results. Even if the products would have produced identical results given the same character strings, they cannot do so now. Making it an "application convention" to strip what appears to be extraneous whitespace (i.e., to figure out what is the "true information", just shifts the burden from a few parsers to all applications and increases the chance of inconsistent treatment). Alternatively, telling the user that he can't put markup or an included element on a line by itself just shifts the burden to him, with even more chance of error if he doesn't have a validating editor. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Mon, 23 Sep 96 21:11:42 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 12:21 PM 9/23/96 -0400, David G. Durand" (David G. Durand wrote: > There are still a some hard questions about how, and whether to try to >normalize the marking of input lines (ie. should we preserve the input >character stream intact, convert to an XML-standard linend convention >before we send data to the application, require a standard linend >convention on XML input, or simply ignore line endings entirely and require >markup for significant linends). But I think the current rules for RS/RE >just don't make the grade. We should not let our decisions about other SGML >features (like includions, comments, and markjed sections) depend on the >handling of this one issue -- we should resolve markup features on their >value as markup, not their ability to preserve line endign comaptibility >with 8879. If we don't have SHORTREF, I don't see that the parser should care what kind of whitespace it read and could just pass everything along to the application. Even the whitespace in element content wouldn't cause the application any problems. It's just going to ignore any extra whitespace anyway unless the stylesheet tells it that it is significant. Even Joe's example of <a> <b>blah</b> <b>blah</b> </a> isn't significant. The stylesheet will determine whether or not <b> starts a new line. I may have forgotten a case or two, but the only place I can think of where the parser *may* need to normalize the whitespace is in CDATA attributes, otherwise it has no idea if the whitespace is significant to the display or not and it shouldn't need to care. I have a slightly different idea of what the "true" data is in a document as compared to Charles. I think the "true" data is anything outside the pointy brackets. 7.6.1 essentially tells typists that they don't have to worry about the trailing newline after some types of markup. I think it's easier to tell them that everything after the MDC is data bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Mon, 23 Sep 1996 18:48:21 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod <papresco@calum.csclub.uwaterloo.ca> Subject: RS/RE quoted data proposal >Disadvantages: > >1) Typists must quote data. >2) Many of today's SGML editing tools have difficulties with > shortrefs. This last disadvantage could be mitigated if we provide two syntaxes for for pseudo-element tagging. An SGML-editor friendly one like this: <P><C>abcdefgh</C><EM><C>ijklmnopq</C></EM></P> and a a typist-friendly alias like this: <P>"abcdefgh"<EM>"ijklmnopq"</EM></P> As long as XML consumers are either XML or SGML compatible, they should have no problem with either syntax. Paul Prescod Date: Mon, 23 Sep 1996 20:35:19 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: questions about element declarations At 07:31 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >The proposals on the table seem to me to raise several questions about >element declarations which we haven't yet discussed here: > >* Should XML retain SGML's prohibition on multiple declarations for the >same element (11.2.1)? I assume that the purpose of this restriction is to make it somewhat possible to limit author's abilities to modify centrally-defined declaration sets. If that's so, then I think we should keep the restriction in XML. The issue of whether or not this is an appropriate restriction should be left for the SGML revision, I think (I personally think it's not a useful restriction, but it's largely a matter of policy, not a technical issue, so we probably shouldn't argue it here). >* Should XML restrict the use of #PCDATA to content models of the form >(#PCDATA), or of the form (#PCDATA | A | B | ...)*, as a way of avoiding >the mixed-content trap (11.2.4) and/or of simplifying RE handling >(7.6.1)? I think this would be a good thing (assuming we allow mixed content at all). SGML effectively requires this anyway, so why not formalize it? >* Should XML prohibit the use of inclusion and exclusion exceptions in >element declarations? (11.2.4, 11.2.5)? Yes. Exclusions are, with very rare exceptions, a bad thing, causing much more trouble than they're worth. Inclusions, while expressing a useful semantic, also cause more trouble then they're worth and require DTD processing. >* Should XML forbid use of the '&' connector in content models >(11.2.4.1)? How much does AND complicate validation? I've seen some statements to the effect that it complicates it quite a bit, but I have no way to evaluate these claims. I think AND expresses a useful semantic, so I would say keep it unless the validation cost is too high. >* Should XML allow nondeterministic content models (11.2.4.3)? Again, how much does this complicate validation? I'm not ambiguity expert, but could the problem be solved simply by stipulating that a token is always matched to the first place in the content model it can match, without lookahead? Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Mon, 23 Sep 1996 21:51:10 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: some topics we haven't yet discussed At 07:15 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >* Should XML use the markup-declaration syntax described by ISO 8879 >clauses 10-11, or should XML define a specialized document type and let >its markup declarations use the document-instance syntax, as proposed by >MGML? No. I see no reason to try to define a new notation for declarations at this time. It is a *non trivial* exercise to define a document type or architecture for declarations, partly because the additional expressive power of SGML gives you a lot more options. Having spent quite a few hours putting together a DTD for DTDs that attempts to provide some useful features beyond simply reflecting the productions, I know that such an effort will either be unsatisfactory or take a long time (or both): it will either be a brain-dead transliteration of the productions (not useful, in my opinion) or it will be a time-consuming document type design-by-committee effort of the sort many of us have been involved with (and about which Eve has written an excellent book). I say no. The declaration syntax is understood and easy to teach. It's not hard to parse and it's compact. >(Using the same syntax for declarations and instances cuts the size of >the grammar approximately in half. It also reflects a firm belief that >structured information belongs in SGML documents.) While I certainly agree with the last sentence, in principle, I think the counter arguments are compelling in this case. If such a thing is to be developed as a standard, it needs to be developed with careful consideration of the implications the design will have. It's not something we can do quickly. >* Should XML change the delimiter-in-context rules to require the STAGO >and ERO strings to be escaped whenever they are not to be recognized as >delimiters (9.6)? This seems like a good idea. It is easy to use short references to define escape sequences for these, e.g.: "~<", "~&", "~~" where "~" is the "escape" character (but is really just the first character of a two-character short reference delimiter). I think a shortref approach is better than MSO, for the reasons given below. This doesn't break SGML, only requires quoting where it might not otherwise be required (but where people usually do it anyway). >* Should XML use MSOCHAR, MSSCHAR, and MSICHAR strings (9.7)? No--escaping is easier and safer using shortrefs. >* Should XML require system and public identifiers to be FORMAL (13.5)? I think the better questions are: A. Should public identifiers be FORMAL? B. Should XML support formal system identifiers? Formal system identifiers are defined in the forthcoming TC to the HyTime standard and would be very useful in a Web context because of the additional information they can provide for storage managers. I see no reason not to use FORMAL public identifiers. An informal public identifier is no better than a system identifier really. Formal public identifiers are functionally equivalent to URNs (to the degree I correctly understand URNs). In fact, URNs and FPIs could be identical were it not for an unfortunate syntax clash (both use "//" in different and unreconcilable ways). Note that the formal system identifier spec includes "URL" as one of the starter set of storage managers, so it already anticipates interoperation with the Web. >* Should XML restrict comment declarations to a single comment (10.3)? Or should we pretend that comment open is "<!--" and comment close is "-->"? While SGML *should* have different delimiter roles for comment open and comment close (and probably will after the revision), it doesn't today and this is an area where anticipating the standard would make XML documents incompatible with existing SGML processors. Thus, we *can't* do something like make "/*" COMO and "*/" COMC, because those delimiter roles don't exist in SGML. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Mon, 23 Sep 1996 21:51:13 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: questions about attribute declarations At 07:33 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >Questions about attribute declarations I'd like to see discussed while >the voting booth is still open: > >* Should XML retain SGML's prohibition on multiple attribute-list >declarations for the same element (11.3.1) or on multiple declarations >for the same attribute (11.3.2)? While I like having multiple attribute list declarations for the same element type, I think that we should not do it for XML, only because it removes one more thing that would have to be mangled to go from XML to SGML. >* Should XML change the set of types available for attributes? >E.g. by suppressing NAME(S), NUMBER(S), NMTOKEN(S), NUTOKEN(S) and >adding constraints in the form of regular expressions, ISO dates, >language-code, external-id, typed IDREF, ... (7.9.4, 11.3.3) I'm not sure what the value of suppressing the listed types is. I don't see that they add significant complexity and they are useful. The only way we could do lexical type constraints would be to do what HyTime does with Lextype, otherwise we break SGML. The HyTime lextype mechanism lets you use any expression language you want, but we'd still have to define one or pick one. Seems like a big can of worms that would only complicate validation. >* Should XML remove SGML's prohibition on multiple ID or NOTATION >attributes on the same element (11.3.3)? I could see an argument for having multiple IDs for a single element (different names used in different domains), although I'm hard pressed to see this as so compelling that we would accept the cost of incompatibility (when translating to SGML, which ID attribute would you use?). For notations, it normally seems meaningless to consider the content of an element to be in more than one notation at the same time, but I could see that there might be times where this is the case. But again, it doesn't seem compelling relative to the cost of incompatibility. These are certainly good issues for the SGML revision, though. >* Should XML allow more than one enumerated type (name-group declared >value) to contain the same possible value (11.3.3)? If we don't allow shorttag, then I don't see a problem here because the attribute can always be redeclared as "NAMES" with a corresponding lexical model, so there's not a serious compatibility issue. >* Should XML define new specific methods of inferring values for >attributes with no attribute-value specifications (11.3.4)? E.g. >INHERITED, to signify that the value is taken from the attribute of the >same name (and type) on the smallest enclosing element with such an >attribute. No. There are too many possible variations in how you might want inheritance to work. To make this complete, you need a way to say, for each attribute of each element type, what the precise inheritance rules are, something like HyTime's default value list. Unless we want to accept default value list, I think such a mechanism is beyond the scope of XML. It's something we should definitely think about for the SGML revision, though. Given that DTD-less parsing means you don't know which attributes are #IMPLIED, it seems like the potential for unintended side effects is very high if you do define an inheritance rule or rules. Having said that, it might be useful to define some conventional attribute form names that have defined inheritance rules, e.g. "language". Having conventional attribute names would require some sort of renaming mechanism such as the "HyNames" mechanism used in HyTime and formalized as the "ArcNamrA" attribute in architectures (see http://www.jclark.com/sp/archform.htm for a brief description of the architecture control attributes). >* When given an incomplete DTD, should XML parsers assume all attributes >are implicity declared CDATA, or should some attribute names cause other >assumptions (e.g. attributes with the same name as some type of declared >value might be assumed to have that declared value)? While Martin Bryan doesn't like it, I think it's reasonable to define a set of conventional attribute names: ID, ENTITY, NOTATION, IDREF(S). Behaves the way most people would expect them to behave and removes the need to declare the attributes in these cases. Doesn't prevent declaring attributes with other names but does prevent using these attribute names for something other than their conventional interpretation. A renaming mechanism could be used to map your own names to these conventional names, e.g.: <MyXMLEelement name=foo XMLNames="ID name"> When there's a DTD, the XMLNames (the "architecture renaming attribute") could be fixed, but when there's no DTD, you'd have to specify it. (Which raises the posibility that a careless author could do something different on two different elements: <MyXMLEelement name=foo XMLNames="ID name"> <MyXMLEelement uniqueID=foo XMLNames="ID uniqueID"> Ugh. Would probably have to make it a reportable XML error to map the same conventional attribute name to different instance names in the same document.) Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Mon, 23 Sep 96 20:08:54 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: RS/RE: basic questions At 10:12 AM 9/23/96 -0600, William D. Lindsey wrote: >I'll add another of my own -- > >5) Precisely what the tool builders, whose support is fundamental to > XML's success, are used to seeing in computer languages. This is a bit misleading since we're discussing a markup language rather than a programming language. The only markup language that I know of that uses the "quoted-text" model is MIF (Maker Interchange Format). DSR (Digital Standard Runoff) does require that *most* markup be at the beginning of a line, sort of the inverse of the quoted-text proposal. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Mon, 23 Sep 1996 22:11:57 -0900 To: w3c-sgml-wg@w3.org From: "W. Eliot Kimber" <kimber@passage.com> Subject: Principles for SGML Compatibility These may already have been articulated, but in responding to Michael's recent notes, I realized that I was applying the following rules for determining when something was or was not acceptible WRT to SGML compatability. I wanted to state them here to make sure these are in line with the official position and/or the majority view: 1. Processing of XML documents as SGML may require either creation of a DTD or transformation from XML-specific declarations to the equivalent SGML declarations. 2. Processing of XML documents as SGML must *never* require transformation of the document instance. In other words, XML instances must be processible as SGML as us once an appropriate DTD has been provided. 3. That while we are accepting the possibility of a transformation of an XML declaration set to an SGML declaration set, we should endeavor to limit the complexity of that transformation as much as possible, with the ideal that XML declarations are identical to SGML declarations (but may be a subset of the declarations required or allowed by SGML). 4. Processing of XML documents as SGML may require any or all of the optional SGML features omittag, shorttag, and shortref (I'm assuming that datatag is not useful). Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Mon, 23 Sep 1996 22:11:54 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: questions about entities and entity declarations At 07:36 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >Questions about entity structure and entity declarations. > >* should XML require all entities to be synchronous with the document's >logical structure? While I firmly believe in SGML's complete separation of storage from semantics (and thus, believe that there should be no required restriction on the alignment of elements with entities), requiring alignment of entities with elements in XML is probably a useful simplification that does not introduce any incompatibilities with SGML (in the XML-to-SGML direction). [...] > * whether XML should retain 'bracketed text' entities (STARTTAG, > ENDTAG, MS, MD) (10.5.4) These are mostly useful for doing short reference maps (although they could be useful for doing markup examples). I would have no objection to dropping them. >* should XML prescribe the use of an ENTITY-END character as the >canonical method of handling entity boundaries, as a way of simplifying >exposition and implementation (6.2.2)? I'm not sure what this question means. There is no ENTITY END *character* in SGML--it is a signal from the entity manager to the parser. >* should XML retain or relax SGML's prohibition on ENTITY attributes >referring to SGML text entities (7.9.4.3)? Retain. SGML text entities have no meaningful existence except as fragements of SGML document strings, therefore it cannot make sense to refer to one from an entity attribute. I would suggest that XML should further restrict NDATA, (external) CDATA, and SUBDOC entity references to ENTITY attributes. >* if XML makes DTDs optional and allows partial DTDs, what must or may a >parser do when it encounters references to undeclared entities (9.4)? >Should XML declare any set of entities automatically? Hmmm. Good question. >* if XML uses ISO 10646, should there be a special form of character >reference using hexadecimal, not decimal, numbers, since most references >to ISO 10646 and Unicode use hex, not decimal (9.5)? Such references would make processing of XML documents with SGML tools impossible without preprocessing. It would probably be useful for SGML to allow hexidecimal numeric character references, though. >(So references to schwa could take a form like &u0259; or &x0259;, not >*#601;, which is rather error-prone, given that nothing in the Unicode >documentation gives decimal numbers for the character positions.) Don't you have a hex-to-decimal calculator? :-) >* Should XML remove SGML's prohibition on ENTITY attributes for >notations (11.4.1)? Do you mean that XML should *allow* ENTITY attributes for notations (also, IDREF(S) and NOTATION)? I would say yes, because it only affects the DTD and it's something that will almost certainly be in the revised SGML. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" From: lee@sq.com Date: Tue, 24 Sep 96 00:15:01 EDT To: kimber@passage.com, w3c-sgml-wg@w3.org Subject: Re: questions about element declarations What follows is a little lengthy, but I hope that it will be of use. I should say at the outset that in saying that SGML can't or doesn't do something, I am stating (what I believe to be) a fact -- that is, there is no implied criticism. We must work with what we have, and not ask for it to have been otherwise. >> * Should XML restrict the use of #PCDATA to content models of the form >> (#PCDATA), or of the form (#PCDATA | A | B | ...)*, as a way of avoiding >> the mixed-content trap (11.2.4) and/or of simplifying RE handling (7.6.1)? > [Eliot...] > I think this would be a good thing (assuming we allow mixed content at > all). SGML effectively requires this anyway, so why not formalize it? Agreed. >> * Should XML prohibit the use of inclusion and exclusion exceptions in >> element declarations? (11.2.4, 11.2.5)? > > Yes. Exclusions are, with very rare exceptions, a bad thing, causing much > more trouble than they're worth. Inclusions, while expressing a useful > semantic, also cause more trouble then they're worth and require DTD > processing. I don't agree here. Some things are hard to do in other ways. Exclusions are good for preventing recursive inclusions. I noticed that in the GCAPAPER DTD, the only way to get text in a caption of a figure was to use HIGHLIGHT, which was an inclusion... but I think it's a mistake. Inclusions are hard to use. >> * Should XML forbid use of the '&' connector in content models >> (11.2.4.1)? > > How much does AND complicate validation? There are essentially three approaches I've seen. (1) build an NDFA ind compile down to a DFA using the normal published algrithms. If there are n terms and'd together, you get n! states to handle them. This means that using a & b & c & d & e & f uses 1 * 2 * 3 * 4 * 5 * 6 = 720 states. But if the model was more complex, e.g. (A,B,C) & d & e & f you are likely to get many more states than that, because you have to account for all the possible insertion points. (2) treat it as an or group and then do a check afterwards (requires either lookahead or, probably more efficiently if most documents are valid, backtracking) (3) modified DFA, essentially using annotations (in the graph theory sense) to note where such intrusions may occur, and pushing another automaton to deal with them. Probably there are other ways. Number (3) is the most efficient of the three, as far as I can tell, but I don't know a good way to deal with this. Unfortunately, as far as I know, there is no way to express (A,B,C) & d* to mean that (A, d, B, d, C) is acceptable. We ended up using inclusions and exceptions to model the HTML HEAD in the IETF HTML WG, because and-groups weren't expressive enough. A lot of the difficulty of SGML content models stems from (1) the lack of non-terminal symbols -- e.g. consider the YACC Chapter: TITLE p_list ; p_list: P | P p_list ; Here, p_list is a non-terminal that would never appear in the input. The (D S) problem recently posted to comp.text.sgml is easily solved using this paradigm, but adds the cost of having a shift-reduce stack. Another way to look at this is to say that SGML doesn't offer an expression language for recognising text. SHORTREF and DATATAG are not general tools for this. For example, you can't specify the syntax of a C expression using SGML as you can with YACC. (2) the lack of scope consider: <!Element (TITLE within Bibliography) - - (#PCDATA|SmallCaps) > <!Element (TITLE) - - (#PCDATA|FnRef|Emphasis|SmallCaps) > You could use exclusions to do this: <!Element Bibliography - - (....) -(FnRef|Emphasis)> and then adding Emphasis and FnRef as inclusions back where they made sense, e.g. in Bibliography.Remarks in an annotated bibliography. (3) the ambiguity rules Consider a report that may have Scope, Requirements, Proposal and Summary, but may have any number of other sections between them: <!Element Report - - ( Scope?, Section*, Requirements?, Section*, --* actually this is illegal *-- Proposal?, Section*, --* actually comments are illegal here too! *-- Summary? ) > Well, this is illegal because of the multiple sections. Of course, the model could cause rather large amounts of look-ahead, too. In many applications that's fine. But for delivering documents over the internet, avoiding look-ahead may be a good thing, reducing the delay before The User Sees The First Screen. Another way to avoid lookahead is to give up the notion that you need to distinguish between a Section in any of the different groups for any reason other than to match the content model. To me, this goes along with dropping the ambiguity rule. Then you cando this withoug look-ahead, as described below. I imagine that the bug forbidding comments in model groups will be fixed in the next update (assuming it's just a bug). Mail me privately on this rather thanbothering the list; suffice it here to note that a comment is not in fact a special case of whitespace, and is not invisible to the parser, the two more usual choices. Now, it is not clear to me that XML can or should address these issues. But if you are talking about removing any one feature, you must understand clearly (as I am sure you do, Eliot, but we are not all Eliots) how that affects the expressivity of the language. Posix regular expressions allow e\{n,m\} to represent between n and m occurrences (inclusive) of expression e. With that, you can eliminate "?", and if you have a notation for infinity, you can also eliminate * and +. But I would not claim that CHAPTER: (TITLE{0,1}, P{1, LOTS}) is sufficiently clearer than CHAPTER: (TITLE?, P*) to justify the simplification. On the other hand, you cannot simulate \{...\} in terms of + and * without getting into some very ugly situations. In the same way, getting rid of inclusions and exclusions will reduce the set of expressible content models, as will removing &. So the important question is whether the loss is worth the gain. >> * Should XML allow nondeterministic content models (11.2.4.3)? > > Again, how much does this complicate validation? I'm not ambiguity expert, > but could the problem be solved simply by stipulating that a token is > always matched to the first place in the content model it can match, > without lookahead? Yes, you could do that, but my content model above would then never work, because if you omitted a Requirements, you'd be stuck on the first Section*, and the Proposal would be an error. I think that my Report can be matched with no lookahead if you don't care about distinguishing sections in the various groups, but you'd have to use a somewhat more sophisticated approach, I'm afraid, than shortest match. An NDFA -> DFA conversion ala Aho et. al. would do it. Here's the model again, followed by the reduced DFA: <!Element Report - - ( Scope?, Section*, Requirements?, Section*, Proposal?, Section*, Summary? )> start: Section -> start Scope -> seenScope Requirements -> seenRequirements Proposal -> seenProposal Summary -> seenSummary seenScope: Section -> seenScope Requirements -> seenRequirements Proposal -> seenProposal Summary -> seenSummary seenRequirements: Section -> seenRequirements Proposal -> seenProposal Summary -> seenSummary seenProposal: Section -> seenProposal Summary -> seenSummary seenSummary: EOF -> accept * -> error So although this is ambiguous in SGML terms, there is no look-ahead needed. Question to brave sould who made it to here: are there any cases where look-ahead _would_ be needed? If so, do they matter? Is it a problem with common prefixes, as in (A B B)* A B C No, I can reduce that to (A B) (B->start | C). Hmmm. Would it help to change the rule for XML to be look-ahead of more than one token may not be practical. Therefore, the parser may forbid content models that require it. Parsers accepting such a model shall produce a warning; the parser may provide an option to disable such warnings, but the default must be to issue the warning. ? If ambiguous content models are allowed, does that affect + - and &, and if so how? If scoping rules were introduced, could + and - be eliminated? What is the expressive loss in elmiminating "&"? I hope this rather lengthy analysis helps. Lee Date: Mon, 23 Sep 1996 22:14:16 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org Subject: Discussion of entities To all who heeded my request to hold off discussing entities for a few days: Thank you. Well-known figures in the SGML field have compared this kind of job to herding cats, but for a bunch of notorious individualists you are proving to be a wonderfully cooperative group of people. I have told our co-editors, Michael Sperberg-McQueen and Tim Bray, that the question of what can be discussed in this regard is up to their judgement. They have decided that they are ready to open up the limited set of questions regarding entity structure and entity declarations listed in Michael's posting titled "questions about entities". This discussion is clearly necessary for the production of even a minimal XML specification. I only ask that you all continue to maintain a tight focus on the questions as stated and try not to wander off into the Wilderness of Linking, which belongs to a later phase of this effort. Jon Date: Tue, 24 Sep 96 00:16:17 CDT To: w3c-sgml-wg@w3.org From: Robert Streich <streich@slb.com> Subject: Re: questions about element declarations At 07:31 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >* Should XML retain SGML's prohibition on multiple declarations for the >same element (11.2.1)? Yes. It should be an error. What would be the advantage of keeping them? If you want to be able to redefine elements in the instance, this would add some flexibility, but I wonder if it isn't flexibility that would only lead to confusion. If you allow them, then you'd want to discard any redefinition of the element, otherwise you force the parser to go back and rebuild that whole structure. I'd also ask that it be made optional so that I could turn it off. I wouldn't want anyone to be able to redefine elements in the instance. >* Should XML restrict the use of #PCDATA to content models of the form >(#PCDATA), or of the form (#PCDATA | A | B | ...)*, as a way of avoiding >the mixed-content trap (11.2.4) and/or of simplifying RE handling >(7.6.1)? This could be a non-issue depending on what the decision is on how we're going to deal with RS/RE in general. >* Should XML prohibit the use of inclusion and exclusion exceptions in >element declarations? (11.2.4, 11.2.5)? Inclusions can go away. Exclusions can be useful and it would be nice if we could retain them. >* Should XML prohibit content-model references to undeclared elements >(11.2.4)? It's really convenient to decompose element declarations in the DTD, but I could live without it if it'll save a chunk of code for the parser writer. However, it would make a typical customization scheme difficult wherein some "local" elements are declared in the internal subset. If multiple element declarations are disallowed, it could make it impossible. We don't use this scheme, however, so I don't care. >* Should XML forbid use of the '&' connector in content models >(11.2.4.1)? This is another one that I could live without if it will make a significant savings to the parser writer. It'd force some rather gnarly looking content models, but I'm much less concerned about inconvenience in the DTD than I am in the instance. >* Should XML allow nondeterministic content models (11.2.4.3)? No. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Tue, 24 Sep 96 00:16:24 CDT To: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> From: Robert Streich <streich@slb.com> Subject: Re: questions about attribute declarations Cc: W3C SGML Working Group <w3c-sgml-wg@w3.org> Charles, didn't you once describe a bunch of impending changes to 8879 in attribute definitions? Could you elaborate on the pertinent ones here? I'd suggest that we stick within whatever changes Charles feels will be in the revision to 8879. I'm comfortable with "jumping ahead" here as Tim put it. If anything additional is needed, then let's try to stick with a meta-DTD approach wherein some of the attribute names are significant. I would love to have more than one ID on an element for example, but we could also add "XML-id" as an attribute name. An SGML editor would have to understand the semantics of the XML meta-DTD to really validate the doc, but at least it wouldn't preclude the use of a stock SGML editor. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: questions about element declarations Date: Tue, 24 Sep 1996 09:13:41 +0100 At 06:35 AM 24/9/96 +0100, W. Eliot Kimber wrote: >>* Should XML prohibit the use of inclusion and exclusion exceptions in >>element declarations? (11.2.4, 11.2.5)? > >Yes. Exclusions are, with very rare exceptions, a bad thing, causing much >more trouble than they're worth. Inclusions, while expressing a useful >semantic, also cause more trouble then they're worth and require DTD >processing. Exclusions are very useful for prohibiting nesting of objects in OR groups that are shared, such as those typically found in %text; parameter entites. For example: <!ENTITY % text (#PCDATA|a|b|c)> <!ELEMENT p (%text;)> <!ELEMENT a (%text;) -(a)> I cannot agree that they are "a bad thing"> ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: w3c-sgml-wg@w3.org From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: RS/RE: basic questions Date: Tue, 24 Sep 1996 09:13:53 +0100 One last plea for sanity on the RE/RS front before I flee the country. May I start by endorsing Lee's plea: <p>Some will authors need <em>simple</em> solutions.</p> I cannot see your average novelist adopting XML if he has to use a standard word processor type in: <P>He said "It stinks" as he picked up the ½" bar</p> or <p><c>He said </c><q><c>It stinks</c></q><c>as he....</c></p> So we are saying that he needs either: a) a tool to convert from WP to XML form b) an XML-knowledgeable editor c) an SGML editor that has an XML output tool. If the last option is used the editor will presumably use the rules in 7.6.1 and then convert the result to XML. The XML editor will need some rules to know how to process the pressing of a Return key immediately adjacent to a markup tag. What these rules will be other than those in 7.6.1 mystifies me somewhat. What really mystifies me, though, is the rules that will be applied to convert WP to XML. Can anyone suggest a simple set of rules for converting existing files into XML that does not require the complexity of the rules in 7.6.1? The alternative solution says that all RE/RS codes are whitespace. This presumes that there will be some rule in the style sheet that distinguishes a) when RE should be treated as hard line ends b) which elements should and should not have leading and/or trailing spaces. Providing mechanisms for both of these options are available in ALL browsers I have not problems with this approach, but if one browser leaves in leading spaces, and another does not, then we are in trouble. I typically use leading spaces after REs for all computer code so that it is indented from the text. I need a mechanism whereby elements with code in both retain my line breaks and retain the leading spaces. Without this I cannot use XML. Martin Bryan ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: lee@sq.com, kimber@passage.com, w3c-sgml-wg@w3.org From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: questions about element declarations Date: Tue, 24 Sep 1996 09:13:55 +0100 Lee wrote: >If scoping rules were introduced, could + and - be eliminated? I think the real question is "Would the scoping rules be any easier for users to understand than the rules for inclusions?". I doubt it somehow. >What is the expressive loss in elmiminating "&"? If you have an & group with more that 3 options the number of expansions you have to do makes the DTD unreadable/incomprehensible. If you use repeatable OR groups instead you cannot ensure that only one occurrence of each element occurs. Forcing people to enter things in a fixed sequence is the only real alternative, and this constrains users to thinking in the same sequence as the DTD writer, which can put them off using the DTD. I know & is hard on the parser writers, but without user-friendly logically controlled input there is no need for logically controlled output. ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: questions about attribute declarations Date: Tue, 24 Sep 1996 09:13:47 +0100 At 07:51 AM 24/9/96 +0100, W. Eliot Kimber wrote: Should XML change the set of types available for attributes? >>E.g. by suppressing NAME(S), NUMBER(S), NMTOKEN(S), NUTOKEN(S) and >>adding constraints in the form of regular expressions, ISO dates, >>language-code, external-id, typed IDREF, ... (7.9.4, 11.3.3) > >I'm not sure what the value of suppressing the listed types is. I don't see >that they add significant complexity and they are useful. I'd like to see NAME, NUMBER and NUTOKEN retained, but could live without NMTOKEN and the plural versions. (NAME is important as it is the only way to get over the problems of non-unique IDs!) >While Martin Bryan doesn't like it, I think it's reasonable to define a set >of conventional attribute names: ID, ENTITY, NOTATION, IDREF(S). Behaves >the way most people would expect them to behave and removes the need to >declare the attributes in these cases. Doesn't prevent declaring attributes >with other names but does prevent using these attribute names for something >other than their conventional interpretation. A renaming mechanism could >be used to map your own names to these conventional names, e.g.: > ><MyXMLEelement name=foo XMLNames="ID name"> > >When there's a DTD, the XMLNames (the "architecture renaming attribute") >could be fixed, but when there's no DTD, you'd have to specify it. (Which >raises the posibility that a careless author could do something different >on two different elements: > ><MyXMLEelement name=foo XMLNames="ID name"> ><MyXMLEelement uniqueID=foo XMLNames="ID uniqueID"> > >Ugh. Would probably have to make it a reportable XML error to map the same >conventional attribute name to different instance names in the same document.) As long as the names are remappable I have no problems. My objection is that id has a different meaning in some languages - is does not mean identifier. Similarly a German may prefer DING to ENTITY or ZIECHEN for NOTATION. Why force XML markup to be in English? (We are becoming xenophobic in our discussions!) ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: some topics we haven't yet discussed Date: Tue, 24 Sep 1996 09:13:44 +0100 At 07:51 AM 24/9/96 +0100, W. Eliot Kimber wrote: >>* Should XML change the delimiter-in-context rules to require the STAGO >>and ERO strings to be escaped whenever they are not to be recognized as >>delimiters (9.6)? > >This seems like a good idea. It is easy to use short references to define >escape sequences for these, e.g.: "~<", "~&", "~~" where "~" is the >"escape" character (but is really just the first character of a >two-character short reference delimiter). I think a shortref approach is >better than MSO, for the reasons given below. This doesn't break SGML, >only requires quoting where it might not otherwise be required (but where >people usually do it anyway). > >>* Should XML use MSOCHAR, MSSCHAR, and MSICHAR strings (9.7)? > >No--escaping is easier and safer using shortrefs. I can't see any difference between defining ~ as a MSSCHAR and defining it as an Escape character (given that we have no short references to decode the strings shown above, and will therefore rely on application conventions to define the escape mechanism). Incidentally, I prefer the tilde to any other escape character. >>* Should XML restrict comment declarations to a single comment (10.3)? > >Or should we pretend that comment open is "<!--" and comment close is "-->"? Yes please > >While SGML *should* have different delimiter roles for comment open and >comment close (and probably will after the revision), it doesn't today and >this is an area where anticipating the standard would make XML documents >incompatible with existing SGML processors. Thus, we *can't* do something >like make "/*" COMO and "*/" COMC, because those delimiter roles don't >exist in SGML. > >Cheers, > >E. >-- >W. Eliot Kimber (kimber@passage.com) >Senior SGML Consultant and HyTime Specialist >Passage Systems, Inc., (512)339-1400 >10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) >366-0320 (fax) >2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) >http://www.passage.com (work) http://www.drmacro.com (home) >"If I never had existed, would you still remember me?..." > --Austin Lounge Lizards, "1984 Blues" > > > ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Martin Bryan <mtbryan@sgml.u-net.com> Subject: Re: questions about entities and entity declarations Date: Tue, 24 Sep 1996 09:13:50 +0100 At 08:11 AM 24/9/96 +0100, W. Eliot Kimber wrote: >At 07:36 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >>* Should XML remove SGML's prohibition on ENTITY attributes for >>notations (11.4.1)? > >Do you mean that XML should *allow* ENTITY attributes for notations (also, >IDREF(S) and NOTATION)? I would say yes, because it only affects the DTD >and it's something that will almost certainly be in the revised SGML. 11.4.1 refers to data entities, which are only associated with Entity declarations. It does not make sense to allow an entity to reference another entity, either directly or via an ID. As the notation of the entity has to be specified after the NDATA keyword it does not make sense to have a possibly conflicting notation attribute. ---- Martin Bryan, The SGML Centre, Churchdown, Glos. GL3 2PU, UK Phone/Fax: +44 1452 714029 WWW home page: http://www.u-net.com/~sgml/ Date: Tue, 24 Sep 1996 07:51:53 -0500 From: Len Bullard <cbullard@HiWAAY.net> To: Robert Streich <streich@slb.com> CC: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Robert Streich wrote: > > I have a slightly different idea of what the "true" data is in a document > as compared to Charles. I think the "true" data is anything outside the > pointy brackets. 7.6.1 essentially tells typists that they don't > have to worry about the trailing newline after some types of markup. I > think it's easier to tell them that everything after the MDC is data Huh? Is that an overstatement? For example, prompt strings in attribute values while not my idea of a good practice are a common one. They are displayable text and are inside the brackets. len bullard Date: Tue, 24 Sep 1996 06:40:31 +0000 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Tim Bray <tbray@textuality.com> Subject: Syntax of XML markup declarations At 09:51 PM 23/09/96 -0900, W. Eliot Kimber wrote: >I see no reason to try to define a new notation for declarations at this >time. It is a *non trivial* exercise to define a document type or >architecture for declarations, partly because the additional expressive >power of SGML gives you a lot more options. Having spent quite a few hours >putting together a DTD for DTDs that attempts to provide some useful >features beyond simply reflecting the productions, I know that such an >effort will either be unsatisfactory or take a long time (or both): it will >either be a brain-dead transliteration of the productions (not useful, in >my opinion) or it will be a time-consuming document type >design-by-committee effort of the sort many of us have been involved with >(and about which Eve has written an excellent book). > >I say no. The declaration syntax is understood and easy to teach. It's not >hard to parse and it's compact. I disagree strongly. It seems to me a huge gain, in writing the spec, in reading the spec, and in implementing the system, to have one syntax for data and metadata to the extent possible. Eliot is right to say that writing the meta-DTD is hard (I did one); and that most of it is a straightforward mapping from DTD syntax into SGML; and that DTD syntax is more compact than SGML for this purpose. I disagree, however, that it would be unsatisfactory or brain-dead. As for the design-by-committee problem, we have that one anyhow, like it or not. Of course I am biased in favour of the metalanguage unification because I went through all the work of doing it, in MGML. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Tue, 24 Sep 1996 06:40:27 +0000 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: Tim Bray <tbray@textuality.com> Subject: Eliot's issues: & and determinism At 08:35 PM 23/09/96 -0900, W. Eliot Kimber wrote: BTW, I agree with all of EK's remarks, except as indicated >>* Should XML forbid use of the '&' connector in content models >>(11.2.4.1)? > >How much does AND complicate validation? I've seen some statements to the >effect that it complicates it quite a bit, but I have no way to evaluate >these claims. You *can* turn an &-constraint into a regular expression, but you get very fast (Steve said exponential, I'm not sure that's precisely true, but it's fast) increase in the size of the regular expression. Supporting this would have added significantly to the size of the MGML parser. I'd say lose it, it can be lived without. >>* Should XML allow nondeterministic content models (11.2.4.3)? > >Again, how much does this complicate validation? I'm not ambiguity expert, >but could the problem be solved simply by stipulating that a token is >always matched to the first place in the content model it can match, >without lookahead? Yes, it substantially complicates the parser. It turns out, as Peter Sharpe explained to me, you can do a clever trick while turning nDFA's into DFA's to spot these; but it's not obvious to our target XML programmer I think. Still, changing this would be a pretty egregious conflict with 8879. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "W. Eliot Kimber" <kimber@passage.com> Cc: W3C SGML Working Group <w3c-sgml-wg@w3.org> Subject: Re: some topics we haven't yet discussed Date: Tue, 24 Sep 1996 14:08:37 GMT Michael Sperberg-McQueen wrote: >(Using the same syntax for declarations and instances cuts the size of >the grammar approximately in half. It also reflects a firm belief that >structured information belongs in SGML documents.) I believe in the second sentence, but the first one sounds like a red herring to me, because: 1. The implementation code isn't cut in half, or anything like it. You still have to process the semantics of the declarations. 2. The grammar may be reduced, but now you have to define and document a DTD. 3. Most users, and many experts, have trouble distinguishing tags, from elements, from element types. If we use elements to define elements, the confusion will only get worse. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: bosak@atlantic-83.eng.sun.com (Jon Bosak) Cc: w3c-sgml-wg@w3.org Subject: Re: Discussion of entities Date: Tue, 24 Sep 1996 14:27:17 GMT On Mon, 23 Sep 1996 22:14:16 -0700, bosak@atlantic-83.Eng.Sun.COM (Jon Bosak) wrote: >To all who heeded my request to hold off discussing entities for a few >days: Thank you. Well-known figures in the SGML field have compared >this kind of job to herding cats, but for a bunch of notorious >individualists you are proving to be a wonderfully cooperative group >of people. Here is a complete proposal for entities. I think it has the virtue of identifying all of the options we have, so we can decide whether we like or dislike each one. It also has the virtue of correcting the many errors in previous postings on entities. ................................ Default entities are not permitted. Internal entities defined with "data text" or "bracketed text" are not permitted. Other internal entities are the same as for full SGML. There are three issues to resolve in external entity management for XML: 1. Types of entity text XML(SGML) text entities are permitted. They are accessed and parsed *in context* by the XML parser. CDATA, SDATA, and NDATA data entities are permitted. Any data content notation can be used. The entities are accessed and processed at the discretion of the application. XML(SGML) SUBDOC entities are permitted. They are accessed and parsed *in their own context* at the discretion of the application. XML(SGML) document entities are CDATA entities declared with the XML(SGML) notation. They are accessed and parsed *in their own context* at the discretion of the application. 2. Forms of reference XML(SGML) text entities are referenced by a parameter entity reference or a general entity reference. Data entities, XML(SGML) SUBDOC entities, and XML(SGML) document entities are referenced only by attribute values, not by general entity references. 3. Forms of address Entities can be declared with PUBLIC ids or Formal System IDs, but not with omitted system identifiers. PUBLIC ids are resolved using the SGML Open catalog with Formal System IDs on the "right side" of the catalog entries. The set of storage managers and attributes allowed in FSIs (in both declarations and catalog entries) is for further discussion. Some obvious candidates are URL and MIME. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Gavin Nicol <gtn@ebt.com> Date: Tue, 24 Sep 1996 14:25:18 GMT To: U35395@UICVM.CC.UIC.EDU CC: w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations >* should XML require all entities to be synchronous with the document's >logical structure? Yes. >* if XML makes DTDs optional and allows partial DTDs, what must or may a >parser do when it encounters references to undeclared entities (9.4)? >Should XML declare any set of entities automatically? Step 1. Look for the entities in a catalog by name: &bar; --> ENTITY "bar" "http://www.ebt.com/bar.sgml" Step 2. Look in a position relative to the document instance for a file of the same name with an .sgml or .sgm or no extension >* if XML uses ISO 10646, should there be a special form of character >reference using hexadecimal, not decimal, numbers, since most references >to ISO 10646 and Unicode use hex, not decimal (9.5)? Use Rick's optimised form: &UXXXXXXXX; Date: Tue, 24 Sep 1996 08:35:25 -0900 To: Martin Bryan <mtbryan@sgml.u-net.com>, w3c-sgml-wg@w3.org From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: RS/RE: basic questions At 09:13 AM 9/24/96 +0100, Martin Bryan wrote: >I cannot see your average novelist adopting XML if he has to use a standard >word processor type in: > ><P>He said "It stinks" as he picked up the ½" bar</p> > >or > ><p><c>He said </c><q><c>It stinks</c></q><c>as he....</c></p> What about simply quoting quotes (where unquoted quotes are PCDATA delimiters): <p>"He said ~"It stinks~" as he...."</p> Legions of TeX typists do more or less the same every day... ['~"' is shortrefed to '"' as a CDATA entity for SGML processing.] Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 1996 08:35:27 -0900 To: w3c-sgml-wg@w3.org From: "W. Eliot Kimber" <kimber@passage.com> Subject: RE/RS Options: Trying to Focus I'm trying to make some sense of the RE/RS issue and not having much luck. What follows is an attempt to state as tersely and precisely as possible what the issue are and what the alternatives are. As James put it so well, the problem is mixed content: if you allow mixed content, then you must provide some mechanism for distinquishing data record ends from non-data record ends. If you don't allow mixed content, then there is no problem, but then you have the problem of explicitly delimiting character data content, which currently neither SGML nor HTML require. Note too that it's not just an RE/RS problem, but an SSEP problem. The proposals, as I understand them, are: A. Disallow mixed content. Furthermore, disallow PIs and markup declarations in character data (with the possible exception of CDATA and RCDATA marked sections). This requires an element type whose only semantic is to contain character data. Short references can be used to enable using a single character to quote character data. This solves the problem by making it clear when record ends are in character data context. Record end handling rules are not changed in any way. Assumes that inclusions are not allowed (at least within the character data containing element), thus avoiding the "record ends following included subelements are not taken as data" rule. B. Treat XML documents as a single record by mapping RS and RE to character codes that cannot occur in documents. There are *no* record ends. This has the disadvantage that some other mechanism must be used to indicate data record ends, one that must be understood and processed by rendering systems, thus raising the likely possibility that different tools will provide different results for the same input data. It also has the problem that many SGML tools do not support this kind of remapping, making it difficult or impossible to process XML documents as SGML. It would also require transformation of SGML documents before they could be processed accurately as XML documents. This also doesn't solve the SSEP problem generally. C. Treat all record ends as data. This requires that authors must do things like put record ends before tag closes in order to format their markup on multiple lines. It also means that SGML documents can't be made into XML documents simply by quoting character data but must move all SSEP inside of markup. Talk about making 5-line Perl hacking harder. Note that if we want to allow DTD-less parsing, we can't use the SGML rules as-is and keep mixed content because without the DTD you have no way to knowing when you're in element content and when you're in mixed content (for a dramatic example of this problem, create an SGML document with lots of SSEP in element context then view it with Panorama with and without the DTD). This also means that there's no way to define a "simpler" set of RE/RS rules and keep mixed content, because you'll have the same problem. My conclusion is that eliminating mixed content by quoting character data is the simplest solution overall and retains the most compatibility with SGML as is. While quoting may seem unnatural to those of us who grew up typing SGML markup (it was to me when I put together some examples), I don't think it will be hard for newcomers to learn and it should be easy for SGML editors to add the quotes as an export option. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 1996 08:35:20 -0900 To: Martin Bryan <mtbryan@sgml.u-net.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: questions about entities and entity declarations At 09:13 AM 9/24/96 +0100, Martin Bryan wrote: >At 08:11 AM 24/9/96 +0100, W. Eliot Kimber wrote: >>At 07:36 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: > >>>* Should XML remove SGML's prohibition on ENTITY attributes for >>>notations (11.4.1)? >> >>Do you mean that XML should *allow* ENTITY attributes for notations (also, >>IDREF(S) and NOTATION)? I would say yes, because it only affects the DTD >>and it's something that will almost certainly be in the revised SGML. > >11.4.1 refers to data entities, which are only associated with Entity >declarations. It does not make sense to allow an entity to reference another >entity, either directly or via an ID. As the notation of the entity has to >be specified after the NDATA keyword it does not make sense to have a >possibly conflicting notation attribute. There are several cases where it *does* make sense. For example, with formal system identifiers, I can indicate that one entity is contained by another entity (container storage manager). Thus, the contained entity needs to refer to the container entity, which it can do most naturally with an ENTITY attribute. Who are we to say when it is or isn't meaningful to relate one entity to another using data attributes. Another case would be relating a graphic to alternate forms of itself by refering directly to those entities with data attributes. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 1996 08:35:23 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: some topics we haven't yet discussed At 09:13 AM 9/24/96 +0100, Martin Bryan wrote: >At 07:51 AM 24/9/96 +0100, W. Eliot Kimber wrote: > >>>* Should XML change the delimiter-in-context rules to require the STAGO >>>and ERO strings to be escaped whenever they are not to be recognized as >>>delimiters (9.6)? >> >>This seems like a good idea. It is easy to use short references to define >>escape sequences for these, e.g.: "~<", "~&", "~~" where "~" is the >>"escape" character (but is really just the first character of a >>two-character short reference delimiter). I think a shortref approach is >>better than MSO, for the reasons given below. This doesn't break SGML, >>only requires quoting where it might not otherwise be required (but where >>people usually do it anyway). >> >>>* Should XML use MSOCHAR, MSSCHAR, and MSICHAR strings (9.7)? >> >>No--escaping is easier and safer using shortrefs. > >I can't see any difference between defining ~ as a MSSCHAR and defining it >as an Escape character (given that we have no short references to decode the >strings shown above, and will therefore rely on application conventions to >define the escape mechanism). Incidentally, I prefer the tilde to any other >escape character. The difference is that MSO and MSI remain as data characters, whereas the shortrefs simply become the characters, which can be declared as CDATA entities: <!ENTITY srLT CDATA "<" > <!ENTITY srAMP CDATA "&" > <!ENTITY srESC CDATA "%" > <!ENTITY srLSBRK CDATA "[" > <!ENTITY srRSBRK CDATA "]" > <!ENTITY pseudoS "<pseudo>" > <!ENTITY pseudoE "</pseudo>" > <!SHORTREF XMLmap "~<" srLT "~&" srAMP "~~" srESC "~[" srLSBRK "~]" srRSBRK > Note that this markup is used to process XML as SGML--it isn't need by XML itself, which simply defines the escape sequences as escape sequences. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 1996 08:35:17 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: questions about element declarations At 09:13 AM 9/24/96 +0100, Martin Bryan wrote: >At 06:35 AM 24/9/96 +0100, W. Eliot Kimber wrote: > >>>* Should XML prohibit the use of inclusion and exclusion exceptions in >>>element declarations? (11.2.4, 11.2.5)? >> >>Yes. Exclusions are, with very rare exceptions, a bad thing, causing much >>more trouble than they're worth. Inclusions, while expressing a useful >>semantic, also cause more trouble then they're worth and require DTD >>processing. > >Exclusions are very useful for prohibiting nesting of objects in OR groups >that are shared, such as those typically found in %text; parameter entites. >For example: > ><!ENTITY % text (#PCDATA|a|b|c)> > ><!ELEMENT p (%text;)> ><!ELEMENT a (%text;) -(a)> I consider this use of exclusions to be *VERY BAD* because it is just a way to *avoid typing* and does not express a true semantic. If there is a semantic distinction between having A and not having A it should be reflected in the content models directly. You can do this by further subdividing the parameter entities or by simply making the content model explicit, e.g.: <!ENTITY % textpart1 "a" > <!ENTITY % textpart2 "b|c|#PCDATA" > <!ENTITY % text "%textpart1 | %textpart2" > <!ELEMENT P (%text;)* > <!ELEMENT A (%textpart2;)* > Exclusions have too many unwanted (and unobvious) side effects and make DTD-less parsing difficult or impossible. For example, what if the content model of B or C includes A? By using exclusion you have prevented the use of any B or C instances that happen to include A when they are in an A context, which may or may not be your intent (but it seems like a very rare case and, I would argue, if it is the case, then there is a semantic distinction between the A allowed in B and C and the A that is a peer to B and C in P, which, if made, also solves the problem without the need for exclusions). In all of my experience, I have only come upon *one* case where exclusions solved a real problem, and I was able to avoid the use of exclusions in that case. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 1996 09:25:08 -0600 From: "William D. Lindsey" <blindsey@bdmtech.com> To: lee@sq.com Cc: w3c-sgml-wg@w3.org Subject: Re: RS/RE quoted data proposal lee@sq.com writes: > > > > <P>"abcdefgh"<EM>"ijklmnopq"</EM></P> > > The latter syntax is certainly not typist friendly. > What happens if you omit a quote? > Since it is SGML, you can check it with a validating parser. In this example, nsgmls will report an error immediately after any one of the quotes you omit. -Bill -- William D. Lindsey blindsey@bdmtech.com +1 (303) 672-8954 Date: Tue, 24 Sep 1996 11:29:13 -0400 To: Charles@sgmlsource.com From: Murray Maloney <murray@sq.com> Subject: Re: some topics we haven't yet discussed Cc: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> At 02:08 PM 24-09-96 GMT, Charles F. Goldfarb wrote: >Michael Sperberg-McQueen wrote: > >>(Using the same syntax for declarations and instances cuts the size of >>the grammar approximately in half. It also reflects a firm belief that >>structured information belongs in SGML documents.) > >I believe in the second sentence, but the first one sounds like a red herring to >me, because: > >1. The implementation code isn't cut in half, or anything like it. You still >have to process the semantics of the declarations. Ya, but Michael was not commenting on the implementations, only the grammar. > >2. The grammar may be reduced, but now you have to define and document a DTD. Yes, exactly. That is what I would like to be able to do. SoftQuad developed an internal tool -- DTDocumenter -- to read a DTD and transform into a SGML instance with its own DTD just so that we could document a DTD. If the DTD were in standard SGML instance syntax, then we could use Panorama to display the DTD with whatever stylesheet we wanted to document it. A DTD for DTDs could also include a formal way to include a human readable description of the element and attribute declarations. That would be a good thing! > >3. Most users, and many experts, have trouble distinguishing tags, from >elements, from element types. If we use elements to define elements, the >confusion will only get worse. The fact that so many users have trouble distinguishing these things should be telling us that these distinctions are not very important to them. Honestly, I don't see why they should be important to users. Users want something that they can understand and SGML is not something that can be easily understood. Before I get flamed for saying that, please allow me to elaborate on my position. I am not an expert on SGML by any stretch of the imagination. A brief conversation with any of the real experts in the crowd who know me will confirm that. I have been lurking on this list until now because I did not want to slow you all down with ill-informed questions. But... I want to use something that I can understand. To some extent, SGML is that. That is, to the extent that I need to understand it, SGML is understandable. My failure to understand starts when I can't do something that I think should be obvious or when I get a result that I don't expect. (RS/RE rules are a perfect example of that.) Now, back to DTDs. I want someting that I can work with in one tool. I want to be able to read and write a DTD. I want to be able to declare an element type and its attributes, and I want to be able to explain the whys and wherefores right there in the DTD in such a way that a downstream processor can produce reasonable documentation. I also want to be able to _add_ attributes to an element type. (That is, I want to be able to have multiple attribute declarations that affect the same element type. And no , I don't think that parameter entities are good enough.) Thanks for taking the time to consider what I have to say. Please carry on. This is the most valuable series of discussions that I have ever encountered in this community. > Murray Maloney murray@sq.com Technical Director http://www.softquad.com SoftQuad Inc. P.O. Box 2025 Phone: +1 416 544-9000 x2219 20 Eglinton Avenue West, 12th Floor Fax: +1 416 544-0300 Toronto, Ontario, Canada M4R 1K8 Date: Tue, 24 Sep 1996 10:09:22 -0900 To: W3C SGML Working Group <w3c-sgml-wg@w3.org> From: "W. Eliot Kimber" <kimber@passage.com> Subject: Re: some topics we haven't yet discussed At 11:29 AM 9/24/96 -0400, Murray Maloney wrote: [...] >Now, back to DTDs. I want someting that I can work with in >one tool. I want to be able to read and write a DTD. I want >to be able to declare an element type and its attributes, >and I want to be able to explain the whys and wherefores >right there in the DTD in such a way that a downstream >processor can produce reasonable documentation. I also >want to be able to _add_ attributes to an element type. >(That is, I want to be able to have multiple attribute >declarations that affect the same element type. And no , >I don't think that parameter entities are good enough.) I think we're all pretty much in aggreement that using SGML markup for representing a document type is probably a good thing. What Charles and I are saying is that XML is not the venue to do it because the design must be done with a degree of care that would make the effort take longer than we have to do the XML design. There are a lot of different ways such a thing could be done and a lot of subtle implications of doing it. It's not even a certainty that it is a good thing in the long run. I certainly apply the principle of representing non-SGML things with SGML (i.e., my DTD for VRML), but it's easy to it unilaterally and for your own purposes, as everyone whose already done their own DTD for DTDs has demonstrated. Doing something that will satisfy the requirements and constraints of a user base as large and varied as the SGML/XML/HTML community is a very different matter. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Tue, 24 Sep 96 10:51:11 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: Re: questions about element declarations To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> On Mon, 23 Sep 1996 22:37:30 -0400 Eliot Kimber said: >At 07:31 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >> ... >>* Should XML forbid use of the '&' connector in content models >>(11.2.4.1)? > >How much does AND complicate validation? I've seen some statements to >the effect that it complicates it quite a bit, but I have no way to >evaluate these claims. > >I think AND expresses a useful semantic, so I would say keep it >unless the validation cost is too high. These are the issues as I understand them; I hope those with better training than mine will expand on this topic and correct my errors. If AND is not allowed, then a validating parser can construct a finite-state automaton for checking the input, using a very simple method. If the content model is deterministic, the FSA will be deterministic, and the checking can run very fast using very simple code. (Full discussion of this process and the underlying math in some technical reports by Anne Brueggemann-Klein, then of Freiburg, in 1991.) If AND is allowed, there are essentially three implementation approaches: - translate the AND group into an equivalent regular expression - use a mechanism based on a finite-state automaton, but augmented to handle certain simple operations when certain arcs are traversed - use some other ad hoc technique Translating the AND group into a regular expression is fine when the AND group has only a few children. When there are five tokens in the group, however, the corresponding regular expression will have 120 SEQ groups, and 600 states; a ten-element AND group produces 3.6 million SEQ groups and 36 million states. I doubt many implementors are willing to bet that they'll never see a 10-element AND group. It's possible to represent an AND group as a starred OR-group, if you augment the FSA with simple actions, so that arcs leaving individual items in the AND group set a bit saying their item has been visited, and arcs leaving the AND group as a whole check to make sure that all elements in the AND group have been visited; it gets slightly hairy when you have star or plus operators on the AND group, but I think it's possible -- but it's complicated enough to be significantly slower than the simple FSA. There are, I am sure, other ad hoc techniques people use. It's possible, for example, to retain the parse tree for the content model and interpret it directly as a way of handling validation; this takes less analysis and setup while parsing the DTD, but looks to be much slower in actual use (at least, the way I prototyped it). Eliot is right that AND is a very concise way of specifying certain requirements, where it applies. Since removing AND does not actually change the expressive power of the DTD language, however, anyone who urgently needs it can get the required effect without it. It seems to me on the whole that removing AND makes it much simpler to write validation code; some other people seem to agree. On the whole, I lean toward removing it: big gain, very little loss in clarity or flexibility, no loss at all in expressive power. >>* Should XML allow nondeterministic content models (11.2.4.3)? > >Again, how much does this complicate validation? I'm not ambiguity >expert, but could the problem be solved simply by stipulating that a >token is always matched to the first place in the content model it >can match, without lookahead? The computer scientists should probably speak up on this one. One reason for removing the restriction is not that it complicates validation but that it needlessly restricts the languages that can be defined by content models. As discussed this summer on c.t.s., the determinism rules make it impossible to define a content model equivalent to ((a, b)*, a?). But this regular expression is certainly parseable without lookahead; as Arjun Ray pointed out on c.t.s., it's an LL(1) language. If the rules about determinism help make simpler implementations possible, let's discuss the tradeoffs between that simplification and the loss of expressive power they entail. But I confess that I don't see that the restriction really simplifies implementation of a validating parser that much -- or conversely that lifting the restriction really makes life harder for validating parsers. I could be wrong; it wouldn't be the first time. But I'd like to understand what advantages are seen in the restriction. One could also argue against lifting this restriction on the ground that the restriction isn't hard to live with, and it's better to deviate from 8879 as little as possible. That's a fair political argument, and one I'm amenable to. But evaluating it requires a dispassionate examination of the technical issues first. This particular change does not, in any case, have any effect whatever on our ability to generate, for any given XML instance, a valid SGML prolog that will cause an SGML parser to produce the same ESIS as an XML parser, which is, I think, the level of compatibility that matters most. -C. M. Sperberg-McQueen Date: Tue, 24 Sep 96 11:41:47 CDT From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> Subject: Re: questions about entities and entity declarations To: "W. Eliot Kimber" <kimber@passage.com>, W3C SGML Working Group <w3c-sgml-wg@w3.org> On Tue, 24 Sep 1996 00:28:04 -0400 Eliot Kimber said: >At 07:36 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: >>* should XML prescribe the use of an ENTITY-END character as the >>canonical method of handling entity boundaries, as a way of simplifying >>exposition and implementation (6.2.2)? > >I'm not sure what this question means. There is no ENTITY END >*character* in SGML--it is a signal from the entity manager to the >parser. Clause 6.2.2 says in part: "A system can represent an Ee in any manner that will allow it to be distinguished from SGML characters. NOTE -- For example, an Ee could be represented by the bit combination of a non-SGML character, if any have been assigned." Sorry for introducing confusion by using the term "an ENTITY-END character" -- I meant, of course, "the bit pattern of a non-SGML character" in the sense of the note. If, for example, control-Z is declared a NONSGML character, it is easy to describe the behavior of EE by saying the entity manager inserts a control-Z at the end of an entity; this helps ensure that the parser doesn't falsely recognize < as STAGO; the parser or someone downstream eventually strips all control-Zs; implementations can do whatever they like, as long as they behave *as if* this were what they did. I think describing EE behavior in terms of an EE character is likely to be significantly simpler than describing it in terms of a non-character signal, particularly to programmers weaned on C's treatment of newline and EOF. It may also simplify implementation. >>* should XML retain or relax SGML's prohibition on ENTITY attributes >>referring to SGML text entities (7.9.4.3)? > >Retain. SGML text entities have no meaningful existence except as >fragements of SGML document strings, therefore it cannot make sense to >refer to one from an entity attribute. This logic eludes me completely. The premise is false, since meaningful existence can be defined by an application in its own terms; an application doesn't need our permission to assign meaning to a text entity. And even if the premise were true, the conclusion doesn't follow. I might wish to point to an external entity which contains an alternative rendition text for the element, which has a fragment of an SGML document which can meaningfully be substituted for the content of the element. Where is the problem? >>* if XML uses ISO 10646, should there be a special form of character >>reference using hexadecimal, not decimal, numbers, since most references >>to ISO 10646 and Unicode use hex, not decimal (9.5)? > >Such references would make processing of XML documents with SGML tools >impossible without preprocessing. It would probably be useful for SGML to >allow hexidecimal numeric character references, though. > >>(So references to schwa could take a form like &u0259; or &x0259;, not >>*#601;, which is rather error-prone, given that nothing in the Unicode >>documentation gives decimal numbers for the character positions.) > >Don't you have a hex-to-decimal calculator? :-) Yes, I do; that's how I know its use is error-prone. Use of hex references requires no more preprocessing than we've already decided on, namely preparing an appropriate prolog, which would in this case contain at least <!ENTITY u0259 'ə'> Date: Tue, 24 Sep 96 12:32:48 CDT From: paul@arbortext.com (Paul Grosso) To: w3c-sgml-wg@w3.org Subject: Re: questions about element declarations > From: Michael Sperberg-McQueen <U35395@UICVM.CC.UIC.EDU> > > * Should XML prohibit the use of inclusion and exclusion exceptions > in element declarations? (11.2.4, 11.2.5)? While we can prohibit exceptions in XML declarations, when a document instance contains an element for which there is no declaration, it seems to me that the obvious "default declarations" for an XML tool to assume (Michael's Waterloo-like ideas) are an ANY sort of declaration for the element *and* the equivalent of an inclusion of this element on the document element's declaration. I'm not sure if this has a bearing on a decision of what we allow in XML, but I want to be sure. For example, if I'm given an XML document and partial declarations equivalent to the following: <!DOCTYPE doc [ <!ELEMENT doc (title, section*)> <!ELEMENT section (para*)> <!ELEMENT title (#PCDATA)> <!ELEMENT para (#PCDATA)> ]> <doc> <title>The Title Chapter Title
    abcMNOxyz.
    then my XML editor would probably intuit and use the following DTD: 1. Am I right that the logical equivalent of what I describe is what's expected by an XML editing tool? 2. Does this have any implications on whether we allow exceptions in XML, or is the use of inclusions in my example restricted to SGML systems and implementation issues? Date: Tue, 24 Sep 96 12:32:07 CDT From: paul@arbortext.com (Paul Grosso) To: w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations > From: Charles@sgmlsource.com (Charles F. Goldfarb) > > . . . > > 2. Forms of reference > > XML(SGML) text entities are referenced by a parameter entity reference or > a general entity reference. > > Data entities, XML(SGML) SUBDOC entities, and XML(SGML) document entities > are referenced only by attribute values, not by general entity references. > Charles, by this do you mean to say that entity references to SDATA entities (which are one kind of data entity) cannot be in #PCDATA and can only be the values of attributes with declared value of ENTITY? Doesn't this mean that all those various ISO character reference sets would be unusable as defined? Are you suggesting that instead all these entities should be redeclared as text entities with replacement text that is a character reference (I assume you're suggesting we continue to allow character references)? Date: Tue, 24 Sep 96 12:31:27 CDT From: paul@arbortext.com (Paul Grosso) To: w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations > From: Gavin Nicol > > > From: Michael Sperberg-McQueen > > > > * if XML makes DTDs optional and allows partial DTDs, what must or may > > a parser do when it encounters references to undeclared entities (9.4)? > > Should XML declare any set of entities automatically? > > Step 1. Look for the entities in a catalog by name: > > &bar; --> ENTITY "bar" "http://www.ebt.com/bar.sgml" > > Step 2. Look in a position relative to the document instance for > a file of the same name with an .sgml or .sgm or no extension Provided &bar; is an external entity (e.g., not an internal text entity), I agree in principle with Gavin. (If/when catalogs allow a BASE entry, the Step 2 search might be relative to something other than the doc instance.) > From: Charles@sgmlsource.com (Charles F. Goldfarb) > > Default entities are not permitted. > > . . . > > 3. Forms of address > > Entities can be declared with PUBLIC ids or Formal System IDs, but not > with omitted system identifiers. I've actually wanted many times to be able to do the equivalent of defining the Default entity to be the omitted system identifier, so that undeclared entities would end up being processed as per Gavin's suggestion. In fact, in SGML I *can* do that now (or, I can try), but there is only one default, so I have to have the same default for all undeclared entities, and if it is supposed to work for undeclared NDATA entities, it has to have a notation which is a problem unless all my undeclared entities are of the same notation. This is something I'd like to see addressed in the SGML rev. But for XML, I suggest we just define the behavior suggested by Gavin. Date: Tue, 24 Sep 96 12:57:41 CDT From: paul@arbortext.com (Paul Grosso) To: w3c-sgml-wg@w3.org Subject: Re: questions about element declarations > From: Michael Sperberg-McQueen > > >>* Should XML allow nondeterministic content models (11.2.4.3)? > > > >Again, how much does this complicate validation? I'm not ambiguity > >expert, but could the problem be solved simply by stipulating that a > >token is always matched to the first place in the content model it > >can match, without lookahead? > > . . . > > One could also argue against lifting this restriction on the ground that > the restriction isn't hard to live with, and it's better to deviate from > 8879 as little as possible. That's a fair political argument, and one > I'm amenable to. But evaluating it requires a dispassionate examination > of the technical issues first. This particular change does not, in any > case, have any effect whatever on our ability to generate, for any given > XML instance, a valid SGML prolog that will cause an SGML parser to > produce the same ESIS as an XML parser, which is, I think, the level of > compatibility that matters most. Michael, I hear you saying that, given an XML document without its declarations, there exists a set of SGML declarations that will allow my SGML parser to generate the same ESIS as an XML parser. However, if someone sends the XML declarations, I should no longer be free to develop an arbitrary DTD with which to edit the document, so if I am going to use an SGML editor to edit this XML document, I'll need to convert the XML declarations into an equivalent set of valid SGML declarations. Therefore, my question/concern is: if we allow in XML content models that are not allowed in SGML and I receive an XML instance *with its declarations*, is there a simple algorithm to convert the XML declarations into SGML declarations so that I can process the document with my SGML tools? Date: Tue, 24 Sep 1996 12:13:00 -0900 To: W3C SGML Working Group From: "W. Eliot Kimber" Subject: Re: questions about entities and entity declarations At 11:41 AM 9/24/96 CDT, Michael Sperberg-McQueen wrote: [..] >I think describing EE behavior in terms of an EE character is >likely to be significantly simpler than describing it in terms of >a non-character signal, particularly to programmers weaned on C's >treatment of newline and EOF. It may also simplify implementation. Makes sense to me, I think (I never thought about it before). >>>* should XML retain or relax SGML's prohibition on ENTITY attributes >>>referring to SGML text entities (7.9.4.3)? >> >>Retain. SGML text entities have no meaningful existence except as >>fragements of SGML document strings, therefore it cannot make sense to >>refer to one from an entity attribute. > >This logic eludes me completely. The premise is false, since meaningful >existence can be defined by an application in its own terms; an >application doesn't need our permission to assign meaning to a text >entity. And even if the premise were true, the conclusion doesn't >follow. I might wish to point to an external entity which contains >an alternative rendition text for the element, which has a fragment >of an SGML document which can meaningfully be substituted for the >content of the element. As far as *SGML* is concerned, text entities have no separate existence--they cannot be parsed or validated in isolation, only in the context in which they are referenced. If an application feels it can do something with a text entity, then simply declare it as a data entity and refer to that entity rather than the text entity. Data entities, because they are processed outside of an SGML-defined context, must therefore be self contained. Or, said another way, each data entity will result in a separate grove, while text entities do not in normal SGML processing. I suppose you could solve the problem by saying that text entities have the implicit data content notation "SGMLfrag" (SGML fragment), equivalent to these declarations: You could certainly generate the necessary declarations for SGML processing. [...] >Use of hex references requires no more preprocessing than we've already >decided on, namely preparing an appropriate prolog, which would in >this case contain at least > > I see that I misunderstood the proposal. I thought you meant allowing hexidecimal numeric character references, not defining an entity set where the names happen to be hexadecimal equivalents. I think the latter is a darn good idea. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" To: w3c-sgml-wg@w3.org Subject: Re: RS/RE: basic questions Date: Tue, 24 Sep 1996 12:23:10 -0700 From: Joe English Robert Streich wrote: > [...] > Even the whitespace in element content wouldn't cause the > application any problems. It's just going to ignore any extra whitespace > anyway unless the stylesheet tells it that it is significant. > > Even Joe's example [below] > isn't significant. The stylesheet will determine whether or not starts > a new line. I think you're correct that _most_ applications will be able to treat all syntactic whitespace the same -- browsers and indexers for instance -- but there may be some circumstances where the distinction between SEPCHARs in element and mixed content is still important. One of the potential advantages of XML over HTML is the ability to create hyperlinks to any part of any document with HyTime or HyTime-like locators. (I suppose this is also possible in theory for HTML, but since the vast majority of Web pages are not valid SGML it wouldn't be very useful unless all parsers had identical error-recovery strategies.) For this to work, it's essential that all XML parsers can construct equivalent groves. For instance, in: > > blah > blah > the tree location address of the second 'B' element can change depending on whether 'A' has mixed content or element content. Then again, this might not be an issue either. TEI-style 'XPTR's could be used instead, and they wouldn't break. --Joe English jenglish@crl.com Date: Tue, 24 Sep 1996 14:10:23 -0700 From: eduardo.gutentag@Eng (Eduardo Gutentag) To: w3c-sgml-wg@w3.org Subject: Re: RE/RS Options: Trying to Focus ----- Begin Included Message ----- >From kimber@passage.com Tue Sep 24 07:55:59 1996 I'm trying to make some sense of the RE/RS issue and not having much luck. [... deleted ...] My conclusion is that eliminating mixed content by quoting character data is the simplest solution overall and retains the most compatibility with SGML as is. While quoting may seem unnatural to those of us who grew up typing SGML markup (it was to me when I put together some examples), I don't think it will be hard for newcomers to learn and it should be easy for SGML editors to add the quotes as an export option. Cheers, E. ----- End Included Message ----- I must confess that I'm sending this with some trepidation, since, like Murray, "I am not an expert on SGML by any stretch of the imagination. A brief conversation with any of the real experts in the crowd who know me will confirm that." However, like others in the crowd, I'm quite uncomfortable with the proposal to "eliminate mixed content by quoting character data". It seems to me this is akin to punishing the data for what the RE is doing. Why not look at it from a different angle? If I understand correctly, it all boils down to how to deal with cases like:

    Listen to my heart beat. And beat and beat and beat.

    so the application sees "Listen to my heart beat And beat and beat and beat" instead of " Listen to my heart beat. And beat and beat and beat. " I would like to propose that instead of quoting the data to achieve the desired results, the REs that produce extra new lines or whitespace, if any, in the application's output should be escaped.

    \ Listen to my heart beat. \ And beat and beat and beat.\

    This would not only avoid a few cases of RSI (for those brave enough to use vi or emacs -- and believe me, whatever someone in this list does is not indicative of what the population at large would) but it is eminently teachable and explainable. I am sure that there are problems with this proposal (such as what is the appropriate escape character), but it's certainly simpler to understand - most writers should be familiar with the concept of escaping REs, most developers should be able to deal with it without a second thought. ******************************************************************************* Eduardo Gutentag Sun Microsystems Inc (SunSoft) AnswerBook Publishing/Online Information A sponsor of the Davenport Group e-mail: eduardo@eng.Sun.COM Phone: (415) 786-5498 fax: (415) 786-5727 2550 Garcia Ave, MPK17-102, Mountain View, CA, 94043-1100 ******************************************************************************* Date: Tue, 24 Sep 1996 16:27:29 -0500 From: Len Bullard To: post-comp-text-sgml@techno.com CC: w3c-sgml-wg@w3.org Subject: BANG! or how I gave up SGML and learned to love the browser I guess it isn't exactly relevant, but in case anyone wonders why I don't want to get too hung up on grandfathering HTML, here is a verbatim from the HTML Reference at the Microsoft site delimited by the asterisks: *********************** ! specifies that enclosed text is an author's comment. Any text between the lines will not print. You can include multiple lines of text between the start-tag and end-tag. Example ********************** Sigh... len bullard lockheed martin Date: Tue, 24 Sep 96 17:16:22 CDT From: Michael Sperberg-McQueen Subject: RE rules - summary of corrections To: W3C SGML Working Group After some offline discussions with Charles Goldfarb and James Clark, I have revised my attempt at a restatement of the RE rules of clause 7.6.1., and will re-post it, for the edification of the participants, in a moment. Mostly the changes make it clearer, I hope, but there was one outright error, namely the claim that everything in an SGML document is either markup or content -- since the start- and end-tags of subelements are both markup and part of the content of the parent element, the correct opposition is not markup and content, but markup and non-markup, or markup, data, and separators. In sum, the rules prescribed by 8879 are these. RE is insignificant (i.e. not passed to any downstream application, not part of the XML grove plan) when it occurs in any of the following patterns: start-tag nondata* RE RE nondata* end-tag RS nondata+ RE where non-data is defined this way: nondata ::= comment declaration | processing instruction | character reference | entity reference | entity-end | marked section declaration | included subelement | short reference | shortref use declaration | link set use declaration The element Q contains no REs in any of the following cases: Listen to my heart beat. This is the simple case: RE adjacent to a start-tag or end-tag. Listen to my heart beat Listen to my heart beat. This is the one case I can think of where the first RE is not actually adjacent to the start-tag. -C. M. Sperberg-McQueen Date: Tue, 24 Sep 96 18:27:21 CDT From: Michael Sperberg-McQueen Subject: revised restatement of the RE rules To: W3C SGML Working Group Here is my restatement of the RE rules, as revised after consultation with Charles Goldfarb, James Clark, sgmls, and nsgmls. It must be admitted that these four sources did not agree on all points as regards the proper treatment and interpretation of the extended examples at the end of the discussion. In view of our agreed need for explicit, well documented, and well understood rules for RE handling, the diversity of views seemed to me suggestive of a need for simplification, both in XML and in the revision of 8879. -C. M. Sperberg-McQueen ------- RS is significant only if it's markup -- since it can be markup only in a shortref, it's of no interest to XML. For our purposes, RS is always ignored, period. SGML documents consist of data interspersed with markup. If an RE occurs in an SGML document, it's either in markup or outside it; there's no place else for it to be. In markup, RE is easy; an RE can occur only: - as a separator (in declarations or tags), in which case it's ignored - within a PI, in which case it's passed through to the application - within a literal in an attribute value specification (in a tag or entity declaration or attlist declaration), in which case it's replaced by a SPACE before the attribute value is passed to the application - within a literal in an entity declaration, in which case its treatment is determined when the entity is expanded - as the refc delimiter on a reference, in which case it's eaten by the parser - within a comment, in which case it's part of the comment Outside of markup, RE can occur: - in element content, between subelements, in which case it's ignored because it's a separator, not data - in mixed content or (replaceable) character data; it is here that 8879 treats some REs as data and others as having been caused by markup and thus insignificant (the standard uses the term 'ignored' for insignificant REs). Clause 7.6.1 a says "the first RE in an element is ignored if no RS, data, or proper subelement preceded it." Phrased the other way around, and working from the list in clause 7.6 of all the things that can occur in mixed or character data content, this means that when the pattern starttag nondata* RE is encountered, the RE is insignificant, where 'nondata' is defined as nondata ::= comment declaration | shortref use declaration | link set use declaration | processing instruction | character reference | entity reference | marked section declaration | included subelement | short reference | entity-end Rule (b) in the same clause says, in effect, that the same applies at the end of an element: "The last RE in an element is ignored if no data or proper subelement follows it." So if the end of an element matches the pattern RE nondata* end-tag the RE is ignored. Rule (c) in the same clause says, that if a record (i.e. the space between an &#RS; and the next &#RE;) is not empty, but contains no data, then the RE is ignored. "An RE that does not immediately follow an RS or RE is ignored if no data or proper subelement intervened." I think this means that if any record containing nothing other than nondata, then its RE is ignored. I.e., in RS nondata+ RE the RE is ignored. The final paragraph of the clause adds another complication: "An RE is deemed to occur immediately prior to the first data or proper subelement that follows it (that is, after any intervening markup declaration, processing instruction, or included subelement)." This allows a parser to handle cases like

    data ... &#RE;

    without having to look ahead past the comment to see whether the comment is followed by an end-tag or by more data: the parser can wait until after the comment to decide what to do with the RE. This has the drawback, however, of making REs wander around migrating past comments and processing instructions in ways not all users are likely to find intuitive. Such migration will generally be invisible in processing >from SGML into some other format, unless the processing instructions are affected by the RE; it will generally be visible after SGML-to-SGML transformations. In summary: RE is ignored in data when the data matches any of the following patterns: starttag nondata* RE RS nondata+ RE RE nondata* end-tag ------- Examples: The element Q contains no REs in any of the following cases: Listen to my heart beat. This is the simple case: RE adjacent to a start-tag or end-tag. Many of the most persuasive examples of 8879's RE rules involve REs adjacent to the tags. Listen to my heart beat Listen to my heart beat. This is the one case I can think of where the first RE is not actually adjacent to the start-tag. RE migration is illustrated by this element: Listen. Listen to my heart beat. The RE after "Listen." is "deemed to occur" after the processing instruction, so the element above is identical in effect to this one: Listen. Listen to my heart beat. The RE originally situated after "Listen." has migrated five lines down, past four comments and a processing instruction. The application of rule (c) is illustrated by the following example: ]>

    Look! this element --has it any visible record boundaries? Listen. Listen hard. (Two-comment-decl version.) Listen. Listen hard. (One-comment version.) Listen. Listen! Listen hard. (Two-comment-decl version.) Listen. Listen! Listen hard. (One-comment version.)

    When parsed by nsgmls, this document produces the following output: (P (Q -Look! this element --has it any visible record boundaries? )Q (Q -Listen. \nListen hard. (Two-comment-decl version.) )Q (Q -Listen. \nListen hard. (One-comment version.) )Q (Q -Listen. \n Listen!\nListen hard. (Two-comment-decl version.) )Q (Q -Listen. Listen!\nListen hard. (One-comment version.) )Q )P C This illustrates the following salient points: - the REs preceded by "" is significant. - the RE after "" is not significant. - the RE after "Listen. " is significant, because the most recent RS in the data was followed by "Listen. " The RS at the start of the second line of the comment is not considered, because it is not data (it's part of the comment). From: "Henry S. Thompson" Date: Wed, 25 Sep 96 00:45:42 BST To: w3c-sgml-wg@w3.org Subject: Re: questions about attribute declarations A minor point, if we're serious about leaping ahead: How about exapnding the syntax of NUMBER a bit, so that at least negative numbers are OK, and arguably decimals as well. It's always seemed odd to me that these are not included, and it regularly gets in my way. ht From: lee@sq.com Date: Tue, 24 Sep 96 19:43:07 EDT To: kimber@passage.com, w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations > At 11:41 AM 9/24/96 CDT, Michael Sperberg-McQueen wrote: > I think describing EE behavior in terms of an EE character is > likely to be significantly simpler than describing it in terms of > a non-character signal, particularly to programmers weaned on C's > treatment of newline and EOF. It may also simplify implementation. Actually, for what it's worth, C does not have a end of file character. The standard library (part of C now with ANSI C) makes the various I/O functions return an out-of-band signal when tehre's no more data. read() returns 0 bytes, getchar() returns 0xFFFFFFFF or -1 (doesn't fit in a char, although ISO10646 might cause problems) and so on. This is exactly like End of Entity, as I understand the latter to be. Eliot wrote: > I see that I misunderstood the proposal. I thought you meant allowing > hexidecimal numeric character references, not defining an entity set where > the names happen to be hexadecimal equivalents. I think the latter is a > darn good idea. Of course, without a #DEFAULT entity, you'd need an awful lot of entity declarations. In practice in XML, you'd create them on the fly. But what would an SGML system do? I suppose an XML to SGML converter, if such existed, could write an entities.sgml file containing declarations for them all, or coudl convert them to decimal char refs. Lee From: lee@sq.com Date: Tue, 24 Sep 96 20:38:17 EDT To: U35395@UICVM.CC.UIC.EDU, w3c-sgml-wg@w3.org Subject: Re: revised restatement of the RE rules You know, if XML had the compiler-theory concept of phases, there would be no problem whatsoever with &#RE; because the lexical analyser would eat the comments entirely, so that the parser wouldn't ever see comments at all. This is trivial to implement and is how almost all modern languges do it. It is unfortunate that , for use within the document, comments would have been much simpler. E.g. consider a parser reading a token at a time... getToken(startMode) { -- handle comments: if input is "<--" -- start comment then eat up to "-->" endif if (inTextMode) { -- handle text if input is MDO then do stuff endif } else if inMarkupMode { -- handle stuff in markup mode } else { . . .} } with this model, which I hope is clear from the incomplete code sketch (the mode names are not meant to correspond to SGML modes in the example), the parser would call getToken() repeatedly to read a file... but comments are never returned, and hence not seen by the parser. Obviously an editor can't use this model directly -- just as editors for the C programming language have to retain comments today (e.g. Visual C, Turbo C++, Brief, etc). I suggest that it be made clear that in XML, all comments are elided entirely before other parsing begins. This does not mean that you have to have multiple passes; it can be implemented concurrently very efficiently using the mechanism I've outlined above. Now,

    A B

    is clearly exactly the same as

    A B

    which I propose be considered the same as

    A B

    by considering any non-zero amount of whitespace to be exactly the same as a single space -- again, as in just about all other modern languages. Whitespace is any sequence of space, tab, form feed, newline, carriage return, and vertical tab (does anyone still use that??). Note that this makes

    A

    distinct from

    A

    but that is a necessary consequence of not requiring the use of a DTD. It would be possible to use a syntactic distinction between what I shall call formes and sticks. By a forme I mean an element that cannot contain text directly, rather like element context in SGML. By a stick, I mean an element that can contain text directly. For example only, suppose that the names of sticks are to be in lower case (in SGML, you'd have to change NAMECASE (yes?) in the SGML declaration to do this).

    An important distinction

    becomes

    An important distinction

    I am not proposing this mechanism for XML. I am describing it so as to make clear that if you don't read the DTD, you have to be able to deduce all the context you need by inspection. If you need varying behaviour >from your tokeniser or parser in different contexts, those contexts must be visible syntactically. But it is simplest to do away with such things altogether. Have a style sheet property telling you to discard leading/trailing spaces in an element if you must, and make the rules the same anywhere. Note that, just as in TeX, you would not be able to include unquoted program source code with this proposal. main() { print("hello world\n"); } might become main()&nl; {&nl; &tab;print("hello world\n");&nl; }&nl; But this is not so bad. How many troff-set books have I seen where this program is printed as main() { print("hello world0); } because the \n wasn't escaped? Lee From: lee@sq.com Date: Tue, 24 Sep 96 20:56:03 EDT To: ht@cogsci.ed.ac.uk, w3c-sgml-wg@w3.org Subject: Re: questions about attribute declarations > A minor point, if we're serious about leaping ahead: How about > exapnding the syntax of NUMBER a bit, so that at least negative > numbers are OK, and arguably decimals as well. It's always seemed odd > to me that these are not included, and it regularly gets in my way. Is this in scope? If so... * you would need to handle decimals like 30 000 000,512 or 30.000.000,512 or 30 000,000·512 (that's a raised dot) * is .3 allowed? How about -.2e-.1 (in FORTRAN-style EXP notation)? I'd rather see NUMBER left as it is in SGML, dropped from XML, and replaced with something like a simplified HyLex: (the 8-bit character is a pound-sterling, perhaps I should've used an entity!) This doesn't cover multilingual documents, but at least it doesn't wire a single number representation into a standard either. This could eaily be extended to content: where /.../ is like PCDATA except matching the given regexp. For conversion to SGML, you'd replace /.../ with PCDATA for an element and with CDATA for an attribute. That's a down-conversion, of course, although you could also add HyLex expressions I suppose.... Lee To: W3C SGML Working Group Subject: Re: revised restatement of the RE rules Date: Tue, 24 Sep 1996 17:53:02 -0700 From: Joe English Michael Sperberg-McQueen wrote: > Here is my restatement of the RE rules Thank you for providing that excellent summary. Please consider posting it to comp.text.sgml too; it deserves a wider audience. And... now that you've put it that way... the RS/RE rules don't look all that hard to implement after all... > Clause 7.6.1 a says "the first RE in an element is ignored if no RS, > data, or proper subelement preceded it." Right before that it says "An RE remaining *after replacement of all references and recognition of markup* is treated as data unless...", &c, so in the production: > nondata ::= comment declaration > | shortref use declaration > | link set use declaration > | processing instruction > | character reference > | entity reference > | marked section declaration > | included subelement > | short reference > | entity-end 'character reference', 'entity reference', and 'short reference' are never seen by this "phase" of the parser. (A quick test with SGMLS confirms that record-ends after references that expand into data are not discarded). (Earlier) > RS is significant only if it's markup -- since it can be markup only in > a shortref, it's of no interest to XML. For our purposes, RS is always > ignored, period. I believe that RS can also slip through when it appears inside a processing instruction (productions 44, 45, 47, 48, 50, and 51...). (I only know this because it crashed Cost once...) --Joe English jenglish@crl.com Date: Tue, 24 Sep 1996 18:57:18 -0700 (PDT) From: Bill Smith Subject: Re: revised restatement of the RE rules To: w3c-sgml-wg@w3.org Michael Sperberg-McQueen wrote: > Here is my restatement of the RE rules, as revised after consultation > with Charles Goldfarb, James Clark, sgmls, and nsgmls. It must be > admitted that these four sources did not agree on all points as regards > the proper treatment and interpretation of the extended examples at the > end of the discussion. In view of our agreed need for explicit, well > documented, and well understood rules for RE handling, the diversity of > views seemed to me suggestive of a need for simplification, both in XML > and in the revision of 8879. I might choose somewhat stronger language but I agree with the sentiment. Since we're having such difficulty with RS/RE, I propose declaring that they do not exist. If XML doesn't specify records, then there are no special rules to deal with their starting and ending points - end of problem. RS and RE are anachronisms and are not necessary to specify a markup language. What are the errors/problems with this approach? I recognize that 8879 compatibility may be lost but what else is wrong? I'm looking for substantive technical issues with David Durand's "one record proposal". Bill Date: Tue, 24 Sep 1996 23:59:23 +0000 To: paul@arbortext.com (Paul Grosso), w3c-sgml-wg@w3.org From: Tim Bray Subject: Re: questions about element declarations At 12:32 PM 24/09/96 CDT, Paul Grosso wrote: >For example, if I'm given an XML document and partial declarations >equivalent to the following: ... [I've moved Paul's example down because it's really not what I'm addressing] ... >1. Am I right that the logical equivalent of what I describe is what's > expected by an XML editing tool? I think it is outside the scope of this work to define what the expectations should be for an XML editing tool. I don't think that experience or wisdom exists in this group or the world at large to understand that question, let alone write it into a standard. My personal bias would be that if there is no DTD, all the tool could possibly do would be to test "well-formed-ness", i.e. no syntax errors nor mis-nested elements. If there is a DTD, the problem is well-understood. Trying to adduce grammars from instances is very far from easy and of questionable utility. -Cheers, Tim > > > > >]> > >The Title > >Chapter Title >
    >abcMNOxyz. >
    >
    >
    > >then my XML editor would probably intuit and use the following DTD: > > > > > > > > > Date: Tue, 24 Sep 1996 23:59:19 +0000 To: W3C SGML Working Group From: Tim Bray Subject: To EE or not to EE I cannot for the life of me see why the XML spec should mention the EE marker. The spec must require the parser to inform the application fully of the "entity stack", including the lengths of the entities *and* the replaced references - doing it via a magic character is perhaps a good way to implement this, but good enough to wire it into the standard? Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 Date: Tue, 24 Sep 1996 23:59:28 +0000 To: w3c-sgml-wg@w3.org From: Tim Bray Subject: RE/RS Options: One bottom line There is no desire here to grandfather existing HTML, but, given the fact that Microsoft IE already does stylesheets, and Netscape says they will too, if they just start applying style attributes when attached to non-HTML tags, then we are VERY VERY VERY close to being able to send generic SGML to 50 million desktops *as they sit* and get good results. Thus, the potential gains would be enormous if the following, given an appropriate DTD, were valid XML:

    The verbal leitmotif "Listen to my heart beat" is sprinkled throughout Laurie Anderson's mid-Eighties work.

    Whatever has to be done to achieve this and retain the maximal amount of RS/RE handling and 8879 compliance, we just have to do. Otherwise we're wasting our time. Cheers, Tim Bray tbray@textuality.com http://www.textuality.com/ +1-604-488-1167 From: Paul Prescod Subject: Re: revised restatement of the RE rules To: bill.smith@Eng Date: Wed, 25 Sep 1996 05:49:50 -0400 (EDT) Cc: w3c-sgml-wg@w3.org > > Since we're having such difficulty with RS/RE, I propose declaring that they do > not exist. If XML doesn't specify records, then there are no special rules to > deal with their starting and ending points - end of problem. RS and RE are > anachronisms and are not necessary to specify a markup language. > > What are the errors/problems with this approach? I recognize that 8879 > compatibility may be lost but what else is wrong? I'm looking for substantive > technical issues with David Durand's "one record proposal". Okay, Let's say I'm Jane WebMeister. Does the "one record proposal" mean that this:
    1234
    1234
    1234
    Is going to generate a parse tree with many newlines in it, which will probably choke a parser that _does_ use a DTD, because is element content? If so, I far prefer the quoted-delimeter proposal that removes the RS/RE without sacrificing the author's ability to reorganize their XML source using whitespace. Paul Prescod Date: Wed, 25 Sep 96 06:42:43 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: RS/RE: basic questions At 07:51 AM 9/24/96 -0500, Len Bullard wrote: >Robert Streich wrote: >> > >> I have a slightly different idea of what the "true" data is in a document >> as compared to Charles. I think the "true" data is anything outside the >> pointy brackets. 7.6.1 essentially tells typists that they don't >> have to worry about the trailing newline after some types of markup. I >> think it's easier to tell them that everything after the MDC is data > >Huh? Is that an overstatement? For example, prompt strings in >attribute values while not my idea of a good practice are a common >one. They are displayable text and are inside the brackets. I should have put that better, Len. I was only referring to the data content. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Wed, 25 Sep 96 07:43:29 CDT To: w3c-sgml-wg@w3.org From: Robert Streich Subject: Re: Syntax of XML markup declarations At 06:40 AM 9/24/96 +0000, Tim Bray wrote: >I disagree strongly. It seems to me a huge gain, in writing the spec, >in reading the spec, and in implementing the system, to have one syntax >for data and metadata to the extent possible. Eliot is right to say >that writing the meta-DTD is hard (I did one); and that most of it is >a straightforward mapping from DTD syntax into SGML; and that DTD >syntax is more compact than SGML for this purpose. I disagree, however, >that it would be unsatisfactory or brain-dead. As for the design-by-committee >problem, we have that one anyhow, like it or not. > >Of course I am biased in favour of the metalanguage unification because >I went through all the work of doing it, in MGML. I agree with both Michael and Tim on this issue, but it seems that the most difficult part will be to decide what to do with the declarations inside the doc's internal subset. For external DTDs, no problem, I'm only concerned about the instance. But since the internal subset is part of the instance, how can you make it a valid SGML document with declarations in another syntax? It would seem that you would have to make *all* declarations external, including those that are instance-specific. However, you're still stuck with referencing the external declarations in such a way that an SGML parser can parse the reference and know what it means. Unless you assume that preprocessing is always required to get an SGML document. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 From: Gavin Nicol Date: Wed, 25 Sep 1996 13:08:11 GMT To: bill.smith@Eng CC: w3c-sgml-wg@w3.org Subject: Re: revised restatement of the RE rules >What are the errors/problems with this approach? I recognize that 8879 >compatibility may be lost but what else is wrong? I'm looking for substantive >technical issues with David Durand's "one record proposal". I do not think compatability is lost at all. One can create a declaration today that would give the proposed behaviour. From: Gavin Nicol Date: Wed, 25 Sep 1996 13:06:28 GMT To: lee@sq.com CC: ht@cogsci.ed.ac.uk, w3c-sgml-wg@w3.org Subject: Re: questions about attribute declarations >I'd rather see NUMBER left as it is in SGML, dropped from XML, and >replaced with something like a simplified HyLex: > Salary /^[$£][0-9]+(\.[0-9][0-9])?/ #REQUIRED > > > >(the 8-bit character is a pound-sterling, perhaps I should've used an entity!) > >This doesn't cover multilingual documents, but at least it doesn't wire a >single number representation into a standard either. > >This could eaily be extended to content: > > (/^[A-Z][A-Z]+[0-9]?( [0-9]+[A-Z]+)?$/) > > > > (/[:upper:]/|smalls)* > > > > > (/[:lower:]/|CAPITALS)* > > > >where /.../ is like PCDATA except matching the given regexp. This is something I've also wanted for a long time. Having this would make it easy to validate not only thr structure, but also validate the lexical structure of the data: great for database applications. The typical response to this is that "this is an application issue". I should not that the "doesn't cover multilingual" can be removed easily by fixing the coded character set, and then defining a set of standard character classes based on that (like DIGIT, NAMESTART etc). Differences in numeric formats and whatnot would make it somewhat complicated in the general case, but in cases where there would be a mixture (for example, HTML) it would be better to require a normalised form, as I proposed to HTML-WG a long time ago. Note, this is still just a DTD issue and application convention. From: lee@sq.com Date: Wed, 25 Sep 96 09:38:35 EDT To: bill.smith@Eng, papresco@calum.csclub.uwaterloo.ca Subject: Re: revised restatement of the RE rules Cc: w3c-sgml-wg@w3.org > Okay, Let's say I'm Jane WebMeister. > Does the "one record proposal" mean that this: > >
    > [...] > Is going to generate a parse tree with many newlines in it, which will > probably choke a parser that _does_ use a DTD, because
    1234
    is element > content? I think this is an irrelevant point. A parser creates its own parse tree. A parser that puts newlines in the parse tree (presumably as text nodes) will have them, and this will in no way affect a parser that does not put them there. THe parse tree is an in-memory structure. In any case, ``Jane WebMeister'' is probably a human and has no need to look inside the software at the parse tree. Lee From: Gavin Nicol Date: Wed, 25 Sep 1996 14:39:52 GMT To: lee@sq.com CC: bill.smith@Eng, papresco@calum.csclub.uwaterloo.ca, w3c-sgml-wg@w3.org Subject: Re: revised restatement of the RE rules >> Is going to generate a parse tree with many newlines in it, which will >> probably choke a parser that _does_ use a DTD, because
    is element > content? > >I think this is an irrelevant point. I think that in this case, the point is that the ESIS will be different, though it would be trivial to convert the one with extra newlines to the one without. Date: Wed, 25 Sep 1996 12:19:56 -0400 From: Derek Denny-Brown To: w3c-sgml-wg@w3.org on Wed, 25 Sep 1996 14:39:52 GMT) Subject: Re: revised restatement of the RE rules > From: Gavin Nicol > I think that in this case, the point is that the ESIS will be > different, though it would be trivial to convert the one > with extra newlines to the one without. Is it important that the ESIS be the same? If the ESIS is different, then HyTime locations which are based on the ESIS representation will be different for a true SGML parser, versus a XML parser. I worry about this. Is there anything wrong with saying that _all_ REs are significant? It might make documents harder for humans to read, but my understanding what that that was not the point of XML. It would be very easy to understand. REs or something like them are necessary as end-of-line markers, though the syntactic form that an end-of-line takes should be as trivial as possible. Quoting content does help, but doesn't necessarily make it any easier to understand what the parser thinks (unless you have syntax highlighting, or something else to help the reader match quotes). Is there any reason, other than human-readability, why mandating that all REs are significant is bad? What is so bad about

    This is a paragraph

    being different from

    This is a paragraph

    As I see it, there are 3 proposals on the floor: 1) explicit REs, this included Charles' quoting 2) _never_ ignore RE's 3) no REs (subtly different from (2) where REs are treated as data content) I worry about (3), but see little real difference between 1 & 2 except the actual markup. -enjoy, derek --------------------------------------------------------------------------- Derek Denny-Brown | Technical Staff @ TechnoTeacher, Inc. http://www.techno.com/~derdb/ | work-phone: (716) 389-0963 SGML/HyTime/DSSSL/WWW | http://www.techno.com/ Date: Wed, 25 Sep 1996 11:07:26 -0500 From: Len Bullard To: Tim Bray CC: w3c-sgml-wg@w3.org Subject: Re: RE/RS Options: One bottom line Tim Bray wrote: > > There is no desire here to grandfather existing HTML, but, given the fact > that Microsoft IE already does stylesheets, and Netscape says they will too, > if they just start applying style attributes when attached to non-HTML > tags, then we are VERY VERY VERY close to being able to send generic SGML > to 50 million desktops *as they sit* and get good results. > > Thus, the potential gains would be enormous if the following, given an > appropriate DTD, were valid XML: > >

    The verbal leitmotif "Listen to my heart > beat" is sprinkled throughout > Laurie Anderson's mid-Eighties work.

    > > Whatever has to be done to achieve this and retain the maximal amount of > RS/RE handling and 8879 compliance, we just have to do. Otherwise we're > wasting our time. I agree and I see the sense of it. I just don't want to go down the path of trying to keep up with and adjust to all of the variants that DO produce gratuitous incompatibilities with extant SGML basic practices such as the odd comment structure I posted yesterday. I feel that the browser wars are becoming a rice grinder of beta hell. That is competition, and we all understand it I think, but I don't want XML to be the rice too soon. It appears from here that that has made life for the HTML WG and W3C members difficult, and the users as well who are now having to choose the platform and stick with it. As you know, in VRML where one language and one syntax exist, the scripting issue alone is forcing the same decision. So while, yes, it will also happen in XML, no doubt, as far as possible, the base language design should be pristine. Otherwise, I'm also in favor that HTML -> XML should be pretty painless especially for authors. Where they can make the transition almost invisibly, everyone wins. len bullard Date: Wed, 25 Sep 1996 09:54:19 -0700 (PDT) From: Bill Smith Subject: Re: revised restatement of the RE rules To: w3c-sgml-wg@w3.org It's been brought to my attention that the "one record proposal" has several independent "inventors" including David Durand, Rick Jelliffe, Gavin Nicol, and I'm sure others. My apologies for not giving proper attribution. From: "Christopher R. Maden" Date: Wed, 25 Sep 1996 20:57:23 GMT To: w3c-sgml-wg@w3.org Subject: More on RE/RS Several people have proposed defining the RE/RS problem out of existence by defining the RE and RS function charcters as codes that won't occur in entities, via the SGML application for XML. James Clark takes the stand that the definitions of RE and RS are what codes the parser should communicate to the application when they are encountered, so redefining them won't change whether or not they occur. He also posits that lines in an input file from DOS or UNIX should be interpreted as records. That certainly seems a reasonable interpretation, but I can't find anything to that effect in 8879. Clause 7.6.1, "Record Boundaries", defines the rules for RE ignorance or preservation, but doesn't say anything about when the parser generates an RS or RE signal. Charles Goldfarb's commentary thereto (pp. 321+322 of the SGML Handbook) discuss translating lines into records, but that's not normative. The best normative thing I can find is 4.140, "function character identification parameter: A parameter of an SGML declaration that identifies the characters assigned to the RE, RS, and SPACE functions, and allows additional functions to be defined." This suggests that, since characters are assigned to functions, that the characters in the document should assume the roles of these functions; ergo, if non-ocurring characters are the ones assigned to those roles, the function characters never occur. Is that not the intended meaning? If not, what is? I think that, if the RE/RS problem can be redefined out of existence, that it can be very easily handled at the application level. Some have suggested this already; I outlined a proposal in conversation with Gavin Nicol, and he seemed to think it worthwhile. I'll send that in another message if others agree that it is possible for an application of ISO 8879:1986 (not :2001) to define every entity to have a single record. -Chris -- http://www.ebt.com +1.401.421.9550 +1.401.521.2030 One Richmond Square, Providence, RI 02906 USA" NDATA SGML.Geek> Date: Wed, 25 Sep 1996 17:01:53 -0400 To: Derek Denny-Brown , w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: revised restatement of the RE rules At 12:19 PM 9/25/96 -0400, Derek Denny-Brown wrote: >What is so bad about > >

    This is a paragraph

    > >being different from > >

    This is a >paragraph

    Nothing, really, but I'm a little concerned about this:
    1234
    12< /TD>34
    Which is quite unreadable. If you put newlines in, then according to the "document is a single record" proposal an XML parser would interpret them as significant (since it can't know element content from mixed content without a DTD). What is a table-displaying application to do with all of the "data" between table cells and rows? It will reject them as invalid, one expects, and spit up the document (though it validates okay). Or should the application enforce an internal rule that "data is allowed between table cells and rows, but only if the data consists entirely of newlines." Ugh. That's what Charles meant about requiring every application to deal with the problem. >As I see it, there are 3 proposals on the floor: > 1) explicit REs, this included Charles' quoting > 2) _never_ ignore RE's > 3) no REs (subtly different from (2) where REs are treated as data content) > >I worry about (3), but see little real difference between 1 & 2 except >the actual markup. Sorry, I'm not subtle enough to understand the difference between 2 and 3. How can we define all REs as "significant" without treating them as "data content" or reintroducing SGML's RE complexity. Paul Prescod Date: Wed, 25 Sep 1996 17:01:38 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Newlines in element content (i.e TABLES) At 09:38 AM 9/25/96 EDT, lee@sq.com wrote: >I think this is an irrelevant point. > >A parser creates its own parse tree. A parser that puts newlines in the >parse tree (presumably as text nodes) will have them, and this will in no >way affect a parser that does not put them there. THe parse tree is an >in-memory structure. > >In any case, ``Jane WebMeister'' is probably a human and has no need to >look inside the software at the parse tree. Well, Jane WebMeister might be concerned if a) some XML applications "eat" the newlines-that-are-data between table sells, b) some dispay them and c) some report the confusing data-between-cells as an error. Let's be clear here: as I understand this proposal it is the absolute opposite of Charles'. Instead of removing mixed content, it must either make ALL CONTENT MIXED CONTENT or restrict all newlines in "element" content. After all, a parser without a DTD doesn't know what is mixed content and what is element content. As Charles pointed out, parsers will have a hard time telling the true content from the source-formatting (unless we absoutely restricted the source-formatting). Paul Prescod From: lee@sq.com Date: Wed, 25 Sep 96 18:14:52 EDT To: w3c-sgml-wg@w3.org Subject: Re: Newlines in element content (i.e TABLES) > Let's be clear here: as I understand this proposal it is the absolute > opposite of Charles'. Instead of removing mixed content, it must either make > ALL CONTENT MIXED CONTENT or restrict all newlines in "element" content. > After all, a parser without a DTD doesn't know what is mixed content and > what is element content That's correct. For what it's worth, SoftQuad Panorama can display SGML tables with newlines between the tags, even if PCDATA is allowed there. It isn't particularly hard to implement, as far as I can see. Try making P a Panorama table and B a table cell in an HTML file... :-) Note that a parser without a DTD also can't use SHORTREF to interpret quotes. Lee From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Michael Sperberg-McQueen Cc: "W. Eliot Kimber" , W3C SGML Working Group Subject: Re: questions about entities and entity declarations Date: Thu, 26 Sep 1996 00:30:40 GMT On Tue, 24 Sep 96 11:41:47 CDT, Michael Sperberg-McQueen wrote: >This logic eludes me completely. The premise is false, since meaningful >existence can be defined by an application in its own terms; an >application doesn't need our permission to assign meaning to a text >entity. And even if the premise were true, the conclusion doesn't >follow. I might wish to point to an external entity which contains >an alternative rendition text for the element, which has a fragment >of an SGML document which can meaningfully be substituted for the >content of the element. >Where is the problem? The problem is that you have subverted the chief purpose of SGML, which is to preserve the "true" information content. An application is forbidden to do anything that would affect the parsing of the document, otherwise there would be no portability because there would be no guarantee that all applications would see the identical parsed input. SGML text entities are part of the true information; they are parsed *in context*. No other entities are parsed in context, except for the SGML document entity in which everything begins (and which establishes the context in the first place). An application can, of course, treat an attribute value as a replacement for the *parsed data* that the parser found in the element's content, but that only affects the output of the application. Such an attribute should not be labeled an "SGML text entity" because that label identifies text that is parsed *in content*. The attribute should be a CDATA (with SGML notation) or SUBDOC entity, as those are parsed *out of context* at the request of the application. (See my postings on entities for more details.) -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: paul@arbortext.com (Paul Grosso) Cc: w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations Date: Thu, 26 Sep 1996 00:30:42 GMT On Tue, 24 Sep 96 12:32:07 CDT, paul@arbortext.com (Paul Grosso) wrote: >> From: Charles@sgmlsource.com (Charles F. Goldfarb) >> >> . . . >> >> 2. Forms of reference >> >> XML(SGML) text entities are referenced by a parameter entity reference or >> a general entity reference. >> >> Data entities, XML(SGML) SUBDOC entities, and XML(SGML) document entities >> are referenced only by attribute values, not by general entity references. >> > >Charles, by this do you mean to say that entity references to SDATA >entities (which are one kind of data entity) cannot be in #PCDATA and >can only be the values of attributes with declared value of ENTITY? Yes. Internal SDATA entities, in this starter proposal, don't exist at all. They are the kind that typically occur in mixed content. External SDATA entities, like all other external data entities, are parsed *out of context* and should only be invoked by an attribute value. > >Doesn't this mean that all those various ISO character reference sets >would be unusable as defined? Are you suggesting that instead all >these entities should be redeclared as text entities with replacement >text that is a character reference (I assume you're suggesting we >continue to allow character references)? > The ISO character entity *sets* are external entities, but the entity declarations within them declare *internal* SDATA entities. They would have to be redeclared as you say if we are to exclude internal SDATA entities. This proposal only dealt with entities. I haven't seen any sentiment to abolish character references. I would support keeping them. (Obviously, we have to if we are going to exclude internal SDATA entities.) -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: lee@sq.com Cc: w3c-sgml-wg@w3.org Subject: Re: Newlines in element content (i.e TABLES) Date: Thu, 26 Sep 1996 00:47:51 GMT On Wed, 25 Sep 96 18:14:52 EDT, lee@sq.com wrote: > >Note that a parser without a DTD also can't use SHORTREF to interpret quotes. An XML parser only has to know that the quotes (or brackets, or whatever) delimite data. Only an SGML<->XML converter needs to know that they are really short references. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Wed, 25 Sep 1996 20:43:14 -0400 To: lee@sq.com, w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: Newlines in element content (i.e TABLES) At 06:14 PM 9/25/96 EDT, lee@sq.com wrote: >That's correct. For what it's worth, SoftQuad Panorama can display SGML >tables with newlines between the tags, even if PCDATA is allowed there. >It isn't particularly hard to implement, as far as I can see. But should every SGML application have to implement it over and over again? That means that between and within ANY ELEMENT you would have to explicitly look out for "meaningless" newlines. Instead of implementing the handling in the parser (which code we expect to be used over and over again) you must implement it in the application. Then you have to define in your DTD-documentation that newlines in that context are going to be interpreted as "meaningless" which means that we are shifting the documentation and education burden to application designers. >Try making P a Panorama table and B a table cell in an HTML file... :-) > >Note that a parser without a DTD also can't use SHORTREF to interpret quotes. A DTD-less parser doesn't know or care that it is dealing with shortref. It would treat '"' as "PCDATA Start" and "PCDATA End". Paul Prescod From: lee@sq.com Date: Wed, 25 Sep 96 21:11:03 EDT To: Charles@sgmlsource.com, paul@arbortext.com Subject: Re: questions about entities and entity declarations Cc: w3c-sgml-wg@w3.org > The ISO character entity *sets* are external entities, but the entity > declarations within them declare *internal* SDATA entities. They would > have to be redeclared as you say if we are to exclude internal SDATA > entities. I am probably being obtuse here, but how would I refer to (for example) a symbol that has no Unicode or ISO10646 code point? For example, suppose I need an r-cedilla... in SGML I've used SDATA entities for that. I'd be quite happy with an alternative. For example, Pinnacles-style reflections combined with TEI Writing Sets would do more than supplant SDATA entities, I think, as well as shaming those HTML weenies into humble submission :-) In fact, the Reflection idea is generally very useful. Long Note For Those Not Familiar With Pinnacles: You use Pinnacles reflections as follows: (1) create a source element with arbitrary content, and give it an ID (2) have a reflection element that uses IDREF to point to the source ID. (3) the application makes the source appear in place of the corresponding reflection element wherever it appears. One could use an architectural form to specify this behaviour. I call them that because I first encountered them used in the PCIS DTD for the Semiconductor industry that Tommie and Debbie and the erstwil Atlis People wrote. They are like entities that are parsed where they are defined instead of where they are used, and are easier to support in some SGML editors than the document fragment sort of entities that might be used instead. I have never seen them used with TEI WSDs, but I think this is an interesting possibility. Lee From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Michael Sperberg-McQueen Cc: W3C SGML Working Group , jjc@jclark.com Subject: Re: revised restatement of the RE rules Date: Thu, 26 Sep 1996 02:00:09 GMT Everyone seems to be ignoring Michael's immediately preceding posting, which contains all that an XML user will need to know about RE handling, so I'll repeat it here: > RE is insignificant >(i.e. not passed to any downstream application, not part of the >XML grove plan) when it occurs in any of the following patterns: > start-tag nondata* RE > RE nondata* end-tag > RS nondata+ RE >where non-data is defined this way: > nondata ::= comment declaration > | processing instruction > | character reference > | entity reference > | entity-end > | marked section declaration > | included subelement > | short reference > | shortref use declaration > | link set use declaration For the record, both James Clark and myself agree with this statement, which is both concise and formal. Michael then went on to provide some examples: >The element Q contains no REs in any of the following cases: > > Listen to my heart beat. > >This is the simple case: RE adjacent to a start-tag or end-tag. > > > Listen to my heart beat > Listen to my heart beat. > >This is the one case I can think of where the first RE is not >actually adjacent to the start-tag. >-C. M. Sperberg-McQueen -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Wed, 25 Sep 96 22:17:30 CDT To: lee@sq.com From: Robert Streich Subject: Re: questions about entities and entity declarations Cc: Charles@sgmlsource.com, paul@arbortext.com, w3c-sgml-wg@w3.org At 09:11 PM 9/25/96 EDT, lee@sq.com wrote: >Long Note For Those Not Familiar With Pinnacles: > You use Pinnacles reflections as follows: > (1) create a source element with arbitrary content, and give it an ID > (2) have a reflection element that uses IDREF to point to the source ID. > (3) the application makes the source appear in place of the corresponding > reflection element wherever it appears. > One could use an architectural form to specify this behaviour. > > I call them that because I first encountered them used in the PCIS > DTD for the Semiconductor industry that Tommie and Debbie and the > erstwil Atlis People wrote. > > They are like entities that are parsed where they are defined instead > of where they are used, and are easier to support in some SGML editors > than the document fragment sort of entities that might be used instead. > > I have never seen them used with TEI WSDs, but I think this is an > interesting possibility. This is done via the conloc attribute in the all-id list in HyTime. See 6.5.1 of the HyTime standard. bob Robert Streich streich@slb.com Schlumberger voice: 1 512 331 3318 Austin Research fax: 1 512 331 3760 Date: Thu, 26 Sep 1996 08:42:15 +0000 To: lee@sq.com From: James Clark Subject: Re: questions about entities and entity declarations Cc: w3c-sgml-wg@w3.org At 21:11 25/09/96 EDT, lee@sq.com wrote: >> The ISO character entity *sets* are external entities, but the entity >> declarations within them declare *internal* SDATA entities. They would >> have to be redeclared as you say if we are to exclude internal SDATA >> entities. > >I am probably being obtuse here, but how would I refer to (for example) >a symbol that has no Unicode or ISO10646 code point? For example, suppose >I need an r-cedilla... in SGML I've used SDATA entities for that. Use the 10646 private use zone. We'll have to find some way to specify the meanings of code points in the private use zone (maybe XML needs an SGML-declaration equivalent that can do this); but with SDATA entities you equally have to find a way to describe their meaning, so I don't think you lose much by junking them. James From: Charles@sgmlsource.com (Charles F. Goldfarb) To: lee@sq.com Cc: paul@arbortext.com, w3c-sgml-wg@w3.org Subject: Re: questions about entities and entity declarations Date: Thu, 26 Sep 1996 08:22:35 GMT On Wed, 25 Sep 96 21:11:03 EDT, lee@sq.com wrote: >> The ISO character entity *sets* are external entities, but the entity >> declarations within them declare *internal* SDATA entities. They would >> have to be redeclared as you say if we are to exclude internal SDATA >> entities. > >I am probably being obtuse here, but how would I refer to (for example) >a symbol that has no Unicode or ISO10646 code point? For example, suppose >I need an r-cedilla... in SGML I've used SDATA entities for that. > >I'd be quite happy with an alternative. > A symbol that is not in the character set would be like any other graphic; use an external NDATA entity. That gives you an opportunity to declare the notation explicitly (and data attributes for it), which SDATA doesn't provide. Remember that SGML has never required a separate file for external entities. With Formal System Identifiers, we can now formally specify when we are using container Storage Managers that let multiple entities occur in the same storage object (e.g., TAR, PKZIP, MIME). If you don't like using external NDATA for this purpose, we could agree to allow internal SDATA entities in XML. If so, we would probably want a convention, such as the one proposed for PIs, that would identify the notation for the SDATA entity. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Thu, 26 Sep 1996 08:37:30 -0600 From: "William D. Lindsey" To: "W. Eliot Kimber" Cc: W3C SGML Working Group Subject: Re: questions about attribute declarations W. Eliot Kimber writes: > At 07:33 PM 9/23/96 CDT, Michael Sperberg-McQueen wrote: > >Questions about attribute declarations I'd like to see discussed while > >the voting booth is still open: > > > >* Should XML retain SGML's prohibition on multiple attribute-list > >declarations for the same element (11.3.1) or on multiple declarations > >for the same attribute (11.3.2)? > > While I like having multiple attribute list declarations for the same > element type, I think that we should not do it for XML, only because it > removes one more thing that would have to be mangled to go from XML to SGML. One possible motivation for removing the restriction is to make it convenient for a document entity to include declarations for just those DEFAULT and FIXED attributes necessary for meaningful processing in the absence of (the rest of) the DTD. Am I wrong in thinking that the issue of how a parser is to obtain these values has not been properly explored? -Bill -- William D. Lindsey blindsey@bdmtech.com +1 (303) 672-8954 Date: Thu, 26 Sep 1996 11:41:45 -0900 To: w3c-sgml-wg@w3.org From: "W. Eliot Kimber" Subject: Current Status of Discussion on RE/RS Handling The SGML-ERB met yesterday by phone and discussed the RS/RE problem at some length. While we haven't necessarily arrived at concensus, our discussion did result in the following proposal that appears to satisfy the requirement of SGML compatibility while being simple and more or less consistent with existing HTML practice. The rules we came up with are: An XML parser shall interpret white space and record ends in XML documents as follows: 1. All white space, including RS and RE, immediately following start tags and immediately preceding end tags is not significant. 2. All other RS/REs are collapsed to a single space. This approach has the effect that the white space and RS/RE collapsing can be done before or after SGML RE rules are applied without affecting the result. The only place this is not true is record ends followed by one or more PIs followed by data. In SGML, the RE will be considered to have occurred *after* the PIs, whereas in XML it will be considered to have occurred *before* the PIs (there are many who consider this behavior of SGML to be a bug that should be fixed, or at least made optional, in the SGML revision). This approach also requires that truly significant record ends in data must be escaped in some way. Cheers, Eliot -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" Date: Thu, 26 Sep 1996 14:09:01 -0400 To: w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: Newlines in element content (i.e TABLES) Having finally finished reading all the back mail, I'm ready to try to jump back into the discussion. At 8:43 PM 9/25/96, Paul Prescod wrote: >At 06:14 PM 9/25/96 EDT, lee@sq.com wrote: >>That's correct. For what it's worth, SoftQuad Panorama can display SGML >>tables with newlines between the tags, even if PCDATA is allowed there. >>It isn't particularly hard to implement, as far as I can see. > >But should every SGML application have to implement it over and over again? >That means that between and within ANY ELEMENT you would have to explicitly >look out for "meaningless" newlines. Instead of implementing the handling in >the parser (which code we expect to be used over and over again) you must >implement it in the application. > >Then you have to define in your DTD-documentation that newlines in that >context are going to be interpreted as "meaningless" which means that we are >shifting the documentation and education burden to application designers. The example under consideration was the table:
    1234
    12< /TD>34
    and the desire to format it, thus:
    1234
    1234
    Since in the DTD-less case we don't know that the table is element content, we are unable to remove the whitespace. Now, we could fix this by having DTD-less processing differ from DTD-ful processing, but I agree with most of you (I expect) that this is a bad idea. But, why not format the table like this:
    1234
    12< /TD>34
    This looks a little weird, but you already have to do this in netscape (at least for TD elements) because in tables leading whitespace is not ignored. And if we have a further "application" convention that whitespace is ignored according to common convention, except when a stylesheet requests verbatim processing for an element. I'd rather add a "verbatim" declaration to the DTD, come down to it. Incompatibility with 8879 is actually not an issue here, anyway, as the entity manager is _never_ required to report RS/RE to a parser. If current entity managers insist on recognizing CR and LF as record boundaries, then we should live with the incompatibility, and encourage the development of simpler entity managers that are not so "obliging". The RS/RE stuff in SGML was supposed to make life easier for taggers. Experience has shown that, arguments about "true content" or not, they do not work, as even SGML experts can disagree about the meaning of the rules. We are in danger of making compatibility with SGML's mis-features a millstone around our necks. >A DTD-less parser doesn't know or care that it is dealing with shortref. It >would treat '"' as "PCDATA Start" and "PCDATA End". This is correct. However, XML would be required to look stupid by quoting things that are already clearly delimited (by tags), and would be permanently, and completely incomaptible with all existing SGML and pseudo-sgml (i.e. HTML) documents. Using quotes to preserve the SGML newlines is like wearing glasses to fix your outdated contact-lense prescription. Ignoring whitespace around markup is an idea that has otlived its usefulness. Let's bury it. > Paul Prescod RE delenda est. -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ From: "Christopher R. Maden" Date: Thu, 26 Sep 1996 18:20:46 GMT To: w3c-sgml-wg@w3.org Subject: Re: Current Status of Discussion on RE/RS Handling [Eliot Kimber for the ERB] > An XML parser shall interpret white space and record ends in XML ^^^^^^ Really? If an XML application is implemented strictly, as an SGML application, then there's separation between the parser and the application, and this step should happen in the application. If the XML application is not implemented in terms of SGML, then this still seems to me to be an application convention. It's only a semantic point, really, but I think it makes a difference: > This approach also requires that truly significant record ends in > data must be escaped in some way. If the whitespace interpretation rules are phrased as an application convention, and not a change in parsing rules, it buys this: o An SGML parser can be an XML parser without any changes. o A DSSSL-based XML application can do the whitespace rules as a grove operation, after an SGML parse. o Record ends need not be escaped. The stylesheet can affect whitespace normalization behavior; if the element is to be styled "verbatim" or "preformatted", then whitespace is not normalized. o Even an SGML application can parse XML with no (or an ANY) DTD; all elements are mixed content, but the extra whitespace between elements is irrelevant, since it'll be normalized away by the application. True, this won't be ESIS-identical to a parse with the DTD, but after application-level normalization, the "true content" will come out to the same thing. The rules adopted by the ERB are good ones, but I think they should be stated as application conventions, not parsing rules, for the benefit of those implementing XML as an SGML application. -Chris -- http://www.ebt.com +1.401.421.9550 +1.401.521.2030 One Richmond Square, Providence, RI 02906 USA" NDATA SGML.Geek> Date: Thu, 26 Sep 1996 13:16:08 -0900 To: w3c-sgml-wg@w3.org From: "W. Eliot Kimber" Subject: Re: Current Status of Discussion on RE/RS Handling At 06:20 PM 9/26/96 GMT, Christopher R. Maden wrote: >[Eliot Kimber for the ERB] > >> An XML parser shall interpret white space and record ends in XML > ^^^^^^ >Really? If an XML application is implemented strictly, as an SGML >application, then there's separation between the parser and the >application, and this step should happen in the application. If the >XML application is not implemented in terms of SGML, then this still >seems to me to be an application convention. It's only a semantic >point, really, but I think it makes a difference: I'll let other members of the ERB correct me if I'm wrong, but I believe our assumption, in fact the whole purpose of this exercise, is to enable the creation of *XML parsers* that are much simpler than SGML parsers but define XML in such a way that it also be processed as SGML without change (to the instance at least, it may be necessary to create a DTD or modify the XML DTD). We want this to be done by an XML parser for the same reason SGML has the record end handling rules--so applications don't have the opportunity to do it wrong. Cheers, E. -- W. Eliot Kimber (kimber@passage.com) Senior SGML Consultant and HyTime Specialist Passage Systems, Inc., (512)339-1400 10596 N. Tantau Ave., Cupertino, CA 95014-3535 (408) 366-0300, (408) 366-0320 (fax) 2608 Pinewood Terrace, Austin, TX 78757 (512) 339-1400 (fone/fax) http://www.passage.com (work) http://www.drmacro.com (home) "If I never had existed, would you still remember me?..." --Austin Lounge Lizards, "1984 Blues" From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Michael Sperberg-McQueen Cc: W3C SGML Working Group Subject: The RE rules in 14 lines Date: Thu, 26 Sep 1996 19:54:25 GMT James Clark and I have prepared this concise definitive specification of the rule for determining insignificant REs in data, with both XML and SGML variants. It is based on Michael Sperberg-McQueen's clever "nondata" formalism, which replaces a great deal of confusing text in 8879. I intend to propose this to WG8 at the November meeting. (Note that for XML, the rule is 14 lines long, 9 of them formal.) For XML and SGML: An RE in data is insignificant (i.e. not passed to an application, which is to say, not part of the grove) when it occurs in any of the following patterns: start-tag nondata* RE RE nondata* end-tag RS nondata+ RE In applying this rule, a reference is transparent; only its replacement is considered. For XML only: nondata ::= comment declaration | processing instruction reference ::= character reference | entity reference For SGML only: nondata ::= comment declaration | processing instruction | marked section declaration start | marked section end | included subelement | shortref use declaration | link set use declaration reference ::= character reference | entity reference | short reference marked section declaration start ::= marked section start , status keyword specification , dso The rule is applied recursively to the data of included subelements. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: Charles@sgmlsource.com (Charles F. Goldfarb) To: "Christopher R. Maden" , Bill Smith Cc: w3c-sgml-wg@w3.org Subject: Re: More on RE/RS Date: Thu, 26 Sep 1996 19:54:23 GMT On Wed, 25 Sep 1996 20:57:23 GMT, "Christopher R. Maden" wrote: >Several people have proposed defining the RE/RS problem out of >existence by defining the RE and RS function charcters as codes that >won't occur in entities, via the SGML application for XML. The standard (and the Handbook) recognize the possibility that records might not exist in some storage formats. In those cases, you can assign RE/RS out of existence. But when records actually exist, you *cannot* evade handling them by not assigning RE/RS and sweeping them under the rug (where "rug" = application or entity manager). If a document is created in an editor that lets you enter tags as text, you must deal with REs. That is because you could be using white space to "format" the source text, and some of that whitespace will be indistinguishable from true information. (See my earlier postings on this point.) If the editor only lets you enter elements through menu selection, it can avoid having any "formatting" REs in its source document. It is for such editors that the standard makes provision for assigning the RE/RS characters out of existence. Bottom line: You can assign the RE/RS characters out of existence, but not the RE/RS problem. It is inherent in all direct entry of markup and undelimited data, not just SGML. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- From: "Christopher R. Maden" Date: Thu, 26 Sep 1996 19:49:20 GMT To: w3c-sgml-wg@w3.org Subject: Re: More on RE/RS [Charles Goldfarb] > The standard (and the Handbook) recognize the possibility that > records might not exist in some storage formats. In those cases, you > can assign RE/RS out of existence. But when records actually exist, > you *cannot* evade handling them by not assigning RE/RS and sweeping > them under the rug (where "rug" = application or entity manager). But when *do* "records actually exist"? There is a perception evident here, and one that I had always taken for granted, that the data between carriage returns (a "line") was to be interpreted as a record. But I can't find anything in 8879 to that effect! IBM mainframes have records; they are definitely records, and nothing else. But in an ASCII text file, from UNIX, DOS, or Macintosh, what constitutes a record, normatively? I think 8879 does not prescribe a particular behavior in this regard. In any case, the fiat of the ERB (per Eliot) seems to have rendered this discussion moot for the purposes of the Working Group. I'm still interested in the subject, but we probably shouldn't clutter the list. Maybe an RE BOF at SGML '96? (-: -Chris -- http://www.ebt.com +1.401.421.9550 +1.401.521.2030 One Richmond Square, Providence, RI 02906 USA" NDATA SGML.Geek> From: "Christopher R. Maden" Date: Thu, 26 Sep 1996 20:00:01 GMT To: w3c-sgml-wg@w3.org Subject: Re: Current Status of Discussion on RE/RS Handling [Eliot Kimber] > I'll let other members of the ERB correct me if I'm wrong, but I > believe our assumption, in fact the whole purpose of this exercise, > is to enable the creation of *XML parsers* that are much simpler > than SGML parsers but define XML in such a way that it also be > processed as SGML without change (to the instance at least, it may > be necessary to create a DTD or modify the XML DTD). Absolutely. An XML parser, implemented without reference to SGML, can follow these rules simply. However, by putting the rules in terms of the parser, it seems that it's difficult for an SGML-based XML application to be compliant with both XML and SGML. Just by changing the rules from a parser rule to an application convention, it becomes possible for an application to be compliant with both SGML and XML. > We want this to be done by an XML parser for the same reason SGML > has the record end handling rules--so applications don't have the > opportunity to do it wrong. Definitely. The rules are very clear; I'm just worried about *where* the rules live. It's slightly picayune, but now is the time to change it if it's to be changed. -Chris -- http://www.ebt.com +1.401.421.9550 +1.401.521.2030 One Richmond Square, Providence, RI 02906 USA" NDATA SGML.Geek> To: w3c-sgml-wg@w3.org Subject: Re: The RE rules in 14 lines Date: Thu, 26 Sep 1996 14:42:50 -0700 From: Joe English Charles@sgmlsource.com (Charles F. Goldfarb) wrote: > > For XML and SGML: > > An RE in data is insignificant (i.e. not passed to an application, > which is to say, not part of the grove) when it occurs in any of the > following patterns: > > start-tag nondata* RE > RE nondata* end-tag > RS nondata+ RE > > In applying this rule, a reference is transparent; only its > replacement is considered. For SGML, should this rule be applied after OMITTAG inference? For example, in: ]> content of x content of y content of z the REs before and after "content of y" are ignored, even though there are no syntactic start- and end- tags matching the above patterns. Also, an RE immediately following the start-tag of an EMPTY element is not discarded: ]> asdf qwerty Perhaps "start of element" and "end of element" would be more appropriate than "start-tag" and "end-tag"? --Joe English jenglish@crl.com Date: Thu, 26 Sep 1996 14:39:14 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: w3c-sgml-wg@w3.org CC: bosak@atlantic-83.eng.sun.com Subject: Re: Current Status of Discussion on RE/RS Handling An implication of the ERB's current RE/RS proposal just posted by Eliot is that extra spaces within the text of a paragraph (for example) would not be collapsed by this rule (though of course they could be by the application). So people who have gotten into the habit of prettying up their source by indenting text (for example) will have to unlearn this habit if the receiving app isn't set up to collapse the extra spaces. Personally (speaking now only as the five-line perl hacker and someone who has to teach this stuff in two-day extension courses), this is fine with me because it's so incredibly easy to explain: you want spaces, put 'em in; you don't want spaces, leave 'em out. But I thought that this aspect should be made explicit. Jon Date: Thu, 26 Sep 1996 14:43:14 -0700 From: bosak@atlantic-83.eng.sun.com (Jon Bosak) To: crm@ebt.com CC: w3c-sgml-wg@w3.org Subject: Re: Current Status of Discussion on RE/RS Handling [Chris Maden:] | The rules adopted by the ERB are good ones The RS/RE rules posted by Eliot have not been formally adopted. This is just an idea that the ERB would like to see examined by the whole group. Jon From: "Christopher R. Maden" Date: Thu, 26 Sep 1996 23:04:15 GMT To: w3c-sgml-wg@w3.org Subject: Re: Current Status of Discussion on RE/RS Handling [Jon Bosak] > An implication of the ERB's current RE/RS proposal just posted by > Eliot is that extra spaces within the text of a paragraph (for > example) would not be collapsed by this rule (though of course they > could be by the application). So people who have gotten into the > habit of prettying up their source by indenting text (for example) > will have to unlearn this habit if the receiving app isn't set up to > collapse the extra spaces. > > Personally (speaking now only as the five-line perl hacker and > someone who has to teach this stuff in two-day extension courses), > this is fine with me because it's so incredibly easy to explain: you > want spaces, put 'em in; you don't want spaces, leave 'em out. But > I thought that this aspect should be made explicit. This is another thing that becomes moot if whitespace normalization is an application convention instead of a parser rule. For non-verbatim elements, spaces can be normalized (in the grove of a DSSSL engine, or before rendering in a simpler browser), and they can be preserved in verbatim elements. I think it simplifies a lot to define it this way, but I'd love to hear counterarguments. -Chris -- http://www.ebt.com +1.401.421.9550 +1.401.521.2030 One Richmond Square, Providence, RI 02906 USA" NDATA SGML.Geek> Date: Thu, 26 Sep 96 18:02:31 CDT From: paul@arbortext.com (Paul Grosso) To: w3c-sgml-wg@w3.org Subject: Re: Current Status of Discussion on RE/RS Handling > From: "W. Eliot Kimber" > > The rules we came up with are: > > An XML parser shall interpret white space and record ends in XML documents > as follows: > > 1. All white space, including RS and RE, immediately following start tags > and immediately preceding end tags is not significant. > > 2. All other RS/REs are collapsed to a single space. Do you mean this as stated, or do you mean all sequences of white space consisting of RSs, REs, and spaces are collapsed to a single space? > > This approach has the effect that the white space and RS/RE collapsing can > be done before or after SGML RE rules are applied without affecting the > result. The only place this is not true is record ends followed by one or > more PIs followed by data. In SGML, the RE will be considered to have > occurred *after* the PIs, whereas in XML it will be considered to have > occurred *before* the PIs (there are many who consider this behavior of > SGML to be a bug that should be fixed, or at least made optional, in the > SGML revision). I'm not so worried about the RE's that migrate around PIs at the moment. What I'm hoping for is that--discounting the case where the relative order of the PI and the RE affects the resulting display--when the user likes what they see presented by an "XML browser" and they then bring up an "SGML browser/viewer" on the document, they'll see the same thing. I don't know how much to weigh cases such as:

    He was over- sensitive.

    which would come out in XML as: He was over-sensitive. and in SGML as: He was over- sensitive. At least if the XML were "XML-normalized" to strip ignored spaces, then the result would be handled the same by both XML and SGML parsers. From: Charles@sgmlsource.com (Charles F. Goldfarb) To: Joe English Cc: w3c-sgml-wg@w3.org Subject: Re: The RE rules in 14 lines Date: Fri, 27 Sep 1996 11:03:10 GMT On Thu, 26 Sep 1996 14:42:50 -0700, Joe English wrote: > >Charles@sgmlsource.com (Charles F. Goldfarb) wrote: >> >> For XML and SGML: >> >> An RE in data is insignificant (i.e. not passed to an application, >> which is to say, not part of the grove) when it occurs in any of the >> following patterns: >> >> start-tag nondata* RE >> RE nondata* end-tag >> RS nondata+ RE >> >> In applying this rule, a reference is transparent; only its >> replacement is considered. > > >Perhaps "start of element" and "end of element" would be >more appropriate than "start-tag" and "end-tag"? > Yes. Sorry, there is no difference in XML but of course there is in SGML. -- Charles F. Goldfarb * Information Management Consulting * +1(408)867-5553 13075 Paramount Drive * Saratoga CA 95070 * USA International Standards Editor * ISO 8879 SGML * ISO/IEC 10744 HyTime Prentice-Hall Series Editor * CFG Series on Open Information Management -- Date: Fri, 27 Sep 1996 12:27:20 +0000 To: "Christopher R. Maden" From: James Clark Subject: Re: More on RE/RS Cc: w3c-sgml-wg@w3.org At 19:49 26/09/96 GMT, Christopher R. Maden wrote: >[Charles Goldfarb] >> The standard (and the Handbook) recognize the possibility that >> records might not exist in some storage formats. In those cases, you >> can assign RE/RS out of existence. But when records actually exist, >> you *cannot* evade handling them by not assigning RE/RS and sweeping >> them under the rug (where "rug" = application or entity manager). > >But when *do* "records actually exist"? There is a perception evident >here, and one that I had always taken for granted, that the data >between carriage returns (a "line") was to be interpreted as a record. >But I can't find anything in 8879 to that effect! IBM mainframes have >records; they are definitely records, and nothing else. But in an >ASCII text file, from UNIX, DOS, or Macintosh, what constitutes a >record, normatively? I think 8879 does not prescribe a particular >behavior in this regard. See the definition of record in 8879 (clause 4.252). I don't think it leaves any doubt that a record is intended to correspond to an input-line. James Date: Fri, 27 Sep 1996 10:43:15 -0400 To: Robert Streich , w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: RS/RE: basic questions At 9:11 PM 9/23/96, Robert Streich wrote: >At 12:21 PM 9/23/96 -0400, David G. Durand" (David G. Durand wrote: >> There are still a some hard questions about how, and whether to try to >>normalize the marking of input lines (ie. should we preserve the input >>character stream intact, convert to an XML-standard linend convention >>before we send data to the application, require a standard linend >>convention on XML input, or simply ignore line endings entirely and require >>markup for significant linends). > >If we don't have SHORTREF, I don't see that the parser should care what >kind of whitespace it read and could just pass everything along to the >application. Even the whitespace in element content wouldn't cause the >application any problems. It's just going to ignore any extra whitespace >anyway unless the stylesheet tells it that it is significant. > I like this, and agree. The hard issue that I'm worried about is the fact that transport mechanisms sometimes mess with line encodings and thus change the byte-sequence. This is an entity-management question, but given the realities of email, for instance, I get nervous about assuming that \n will not mutate into \r\n. This has been the only reason I can see for RS/RE as defined. I never thought that ignoring initial withespace in markup was anything but confusing... It wasn't until I had to spend a couple of hours with the handbook that I even claimed to understand the rules... That seems too high a price to pay for "convenient formatting", especially when there is the angle whitespace in brackets solution: "

    this is a paragraph with no initial or final whitespace

    " -- David. --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 10:43:22 -0400 To: w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: Current Status of Discussion on RE/RS Handling At 11:41 AM 9/26/96, W. Eliot Kimber wrote: .... stuff about genesis of proposal in a phone call deleted ... > >An XML parser shall interpret white space and record ends in XML documents >as follows: > >1. All white space, including RS and RE, immediately following start tags and > immediately preceding end tags is not significant. > >2. All other RS/REs are collapsed to a single space. I like the consistency and simplicity of these rules, but the lack of any way to have a "verbatim" element is a fatal flaw. Pandering to typists is not a good idea, but removing useful functionality for the sole sake of preserving compatibility is far worse. I still have not seen any reason that we ever _need_ to ignore a linend character in an element. Given that, I oppose ignoring them all, just so that we can preserve SGML's bad habit of ignoring some. C showed us the way almost 2 decades ago: '\n' '\r' are whitespace, and characters _like any others_. RE delenda est. -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 10:43:29 -0400 To: Tim Bray , paul@arbortext.com (Paul Grosso), w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: questions about element declarations At 7:59 PM 9/24/96, Tim Bray wrote: >At 12:32 PM 24/09/96 CDT, Paul Grosso wrote: > >>For example, if I'm given an XML document and partial declarations >>equivalent to the following: >... >[I've moved Paul's example down because it's really not what I'm addressing] >... > >>1. Am I right that the logical equivalent of what I describe is what's >> expected by an XML editing tool? > >I think it is outside the scope of this work to define what the expectations >should be for an XML editing tool. I don't think that experience or wisdom >exists in this group or the world at large to understand that question, let >alone write it into a standard. My personal bias would be that if there >is no DTD, all the tool could possibly do would be to test "well-formed-ness", >i.e. no syntax errors nor mis-nested elements. If there is a DTD, the >problem is well-understood. This all seems right to me. You can't validate without a DTD, so you shouldn't try. > Trying to adduce grammars from instances is >very far from easy and of questionable utility. I disagree with this, though. I think that there is a good case to be made for requiring (or at least encouraging) XML processors to generate a DTD when applied to a file lacking one. I think that for a certain sort of "word-processing" DTD for one-off documents, this could be very useful. And for those unwilling to create DTDs, the automatic DTD would at least keep the issue open, and perhaps even make the process less arcane-seeming. Perhaps this should only be a suggestion rather than a requirement. Some software, like a browser, is most frequently run on documents where the user is _not_ an author, and could care less about having a DTD. I changed my mind on this after heariong about how some users use OCLC's Fred to assist the process of DTD development (rather than to patch a DTD onto any old crap). In other words, I expect the kind of DTD-less processing Paul fears (users will indeed do anything they are allowed to do), but I think it may be an opportunity to educate people. I do agree that grammar induction is a hard problem, and would discourage any specification of _what DTD_ a tool should create, except that it should validate the one instance in question. This will allow implementations to compete on the qulaity of grammar induction (if people care about it), and not be too inconvenient for lazy parsers: "create an ANY delcaration for every element you see". >-Cheers, Tim RE delenda est. -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 10:43:26 -0400 To: "Christopher R. Maden" , w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: More on RE/RS At 4:57 PM 9/25/96, Christopher R. Maden wrote: >That certainly seems a reasonable interpretation, but I can't find >anything to that effect in 8879. Clause 7.6.1, "Record Boundaries", >defines the rules for RE ignorance or preservation, but doesn't say >anything about when the parser generates an RS or RE signal. Charles >Goldfarb's commentary thereto (pp. 321+322 of the SGML Handbook) >discuss translating lines into records, but that's not normative. > >The best normative thing I can find is 4.140, "function character >identification parameter: A parameter of an SGML declaration that >identifies the characters assigned to the RE, RS, and SPACE functions, >and allows additional functions to be defined." > >This suggests that, since characters are assigned to functions, that >the characters in the document should assume the roles of these >functions; ergo, if non-ocurring characters are the ones assigned to >those roles, the function characters never occur. Is that not the >intended meaning? If not, what is? It's even easier. If the entity manager does not recognize the functions, then the characters need never be generated. Nothing in the standard _requires_ recognizing the RE/RS functions at all. Defining RS/RE out of the code space is just to avoid wasting code points (which are valuable in 8-bit code-spaces) on something that will never occur. Within a 10646 framework we could even choose a particular code in the private-use area, given the improved number of codepoints available. -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 11:13:40 -0400 To: w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: More on RE/RS At 8:27 AM 9/27/96, James Clark wrote: >See the definition of record in 8879 (clause 4.252). I don't think it >leaves any doubt that a record is intended to correspond to an input-line. Quote of definitional text (emphasis mine): 4.252 Record: A division of an SGML entity bounded by a record start and a record end character, _normally_ corresping to an input line on a text-entry device. That "normally" is a hole that I think I'm justified in driving a truck through. I'll also note that, at least for the Mac, there is no notion of line, if we take Simpletext (supplied with every Mac by Apple) as the System's "text-entry device." In Simpletext CR is a paragraph delimiter, due to automatic word wrapping. This wrapping is not only a feature of most applications, but it is actually built into the system suypplied text-display and handling functions. This is just one example, but the uniformity of file handling in Unix suggests that '\n' is an application convention, and not a system feature (so we are even within our rights to ignore it). Also the notion of text-entry device is vacuous or at least ill-defined in the days of modern windowing systems. Will the Hollerith card never die? >James RE delenda est. -- David --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 12:08:58 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: Current Status of Discussion on RE/RS Handling At 06:02 PM 9/26/96 CDT, Paul Grosso wrote: >I don't know how much to weigh cases such as: > >

    He was over- sensitive.

    >which would come out in XML as: > He was over-sensitive. >and in SGML as: > He was over- sensitive. Well, an SGML tool working with HTML data already has this problem, right? For instance, in HTML, multiple spaces are collapsed, except in PRE. Can't we treat the XML whitespace-in-mixed-content rules as application conventions? Paul Prescod From: lee@sq.com Date: Fri, 27 Sep 96 12:52:45 EDT To: dgd@cs.bu.edu, w3c-sgml-wg@w3.org Subject: Re: More on RE/RS > This is just one example, but the uniformity of file handling in Unix > suggests that '\n' is an application convention, and not a system feature > (so we are even within our rights to ignore it). Yes. It was an early goal of Unix (it's in one of the Bell Tech. Journal Special Issue papers, I think) to avoid record-based files altogether. I think a lot of the RS/RE trouble comes from confusion between the representation and that which is represented. Some people (many people) want to be able to say this is a monospaced example and the columns line up. and have spaces, tabs and newlines be significant in the markup in order to represent that in a ``WYSIWYG'' way. Then people with no background in design can try and lay out their source code or DTDs and align unrelated things at the expense of clarity :-), and neither the receiving application nor the screen or paper layout designer can correct the errors without extensive hand work, it's awful. Rants aside :-), this is an insidious form of minimisation. Insidious because it does not in fact have a fully normalised form! It is one thing to use SHORTREF or DATATAG to map newline to
    or &linebreak; or something. It is quite another to use spaces tabs and newlines to draw ASCII art. If people want to do this, they should use a NOTATION. Perhaps we need a kind of Unix vgrind that takes an SGML instance or DTD fragment and pretty-prints it into an XML representation. If the same were done for C (I have a C --> SMLG (sic) program I wrote that lets you view C in Panorama, for example) and plain text, how important would this issue still be? You can't include C or C++ in SGML without pre-processing anyway, because of int *ip = &i; being not only legal but common, unless you change ERO to be @ or something else saner than &. Same with perl. Unix shell scripts are OK with & because it's not usually followed by a name, although 1>&2; occurs fairly often in both shell and awk. But you do get prog output, which isn't OK in #PCDATA, and even prog Subject: Re: Current Status of Discussion on RE/RS Handling At 10:43 AM 9/27/96 -0400, David G. Durand" (David G. Durand wrote: > I like the consistency and simplicity of these rules, but the lack of >any way to have a "verbatim" element is a fatal flaw. Pandering to typists >is not a good idea, but removing useful functionality for the sole sake of >preserving compatibility is far worse. Isn't a verbatim element just text-entry convenience? Why should it be preserved instead of the ability to use newlines as formatting around tags? Paul Prescod Date: Fri, 27 Sep 1996 13:15:46 -0400 To: w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Regular expressions and content models (was: Re: questions about element declarations) Lee brought up the key issues in content-model ambiguity, and I had a few things to add, in terms of analysis, and finally opinion. At 12:15 AM 9/24/96, lee@sq.com wrote: >>> * Should XML allow nondeterministic content models (11.2.4.3)? >> >> Again, how much does this complicate validation? I'm not ambiguity expert, >> but could the problem be solved simply by stipulating that a token is >> always matched to the first place in the content model it can match, >> without lookahead? > >Yes, you could do that, but my content model above would then never work, >because if you omitted a Requirements, you'd be stuck on the first Section*, >and the Proposal would be an error. > >I think that my Report can be matched with no lookahead if you don't care >about distinguishing sections in the various groups, but you'd have to use >a somewhat more sophisticated approach, I'm afraid, than shortest match. >An NDFA -> DFA conversion ala Aho et. al. would do it. Here's the model >again, followed by the reduced DFA: > > Scope?, Section*, Requirements?, Section*, > Proposal?, Section*, Summary? > )> > >start: > Section -> start > Scope -> seenScope > Requirements -> seenRequirements > Proposal -> seenProposal > Summary -> seenSummary > >seenScope: > Section -> seenScope > Requirements -> seenRequirements > Proposal -> seenProposal > Summary -> seenSummary > >seenRequirements: > Section -> seenRequirements > Proposal -> seenProposal > Summary -> seenSummary > >seenProposal: > Section -> seenProposal > Summary -> seenSummary > >seenSummary: > EOF -> accept > * -> error > >So although this is ambiguous in SGML terms, there is no look-ahead needed. >Question to brave sould who made it to here: are there any cases where >look-ahead _would_ be needed? If so, do they matter? As long as you don't care which part of the content model matches an element the answer is "No". The problems with treating SGML content models as regular expressions are: 1.) &-groups. This is a problem in practice, not principle, because &-groups either require an extended FSA parser, or exponential table space, or nondeterministic implementation. 2.) As you note, SGML requires that the parser match syntactic occurrences in the content model with instances in the document. Any model like (a?, a) is a problem because we don't know _which_ "a" we've seen. construed as a regular expression this is a non-problem because the set of strings defined is the same as than defined by (a, a?) or (a | (a,a)). Equivalence of regular expressions is defined in terms of the sequence of the elements they describe. The only sensible way I know to implement this correctly is to implement content models as NDFAs and then ensure that there is no place where the non-determinism is used. In other words, SGML content models must be parsed as though their structure is a grammar, and the way that they match the input elements is significant. This makes content models that define the same language in different ways inequivalent in SGML terms. This is unforatunate because standard theory looks at the language (what sequences of subelements are permitted) and not the parse. Now, I think there are two problems with the way SGML does things: a) It's a shame that we've invented "almost regular" expressions, and done it in a way that _prevents_ application of existing tools and basic theory. b) It encourages what I call crypto-markup. For instance, a content model like: seems to give more structure to the list than is evident from the markup itself: ........... ..... The ESIS, for instance, does not let a browser or search or analysis program determine that a term and its item are grouped. Even examining the content model doesn't make it as clear as an intermediate element would, without reparsing the element sequence in the ESIS. This all harks back to the grammar/language distinction. We parse as though the _form_ of the content model is important, but then we hide that information from the application (unless it should shoose to read the DTD, and re-parse the elements). If we were to make this information we have actually made a big change: because we've introduced anonymous nodes (content model groups) into the SGML parse tree for the first time. I think that would not be a good idea. Note that if XML follows the lead of Hytime, and says that any content model producing the same sequence of elements is good, we will have removed one of the perennial thorns in the side of DTD designers, and those who have to train them. Conversion from XML "any REGEXP goes" content models to SGML content models is possible (by the theorem proving the equivalence of DFAs and NDFAs and REs). There is a standard algorithm in Hopcroft and Ullman (Intro. to Automata Theory, Languages and Computation) for producing an RE from a DFA, but the expressions it produces are yucky, and would not be good SGML content models. I'm sure that can be fixed up, but I don't have a proof handy. If architectural forms didn't have the restrictions of SGML content models, we could use them to get around the compatibility problem since they are defined in terms of the language generated (document instance) and not SGML's model subgroup matching rules. If you haven't guessed, I'd rather eliminate the ambiguity rules and be able to use a rich and well-understood theory instead of having to help invent a new theory that only allows things we might rather not allow anyway. It's also worth noting that people use naive parsers of ambiguous regular expressions in text-editors every day, and it's not a problem. So the efficiencies that motivate the technical argument for eliminating ambiguity may not be very important anyhow. -- David RE delenda est. --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 13:15:54 -0400 To: W3C SGML Working Group From: "David G. Durand" (David G. Durand) Subject: Opinions on some of Michael's undiscussed topics At 7:15 PM 9/23/96, Michael Sperberg-McQueen wrote: >* Should XML use the markup-declaration syntax described by ISO 8879 >clauses 10-11, or should XML define a specialized document type and let >its markup declarations use the document-instance syntax, as proposed by >MGML? > >(Using the same syntax for declarations and instances cuts the size of >the grammar approximately in half. It also reflects a firm belief that >structured information belongs in SGML documents.) I need to review MGML, but I think this would be great, as I too feel the desire for a meta-DTD everytime a write one. Unfortunately the one time I tried to do one, I did a project-specific one, and so I still don't have the general tool that I need. I agree with Eliot that we might get some things wrong, but the risk of getting something disastrously wrong with this group is slight, I think. And the goal of easy implementability should steer us away from a lot of tempting problems, I think. >* How should XML deal with the need for conditional inclusion of markup >declarations, if XML has no marked sections (10.4.1)? Sentiment seems to lean to allowing them in DTDs. DTD subsets are an open problem. These could also be handled by the conditional inclusion mechanism for instances, if we align the two syntaxes... >* Should XML change the delimiter-in-context rules to require the STAGO >and ERO strings to be escaped whenever they are not to be recognized as >delimiters (9.6)? > >(Some people have proposed this; certainly many user manuals I've seen >prescribe this behavior for users, rather than explaining the d-i-c >rules.) I wouldn't die for this, but I would love it. The contextual rules are just too hard to remember on the fly. And, as James has noted they are another little convenience that the parser has to care about. I'd rather we end up with a language where we could have a real split between lexical analyzer and parser, and eliminating d.i.c. seems to aid that. >* Should XML use MSOCHAR, MSSCHAR, and MSICHAR strings (9.7)? > >(If MSSCHAR is '\', then \< and \& are escaped in the expected way. >Unlike most Unix-style backslash escapes, however, the MSSCHAR is not >removed from the data stream; this means \ characters in existing docs >are safe.) This breaks for strings like the following: \&entity;\ Actually I favor the proposal, but we shouldn't expect magic for existing SGML instances if we accept it. > >* Should XML require system and public identifiers to be FORMAL (13.5)? Yes. I'm inclined even to remove system IDs altogether (even formal ones) and leave them only in CATALOGs. >* Should XML restrict comment declarations to a single comment (10.3)? Sure. I can see no rational justification for multiple comments in a single comment declaration anyway. I'd rather be stronger and allow '--' in comments except when followed by '>'. Otherwise the single comment nature of the markup will make the no '--' rule even harder to justify... -- David RE delenda est. --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 13:56:10 -0400 To: w3c-sgml-wg@w3.org From: "David G. Durand" (David G. Durand) Subject: Re: Current Status of Discussion on RE/RS Handling At 12:56 PM 9/27/96, Paul Prescod wrote: >At 10:43 AM 9/27/96 -0400, David G. Durand" (David G. Durand wrote: >> I like the consistency and simplicity of these rules, but the lack of >>any way to have a "verbatim" element is a fatal flaw. Pandering to typists >>is not a good idea, but removing useful functionality for the sole sake of >>preserving compatibility is far worse. > >Isn't a verbatim element just text-entry convenience? Why should it be >preserved instead of the ability to use newlines as formatting around tags? > > Paul Prescod Yes, that is true, but on a practical note, I cannot see that we are heading for wide acceptance if we rule it out. I was not the first on this list to raise the issue of vebatim-style elements. This is not going to help us make our case to the currently unconvinced. We are taking a common, reasonable, well-established usage, and making it impossible, solely for compatibility with a feature of SGML that isn't popular, easy to explain, or widely understood. -- David RE delenda est --------------------------------------------+-------------------------- David Durand dgd@cs.bu.edu | david@dynamicDiagrams.com Boston University Computer Science | Dynamic Diagrams http://www.cs.bu.edu/students/grads/dgd/ | http://dynamicDiagrams.com/ Date: Fri, 27 Sep 1996 20:14:25 +0000 To: "Christopher R. Maden" From: James Clark Subject: Re: Current Status of Discussion on RE/RS Handling Cc: w3c-sgml-wg@w3.org At 20:00 26/09/96 GMT, Christopher R. Maden wrote: >[Eliot Kimber] > >> I'll let other members of the ERB correct me if I'm wrong, but I >> believe our assumption, in fact the whole purpose of this exercise, >> is to enable the creation of *XML parsers* that are much simpler >> than SGML parsers but define XML in such a way that it also be >> processed as SGML without change (to the instance at least, it may >> be necessary to create a DTD or modify the XML DTD). > >Absolutely. An XML parser, implemented without reference to SGML, can >follow these rules simply. > >However, by putting the rules in terms of the parser, it seems that >it's difficult for an SGML-based XML application to be compliant with >both XML and SGML. Just by changing the rules from a parser rule to >an application convention, it becomes possible for an application to >be compliant with both SGML and XML. If the rules about ignoring white-space are left to the XML application and the application is free to require that those rules are not applied for verbatim elements, then XML tools built on top of SGML parsers will be unable to correctly process some XML documents, namely those that have verbatim elements that include REs that are ignored according to the SGML rules. (An application could get information from the SGML parser about the record-ends it ignored and attempt to undo the ignoring that was done by the SGML parser, but that's not going to be practical in many cases.) The effect would be to prevent most unmodified SGML-based tools from being able reliably to process XML documents. I would say that would be a far worse situation for XML to be in than requiring that a user, in verbatim text, simply replace space and newline by entity references at the same time as they are replacing <, > and & by entity references. James Date: Fri, 27 Sep 1996 16:32:19 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: Current Status of Discussion on RE/RS Handling At 01:56 PM 9/27/96 -0400, David G. Durand" (David G. Durand wrote: >Yes, that is true, but on a practical note, I cannot see that we are >heading for wide acceptance if we rule it out. I was not the first on this >list to raise the issue of vebatim-style elements. This is not going to >help us make our case to the currently unconvinced. We are taking a common, >reasonable, well-established usage, and making it impossible, solely for >compatibility with a feature of SGML that isn't popular, easy to explain, >or widely understood. As Lee says, they may not be as reasonable as they seem. Since when did it become "okay" to encode formatting in an SGML document? And isn't it doubly wrong to put formatting in using characters that are meant to encode structure. (i.e. word breaks and line breaks?) Also, most text has to be escape-encoded for inclusion in an SGML document anyways. Improperly including non-SGML data in an SGML document is a MAJOR source of errors. It may be a good thing to make it a little more difficult. People use verbatim elements because a) SGML "files" usually only have one entity in them, so people who want to keep their documents in a single file don't want multiple entities and b) the handling of an entity with notation "C++" or "Perl" is pretty much up in the air. Maybe we should fix those two problems to make verbatim formatting unncessary instead of bringing back verbatim. Mime seems to have solutions for both. Paul Prescod Date: Fri, 27 Sep 1996 16:24:21 -0400 To: w3c-sgml-wg@w3.org From: Paul Prescod Subject: Re: Current Status of Discussion on RE/RS Handling At 11:04 PM 9/26/96 GMT, Christopher R. Maden wrote: >This is another thing that becomes moot if whitespace normalization is >an application convention instead of a parser rule. For non-verbatim >elements, spaces can be normalized (in the grove of a DSSSL engine, or >before rendering in a simpler browser), and they can be preserved in >verbatim elements. I think it simplifies a lot to define it this way, >but I'd love to hear counterarguments. >From an SGML point of view, XML _will be_ an application and XML's whitespace handling rules will be application conventions. >From an XML point of view, XML's rules are "the rules" and should be enforced by the parser. Paul Prescod