Paul M. Foster (paulf@quillandmouse.com)
Thu, 13 Jan 2000 00:35:59 -0500 (EST)
On Wed, 12 Jan 2000, Aaron Turner wrote:
> On Tue, 11 Jan 2000, Paul M. Foster wrote:
>
> > On Tue, 11 Jan 2000, Kim Lester wrote:
> >
> > >
> > > ODE Summary Report #1
> > > =====================
> >
<snip>
> > Much more cumbersome. I presume the idea is the develop system(s) that
>
> Cumbersome? That's an interesting statement. In what way is it so? From
> the end user perspective our the repository maintainers perspective?
>
Repository maintainer's perspective. I was talking about allowing email
threads and such to be part of the indexed documents.
> > works for both areas, perhaps slightly different ones for each area. My
> > comments will relate to building such a system for local machines, running
> > on their machines.
>
> I won't assume to speak for other groups, but this isn't something the
> LinuxKB had in mind. We're very hard-core central-web repository. It
> just so happens that some of the internal tools we will develop to
> maintain the site also have certain applications for local content
> distribution as on a CDROM. We however have no current plans to provide
> any tools to allow users to search that local content.
>
Yeah, I'm on the LinuxKB list too. I understand that's not your
perspective. However, I think this whole project here is also valuable
from the viewpoint that you could put the engine/software/whatever on the
local system and then the user would have _easy_ access to the docs,
instead of what we have now. Naturally, LinuxKB's engine wouldn't fit/work
locally, but perhaps we could come up with something that would.
> > Next, do we plan to have any humans do any indexing on existing documents,
> > or do we want the "indexer" program to be smart enough to figure all this
> > out?
>
> I've thought about, dreamed of, and racked my brain trying to figure out
> how the hell to do that. I've come up empty. What you're talking about
> literally is an AI that can do context based word matching and ranking.
> Yes this sort of thing has been done. No I don't have a clue how they do
> it. And my guess is that the human brain will do it better for the
> foreseeable future. And there is no reason that the review/edit process
> can't include categorization which would be a part of the doc spec IMHO.
>
I agree humans would be better. But considering the amount of
documentation to sift through, I'm not sure having humans do it is
practical. It's pretty easy to point an indexer at a bunch of docs and say
"index it" and get back keywords and such. It's quite another to
categorize it according to some more esoteric scheme.
> > What should our favored format be? Obviously, opinions vary. However,
> > since we're trying to index information, and since presentation should be
> > less important, I'd vote for XML. It is specifically designed to allow
> > indexing of information, and has incredible flexibility. The writers of
> > documentation needn't worry with what format we use, since we are
> > translating from their format into ours. Real issue is what will make our
> > indexing job the easiest? I'm inclined to think XML. And forget standards
> > for XML. You actually don't need a DTD for XML, but we can come up with
> > one if needed. After all, this is our internal format. Now, if you want
> > others to write to this format, you'll have to make the DTD simple.
>
> Indexing is and likely always will be a manual process. Both SGML and XML
> provide markup to allow you to specifiy the category in a way that a
> program can extract it. Hence from a indexing aspect XML and SGML are on
> equal footing. However, the tool set to work with SGML is much more rich
> which makes me give it a thumbs up over XML.
>
You're probably right, and that seems to be the concensus: SGML > XML. I
don't quite know what DocBook is, but from what I understand it is a
certain DTD under SGML. Yes?
Here's the other problem with SGML/Docbook. If you want this to be served
up in a browser, you'll get a lot more support for doing it in XML than
SGML. Browsers are starting to support XML, but I don't know that they'll
ever support SGML or Docbook. And as far as I can see, the browser is
probably the ideal interface for this stuff.
> > How should the user access the information? I think the standard
> > technology most widely available today is the browser. It works fairly
> > well for this purpose and is pretty flexible. That way we are not building
> > yet another tool. That said, we still will need some sort of cgi or other
> > back end that will eat one format and turn it into another. This also
> > means that we will need to have a [thin] document server running on the
> > local machine.
>
> You shouldn't require some daemon on your local system to read
> documentation IMHO. If you can't export your format to HTML or plain text
> you are wasting your time. Don't expect to have a mini-knowledge-base
> running locally on people's system for sometime if ever.
>
Well, here's the problem. Joe creates program barfoo, and makes his little
man page for it. (And for the sake of argument, let's say the program is
quite old but is being regularly maintained.) When Joe updates his
program, he updates his man pages as well. He sticks it up on Sunsite as a
tarball. Now, the maintainers of RedHat go out and get his program, make a
spec file for it, and wrap it in an RPM, and it goes on their distro.
But let's say that everyone decides docbook is the way to go. Is Joe going
to change his documentation to docbook? Unlikely. Is Red Hat going to take
the time to translate his man pages into docbook before building the
RPM? Also unlikely. Which means that if you want to provide the distros
with a tool to allow them to serve up all the docs in a single way on
someone's machine, sooner or later you're going to have to build a program
that goes out and reads those man files (and HOWTOs and info files,
etc.) and builds some sort of a database or something of them, indexes
them, and provides a way for a browser to serve them up.
Again, this isn't something LinuxKB would have much to do with per se.
<snip>
> > It might be instructive to look at the way documentation is handled for
> > the Debian Linux distribution. If you choose, you can install programs
> > that allow the search and retrieval of all documentation on the system
> > via Apache, Lynx and such. It is _similar_ to what we are talking about,
> > but extremely primitive. For instance, you really need to specify whether
> > you want to view information from the man files, info files or HOWTOs in
> > order to find what you want. Mostly, their solution is a backend that uses
> > off-the-shelf tools to convert man, info and other docs into HTML. As I
> > said, very primitive, but it works.
>
> What you're talking about is specifying in the doc spec a "document type"
> and allowing users to limit their queries by type. If you design your
> system well this is easy.
No no. The way Debian does it, it works out that if you want to look at
something contained in the HOWTOs, you have to go into the HOWTO section
of the documentation. But you may not know whether what you want is in the
HOWTOs or not. That's one reason why the Debian system is klunky. The user
shouldn't have to specify the source for answering his question.
<snip>
> > One thing sorely lacking in Linux distros, which was commented on in
> > previous posts, is the ability to ask, "What programs can I use to look at
> > my mail with?" And "what does X program in Y directory do?" Quite often,
> > these programs don't even have a doc file of any kind with them.
>
> What you're talking about is the ability to search for categories and
> applications in those categories, not actual documents. Again, assuming
> the correct implimentation of the backend, easy. (BTW, I say it's easy
> becuase we've already figured it out how to do that. YMMV)
>
Actually, this is kinda handled by the distros themselves when they
install. They have packages grouped in categories at install time. I just
don't know if the file detailing this hangs around on the system
afterward.
> > Is the plan to index things via some hierarchy, like the way man files are
> > laid out, or the way the Dewey Decimal System works?
>
> Either and/or neither. A flexible categorization system should categorize
> documentation in various ways. Perhaps by user function (admin, end user,
> developer, etc), by document type (kernel doc, man page, software readme,
> etc), or by whatever structure tickles your fancy.
>
Yeah, but you have to set up categories so your humans or indexing
programs know how to do the indexing at the outset. I don't think we need
to be all that flexible as long as we are thorough. Again, the Dewey
Decimal system. Not perfect, but a very good start. (Not that we should
use this. It's just an example of a good categorization system.)
> > If so, then we much
> > exhaustively generate appropriate categories. Then the question is, can
> > our indexing technology be smart enough to figure out from the
> > documentation the categories into which the documentation falls? And what
> > if we further subdivide the content of a single man page? Does the indexer
> > do a good job with that as well? I'd love to see some suggestions on what
> > categories we should use.
>
> So would I :) Alas, give X people 100 documents to categorize, and you'll
> probably get X different ways to categorize them. What I would rather see
> is categorization guidelines (methodology?) like I showed above. Something
> that tells you how to go about categorizing content rather than a fixed
> structure that works today, but not tomorrow.
>
Make it extensible, and it will work tomorrow, too. I'd have to see what
sort of guidelines, specifically, before I could agree that that's all you
need.
<snip>
> > Mention was made earlier about categorization in XML being done in
> > comments. This defeats the purpose of putting it in XML. The tags,
> > attributes and values need to be part of the ultimate XML document in
> > order to be of value in indexing.
>
> Depends on if your repository is XML aware. If your repository/search
> engine chokes on XML (which is most of them) then it doesn't matter if
> categorization info is in native XML or in comment tags as the repoistory
> engine will just parse the raw text of the document to find what it's
> looking for.
>
If that's the case, you might as well put all the docs in straight text
files. The idea should be to get the document to tell you what's important
in it, and how it should be categorized.
<snip>
> > Nothing's worse than calling "man
> > foo" and finding out that GNU has decided they don't want to maintain the
> > man pages anymore, and that the real up-to-date stuff is in info. Argh. I
> > don't like the info format anyway. Just put it all in XML or SGML or
> > something, and let the user access all of it that way. And if they want to
> > type "man foo", then what they get is Lynx handing them an XML page for
> > what they asked for. Of course, I'd like to see most Linux docs
> > completely rewritten so that one doesn't need three PhDs from four
> > universities to understand it. Too radical, I know. </rant>
>
> The issues of man/texinfo shouldn't effect us as we will be maintaining
> one master source which will then be converted in the alternate formats.
> Hence, if you're looking at the HTML or plain text or native format, they
> should all be at the same "version".
>
Hmm. That may be the crux of the problem. Is that what this project is
about? That seems impractical to me, because of the example I gave
above. Joe will update his tarball with his new man page, and you'll need
to be sure you know when it's been updated, so you can translate it into
your format and put it in your repository. At that point, versioning
_does_ become important.
This sounds like LinuxKB to me. You're talking about all docs in one
format, then converted to others for use. But if you have it all in this
cool format, why convert it? Just build or find a program that uses it in
its native format and forget about man pages and such altogether.
Again, this looks like the difference between docs as viewed by LinuxKB
and local docs as viewed by a local user on his local machine. Which
approach is ODE looking at? Or both?
Paul M. Foster
This archive was generated by hypermail 2.0b3 on Wed Jan 12 2000 - 23:35:25 EST