Re: [ode] Re: ODE Summary Report #1


Aaron Turner (aturner@linuxkb.org)
Thu, 13 Jan 2000 18:26:12 -0800 (PST)


On Thu, 13 Jan 2000, Paul M. Foster wrote:
>
> On Wed, 12 Jan 2000, Aaron Turner wrote:
>
> > On Tue, 11 Jan 2000, Paul M. Foster wrote:
> >
> > > On Tue, 11 Jan 2000, Kim Lester wrote:
> > > >
> > > > ODE Summary Report #1
> > > > =====================
> > >
>
> <snip>
>
> > > Much more cumbersome. I presume the idea is the develop system(s) that
> >
> > Cumbersome? That's an interesting statement. In what way is it so? From
> > the end user perspective our the repository maintainers perspective?
>
> Repository maintainer's perspective. I was talking about allowing email
> threads and such to be part of the indexed documents.

Well, yeah, that does complicate things :) But IMHO it's worth it. Not
every repository has to do it the same way, so they can do what their
skill/time allows.
 
> > > works for both areas, perhaps slightly different ones for each area. My
> > > comments will relate to building such a system for local machines, running
> > > on their machines.
> >
> > I won't assume to speak for other groups, but this isn't something the
> > LinuxKB had in mind. We're very hard-core central-web repository. It
> > just so happens that some of the internal tools we will develop to
> > maintain the site also have certain applications for local content
> > distribution as on a CDROM. We however have no current plans to provide
> > any tools to allow users to search that local content.
> >
>
> Yeah, I'm on the LinuxKB list too. I understand that's not your
> perspective. However, I think this whole project here is also valuable
> from the viewpoint that you could put the engine/software/whatever on the
> local system and then the user would have _easy_ access to the docs,
> instead of what we have now. Naturally, LinuxKB's engine wouldn't fit/work
> locally, but perhaps we could come up with something that would.

I'd love to see someone else do that.

> > > Next, do we plan to have any humans do any indexing on existing documents,
> > > or do we want the "indexer" program to be smart enough to figure all this
> > > out?
> >
> > I've thought about, dreamed of, and racked my brain trying to figure out
> > how the hell to do that. I've come up empty. What you're talking about
> > literally is an AI that can do context based word matching and ranking.
> > Yes this sort of thing has been done. No I don't have a clue how they do
> > it. And my guess is that the human brain will do it better for the
> > foreseeable future. And there is no reason that the review/edit process
> > can't include categorization which would be a part of the doc spec IMHO.
> >
>
> I agree humans would be better. But considering the amount of
> documentation to sift through, I'm not sure having humans do it is
> practical. It's pretty easy to point an indexer at a bunch of docs and say
> "index it" and get back keywords and such. It's quite another to
> categorize it according to some more esoteric scheme.

The trick is getting a critical mass of people to index and give them
easy/powerful tool to do the work. I'm working on such a tool now.

> > > What should our favored format be? Obviously, opinions vary. However,
> > > since we're trying to index information, and since presentation should be
> > > less important, I'd vote for XML. It is specifically designed to allow
> > > indexing of information, and has incredible flexibility. The writers of
> > > documentation needn't worry with what format we use, since we are
> > > translating from their format into ours. Real issue is what will make our
> > > indexing job the easiest? I'm inclined to think XML. And forget standards
> > > for XML. You actually don't need a DTD for XML, but we can come up with
> > > one if needed. After all, this is our internal format. Now, if you want
> > > others to write to this format, you'll have to make the DTD simple.
> >
> > Indexing is and likely always will be a manual process. Both SGML and XML
> > provide markup to allow you to specifiy the category in a way that a
> > program can extract it. Hence from a indexing aspect XML and SGML are on
> > equal footing. However, the tool set to work with SGML is much more rich
> > which makes me give it a thumbs up over XML.
> >
>
> You're probably right, and that seems to be the concensus: SGML > XML. I
> don't quite know what DocBook is, but from what I understand it is a
> certain DTD under SGML. Yes?

Yep. You can buy or download the Orielly DocBook manual.

> Here's the other problem with SGML/Docbook. If you want this to be served
> up in a browser, you'll get a lot more support for doing it in XML than
> SGML. Browsers are starting to support XML, but I don't know that they'll
> ever support SGML or Docbook. And as far as I can see, the browser is
> probably the ideal interface for this stuff.

Yep. Right now though, whatever standard you choose you'll need to
convert to HTML for the browsers. Long term, once the tools are there,
I'd love to see XML, as I really think it's the future.

> > > How should the user access the information? I think the standard
> > > technology most widely available today is the browser. It works fairly
> > > well for this purpose and is pretty flexible. That way we are not building
> > > yet another tool. That said, we still will need some sort of cgi or other
> > > back end that will eat one format and turn it into another. This also
> > > means that we will need to have a [thin] document server running on the
> > > local machine.
> >
> > You shouldn't require some daemon on your local system to read
> > documentation IMHO. If you can't export your format to HTML or plain text
> > you are wasting your time. Don't expect to have a mini-knowledge-base
> > running locally on people's system for sometime if ever.
> >
>
> Well, here's the problem. Joe creates program barfoo, and makes his little
> man page for it. (And for the sake of argument, let's say the program is
> quite old but is being regularly maintained.) When Joe updates his
> program, he updates his man pages as well. He sticks it up on Sunsite as a
> tarball. Now, the maintainers of RedHat go out and get his program, make a
> spec file for it, and wrap it in an RPM, and it goes on their distro.
>
> But let's say that everyone decides docbook is the way to go. Is Joe going
> to change his documentation to docbook? Unlikely. Is Red Hat going to take
> the time to translate his man pages into docbook before building the
> RPM? Also unlikely. Which means that if you want to provide the distros
> with a tool to allow them to serve up all the docs in a single way on
> someone's machine, sooner or later you're going to have to build a program
> that goes out and reads those man files (and HOWTOs and info files,
> etc.) and builds some sort of a database or something of them, indexes
> them, and provides a way for a browser to serve them up.

I don't think what we do here will ever replace man pages on Unix, so I'm
less sure about the need for this. There are already man2html converters,
so making the man page available to a browser is a simple task.

> Again, this isn't something LinuxKB would have much to do with per se.

Agreed.

> <snip>
>
> > > It might be instructive to look at the way documentation is handled for
> > > the Debian Linux distribution. If you choose, you can install programs
> > > that allow the search and retrieval of all documentation on the system
> > > via Apache, Lynx and such. It is _similar_ to what we are talking about,
> > > but extremely primitive. For instance, you really need to specify whether
> > > you want to view information from the man files, info files or HOWTOs in
> > > order to find what you want. Mostly, their solution is a backend that uses
> > > off-the-shelf tools to convert man, info and other docs into HTML. As I
> > > said, very primitive, but it works.
> >
> > What you're talking about is specifying in the doc spec a "document type"
> > and allowing users to limit their queries by type. If you design your
> > system well this is easy.
>
> No no. The way Debian does it, it works out that if you want to look at
> something contained in the HOWTOs, you have to go into the HOWTO section
> of the documentation. But you may not know whether what you want is in the
> HOWTOs or not. That's one reason why the Debian system is klunky. The user
> shouldn't have to specify the source for answering his question.

I think we're miss-communicating. I'm saying allow them to limit queries
by the type to reduce the number of hits, not seperate the content so they
can't do a global search against all docs if they don't have a clue where
to look.

> <snip>
>
> > > One thing sorely lacking in Linux distros, which was commented on in
> > > previous posts, is the ability to ask, "What programs can I use to look at
> > > my mail with?" And "what does X program in Y directory do?" Quite often,
> > > these programs don't even have a doc file of any kind with them.
> >
> > What you're talking about is the ability to search for categories and
> > applications in those categories, not actual documents. Again, assuming
> > the correct implimentation of the backend, easy. (BTW, I say it's easy
> > becuase we've already figured it out how to do that. YMMV)
> >
>
> Actually, this is kinda handled by the distros themselves when they
> install. They have packages grouped in categories at install time. I just
> don't know if the file detailing this hangs around on the system
> afterward.

Unfortunately in my experiance the categories are pretty bad. Maybe I'm
just dumb and always look in the wrong place, but I've heard other people
complain as well. Perhaps using the RPM category would be a good starting
point or as the "offical home" for docs that are in multiple places in the
tree.

> > > Is the plan to index things via some hierarchy, like the way man files are
> > > laid out, or the way the Dewey Decimal System works?
> >
> > Either and/or neither. A flexible categorization system should categorize
> > documentation in various ways. Perhaps by user function (admin, end user,
> > developer, etc), by document type (kernel doc, man page, software readme,
> > etc), or by whatever structure tickles your fancy.
> >
>
> Yeah, but you have to set up categories so your humans or indexing
> programs know how to do the indexing at the outset. I don't think we need
> to be all that flexible as long as we are thorough. Again, the Dewey
> Decimal system. Not perfect, but a very good start. (Not that we should
> use this. It's just an example of a good categorization system.)

Indexing programs just follow links. If you have the same doc in 20
different places (I'm exagerating to make a point) you don't want to see
20 different hits for the same doc. If you have this problem, you may
want our patch for Ht://Dig which allows you to de-dup hits based on a
unique identifer in the CGI params.

The hard part is making the UI good enough so that the categorization
structure makes sense to the end user.

> > > If so, then we much
> > > exhaustively generate appropriate categories. Then the question is, can
> > > our indexing technology be smart enough to figure out from the
> > > documentation the categories into which the documentation falls? And what
> > > if we further subdivide the content of a single man page? Does the indexer
> > > do a good job with that as well? I'd love to see some suggestions on what
> > > categories we should use.
> >
> > So would I :) Alas, give X people 100 documents to categorize, and you'll
> > probably get X different ways to categorize them. What I would rather see
> > is categorization guidelines (methodology?) like I showed above. Something
> > that tells you how to go about categorizing content rather than a fixed
> > structure that works today, but not tomorrow.
> >
>
> Make it extensible, and it will work tomorrow, too. I'd have to see what
> sort of guidelines, specifically, before I could agree that that's all you
> need.

Wish I had some :)

> <snip>
>
> > > Mention was made earlier about categorization in XML being done in
> > > comments. This defeats the purpose of putting it in XML. The tags,
> > > attributes and values need to be part of the ultimate XML document in
> > > order to be of value in indexing.
> >
> > Depends on if your repository is XML aware. If your repository/search
> > engine chokes on XML (which is most of them) then it doesn't matter if
> > categorization info is in native XML or in comment tags as the repoistory
> > engine will just parse the raw text of the document to find what it's
> > looking for.
> >
>
> If that's the case, you might as well put all the docs in straight text
> files. The idea should be to get the document to tell you what's important
> in it, and how it should be categorized.

I wouldn't go that far. Many HTML search engines can take advantage of
META tags in HTML.

> <snip>
>
> > > Nothing's worse than calling "man
> > > foo" and finding out that GNU has decided they don't want to maintain the
> > > man pages anymore, and that the real up-to-date stuff is in info. Argh. I
> > > don't like the info format anyway. Just put it all in XML or SGML or
> > > something, and let the user access all of it that way. And if they want to
> > > type "man foo", then what they get is Lynx handing them an XML page for
> > > what they asked for. Of course, I'd like to see most Linux docs
> > > completely rewritten so that one doesn't need three PhDs from four
> > > universities to understand it. Too radical, I know. </rant>
> >
> > The issues of man/texinfo shouldn't effect us as we will be maintaining
> > one master source which will then be converted in the alternate formats.
> > Hence, if you're looking at the HTML or plain text or native format, they
> > should all be at the same "version".
> >
>
> Hmm. That may be the crux of the problem. Is that what this project is
> about? That seems impractical to me, because of the example I gave
> above. Joe will update his tarball with his new man page, and you'll need
> to be sure you know when it's been updated, so you can translate it into
> your format and put it in your repository. At that point, versioning
> _does_ become important.

I don't know if Joe will stop writing man pages and start writing ODE.
If he does then great. If not, then nothing we can do about it.

> This sounds like LinuxKB to me. You're talking about all docs in one
> format, then converted to others for use. But if you have it all in this
> cool format, why convert it? Just build or find a program that uses it in
> its native format and forget about man pages and such altogether.

Actually the LinuxKB is about converting all docs from their native format
to html. This actually sucks because it limits the power of the site, but
since there is not a standard, we're SOL. The ODE is about the opposite,
getting all docs writing in the same format natively. At least that's the
way I see it. Now if everyone uses the ODE, then that's really cool for
the LinuxKB because we can take advantage of those standards (like the
keywords META tag in HTML) to make the site more powerful.

Still though, it's good to convert ODE to something like man, because
everyone has a man page viewer today and will have one for a long long
time. Not everyone will have a ODE viewer for some time though, and it
isn't likely to ever run on lower end systems very well.

> Again, this looks like the difference between docs as viewed by LinuxKB
> and local docs as viewed by a local user on his local machine. Which
> approach is ODE looking at? Or both?

I would assume both. Not doing both is rather dumb IMHO. I'm only
advocating that whatever we do, we do our best to support remote
centralized repositories like the LKB.

--
Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization  http://linuxkb.org/
Because world domination requires quality open documentation.
aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
The difference between `Unstable' and `Usable' is only two characters: NT



This archive was generated by hypermail 2.0b3 on Thu Jan 13 2000 - 21:27:21 EST