Re: ODE Summary Report #1


Aaron Turner (aturner@linuxkb.org)
Wed, 12 Jan 2000 08:54:40 -0800 (PST)


On Tue, 11 Jan 2000, Paul M. Foster wrote:

> On Tue, 11 Jan 2000, Kim Lester wrote:
>
> >
> > ODE Summary Report #1
> > =====================
>
> <snip of excellent summary>
>
> Rather than quoting the prior post and responding point by point, I'll
> just weigh in. No, I'm not associated with any of the doc groups
> mentioned. Just a concerned Linux user. Those of you not in the Linux
> world will hopefully not take offense if I couch things in terms of Linux.
>
> First off, there are two types of ultimate end results here, as I see it.
> First, there is the software that will sit on someone's computer and
> hopefully be shipped with distros to assist the user in finding what he
> wants. Second there is the "web repository" idea. Different thing that
> might encompass not only man pages and such, but also email threads, etc.

I think a lot of people are from a personal perspective only interested in
standardizing the actual documents. I'm not saying that distribution of
content isn't important, but that there appears to be a significant body
of people who don't care that much (mostly out of lack of time I would
expect).

> Much more cumbersome. I presume the idea is the develop system(s) that

Cumbersome? That's an interesting statement. In what way is it so? From
the end user perspective our the repository maintainers perspective?

> works for both areas, perhaps slightly different ones for each area. My
> comments will relate to building such a system for local machines, running
> on their machines.

I won't assume to speak for other groups, but this isn't something the
LinuxKB had in mind. We're very hard-core central-web repository. It
just so happens that some of the internal tools we will develop to
maintain the site also have certain applications for local content
distribution as on a CDROM. We however have no current plans to provide
any tools to allow users to search that local content.
 
> Next, do we plan to have any humans do any indexing on existing documents,
> or do we want the "indexer" program to be smart enough to figure all this
> out?

I've thought about, dreamed of, and racked my brain trying to figure out
how the hell to do that. I've come up empty. What you're talking about
literally is an AI that can do context based word matching and ranking.
Yes this sort of thing has been done. No I don't have a clue how they do
it. And my guess is that the human brain will do it better for the
foreseeable future. And there is no reason that the review/edit process
can't include categorization which would be a part of the doc spec IMHO.
 
> What should our favored format be? Obviously, opinions vary. However,
> since we're trying to index information, and since presentation should be
> less important, I'd vote for XML. It is specifically designed to allow
> indexing of information, and has incredible flexibility. The writers of
> documentation needn't worry with what format we use, since we are
> translating from their format into ours. Real issue is what will make our
> indexing job the easiest? I'm inclined to think XML. And forget standards
> for XML. You actually don't need a DTD for XML, but we can come up with
> one if needed. After all, this is our internal format. Now, if you want
> others to write to this format, you'll have to make the DTD simple.

Indexing is and likely always will be a manual process. Both SGML and XML
provide markup to allow you to specifiy the category in a way that a
program can extract it. Hence from a indexing aspect XML and SGML are on
equal footing. However, the tool set to work with SGML is much more rich
which makes me give it a thumbs up over XML.

> How should the user access the information? I think the standard
> technology most widely available today is the browser. It works fairly
> well for this purpose and is pretty flexible. That way we are not building
> yet another tool. That said, we still will need some sort of cgi or other
> back end that will eat one format and turn it into another. This also
> means that we will need to have a [thin] document server running on the
> local machine.

You shouldn't require some daemon on your local system to read
documentation IMHO. If you can't export your format to HTML or plain text
you are wasting your time. Don't expect to have a mini-knowledge-base
running locally on people's system for sometime if ever.
 
> FWIW, I'd request that we make things Lynx-friendly.

I would agree.

> It might be instructive to look at the way documentation is handled for
> the Debian Linux distribution. If you choose, you can install programs
> that allow the search and retrieval of all documentation on the system
> via Apache, Lynx and such. It is _similar_ to what we are talking about,
> but extremely primitive. For instance, you really need to specify whether
> you want to view information from the man files, info files or HOWTOs in
> order to find what you want. Mostly, their solution is a backend that uses
> off-the-shelf tools to convert man, info and other docs into HTML. As I
> said, very primitive, but it works.

What you're talking about is specifying in the doc spec a "document type"
and allowing users to limit their queries by type. If you design your
system well this is easy.

> Do you want to be able to search the Red Hat package database as
> well? Like, "what package is file X in?"

Already exists today.
 
> One thing sorely lacking in Linux distros, which was commented on in
> previous posts, is the ability to ask, "What programs can I use to look at
> my mail with?" And "what does X program in Y directory do?" Quite often,
> these programs don't even have a doc file of any kind with them.

What you're talking about is the ability to search for categories and
applications in those categories, not actual documents. Again, assuming
the correct implimentation of the backend, easy. (BTW, I say it's easy
becuase we've already figured it out how to do that. YMMV)

> Is the plan to index things via some hierarchy, like the way man files are
> laid out, or the way the Dewey Decimal System works?

Either and/or neither. A flexible categorization system should categorize
documentation in various ways. Perhaps by user function (admin, end user,
developer, etc), by document type (kernel doc, man page, software readme,
etc), or by whatever structure tickles your fancy.

> If so, then we much
> exhaustively generate appropriate categories. Then the question is, can
> our indexing technology be smart enough to figure out from the
> documentation the categories into which the documentation falls? And what
> if we further subdivide the content of a single man page? Does the indexer
> do a good job with that as well? I'd love to see some suggestions on what
> categories we should use.

So would I :) Alas, give X people 100 documents to categorize, and you'll
probably get X different ways to categorize them. What I would rather see
is categorization guidelines (methodology?) like I showed above. Something
that tells you how to go about categorizing content rather than a fixed
structure that works today, but not tomorrow.
 
> As I see it, we're looking at a program or suite of programs that are
> installed on a system, which then go out and index the documentation on
> the system. Perhaps a cron job. And when the user asks about "foo" and
> there is no documentation nor any such program on the computer, the cgi or
> whatever says, "Sorry, you might want to search on the web...."

Maybe some other group would be interested in a local document retrieval
engine like you describe, but I personally find that a much harder and
less powerful solution. You either end up requiring the user to install a
ton of support programs to power the features you want or give them a
simple/stripped down/barely useful at all system that is easy to install.
Maybe I'm over simplifying things, but that's how I see it- I can build a
much more complicated system on a dedicated server than try to make the
same system work on the 10 million different Linux systems around the
world.
 
> Do we plan to include things like the kernel docs in our indexing? That
> is, do we plan to include some of the more arcane, less well-known
> documentation sources in our indexing?

This is more of a repository question than open standards question, so I
expect the answer is that it depends on the repository. Some sites like
the LinuxKB will, others (like the LDP) are less likely. Of course if the
Linux kernel doc maintainers adopt our doc standard, then that is likely
to change. (Any kernel coders listening???)

> Mention was made earlier about categorization in XML being done in
> comments. This defeats the purpose of putting it in XML. The tags,
> attributes and values need to be part of the ultimate XML document in
> order to be of value in indexing.

Depends on if your repository is XML aware. If your repository/search
engine chokes on XML (which is most of them) then it doesn't matter if
categorization info is in native XML or in comment tags as the repoistory
engine will just parse the raw text of the document to find what it's
looking for.

> Language may not be that much of an issue. User defines an environment
> variable that specifies his native language, and we serve it to him in
> that language if it's available.
>
> Don't know how much importance versioning has in all this. If the user is
> using fetchmail 5.0, he probably will not care what fetchmail 4.2
> did. He/she will only be interested in the documentation for what's on his
> system. Maybe I misunderstand what is meant by versioning here.

That is a vaild point. I believe though that versioning was refering to
revision control of the documents themselves.
 
> Personally, I'd like to see all the docs distributed in ONE format and
> skip all the goofy alternate formats.

Reality is that whatever format we choose (cept for plain text) someone,
somewhere won't be able to read it. When you're talking about
documentation, you often have to play to the least common denominator.

> Nothing's worse than calling "man
> foo" and finding out that GNU has decided they don't want to maintain the
> man pages anymore, and that the real up-to-date stuff is in info. Argh. I
> don't like the info format anyway. Just put it all in XML or SGML or
> something, and let the user access all of it that way. And if they want to
> type "man foo", then what they get is Lynx handing them an XML page for
> what they asked for. Of course, I'd like to see most Linux docs
> completely rewritten so that one doesn't need three PhDs from four
> universities to understand it. Too radical, I know. </rant>

The issues of man/texinfo shouldn't effect us as we will be maintaining
one master source which will then be converted in the alternate formats.
Hence, if you're looking at the HTML or plain text or native format, they
should all be at the same "version".

[snip project list]

> I assume that we don't just intend to sit around and build standards. I
> assume that we intend to actually do some coding at some point. Yes?
>
> If we want this thing to take forever, we should be obsessive about
> obtaining everyone's agreement. Otherwise, let's just build a better
> mousetrap, get most people to agree, and let the rest do as they like
> (they'll probably come around anyway).
>
> I know, this will ruffle some feathers. But look, I head a local Linux
> Users Group and own my own business, and I know what it's like to get
> everyone to agree. Forget it. On the other hand, look at things like W3C,
> where they don't ask a lot of people what they think. Yet their standards
> stick. Democracy's overrated. Most open source software was developed (and
> quite successfully) by single individuals who didn't really care if anyone
> agreed with them or not. So let's have a tight group that can make
> decisions and act quickly and decisively. Let's get as much agreement as
> possible, but when the inevitable impasse results, we skip the agreement
> part and just _decide_. Rightness and correctness have very little to do
> with agreement anyway.

I couldn't agree more. There is no way we will ever get everyone to
agree. The best we can offer is majority rule. Yes this requires more
management and some kind of organization. Maybe I'm wrong, but I don't
think that a loose-nit group of people are going to define a standard
without a way to work out differences where two or more groups can't
agree. But of course we can always give it a try. Always easier to
create the structure than to destroy it later.

--
Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization  http://linuxkb.org/
Because world domination requires quality open documentation.
aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
The difference between `Unstable' and `Usable' is only two characters: NT



This archive was generated by hypermail 2.0b3 on Wed Jan 12 2000 - 11:56:04 EST