Miles Efron (mefron@metalab.unc.edu)
Fri, 14 Jan 2000 13:58:44 -0500 (EST)
> The trick is getting a critical mass of people to index and give them
> easy/powerful tool to do the work. I'm working on such a tool now.
Towards this, several folks here at the UNC Open Source Research Team have
been working on a metadata schema and associated tools for Open Source
Documentation. The tools are still pretty rough, but I'd like to hear any
comments etc. earlier rather than later...so here's a little description
of what we're doing:
With Kendall Clark, Ismael Olea, and Victor Ruiz the UNC Open Source
Research Team (http://metalab.unc.edu/osrt) is working on a Dublin
Core-based metadata format to improve access (especialy retrieval and
sorting) to open source software documentation. The idea is that the
metadata conform to known standards, be somewhat independent of the
documents they describe (like say card catalogue records), be simple
enough to be filled out by resource authors (like LSMs), be easily shared,
and enhance searching and retrival of relevant documents. In theory,
someone could do a template based search (something like
http://metalab.unc.edu/linsearch/) and find a How-to on modems in Polish,
or all the documen
ts written by a
given author, etc.
Towards this, we have built a prototype of a template with which authors
may create and edit OpenDoc records
(http://metalab.unc.edu/osrt/ldpcore/template.html). This template is
intended to feed a database of OpenDoc records which can then be used for
resource discovery or arrangement. Keep in mind--the template is not the
only way into this database: authors may submit records singly or in batch
via ftp. Since OpenDoc records exist separately from the resources they
describe, the contents of this database would likely point to resources
scattered all over.
Please feel free to take a look at what we've done so far. As our project
is still in a provisional state, I am really eager to hear any questions
or comments about it. If people are interested in lending a hand, all the
better.
Cheers.
-Miles
__________________________________
Miles Efron
Linux Archivist, Harmless Drudge
mefron@metalab.unc.edu
On Thu, 13 Jan 2000, Aaron Turner wrote:
> On Thu, 13 Jan 2000, Paul M. Foster wrote:
> >
> > On Wed, 12 Jan 2000, Aaron Turner wrote:
> >
> > > On Tue, 11 Jan 2000, Paul M. Foster wrote:
> > >
> > > > On Tue, 11 Jan 2000, Kim Lester wrote:
> > > > >
> > > > > ODE Summary Report #1
> > > > > =====================
> > > >
> >
> > <snip>
> >
> > > > Much more cumbersome. I presume the idea is the develop system(s) that
> > >
> > > Cumbersome? That's an interesting statement. In what way is it so? From
> > > the end user perspective our the repository maintainers perspective?
> >
> > Repository maintainer's perspective. I was talking about allowing email
> > threads and such to be part of the indexed documents.
>
> Well, yeah, that does complicate things :) But IMHO it's worth it. Not
> every repository has to do it the same way, so they can do what their
> skill/time allows.
>
> > > > works for both areas, perhaps slightly different ones for each area. My
> > > > comments will relate to building such a system for local machines, running
> > > > on their machines.
> > >
> > > I won't assume to speak for other groups, but this isn't something the
> > > LinuxKB had in mind. We're very hard-core central-web repository. It
> > > just so happens that some of the internal tools we will develop to
> > > maintain the site also have certain applications for local content
> > > distribution as on a CDROM. We however have no current plans to provide
> > > any tools to allow users to search that local content.
> > >
> >
> > Yeah, I'm on the LinuxKB list too. I understand that's not your
> > perspective. However, I think this whole project here is also valuable
> > from the viewpoint that you could put the engine/software/whatever on the
> > local system and then the user would have _easy_ access to the docs,
> > instead of what we have now. Naturally, LinuxKB's engine wouldn't fit/work
> > locally, but perhaps we could come up with something that would.
>
> I'd love to see someone else do that.
>
> > > > Next, do we plan to have any humans do any indexing on existing documents,
> > > > or do we want the "indexer" program to be smart enough to figure all this
> > > > out?
> > >
> > > I've thought about, dreamed of, and racked my brain trying to figure out
> > > how the hell to do that. I've come up empty. What you're talking about
> > > literally is an AI that can do context based word matching and ranking.
> > > Yes this sort of thing has been done. No I don't have a clue how they do
> > > it. And my guess is that the human brain will do it better for the
> > > foreseeable future. And there is no reason that the review/edit process
> > > can't include categorization which would be a part of the doc spec IMHO.
> > >
> >
> > I agree humans would be better. But considering the amount of
> > documentation to sift through, I'm not sure having humans do it is
> > practical. It's pretty easy to point an indexer at a bunch of docs and say
> > "index it" and get back keywords and such. It's quite another to
> > categorize it according to some more esoteric scheme.
>
> The trick is getting a critical mass of people to index and give them
> easy/powerful tool to do the work. I'm working on such a tool now.
>
> > > > What should our favored format be? Obviously, opinions vary. However,
> > > > since we're trying to index information, and since presentation should be
> > > > less important, I'd vote for XML. It is specifically designed to allow
> > > > indexing of information, and has incredible flexibility. The writers of
> > > > documentation needn't worry with what format we use, since we are
> > > > translating from their format into ours. Real issue is what will make our
> > > > indexing job the easiest? I'm inclined to think XML. And forget standards
> > > > for XML. You actually don't need a DTD for XML, but we can come up with
> > > > one if needed. After all, this is our internal format. Now, if you want
> > > > others to write to this format, you'll have to make the DTD simple.
> > >
> > > Indexing is and likely always will be a manual process. Both SGML and XML
> > > provide markup to allow you to specifiy the category in a way that a
> > > program can extract it. Hence from a indexing aspect XML and SGML are on
> > > equal footing. However, the tool set to work with SGML is much more rich
> > > which makes me give it a thumbs up over XML.
> > >
> >
> > You're probably right, and that seems to be the concensus: SGML > XML. I
> > don't quite know what DocBook is, but from what I understand it is a
> > certain DTD under SGML. Yes?
>
> Yep. You can buy or download the Orielly DocBook manual.
>
> > Here's the other problem with SGML/Docbook. If you want this to be served
> > up in a browser, you'll get a lot more support for doing it in XML than
> > SGML. Browsers are starting to support XML, but I don't know that they'll
> > ever support SGML or Docbook. And as far as I can see, the browser is
> > probably the ideal interface for this stuff.
>
> Yep. Right now though, whatever standard you choose you'll need to
> convert to HTML for the browsers. Long term, once the tools are there,
> I'd love to see XML, as I really think it's the future.
>
> > > > How should the user access the information? I think the standard
> > > > technology most widely available today is the browser. It works fairly
> > > > well for this purpose and is pretty flexible. That way we are not building
> > > > yet another tool. That said, we still will need some sort of cgi or other
> > > > back end that will eat one format and turn it into another. This also
> > > > means that we will need to have a [thin] document server running on the
> > > > local machine.
> > >
> > > You shouldn't require some daemon on your local system to read
> > > documentation IMHO. If you can't export your format to HTML or plain text
> > > you are wasting your time. Don't expect to have a mini-knowledge-base
> > > running locally on people's system for sometime if ever.
> > >
> >
> > Well, here's the problem. Joe creates program barfoo, and makes his little
> > man page for it. (And for the sake of argument, let's say the program is
> > quite old but is being regularly maintained.) When Joe updates his
> > program, he updates his man pages as well. He sticks it up on Sunsite as a
> > tarball. Now, the maintainers of RedHat go out and get his program, make a
> > spec file for it, and wrap it in an RPM, and it goes on their distro.
> >
> > But let's say that everyone decides docbook is the way to go. Is Joe going
> > to change his documentation to docbook? Unlikely. Is Red Hat going to take
> > the time to translate his man pages into docbook before building the
> > RPM? Also unlikely. Which means that if you want to provide the distros
> > with a tool to allow them to serve up all the docs in a single way on
> > someone's machine, sooner or later you're going to have to build a program
> > that goes out and reads those man files (and HOWTOs and info files,
> > etc.) and builds some sort of a database or something of them, indexes
> > them, and provides a way for a browser to serve them up.
>
> I don't think what we do here will ever replace man pages on Unix, so I'm
> less sure about the need for this. There are already man2html converters,
> so making the man page available to a browser is a simple task.
>
> > Again, this isn't something LinuxKB would have much to do with per se.
>
> Agreed.
>
> > <snip>
> >
> > > > It might be instructive to look at the way documentation is handled for
> > > > the Debian Linux distribution. If you choose, you can install programs
> > > > that allow the search and retrieval of all documentation on the system
> > > > via Apache, Lynx and such. It is _similar_ to what we are talking about,
> > > > but extremely primitive. For instance, you really need to specify whether
> > > > you want to view information from the man files, info files or HOWTOs in
> > > > order to find what you want. Mostly, their solution is a backend that uses
> > > > off-the-shelf tools to convert man, info and other docs into HTML. As I
> > > > said, very primitive, but it works.
> > >
> > > What you're talking about is specifying in the doc spec a "document type"
> > > and allowing users to limit their queries by type. If you design your
> > > system well this is easy.
> >
> > No no. The way Debian does it, it works out that if you want to look at
> > something contained in the HOWTOs, you have to go into the HOWTO section
> > of the documentation. But you may not know whether what you want is in the
> > HOWTOs or not. That's one reason why the Debian system is klunky. The user
> > shouldn't have to specify the source for answering his question.
>
> I think we're miss-communicating. I'm saying allow them to limit queries
> by the type to reduce the number of hits, not seperate the content so they
> can't do a global search against all docs if they don't have a clue where
> to look.
>
> > <snip>
> >
> > > > One thing sorely lacking in Linux distros, which was commented on in
> > > > previous posts, is the ability to ask, "What programs can I use to look at
> > > > my mail with?" And "what does X program in Y directory do?" Quite often,
> > > > these programs don't even have a doc file of any kind with them.
> > >
> > > What you're talking about is the ability to search for categories and
> > > applications in those categories, not actual documents. Again, assuming
> > > the correct implimentation of the backend, easy. (BTW, I say it's easy
> > > becuase we've already figured it out how to do that. YMMV)
> > >
> >
> > Actually, this is kinda handled by the distros themselves when they
> > install. They have packages grouped in categories at install time. I just
> > don't know if the file detailing this hangs around on the system
> > afterward.
>
> Unfortunately in my experiance the categories are pretty bad. Maybe I'm
> just dumb and always look in the wrong place, but I've heard other people
> complain as well. Perhaps using the RPM category would be a good starting
> point or as the "offical home" for docs that are in multiple places in the
> tree.
>
> > > > Is the plan to index things via some hierarchy, like the way man files are
> > > > laid out, or the way the Dewey Decimal System works?
> > >
> > > Either and/or neither. A flexible categorization system should categorize
> > > documentation in various ways. Perhaps by user function (admin, end user,
> > > developer, etc), by document type (kernel doc, man page, software readme,
> > > etc), or by whatever structure tickles your fancy.
> > >
> >
> > Yeah, but you have to set up categories so your humans or indexing
> > programs know how to do the indexing at the outset. I don't think we need
> > to be all that flexible as long as we are thorough. Again, the Dewey
> > Decimal system. Not perfect, but a very good start. (Not that we should
> > use this. It's just an example of a good categorization system.)
>
> Indexing programs just follow links. If you have the same doc in 20
> different places (I'm exagerating to make a point) you don't want to see
> 20 different hits for the same doc. If you have this problem, you may
> want our patch for Ht://Dig which allows you to de-dup hits based on a
> unique identifer in the CGI params.
>
> The hard part is making the UI good enough so that the categorization
> structure makes sense to the end user.
>
> > > > If so, then we much
> > > > exhaustively generate appropriate categories. Then the question is, can
> > > > our indexing technology be smart enough to figure out from the
> > > > documentation the categories into which the documentation falls? And what
> > > > if we further subdivide the content of a single man page? Does the indexer
> > > > do a good job with that as well? I'd love to see some suggestions on what
> > > > categories we should use.
> > >
> > > So would I :) Alas, give X people 100 documents to categorize, and you'll
> > > probably get X different ways to categorize them. What I would rather see
> > > is categorization guidelines (methodology?) like I showed above. Something
> > > that tells you how to go about categorizing content rather than a fixed
> > > structure that works today, but not tomorrow.
> > >
> >
> > Make it extensible, and it will work tomorrow, too. I'd have to see what
> > sort of guidelines, specifically, before I could agree that that's all you
> > need.
>
> Wish I had some :)
>
> > <snip>
> >
> > > > Mention was made earlier about categorization in XML being done in
> > > > comments. This defeats the purpose of putting it in XML. The tags,
> > > > attributes and values need to be part of the ultimate XML document in
> > > > order to be of value in indexing.
> > >
> > > Depends on if your repository is XML aware. If your repository/search
> > > engine chokes on XML (which is most of them) then it doesn't matter if
> > > categorization info is in native XML or in comment tags as the repoistory
> > > engine will just parse the raw text of the document to find what it's
> > > looking for.
> > >
> >
> > If that's the case, you might as well put all the docs in straight text
> > files. The idea should be to get the document to tell you what's important
> > in it, and how it should be categorized.
>
> I wouldn't go that far. Many HTML search engines can take advantage of
> META tags in HTML.
>
> > <snip>
> >
> > > > Nothing's worse than calling "man
> > > > foo" and finding out that GNU has decided they don't want to maintain the
> > > > man pages anymore, and that the real up-to-date stuff is in info. Argh. I
> > > > don't like the info format anyway. Just put it all in XML or SGML or
> > > > something, and let the user access all of it that way. And if they want to
> > > > type "man foo", then what they get is Lynx handing them an XML page for
> > > > what they asked for. Of course, I'd like to see most Linux docs
> > > > completely rewritten so that one doesn't need three PhDs from four
> > > > universities to understand it. Too radical, I know. </rant>
> > >
> > > The issues of man/texinfo shouldn't effect us as we will be maintaining
> > > one master source which will then be converted in the alternate formats.
> > > Hence, if you're looking at the HTML or plain text or native format, they
> > > should all be at the same "version".
> > >
> >
> > Hmm. That may be the crux of the problem. Is that what this project is
> > about? That seems impractical to me, because of the example I gave
> > above. Joe will update his tarball with his new man page, and you'll need
> > to be sure you know when it's been updated, so you can translate it into
> > your format and put it in your repository. At that point, versioning
> > _does_ become important.
>
> I don't know if Joe will stop writing man pages and start writing ODE.
> If he does then great. If not, then nothing we can do about it.
>
> > This sounds like LinuxKB to me. You're talking about all docs in one
> > format, then converted to others for use. But if you have it all in this
> > cool format, why convert it? Just build or find a program that uses it in
> > its native format and forget about man pages and such altogether.
>
> Actually the LinuxKB is about converting all docs from their native format
> to html. This actually sucks because it limits the power of the site, but
> since there is not a standard, we're SOL. The ODE is about the opposite,
> getting all docs writing in the same format natively. At least that's the
> way I see it. Now if everyone uses the ODE, then that's really cool for
> the LinuxKB because we can take advantage of those standards (like the
> keywords META tag in HTML) to make the site more powerful.
>
> Still though, it's good to convert ODE to something like man, because
> everyone has a man page viewer today and will have one for a long long
> time. Not everyone will have a ODE viewer for some time though, and it
> isn't likely to ever run on lower end systems very well.
>
> > Again, this looks like the difference between docs as viewed by LinuxKB
> > and local docs as viewed by a local user on his local machine. Which
> > approach is ODE looking at? Or both?
>
> I would assume both. Not doing both is rather dumb IMHO. I'm only
> advocating that whatever we do, we do our best to support remote
> centralized repositories like the LKB.
>
> --
> Aaron Turner, Core Developer http://vodka.linuxkb.org/~aturner/
> Linux Knowledge Base Organization http://linuxkb.org/
> Because world domination requires quality open documentation.
> aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
> The difference between `Unstable' and `Usable' is only two characters: NT
>
>
This archive was generated by hypermail 2.0b3 on Fri Jan 14 2000 - 14:00:20 EST