Introduction to WAIS the Wide Area Information Server Peter Marshall, Manager: Academic Networking The University of Western Ontario Computing and Communications Services 1 Outline o WAIS Concepts o What's Out there? o WAIS at UWO o Searching with SWAIS o Setting up Databases o Problems o Future o For Further Information 2 Concepts: Overview 3 Concepts: Implementations o Clients - Unix * shell commands * curses-based screens * X-windows - MS-DOS: packet-drivers (2) - Windows - Macintosh - VMS: Wollongong; Multinet; Digital's Ultrix Connection - "gopher"s of all kinds! o Servers - Unix systems (all kinds) - Connection Machines - VMS - Macintosh (soon?) 4 Concepts: Who? o over 335 servers o more than 6,000 hosts o greater than 10,000 direct users o plus gopher gateway users o world-wide - USA, Canada - UK, France, Germany, Switzerland - Australia, New Zealand - Israel o (soon) major library catalogue software vendors 5 What's Out There? o An index to journalism periodicals: Does the reporting of John Crosby's speeches pose an ethical dilemma? o ERIC (Educational Resources Information Center) Digests: Does taking notes help students in information reten- tion? o Communications of the ACM (full-text): Show me that recent article on user interfaces to computer programs. o The Gutenberg Project archives: How about checking out a copy of Bront"e's Wuthering Heights? o Poetry archives: Find me a poem about roses. o Human DNA sequences (updated as they are discovered). o Weather satellite pictures (updated every hour): Show me the current ground conditions in North America. 6 o The KJV of the Bible: Who was the guy with all the boils? o The Koran: Is the Garden of Eden mentioned? o FTP'able README files: Where would I find a program that converts scribe to TeX? WAIS at UWO o Still experimental... - mostly unix: /ccs/export to sun3/4, mips - no VMS client at CCS - some trials with PC and Mac clients o Local databases: - Index to Journalism Periodicals - Change Notices - Newsletters - Personal E-Mail 7 Index to Journalism Periodicals o Bibliographic index: 40 journals on jour- nalism o Maintained by UWO Graduate School of Journalism o 10 years cumulative data: >15,000 entries o Fiche sold widely o Experimental WAIS index: Widely used but...Will anyone pay? o Used locally by GSOJ students o other GSOJ databases possible: LFP, re- search papers o changing the update frequency: 6 months - ! 1 month 8 Change Notices o help coordinate changes, spot problems o local usenet news group o poor retrieval mechanisms o tried mh/xmh folders o WAIS much better o example o On-line maintenance manual - Changing the way changes notices are written 9 Other UWO Databases o Newsletters - CCS's Focus, InfoCentre's Info Ex- change - Campus Computer Store Price list - CA*Net Newsletter - LinkLetter o Personal E-Mail 10 Other Possible Indexes o Senate and Board of Governors minutes. o Campus newspapers: student and admin- istration. o Restaurant and movie reviews. o Course outlines and calendars. o Departmental minutes (using a private server). o Usage write-ups Unix man pages and local manuals. o Library catalogues. o Frequently Asked Questions. - from usenet group news.answers - local FAQ one for/with the help desk 11 Searching with swais English, no Venn, no booleans _ easy? 1. Start up the querying program: swais 2. Choose database(s): cursors and t 3. Enter search terms: w, ..., - 4. View ranked "headlines" 5. Choose documents for retrieval: cursors 6. View or save document: t, m or _ 12 Exercise # 1: getting started 1. Start up the swais program. (Type swais-.) 2. Select the journalism.periodicals database for searching. (Move the cursor over it and press the space bar.) 3. Search using Bill Clinton and Gennifer Flowers and read a couple of the entries. (Type w and then the words above to the Keywords: prompt.) 4. Do another search using the keyword Bucharest. (Type w to get back to the Keywords: prompt, control-u to clear the line and then enter Bucharest -.) What do you get back? 5. Deselect the journalism.periodicals database. (Type s to get to the sources display. Move the cursor over the journalism entry and type a space.) 6. Exit from the SWAIS program. (Type q.) 13 Searching: Finding a Data Source o menu on local server? o directory-of-servers 1. initial search of directory-of-servers 2. choose appropriate database(s) 3. add them to your "personal" list: u (use) 4. use them in a search o cache, but no invalidation o database sources fairly stable 14 Exercise # 2: Add a Database 1. Make a directory to store pointers to databases in mkdir "/wais-sources 2. Find the new (1991) CIA world fact book and add it to your personal list of search- able WAIS databases. 15 Searching: Multiple Databases o local arbitration and sorting o coarse grained parallelism 16 Exercise # 3: Searching Multiple Databases 1. Search in the journalism.periodicals and one of the World Fact Book databases (1990 or 1991) for information about Sin- gapore. (Select both databases from the s (source select) screen, use w to get to the Keywords: prompt and enter Singapore.) 2. Mail an entry to yourself _ if you have e-mail. (Move the cursor to one of the returned entries and type the m command. Then enter you e-mail address at the Ad- dress: prompt.) 17 A Personal Database: e-mail 1. Save the e-mail messages that you want to keep in mail folder files. (We'll assume that they are in your ./mail directory.) 2. Path into ./mail and give the command waisindex -nocat -mem 40 -r -d my-mail -t mail * 3. Copy the resulting my-mail.src file into your directory ./wais-sources. Leave all the rest of the my-mail.* files around. 4. Use my-mail.src like any other WAIS database. 5. Periodically re-run the waisindex com- mand to update the indexes as you add more mail messages. 18 Setting up a Public Server 1. Find the data 2. (buy more disk) 3. Massage the data or waisindex 4. (buy even more disk!) 5. Run waisindex on the data. Use append- mode for updates 6. Edit .src file 7. *Set up a Z3950 service (usu. port 210) 8. *Add a line to /etc/inetd to run waisserver 9. *Log file management procedures 10. Register by sending the .src file to wais- directory-of-servers@quake.think.com 19 Problems with WAIS o Implementation Deficiencies - charging: charging your credit card - maximum number of documents re- turned - coding & recompile required to de- scribe a new data format - searching model imprecise and often not adequate * No natural language * no word stemming or variants * no booleans * poor relevance feedback implementa- tions o Architectural Problems - directory scaling - X.500? 20 The Future Bright! Directions: o More flexible and powerful searching: - stripping suffixes like "ing", "ed" and "s" for English databases - boolean searching - more powerful, cross-server relevance feedback - proximity search capabilities, phrasing - better understanding of natural lan- guage queries - non-word searching 21 The Future continued o improved index density for space savings o improved client implementations (espe- cially MS-DOS) o specialized, imbedded client applications o Commercial databases, charging and ac- cess control. $$$!!! o WAIS front-end ! PAT/Lector back- end o Multi-media: sound, video, graphics - music retrieval o compatible commercial Library catalog software 22 For Further Information o See Peter Marshall's paper WAIS: Wide Area Information Server or Anonymous What???: via anonymous ftp on julian as ftp/doc/wais-paper.ps.Z o WAIS bibliography: via anonymous ftp to think.com as wais/bibliography.txt o Newsgroups: comp.infosystems.wais (gen- eral discussion), uwo.comp.wais (develop- ers e-mail list) o See Roberto Sierra's WAIS & Music in wais-discussion archives o Check out gopher and WWW 23 Relevance Feedback: Narrowing the Search o Not really English queries o not really ORs o ranking o using relevant documents to find others o better than Booleans! 24