Dissertation Information for Robert Stafford Hazelton
- Robert Stafford Hazelton
- Library and Information Science
- Case Western Reserve University (USA) (1972)
- William Goffman
- Conrad H. Rawski
- Phyllis A. Richmond
- Prosanta K. Saha
MPACT Status: Fully Complete
Title: Underlying aspects of word frequency distributions
Abstract: Relevance is at the heart of any information retrieval system. The success or failure of a retrieval system can be measured in part by how relevant the retrieved documents are to the queries. In any system employing automated assignment of index terms, the assignment must adequately describe what the document is about if searching based on these terms is to be successful. One method of automatic indexing is based on the word frequency distributions of texts. The interrelationships are revealing with regard to the frequency distributions.
Relevance can be understood as aboutness. Nelson Goodman provided one formalization of absolute aboutness which is intuitively acceptable. One result which can be proved -- based on Goodman's formalization -- is that if a statement K is about X then -K is also about X. Negation in English can be implemented in many ways, but perhaps the simplest approach is to write 'It is not the case that K' for the negation of K. This procedure has the property that it can be applied recursively. By repeated applications one may create a large set of statemments all about X such that none of the documents has the same word frequency distribution as any other in the set.
Aboutness is seen to specify a range, not make positive or negative assertions about elements in the range. Because of this it is possible to persorm reasonable indexing by title analysis without performing any syntactic analysis of the titles. One can defeat local analysis indexing schemes by employing contorted or unexpected grammatical constructions.
Six documents ar epresented all having the same word frequency distribution. These documents were indexed by a set of indexers. Statistical tests performed on the indexes were from the same population. Coupled with the earlier frequency result, this indicates that word frequency distrbution is neither necessary nor sufficient to predict what a document is about.
A Guttman-Lingoes smallest space analysis was perdormed on the indexes. This non-metric analysis of proximities showed that the major classfying factor was the number of index terms assigned to a given document. This corresponded closely to the acceptability of the documents. One can then conclude that a theory of linguistic performance is required to explain the appropriate use of frequency distributions. Becuase of this-- it is likely that the basic biological communication processes are most likely to yield useful information about word frequency distribution.