Dissertation Information for Louise Su
- Louise Su
- Library and Information Science
- Rutgers University (USA) (1991)
- Nicholas Belkin
- Belver Griffith
- Daniel O'Connor
- Tefko Saracevic
- Brent D. Ruben
MPACT Status: Fully Complete
Title: An investigation to find appropriate measures for evaluating interactive information retrieval
Abstract: There is no clear guidance in IR literature about how to evaluate interactive information retrieval performance. Several criteria and measures have been proposed and used in evaluation. There is no agreement about what is a successful IR performance or which are the best existing evaluation measures. This study aims to identify appropriate measures for evaluating interactive IR performance, i.e. evaluative measures that can best reflect system's success in providing help for users' needs or problems.
Twenty measures of IR performance were selected for study, representing four major evaluation criteria: relevance, utility, user satisfaction and efficiency. Four non-performance characteristics were also selected for subgroup analysis. User's judgment of system success in realistic IR situation was used as the devised criterion measure with which all other 20 measures were to be correlated.
Six searchers and 40 end-users were recruited from Rutgers Online Automatic Retrieval Service. A computer search was conducted by a searcher in the normal manner, for each individual problem, in the user's presence and with his/her participation. The end-users were each responsible for the cost of his/her own search.
This study showed that value of search results as a whole is the best single measure of interactive IR performance among the 20 measures selected. It is strongly correlated with other significant correlates of success: user's satisfaction with completeness of search results, and user's satisfaction with precision of the search. Precision, one of the most important traditional measures of effectiveness, was found to be not significantly correlated with success. Users appeared to be more concerned with absolute recall than with precision of the search.
The study also identified two more basic factors which can account for much higher proportion of the total variance than value of search results as a whole alone can. The study suggested 17 new success categories for further investigation. Open issues and future research are also discussed.
MPACT Scores for Louise Su
A = 0
C = 7
A+C = 7
T = 0
G = 0
W = 0
TD = 0
TA = 0
calculated 2008-01-31 06:10:01
Advisors and Advisees Graph