Harry Halpin
and
Johanna Moore
<H.Halpin@ed.ac.uk>
<J.Moore@ed.ac.uk>
School of Informatics
University of Edinburgh
Scotland UK
Hypothesis
The features of a text discovered via automatic event extraction can be used in both natural language understanding and advice generation in the domain of narrative instruction.Question
Does this extra amount of automatically induced structure provide some benefit against more traditional approaches that rely on simply word distribution?
These are investigated in the context of StoryStation, in order to create a fully automated plot analysis agent to improve the writing of students.
We investigate story rewriting task, where a story, the exemplar story, is read to the students, and afterwards the story is rewritten by each student, providing a corpus of rewritten stories.
This tests the students ability to both listen and write, while removing from the student the cognitive load needed to generate a new plot
.Stories are naturally composed on events. These events are composed of the interactions of characters with other entities in particular locales (Propp, 1968).
With recent advances in the accuracy of NLP components, we can now construct pipelines of NLP components that can automatically extract these events on a on a fine-grained level.
These events can then be used to analyze the story on a in general for the teacher, while also providing particular event-based advice for the student.
Work by Mueller (2004) has investigated using reasoning over event structures to understand complex stories. However, Mueller crafts his event representations by hand.
The WRITE system (Burstein et. al, 2003) analyzes student essays using Rhetorical Structure Theory. The errors in the story rewriting task are primarily informational, not intentional, so RST is unsuitable as the primary component.
Graesser and others (2000) used LSA to compare student stories, but this by itself can not provide the specific feedback needed.
The ATLAS system (Rose, 2002) is the most similar, but has as its domain physics stories and does not do non-domain specific event extraction.
.Our previous work was with a smaller corpus (100 stories) and did not test either domain portability or have an advice generation component (Halpin and Moore, 2004).
We collected a corpus of 290 stories from primary schools based on two different exemplar stories.
The stories are highly ungrammatical (only 35 percent could be parsed by a CCG-based parser) and contain many invented words (torlix) and the consistent usage of homonyms("there" for "their"). Many stories just one long run-on sentence.
Since the scale is ordinal, not nominal, we will not use the kappa statistic but instead use Cronbach's alpha and Kendall's taub statistics that penalize ranking errors correctly.
We used three raters, with one chosen as the gold standard. Absolute agreement was relatively low: (58% and 53%)
Taking into account the rating scale, almost all differences were just "one-level" differences.
Between raters there was an alpha statistic of .90 and .87 respectively, and a tau of .74 and .67 respectively. This shows the scheme to be reliable.
An event is simply a minimal subset of predicate argument structure without quantification, and an event structure is a partially-ordered series of events.
| Nils stays in Sweden | stay(t=1, Nils, Sweden) |
| and he is always playing with geese on his mountain. | play(t=1,Nils,geese, mountain) |
| He sees a bird | see(t=2, Nils, bird) |
Motivation: Need a measure of difference between the events in the rewritten story and exemplar story.
The algorithm iterates through the exemplar story's events looking for matches in each event of the rewritten story.
Events are matched by matching their components one at a time, starting with the event name and then matching entities.
Uses WordNet to find out if synonyms were used and uses a now-point to determine the order of events.
Motivation: Need a measure of difference between the events in the rewritten story and exemplar story.
The algorithm iterates through the exemplar story's events looking for matches in each event of the rewritten story.
Events are matched by matching their components one at a time, starting with the event name and then matching entities.
Uses WordNet to find out if synonyms were used and uses a ``now-point'' to determine the order of events.
| Exemplar | Rewritten |
| throw(t=1,Nils, coin) | toss(t=3,coin) |
| Algorithm Result: | synonym, out of order, no match, exact match |
The algorithm produces a feature vector of these results for each rewritten story by encoding these results as integers.
| Machine Learner | Corpus | Features | % Correct |
| k-NN | Adventure | LSA | 47.5 |
| Naive Bayes | Adventure | PLOT | 55.6 |
| k-NN | Thief | LSA | 41.2 |
| Naive Bayes | Thief | PLOT | 45.4 |
Performance comparable to human teachers on Adventure stories (55.6 to 56 percent agreement).
The system is portable across domains, although the performance clearly decreases on the Thief stories. We believe this is due to their increased length, complexity, and characters who are never identified with proper names, only descriptions.
Combining PLOT and LSA did not increase performance
System actually did not mislabel stories very often, and labelled poor stories remarkably correctly.
| Rating | Excellent | Good | Fair | Poor |
| Excellent | 14 | 22 | 0 | 1 |
| Good | 5 | 36 | 0 | 7 |
| Fair | 3 | 20 | 0 | 2 |
| Poor | 0 | 11 | 0 | 39 |
The generation of specific advice uses the results of the plot-comparison algorithm. The ten most frequent events in the corpus are chosen as the important events.
Search the rewritten story event structure for missing events and partial matches of predicates and arguments.
These predicate names fill in the verbs and the arguments fill in nouns in NLG templates.
Example:Given fly(stork):
A teacher helped us produced the following advice rating scheme:
| Rating | % Given |
| Excellent | 0 |
| Good | .35 |
| Fair | .60 |
| Poor | .05 |
The vast majority of advice was acceptable to teachers.
In some cases the specific advice was shown to help provide a "crucial detail" and help "elicit a fact."
The advice received a large amount of qualitative feedback from teachers.
The advice was often "repetitive" and "badly phrased." This is mostly due to errors in event extraction or template errors that could be corrected by full-scale NLG.
The specific advice was also not "too literal" and not "inferential enough." However, the main problem is that understanding stories requires inference that includes world knowledge outside the text and common-sense, and these are very hard problems for NLP and AI in general.
Overall, the system succeeded in producing coarse-grained advice and fine-grained advice from extracted events that was judged to be acceptable to teachers.
Event extraction can provide the needed structure not provided by "bags-of-words" techniques for some tasks in NLP.
However, extracting events from text is fraught with error, particularly in the ungrammatical and informal domain used in this experiment.
Further work on prepositions, embedded speech, semantic role labels, and improved temporal representations could help.
Advice generation needs to go beyond templates to ask directed questions and look at whole groups of events.
Inference with some degree of simulated world knowledge would be useful, or so would having a teacher summarize the "point" of the story.