Event Extraction in a Plot Advice Agent

Harry Halpin and Johanna Moore
<H.Halpin@ed.ac.uk> <J.Moore@ed.ac.uk> School of Informatics
University of Edinburgh
Scotland UK




StoryStation COLING-ACL Sydney Austrailia, July 2006

Introduction

Hypothesis

The features of a text discovered via automatic event extraction can be used in both natural language understanding and advice generation in the domain of narrative instruction.

Question

Does this extra amount of automatically induced structure provide some benefit against more traditional approaches that rely on simply word distribution?

The Story Rewriting Task

These are investigated in the context of StoryStation, in order to create a fully automated plot analysis agent to improve the writing of students.

We investigate story rewriting task, where a story, the exemplar story, is read to the students, and afterwards the story is rewritten by each student, providing a corpus of rewritten stories.

This tests the students ability to both listen and write, while removing from the student the cognitive load needed to generate a new plot

.

Requirements

Why Events?

Stories are naturally composed on events. These events are composed of the interactions of characters with other entities in particular locales (Propp, 1968).

With recent advances in the accuracy of NLP components, we can now construct pipelines of NLP components that can automatically extract these events on a on a fine-grained level.

These events can then be used to analyze the story on a in general for the teacher, while also providing particular event-based advice for the student.

Previous Work

Work by Mueller (2004) has investigated using reasoning over event structures to understand complex stories. However, Mueller crafts his event representations by hand.

The WRITE system (Burstein et. al, 2003) analyzes student essays using Rhetorical Structure Theory. The errors in the story rewriting task are primarily informational, not intentional, so RST is unsuitable as the primary component.

Graesser and others (2000) used LSA to compare student stories, but this by itself can not provide the specific feedback needed.

The ATLAS system (Rose, 2002) is the most similar, but has as its domain physics stories and does not do non-domain specific event extraction.

.

Our previous work was with a smaller corpus (100 stories) and did not test either domain portability or have an advice generation component (Halpin and Moore, 2004).

Corpus

We collected a corpus of 290 stories from primary schools based on two different exemplar stories.

  1. An episode of ``The Wonderful Adventures of Nils'' by Selma Lagerloff (160 stories)
  2. The second a re-telling of ``The Treasure Thief'' by Herodotus (130 stories).

The stories are highly ungrammatical (only 35 percent could be parsed by a CCG-based parser) and contain many invented words (torlix) and the consistent usage of homonyms("there" for "their"). Many stories just one long run-on sentence.

Sample Story

Nils was settling down to watch the sunrise when he saw a beautiful magical stork. It landed it said would like come on a Journey with me. Then Nils thought about the geese he said to himself they will be alright. So he jumped on to the stork's back and he turned into an elf and off the stork went it flew by the sea the moon made a silvery path through the sea. So the stork landed Nils found a green coin so he picked it up he thought it was super but he thought it was worth nothing so he threw it away he turned round and saw the best city he could ever imagine he saw large gates about five times taller than him. He went past two tall guards he used all his strength and energy to open the gates. He found a market where he found clocks, gold jugs, cloths and the man said take as much as you want for a coin so he said one minute so ran back to the coin picked it up he turned round and city wasn't there so woke the stork up and went back to the mountains and he was staring at the moon.

Rating Scale

  1. Excellent: An excellent story shows that the student has ``read beyond the lines'' and demonstrates a deep understanding of the story, using inference to grasp points that may not have been explicit in the story. The student should be able to retrieve all the important links, and not all the details, but the right details.
  2. Good: A good story shows that the student understood the story and has ``read between the lines." The student recalls the main events and links in the plot. However, the student shows no deep understanding of the plot and does not make use of inference. This can often be detected by the student leaving out an important link or emphasizing the wrong details.
  3. Fair: A fair story shows that student has listened to the story but not understood the story, and so is only trying to repeat what they have heard. This is shown by the fact that the fair story is missing multiple important links in the story, including a possibly vital part of the story.
  4. Poor: A poor story shows the student has had trouble listening to the story. The poor story is missing a substantial amount of the plot, with characters left out and events confused. The student has trouble connecting the parts of the story.

Intercoder Agreement

Since the scale is ordinal, not nominal, we will not use the kappa statistic but instead use Cronbach's alpha and Kendall's taub statistics that penalize ranking errors correctly.

We used three raters, with one chosen as the gold standard. Absolute agreement was relatively low: (58% and 53%)

Taking into account the rating scale, almost all differences were just "one-level" differences.

Between raters there was an alpha statistic of .90 and .87 respectively, and a tau of .74 and .67 respectively. This shows the scheme to be reliable.

Plot Advice Agent

  1. Extract Events from rewritten story
  2. Use Plot Comparison Algorithm to compare rewritten events to the events from the exemplar story.
  3. Use the machine-learner to assign a general rating to the student by using the absence or presence of events as their features
  4. Produce advice based on this rating for the student. Alert teacher if rating is Poor.
  5. Produce a limited number of event-based specific advice from the story.

Event Extraction

An event is simply a minimal subset of predicate argument structure without quantification, and an event structure is a partially-ordered series of events.

  1. Tokenize the words: LT-TTT
  2. Detect sentence boundaries: LT-TTT
  3. Tag words with Penn Treebank POS with max entropy tagger
  4. Resolve pronouns using the shallow CogNIAC rule-based algorithm
  5. Extract chunks from each sentence: Cass
  6. Convert each chunk to event using rules (McNeill, 2006)

Example

Nils stays in Swedenstay(t=1, Nils, Sweden)
and he is always playing with geese on his mountain.play(t=1,Nils,geese, mountain)
He sees a birdsee(t=2, Nils, bird)

Plot Comparison Algorithm

Motivation: Need a measure of difference between the events in the rewritten story and exemplar story.

The algorithm iterates through the exemplar story's events looking for matches in each event of the rewritten story.

Events are matched by matching their components one at a time, starting with the event name and then matching entities.

Uses WordNet to find out if synonyms were used and uses a now-point to determine the order of events.

Plot Comparison Algorithm

Motivation: Need a measure of difference between the events in the rewritten story and exemplar story.

The algorithm iterates through the exemplar story's events looking for matches in each event of the rewritten story.

Events are matched by matching their components one at a time, starting with the event name and then matching entities.

Uses WordNet to find out if synonyms were used and uses a ``now-point'' to determine the order of events.

Plot Comparison Example

ExemplarRewritten
throw(t=1,Nils, coin)toss(t=3,coin)
Algorithm Result:synonym, out of order, no match, exact match

The algorithm produces a feature vector of these results for each rewritten story by encoding these results as integers.

Automatic Rating Results

Machine LearnerCorpusFeatures% Correct
k-NNAdventureLSA47.5
Naive BayesAdventurePLOT55.6
k-NNThiefLSA41.2
Naive BayesThiefPLOT45.4

Result Analysis

Performance comparable to human teachers on Adventure stories (55.6 to 56 percent agreement).

The system is portable across domains, although the performance clearly decreases on the Thief stories. We believe this is due to their increased length, complexity, and characters who are never identified with proper names, only descriptions.

Combining PLOT and LSA did not increase performance

System actually did not mislabel stories very often, and labelled poor stories remarkably correctly.

Naive Bayes Confusion Matrix

RatingExcellentGoodFairPoor
Excellent 14 22 0 1
Good 5 36 0 7
Fair 3 20 0 2
Poor 0 11 0 39

Coarse-Grained Advice Generation

An Excellent story needs less advice than a Good story, and so on.The first piece of advice is based on the raw rating given by the machine-learner, and the rest are specific. We produce 7 advice statements for Poor stories, 5 for Fair, 3 for Good, and only 1 (the general advice) for Excellent Stories. A number of canned text statements offering general advice were created for each rating class.
  • Excellent: "Great work! You really understood the story"
  • Good: "It's very good! I only have a few pointers."
  • Fair: "There are a few crucial elements you forgot. Want to read the story again?"
  • Poor: "Let's get help from the teacher."

Fine-Grained Advice Generation

The generation of specific advice uses the results of the plot-comparison algorithm. The ten most frequent events in the corpus are chosen as the important events.

Search the rewritten story event structure for missing events and partial matches of predicates and arguments.

These predicate names fill in the verbs and the arguments fill in nouns in NLG templates.

Example:Given fly(stork):

  • If has neither fly or stork: "Did the stork fly?"
  • If only has stork: "Can you tell me more about the stork?"
  • If the event is out of order: "I think something with the stork happened earlier in the story."

Advice Rating

A teacher helped us produced the following advice rating scheme:

  1. Excellent: The advice was suitable for the story, and helped the student gain insight into the story.
  2. Good: The advice was suitable for the story, and would help the student.
  3. Fair: The advice was suitable, but could have been phrased better.
  4. Poor: The advice really didn't make sense and would only confuse the student further.

Generated Advice Results

Rating% Given
Excellent0
Good.35
Fair.60
Poor.05

Generated Advice Results

The vast majority of advice was acceptable to teachers.

In some cases the specific advice was shown to help provide a "crucial detail" and help "elicit a fact."

The advice received a large amount of qualitative feedback from teachers.

The advice was often "repetitive" and "badly phrased." This is mostly due to errors in event extraction or template errors that could be corrected by full-scale NLG.

The specific advice was also not "too literal" and not "inferential enough." However, the main problem is that understanding stories requires inference that includes world knowledge outside the text and common-sense, and these are very hard problems for NLP and AI in general.

Conclusion and Future Work

Overall, the system succeeded in producing coarse-grained advice and fine-grained advice from extracted events that was judged to be acceptable to teachers.

Event extraction can provide the needed structure not provided by "bags-of-words" techniques for some tasks in NLP.

However, extracting events from text is fraught with error, particularly in the ungrammatical and informal domain used in this experiment.

Further work on prepositions, embedded speech, semantic role labels, and improved temporal representations could help.

Advice generation needs to go beyond templates to ask directed questions and look at whole groups of events.

Inference with some degree of simulated world knowledge would be useful, or so would having a teacher summarize the "point" of the story.