A Multi-Strategy Approach to Parsing of Grammatical Relations in Child Language Transcripts Kenji Sagae Language Technologies Institute Carnegie Mellon.

A Multi-Strategy Approach to A Multi-Strategy Approach to Parsing of Grammatical Relations Parsing of Grammatical Relations

in Child Language Transcriptsin Child Language Transcripts

Kenji SagaeLanguage Technologies Institute

Carnegie Mellon University

Thesis Committee:

Alon Lavie, co-chair

Brian MacWhinney, co-chair

Lori Levin

Jaime Carbonell

John Carroll, University of Sussex

2

Natural Language Parsing:Natural Language Parsing:Sentence Sentence → Syntactic Structure→ Syntactic Structure

• One of the core problems in NLP

Input: The boy ate the cheese sandwich

Output:

(S (NP (Det The) (N boy))

(VP (V ate) (NP (Det the) (N cheese) (N sandwich))))

(ROOT (predicate eat) (surface ate) (tense past) (category V) (SUBJ (category N) (agreement 3s)

(surface boy) (DET (surface the)

(category Det)))(OBJ (category N) (definite +) (DET (surface the)

(category Det)) (predicate sandwich) (surface sandwich) (MOD (category N)

(surface cheese) (predicate cheese))))

((1 2 The DET) (2 3 boy SUBJ) (3 0 ate ROOT) (4 6 the DET) (5 6 cheese MOD) (6 3 sandwich OBJ))

Grammatical Relations (GRs)• Subject, object, adjunct, etc.

3

Using Natural Language ProcessingUsing Natural Language Processingin Child Language Researchin Child Language Research

• CHILDES Database (MacWhinney, 2000)– 200 megabytes of child-parent dialog transcripts– Part-of-speech and morphology analysis

• Tools available• Not enough for many research questions

– No syntactic analysis

• Can we use NLP to analyze CHILDES transcripts?– Parsing– Many decisions: representation, approach, etc.

4

Parsing CHILDES: Parsing CHILDES: Specific and General MotivationSpecific and General Motivation

• Specific task: automatic analysis of syntax in CHILDES corpora– Theoretical importance (study of child language

development)– practical importance (measurement of syntactic

competence)

• In general: Develop techniques for syntactic analysis, advance parsing technologies– Can we develop new techniques that perform better

than current approaches?• Rule-based• Data-driven

5

Research ObjectivesResearch Objectives

• Identify a suitable syntactic representation for CHILDES transcripts– Must address the needs of child language research

• Develop a high accuracy approach for syntactic analysis of spoken language transcripts– parents and children at different stages of language

acquisition

• The plan: a multi-strategy approach– ML: ensemble methods– Parsing: several approaches possible, but

combination is an underdeveloped area

6

Research ObjectivesResearch Objectives

• Develop methods for combining analyses from different parsers and obtain improved accuracy– Combining rule-based and data-driven approaches

• Evaluate the accuracy of developed systems

• Validate the utility of the resulting systems to the child language community– Task-based evaluation: Automatic measurement of

grammatical complexity in child language

7

Overview of the Multi-Strategy ApproachOverview of the Multi-Strategy Approachfor Syntactic Analysisfor Syntactic Analysis

Transcripts

Parser A

Parser B

Parser C

Parser D

Parser E

ParserCombination

SYNTACTICSTRUCTURES

8

Thesis StatementThesis Statement

• The development of a novel multi-strategy approach for syntactic parsing allows for identification of Grammatical Relations in transcripts of parent-child dialogs at a higher level of accuracy than previously possible

• Through the combination of different NLP techniques (rule-based or data-driven), the multi-strategy approach can outperform each strategy in isolation, and produce significantly improved accuracy

• The resulting syntactic analysis are at a level of accuracy that makes them useful to child language research

9

OutlineOutline

• The CHILDES GR scheme

• GR Parsing of CHILDES transcripts

• Combining different strategies

• Automated measurement of syntactic development in child language

• Related work

• Conclusion

10

CHILDES GR SchemeCHILDES GR Scheme(Sagae, MacWhinney and Lavie, 2004)(Sagae, MacWhinney and Lavie, 2004)

• Grammatical Relations (GRs)– Subject, object, adjunct, etc.– Labeled dependencies

• Addresses needs of child language researchers– Informative and intuitive, basis for DSS and IPSyn

Dependent Head

Dependency Label

11

CHILDES GR Scheme Includes Important CHILDES GR Scheme Includes Important GRs for Child Language StudyGRs for Child Language Study

12

OutlineOutline





• Related word

• Conclusion

• Evaluation• Data• Rule-based GR parsing• Data-driven GR parsing

13

The Task:The Task:Sentence Sentence → GRs→ GRs

• Input: We eat the cheese sandwich

• Output:

14

Evaluation of GR ParsingEvaluation of GR Parsing

• Dependency accuracy

• Precision/Recall of GRs

15

Evaluation: Evaluation: Calculating Dependency AccuracyCalculating Dependency Accuracy

1 2 3 4 5

1 2 We SUBJ

2 0 eat ROOT

3 5 the DET4 5 cheese

MOD5 2 sandwich

SUBJ

16

Evaluation: Evaluation: Calculating Dependency AccuracyCalculating Dependency Accuracy

1 2 3 4 5

1 2 We SUBJ

2 0 eat ROOT


MOD5 2 sandwich

SUBJ

1 2 We SUBJ

2 0 eat ROOT

3 4 the DET4 2 cheese OBJ5 2 sandwich

PRED

Accuracy = number of correct dependenciestotal number of dependencies

= 2 / 5 = 0.40

40%

GOLD PARSED

17

Evaluation:Evaluation:Precision and Recall of GRsPrecision and Recall of GRs

• Precision and recall are calculated separately for each GR type

• Calculated on aggregate counts over entire test corpus

• Example: SUBJ

Precision = # SUBJ matches between PARSED and GOLD Total number of SUBJs in PARSED

Recall = # SUBJ matches between PARSED and GOLD Total # of SUBJs in GOLD

F-score = 2 ( Precision × Recall ) Precision + Recall

18

Evaluation:Evaluation:Precision and Recall of GRsPrecision and Recall of GRs

Precision = # SUBJ matches between PARSED and GOLD

Total number of SUBJs in PARSED

= 1 / 2 = 50%

1 2 We SUBJ

2 0 eat ROOT


MOD5 2 sandwich OBJ

1 2 We SUBJ

2 0 eat ROOT

3 4 the DET4 2 cheese OBJ5 2 sandwich

SUBJ

GOLD PARSED

Recall = # SUBJ matches between PARSED and GOLD Total # of SUBJs in GOLD

= 1 / 1 = 100%

F-score = 2 ( Precision × Recall ) Precision + Recall

= 2(50×100) / (50+100) = 66.67

19

OutlineOutline






20

CHILDES Data: the Eve CorpusCHILDES Data: the Eve Corpus(Brown, 1973)(Brown, 1973)

• A corpus from CHILDES– Manually annotated with GRs

• Training: ~ 5,000 words (adult)

• Development: ~ 1,000 words– 600 adult, 400 child

• Test: ~ 2,000 words– 1,200 adult, 800 child

21

Not All Child Utterances Have GRsNot All Child Utterances Have GRs

• Utterances in training and test sets are well-formed

I need tapioca in the bowl.

That’s a hat.

In a minute.

• What about

* Warm puppy happiness a blanket.

* There briefcase.

? I drinking milk.

? I want Fraser hat.

• Separate Eve-child test set (700 words)

22

The WSJ Corpus (Penn Treebank)The WSJ Corpus (Penn Treebank)

• 1 million words • Widely used

– Sections 02-21: training– Section 22: development– Section 23: evaluation

• Large corpus with syntactic annotation– Out-of-domain

• Constituent structures– Convert to unlabeled dependencies using head-

percolation table

23

OutlineOutline






24

Rule-Based ParsingRule-Based Parsing

• The parser’s knowledge is encoded in manually written rules– Grammar, lexicon, etc.

• Only analyses that fit the rules are possible

• Accurate in specific domains, difficult to achieve wide coverage in open domain– Coverage, ambiguity, domain knowledge

25

Rule-Based Parsing of CHILDES dataRule-Based Parsing of CHILDES data(Sagae, Lavie & MacWhinney, 2001, 2004)(Sagae, Lavie & MacWhinney, 2001, 2004)

LCFlex (Rosé and Lavie, 2001) Rules: CFG backbone augmented with unification

constraints Manually written, 153 rules

Robustness Limited insertions:

[Do] [you] want to go outside?

Limited skipping:No um maybe later.

PCFG disambiguation model Trained on 2,000 words

26

High Precision from a Small GrammarHigh Precision from a Small Grammar

• Eve test corpus– 2,000 words

• 31% of the words can be parsed• Accuracy (over all 2,000 words): 29%• Precision: 94%• High Precision, Low Recall

• Improve recall using parser’s robustness– Insertions, skipping– Multi-pass approach

27

Robustness and Multi-Pass ParsingRobustness and Multi-Pass Parsing

• No insertions, no skipping31% parsed, 29% recall, 94% precision

• Insertion of NP and/or auxiliary38% parsed, 35% recall, 92% precision

• Skipping of 1 word52% parsed, 47% recall, 90% precision

• Skipping of 1 word, insertion of NP, aux63% parsed, 55% recall, 88% precision

28

Use Robustness to Improve RecallUse Robustness to Improve Recall

0

10

20

30

40

50

60

70

80

90

100

none insert NP/aux skip 1 word insert/skip

precision

recall

f-score

29

OutlineOutline






30

Data-driven ParsingData-driven Parsing

• Parser learns from a corpus of annotated examples

• Data-driven parsers are robust

• Two approaches– Existing statistical parser– Classifier-based parsing

31

Accurate GR Parsing with Accurate GR Parsing with Existing Resources (Mostly)Existing Resources (Mostly)

• Large training corpus: Penn Treebank (Marcus et al., 1993)– Head-table converts constituents into dependencies

• Use an existing parser (trained on the Penn Treebank)– Charniak (2000)

• Convert output to unlabeled dependencies

• Use a classifier for dependency labeling

32

Unlabeled Dependency IdentificationUnlabeled Dependency Identification

We eat the cheese sandwich

sandwich

eat

eat

33

Domain IssuesDomain Issues

• Parser training data is in a very different domain– WSJ vs Parent-child dialogs

• Domain specific training data would likely be better

• Performance is acceptable– Shorter, simpler sentences– Unlabeled dependency accuracy

• WSJ test data: 92%• Eve test data: 90%

34

Dependency LabelingDependency Labeling

• Training data is required– Eve training set (5,000 words)

• Labeling dependencies is easier than finding unlabeled dependencies

• Use a classifier– TiMBL (Daelemans et al., 2004)– Extract features from unlabeled dependency structure– GR labels are target classes

35

Dependency LabelingDependency Labeling

36

Features Used for GR LabelingFeatures Used for GR Labeling

• Head and dependent words– Also their POS tags

• Whether the dependent comes before or after the head

• How far the dependent is from the head

• The label of the lowest node in the constituent tree that includes both the head and dependent

37

Features Used for GR LabelingFeatures Used for GR Labeling

Consider the words “we” and “eat”

Features: we, pro, eat, v, before, 1, S

Class: SUBJ

38

Good GR Labeling Results Good GR Labeling Results with Small Training Setwith Small Training Set

• Eve training set– 5,000 words for training

• Eve test set– 2,000 words for testing

• Accuracy of dependency labeling (on perfect dependencies): 91.4%

• Overall accuracy (Charniak parser + dependency labeling): 86.9%

39

Some GRs Are Easier Than OthersSome GRs Are Easier Than Others

• Overall accuracy: 86.9%

• Easily identifiable GRs– DET, POBJ, INF, NEG: Precision and recall above

98%

• Difficult GRs– COMP, XCOMP: below 65%

• I think that Mary saw a movie (COMP)• She tried to see a movie (XCOMP)

40

Precision and Recall of Specific GRsPrecision and Recall of Specific GRs

GR Precision Recall F-score

SUBJ 0.94 0.93 0.93

OBJ 0.83 0.91 0.87

COORD 0.68 0.85 0.75

JCT 0.91 0.82 0.86

MOD 0.79 0.92 0.85

PRED 0.80 0.83 0.81

ROOT 0.91 0.92 0.91

COMP 0.60 0.50 0.54

XCOMP 0.58 0.64 0.61

41

Parsing with Domain-Specific DataParsing with Domain-Specific Data

• Good results with a system based on the Charniak parser

• Why domain-specific data?– No Penn Treebank– Handle dependencies natively– Multi-strategy approach

42

Classifier-Based ParsingClassifier-Based Parsing(Sagae & Lavie, 2005)(Sagae & Lavie, 2005)

• Deterministic parsing– Single path, no backtracking– Greedy– Linear run-time

• Simple shift-reduce algorithm– Single pass over the input string

• Variety: Left-to-right, right-to-left (order matters)

• Classifier makes parser decisions– Classifier not tied to parsing algorithm

• Variety: Different types of classifiers can be used

43

A Simple, Fast and Accurate ApproachA Simple, Fast and Accurate Approach

• Classifier-based parsing with constituents– Trained and evaluated on WSJ data: 87.5%– Very fast, competitive accuracy

• Simple adaptation to labeled dependency parsing– Similar to Malt parser (Nivre, 2004)– Handles CHILDES GRs directly

44

GR Analysis with Classifier-Based ParsingGR Analysis with Classifier-Based Parsing

• Stack S– Items may be POS-tagged words or

dependency trees– Initialization: empty

• Queue W– Items are POS-tagged words– Initialization: Insert each word of the input

sentence in order (first word is in front)

45

Shift and Reduce ActionsShift and Reduce Actions

• Shift– Remove (shift) the word in front of queue W– Insert shifted item on top of stack S

• Reduce– Pop two topmost item from stack S– Push new item onto stack S

• New item forms new dependency• Choose LEFT or RIGHT• Choose Dependency Label

46

Parser DecisionsParser Decisions

• Shift vs. Reduce

• If Reduce– RIGHT or LEFT– Dependency label

• We use a classifier to make these decisions

47

Classes and FeaturesClasses and Features

• Classes– SHIFT– LEFT-SUBJ– LEFT-JCT– RIGHT-OBJ– RIGHT-JCT– …

• Features: derived from parser configuration– Crucially: two topmost items in S, first item in W– Additionally: other features that describe the current

configuration (look-ahead, etc)

48

Parsing CHILDESParsing CHILDESwith a Classifier-Based Parserwith a Classifier-Based Parser

• Parser uses SVM• Trained on Eve training set (5,000 words)• Tested on Eve test set (2,000 words)

• Labeled dependency accuracy: 87.3%– Uses only domain-specific data– Same level of accuracy as GR system based on

Charniak parser

49



SUBJ 0.97 0.98 0.98

OBJ 0.89 0.94 0.92

COORD 0.71 0.76 0.74

JCT 0.78 0.88 0.83

MOD 0.94 0.87 0.91

PRED 0.80 0.83 0.82

ROOT 0.95 0.94 0.94

COMP 0.70 0.78 0.74

XCOMP 0.93 0.82 0.87

50



SUBJ 0.97 0.98 0.98 0.93

OBJ 0.89 0.94 0.92 0.87

COORD 0.71 0.76 0.74 0.75

JCT 0.78 0.88 0.83 0.86

MOD 0.94 0.87 0.91 0.85

PRED 0.80 0.83 0.82 0.81

ROOT 0.95 0.94 0.94 0.91

COMP 0.70 0.78 0.74 0.54

XCOMP 0.93 0.82 0.87 0.61

51

OutlineOutline





• Related Work

• Conclusion

• Weighted voting• Combination as parsing• Handling young child utterances

52

Combine Different Parsers Combine Different Parsers to Get More Accurate Resultsto Get More Accurate Results

• Rule-based

• Statistical parsing + dependency labeling

• Classifier-based parsing– Obtain even more variety

• SVM vs MBL• Left-to-right vs right-to-left

53

Simple (Unweighted) VotingSimple (Unweighted) Voting

• Each parser votes for each dependency

• Word-by-word

• Every vote has the same weight

54


He eats cake

Parser A Parser B Parser C1 2 He SUBJ 1 2 He SUBJ 1 3 He SUBJ 2 0 eats CMOD 2 0 eats ROOT 2 0 eats

ROOT3 1 cake OBJ 3 1 cake OBJ 3 2 cake OBJ

GOLD1 2 He SUBJ2 0 eats ROOT3 2 cake OBJ

55


He eats cake




VOTED1 2 He SUBJ2 0 eats ROOT3 1 cake SUBJ

He eats cake




VOTED

56


He eats cake




VOTED1 2 He SUBJ2 0 eats ROOT3 1 cake OBJ

He eats cake




VOTED

57

Weighted VotingWeighted Voting

• Each parser has a weight– Reflects confidence in parser’s GR

identification

• Instead of adding number of votes,add the weight of votes

• Takes into account that some parsers are better than others

58


He eats cake

Parser A (0.4) Parser B (0.3) Parser C (0.8)1 2 He SUBJ 1 2 He SUBJ 1 3 He SUBJ 2 0 eats CMOD 2 0 eats ROOT 2 0 eats




VOTED

59

Label-Weighted VotingLabel-Weighted Voting

• Not just one weight per parser, but

one weight for each GR for each parser

• Takes into account specific strengths of each parser

60

Label-Weighted VotingLabel-Weighted Voting

He eats cake

Parser A Parser B Parser C1 2 He SUBJ (0.7) 1 2 He SUBJ (0.8) 1 3 He SUBJ (0.6)

2 0 eats CMOD (0.3) 2 0 eats ROOT (0.9) 2 0 eats ROOT(0.7)

3 1 cake OBJ (0.5) 3 1 cake OBJ (0.3) 3 2 cake OBJ (0.9)



VOTED

61

Voting Produces Very Accurate ResultsVoting Produces Very Accurate Results

• Parsers– Rule-based– Statistical based on Charniak parser– Classifier-based

• Left-to-right SVM• Right-to-left SVM• Left-to-right MBL

• Simple Voting: 88.0%• Weighted Voting: 89.1%• Label-weighted Voting: 92.1%

62



SUBJ 0.98 0.98 0.98

OBJ 0.94 0.94 0.94

COORD 0.94 0.91 0.92

JCT 0.87 0.90 0.88

MOD 0.97 0.91 0.94

PRED 0.86 0.89 0.87

ROOT 0.97 0.96 0.96

COMP 0.75 0.67 0.71

XCOMP 0.90 0.88 0.89

63

OutlineOutline






64

Voting May Not Produce Voting May Not Produce a Well-Formed Dependency Treea Well-Formed Dependency Tree

• Voting on a word-by-word basis

• No guarantee of well-formedness

• Resulting set of dependencies may form a graph with cycles, or may not even be fully connected– Technically not fully compliant with CHILDES

GR annotation scheme

65

Parser Combination as ReparsingParser Combination as Reparsing

• Once several parsers have analyzed a sentence, use their output to guide the process of reparsing the sentence

• Two reparsing approaches– Maximum spanning tree– CYK (dynamic programming)

66

Dependency Parsing as Search for Dependency Parsing as Search for Maximum Spanning TreeMaximum Spanning Tree

• First, build a graph– Each word in input sentence is a node– Each dependency proposed by any of the parsers is

an weighted edge– If multiple parsers propose the same dependency,

add weights into a single edge

• Then, simply find the MST– Maximizes the votes– Structure guaranteed to be a dependency tree– May have crossing branches

67

Parser Combination with the CYK AlgorithmParser Combination with the CYK Algorithm

• The CYK algorithm uses dynamic programming to find all parses for a sentence given a CFG– Probabilistic version finds most probable parse

• Build a graph, as with MST• Parse the sentence using CYK

– Instead of a grammar, consult the graph to determine how to fill new cells in the CYK table

– Instead of probabilities, we use the weights from the graph

68



SUBJ 0.98 0.98 0.98

OBJ 0.94 0.94 0.94

COORD 0.94 0.91 0.92

JCT 0.87 0.90 0.88

MOD 0.97 0.91 0.94

PRED 0.86 0.89 0.87

ROOT 0.97 0.97 0.97

COMP 0.73 0.89 0.80

XCOMP 0.88 0.88 0.88

69

OutlineOutline






70

Handling Young Child Utterances withHandling Young Child Utterances withRule-Based and Data-Driven ParsingRule-Based and Data-Driven Parsing

• Eve-child test set:I need tapioca in the bowl.

That’s a hat.

In a minute.

* Warm puppy happiness a blanket.

* There briefcase.

? I drinking milk.

? I want Fraser hat.

71

Three Types of Sentences in One CorpusThree Types of Sentences in One Corpus

• No problem– High accuracy

• No GRs– But data-driven systems will output GRs

• Missing words, agreement errors, etc.– GRs are fine, but a challenge for data-driven

systems trained on fully grammatical utterances

72

To Analyze or Not To Analyze:To Analyze or Not To Analyze:Ask the Rule-Based ParserAsk the Rule-Based Parser

• Utterances with no GRs are annotated in test corpus as such

• Rule-based parser set to high precision– Same grammar as before

• If sentence cannot be parsed with the rule-based system, output No GR.– 88% Precision, 89% Recall– Sentences are fairly simple

73

The Rule-Based Parser also The Rule-Based Parser also Identifies Missing WordsIdentifies Missing Words

• If the sentence can be analyzed with the rule-based system, check if any insertions were necessary– If inserted be or possessive marker ’s, insert

the appropriate lexical item in the sentence

• Parse the sentence with data-driven systems, run combination

74

High Accuracy Analysis of High Accuracy Analysis of Challenging UtterancesChallenging Utterances

• Eve-child test– No rule-based first pass: 62.9% accuracy

• Many errors caused by GR analysis of words with no GRs

– With rule-based pass: 88.0% accuracy

• 700 words from Naomi corpus– No rule-based: 67.4%– Rule-based, then combo: 86.8%

75

OutlineOutline





• Related work

• Conclusion

76

Index of Productive Syntax (IPSyn)Index of Productive Syntax (IPSyn)(Scarborough, 1990)(Scarborough, 1990)

• A measure of child language development

• Assigns a numerical score for grammatical complexity

(from 0 to 112 points)

• Used in hundreds of studies

77

IPSyn Measures Syntactic DevelopmentIPSyn Measures Syntactic Development

• IPSyn: Designed for investigating differences in language acquisition– Differences in groups (for example: bilingual children)– Individual differences (for example: delayed language

development)– Focus on syntax

• Addresses weaknesses of Mean Length of Utterance (MLU)– MLU surprisingly useful until age 3, then reaches ceiling (or

becomes unreliable)

• IPSyn is very time-consuming to compute

78

Computing IPSyn (manually)Computing IPSyn (manually)

• Corpus of 100 transcribed utterances– Consecutive, no repetitions

• Identify 56 specific language structures (IPSyn Items)– Examples:

• Presence of auxiliaries or modals• Inverted auxiliary in a wh-question• Conjoined clauses• Fronted or center-embedded subordinate clauses

– Count occurrences (zero, one, two or more)

• Add counts

79

Automating IPSynAutomating IPSyn

• Existing state of manual computation– Spreadsheets– Search each sentence for language structures– Use part-of-speech tagging to narrow down the

number of sentences for certain structures• For example: Verb + Noun, Determiner + Adjective + Noun

• Automatic computation is possible with accurate GR analysis– Use GRs to search for IPSyn items

80

Some IPSyn Items Require Syntactic Analysis for Some IPSyn Items Require Syntactic Analysis for Reliable RecognitionReliable Recognition

(and some don’t)(and some don’t)

• Determiner + Adjective + Noun• Auxiliary verb• Adverb modifying adjective or nominal• Subject + Verb + Object• Sentence with 3 clauses• Conjoined sentences• Wh-question with inverted auxiliary/modal/copula• Relative clauses• Propositional complements• Fronted subordinate clauses• Center-embedded clauses

81

Automating IPSyn with Automating IPSyn with Grammatical Relation AnalysesGrammatical Relation Analyses

• Search for language structures using patterns that involve POS tags and GRs (labeled dependencies)

• Examples

– Wh-embedded clauses: search for wh-words whose head (or transitive head) is a dependent in a GR of types [XC]SUBJ, [XC]PRED, [XC]JCT, [XC]MOD, COMP or XCOMP

– Relative clauses: search for a CMOD where the dependent is to the right of the head

82

Evaluation DataEvaluation Data

• Two sets of transcripts with IPSyn scoring from two different child language research groups

• Set A– Scored fully manually– 20 transcripts– Ages: about 3 yrs.

• Set B– Scored with CP first, then manually corrected– 25 transcripts– Ages: about 8 yrs.

(Two transcripts in each set were held out for development and debugging)

83

Evaluation Metrics: Evaluation Metrics: Point DifferencePoint Difference

• Point difference

– The absolute point difference between the scores provided by our system, and the scores computed manually

– Simple, and shows how close the automatic scores are to the manual scores

– Acceptable range• Smaller for older children

84

Evaluation Metrics:Evaluation Metrics:Point-to-Point AccuracyPoint-to-Point Accuracy

• Point-to-point accuracy

– Reflects overall reliability over each scoring decision made in the computation of IPSyn scores

– Scoring decisions: presence or absence of language structures in the transcript

Point-to-Point Acc = C(Correct Decisions)

C(Total Decisions)

– Commonly used for assessing inter-rater reliability among human scorers (for IPSyn, about 94%).

85

ResultsResults

• IPSyn scores from

– Our GR-based system (GR)

– Manual scoring (HUMAN)

– Computerized Profiling (CP)• Long, Fey and Channell, 2004

86

GR-based IPSyn Is Quite AccurateGR-based IPSyn Is Quite Accurate

System Avg. Point Difference to HUMAN

Point-to-point Reliability (%)

GR (total) 3.3 92.8

CP (total) 8.3 85.4

GR (set A) 3.7 92.5

CP (set A) 6.2 86.2

GR (set B) 2.9 93.0

CP (set B) 10.2 84.8

87

GR-Based IPSyn Close to Human ScoringGR-Based IPSyn Close to Human Scoring

• Automatic scores very reliable

• Validates usefulness of– GR annotation scheme– Automatic GR analysis

• Validates analysis over a large set of children of different ages

88

OutlineOutline





• Related work

• Conclusion

89

Related WorkRelated Work

• GR schemes, GR evaluation:– Carroll, Briscoe & Sanfilippo,

1998– Lin, 1998– Yeh, 2000– Preiss, 2003

• Rule-based robust parsing– Heeman & Allen, 2001– Lavie, 1996– Rosé & Lavie, 2001

• Parsing– Carroll & Briscoe, 2002– Briscoe & Carroll, 2002– Buchholz, 2002– Tomita, 1987

– Magerman, 1995– Ratnaparkhi, 1997– Collins, 1997– Charniak, 2000

• Deterministic parsing– Yamada & Matsumoto,

2003– Nivre & Scholz, 2004

• Parser Combination– Henderson & Brill, 1999– Brill & Wu, 1998– Yeh, 2000– Sarkar, 2001

• Automatic measurement of grammatical complexity– Long, Fey & Channell,

2004

90

OutlineOutline





• Related work

• Conclusion

91

Major ContributionsMajor Contributions

• An annotation scheme based on GRs for syntactic structure in CHILDES transcripts

• A linear-time classifier-based parser for constituent structures

• The development of rule-based and data-driven approaches to GR analysis– Precision/recall trade-off using insertions and skipping– Data-driven GR analysis using existing resources

• Charniak parser, Penn Treebank

– Parser variety in classifier-based dependency parsing

92

Major Contributions (2)Major Contributions (2)

• The use of different voting schemes for combining dependency analyses– Surpasses state-of-the-art in WSJ dependency

parsing– Vastly outperforms individual parsing approaches

• A novel reparsing combination scheme– Maximum spanning trees, CYK

• An accurate automated tool for measurement of syntactic development in child language– Validates annotation scheme and quality of GR

analyses

93

Possible Future DirectionsPossible Future Directions

• Classifier-based parsing– Beam search keeping linear time– Tree classification (Kudo & Matsumoto, 2004)

• Parser combination– Parser variety, reparsing combination with constituent

trees• Automated measurement of grammatical

complexity– Take precision/recall into account– A data-driven approach to replace search rules

• Other languages

94

95

96

97

98

More on Dependency VotingMore on Dependency Voting

• On WSJ data: 93.9% unlabeled accuracy• On Eve data

– No RB: 91.1% • COMP: 50%

– No charn, No RB: 89.1%• COMP: 50%, COORD: 84%, ROOT: 95%

– No charn: 90.5%• COMP: 67%

– No RL, no MBL: 91.8%

99

Full GR ResultsFull GR Results

• XJCT ( 2 / 2) : 1.00 1.00 1.00• OBJ ( 90 / 91) : 0.95 0.96 0.95• NEG ( 26 / 25) : 1.00 0.96 0.98• SUBJ ( 180 / 181) : 0.98 0.98 0.98• INF ( 19 / 19) : 1.00 1.00 1.00• POBJ ( 48 / 51) : 0.92 0.98 0.95• XCOMP ( 23 / 23) : 0.88 0.88 0.88• QUANT ( 4 / 4) : 1.00 1.00 1.00• VOC ( 2 / 2) : 1.00 1.00 1.00• TAG ( 1 / 1) : 1.00 1.00 1.00• CPZR ( 10 / 9) : 1.00 0.90 0.95• PTL ( 6 / 6) : 0.83 0.83 0.83• COORD ( 33 / 33) : 0.91 0.91 0.91• COMP ( 18 / 18) : 0.71 0.89 0.80• AUX ( 74 / 78) : 0.94 0.99 0.96• CJCT ( 6 / 5) : 1.00 0.83 0.91• PRED ( 54 / 55) : 0.87 0.89 0.88• DET ( 45 / 47) : 0.96 1.00 0.98• MOD ( 94 / 89) : 0.97 0.91 0.94• ROOT ( 239 / 238) : 0.97 0.96 0.96• PUNCT ( 286 / 286) : 1.00 1.00 1.00• COM ( 45 / 44) : 0.93 0.91 0.92• ESUBJ ( 2 / 2) : 1.00 1.00 1.00• CMOD ( 3 / 3) : 1.00 1.00 1.00• JCT ( 78 / 84) : 0.85 0.91 0.88

100


He eats cake

Parser A (0.4) Parser B (0.3) Parser C (0.8)1 2 He SUBJ 1 2 He SUBJ 1 3 He SUBJ 2 0 eats CMOD 2 0 eats ROOT 2 0 eats




VOTED

0.80.7

101


He eats cake

Parser A Parser B Parser C1 2 He SUBJ (0.7) 1 2 He SUBJ (0.8) 1 3 He SUBJ (0.6)

2 0 eats CMOD (0.3) 2 0 eats ROOT (0.9) 2 0 eats ROOT(0.7)

3 1 cake OBJ (0.5) 3 1 cake OBJ (0.3) 3 2 cake OBJ (0.9)



VOTED

0.61.5

A Multi-Strategy Approach to Parsing of Grammatical Relations in Child Language Transcripts Kenji Sagae Language Technologies Institute Carnegie Mellon.

Documents

syntactic parsing

parsing childes

megabytes of child

child languageoverview

child dialogs

s surface boy det

novel multistrategy

multistrategy approachml