Natural Language Generation

PowerPoint Presentation

Natural Language GenerationSaurabh Chanderiya (07005004)Abhimanyu Dhamija (07005024)E K Venkatesh (07005031)G Hrudil (07005032)B Vinod Kumar (07d05018)

Guide: Prof. Pushpak Bhattacharya1OutlineWhat is Natural Language Generation?MotivationStages in NLGApplications of NLGEvaluation TechniquesConclusion2What is Natural Language Generation?Natural Language Generation (NLG) is the subfield of artificial intelligence and computational linguistics that focuses on computer systems that can produce understandable texts in English or other human languages[Reiter and Dale, 2000]

3What is Natural Language Generation?Convert computer based representation into natural language representation (opposite of NLU)

Machine representation comprises of some form of computerized dataExamples:A database of daily temperature values in a cityAn ontology A collection of fairy tales4Natural Language TextNLGData/ Machine-representationNLUKey Elements in NL GenerationMany choices available an NLG system needs to choose the most appropriate oneExample: Denoting value-changethe temperature rose increase in valuethe temperature plummeted drastic increase in valuethe rain got heavier again increase in value, but different context [Wikipedia]Meeting the communication goals so that the generated text is understandable to the target reader

5Motivating Example1Suppose you are asked to write an article on IIT Bombay.How do you proceed?Step 1: What all should I write about? How should I organize it?History, Students, Professors, Gymkhana, Mood Indigo Start with description of gymkhana or history Step 2: What should my style be?Editorial, Prose, Poetry Simple words Step 3: Pen it down6Motivating Example2We have just identified the key stages in Natural Language Generation

7Motivating Example3We have just identified the key stages in Natural Language Generation

Step 1: What all should I write about? How should I organize it?History, Students, Professors, Gymkhana, Mood Indigo Start with description of gymkhana or history

TEXT PLANNING (Content Determination and Document Structuring)

8Motivating Example4We have just identified the key stages in Natural Language Generation

Step 2: What should my style be?Editorial, Prose, Poetry Simple words

MICROPLANNING (Lexical Choice, Referring Expression Generation, Aggregation)

9Motivating Example5We have just identified the key stages in Natural Language Generation

Step 3: Pen it down

REALIZATION

10NLG Systems ArchitectureControl DataRealizationDocument PlanningMicro -planningInput DataOutput DataContent DeterminationDocument StructuringLexical ChoiceReferring ExpressionsAggregation11Stages in NLGThe following different stages of Natural Language Generation can be identified:Content DeterminationDocument StructuringLexical ChoiceReferring Expression GenerationAggregationRealizationEach of these is considered in detail in the next few slides12Content Determination1Deciding what information to mention in the textExample:[Wikipedia]NLG system to summarize information about sick babies has the following information:The baby is being given morphine via an IV dropThe baby's heart rate shows bradycardias (temporary drops)The baby's temperature is normalThe baby is crying13Content Determination2Factors affecting the decision could beCommunicative goal the purpose of the text and the readerA diagnosing doctor would be interested in heart rate while a parent would want to know if the baby is crying or notSize and level of detailA formal report about the patient vs. an SMS to the doctorHow unusual the information isIs it important to mention that the babys temperature is normal?14Content Determination3Techniques employedSchemas predefined templates which explicitly specify what information is to be includedBased on Rhetorical PredicatesRhetorical predicates specify the role that is played by each utterance in the textExample:Mary has a pink coatAttributiveOther rhetorical predicates: Particular illustration, evidence, inference etc.[McKeown, 1985] 15Content Determination4Example Schema using Rhetorical PredicatesIdentification Schema (for providing definitions)[McKeown, 1985]Identification (class & attribute)AttributiveParticular IllustrationSample text generated from this schema could beMumbai is an important economic region in Maharashtra. There are many textile mills in Mumbai. Bombay Dyeing is among the noteworthy textile mills.16Content Determination5Explicit Reasoning ApproachesExample: Plot generation using case based reasoning[B. Daz-Agudo et. al, 2004]Case based reasoning characterized by: retrieve, reuse, revise, retainBuild cases from a set of stories similar to identifying features that constitute the storyOntology for the fairy tale worldAccept query from user regarding features of the new plot to be generated17Content Determination6Example: Plot generation using case based reasoning (contd.)Retrieve similar case similarity calculated on the basis of distance in the ontologyResolve dependencies ask user for further input if neededGenerate plot18Content Determination7Sample run:Query: princess, murder, interdiction, interdiction violated, competition, test of heroStory number 113 (Swan Geese) returned based on similarityPerform substitutionsGenerate plot19Document Structuring1Decide the order and grouping of sentences in a generated textExample:John went to the shop.John bought an apple.Now consider:John bought an apple.John went to the shop.The first case seems more coherent than the second. Thus, sentence structuring is important.

20Document Structuring2AlgorithmsSchema based approachCorpus based Approach [M Lapata, 2003]P(S1 Sn) = P(S1) * P(S2|S1) * P(S3|S1,S2) * *P(Sn|S1 Sn-1)(assuming dependence only on previous sentence) P(S1 Sn) = P(S1) * P(S2|S1) * P(S3|S2) * * P(Sn|Sn-1)(using features to represent sentences) P(S1 Sn) = P((a, a, , a) | (a, a))(assuming independence of features and approximating P(S|S) from the Cartesian product S x S)P(S|S) = {P(a|a)} where jS and k S (estimate prob. using counts, construct directed weighted graph (sentences as nodes and probabilities as edge weights) and obtain approximate solution)21Aggregation1Aggregation is a subtask of Natural language generation, which involves merging syntactic constituents (such as sentences and phrases) togetherExample: John went to the shop. John bought an apple. John went to the shop and bought an apple.Could be syntactic or conceptualExample of conceptual: replacing Saturday and Sunday by weekendAggregation algorithms must do two things:Decide when two constituents should be aggregatedDecide how two constituents should be aggregated, and create the aggregated structure22Post-editing2Identity between different word-groupsLemma identity: two different words belong to the same inflectional paradigmForm identity: two words have the same spelling/ sound and are lemma-identicalCo-referentiality: two words/constituents denote the same entity or entities in the external context, i.e. have the same reference[Karin Harbusch et. al, 2009] 2324[Karin Harbusch et. al, 2009] Lexical choice1Lexical choice involves choosing the content words (nouns, verbs, adjectives, adverbs) in a generated text.The simplest type of lexical choice involves mapping a domain concept to a word.Lexical choice modules must be informed by linguistic knowledge of how the system's input data maps onto words. This is a question of semantics, but it is also influenced by syntactic factors and pragmatic factors.3 factors to look for:GenrePeople perceive different words differentlyHow language relates to the non-linguistic world25Humans perception about words3[Rohit Parikh, 1994]By evening: has different meaningDifferent dialectsChoosing between near-synonymous wordsIt has been suggested that utility theory be applied to word choice. In other words, if we know (1) the probability of a words being correctly interpreted or misinterpreted and (2) the benefit to the user of correct interpretation and the cost of mis-interpretation, then we can compute an overall utility to the user of using the word.26Referring expression generation1This the second last stage in natural language generationThis involves creating referring expressions (noun phrases) that identify specific entities to the readerExample: He told the tourist that rain was expected tonight in Southern Scotland.He, the tourist, tonight and Southern Scotland are reference expressions27Criteria for good referents2Ideally, a good referring expression should satisfy a number of criteria:Referential success: It should unambiguously identify the referent to the reader.Ease of comprehension: The reader should be able to quickly read and understand it.Computational complexity: The generation algorithm should be fastNo false inferences: The expression should confuse or mislead the reader by suggesting false implications or other pragmatic inferences.[Wikipedia]28Kinds of Referring Expressions3Proper noun-nounDefinite Noun PhrasesSpatialTemporal ReferenceDifferent Algorithmic modelsGraph-Based Generation of Referring Expressions [Krahmer, et. al. 2003]Centering theory uses ranking [Poesio et. al, 2004 ]Generating Approximate Geographic Descriptions [Turner et. al, 2009]29Realization1Realization deals with creating the actual text from the abstract representationRealization involves three kinds of processing:Syntactic realization decide order of components, add function words etc.Example: in English, Subject usually precedes the verbMorphological realization compute inflected formsExample: plural(woman) == womenOrthographic realization Capitalization of first letter, punctuations etc.Realization systems: simplenlg, kpml etc.30Realization2SIMPLENLGa simple NLG library for Java for generating grammatically correct English sentences Sample code: SPhraseSpec p = nlgFactory.createClause(); p.setSubject("Mary"); p.setVerb("chase"); p.setObject("the monkey"); String output2 = realiser.realiseSentence(p); System.out.println(output2);Output:Mary chases the monkey[http://code.google.com/p/simplenlg/wiki/Section1]31Applications of NLG1Present information in more convenient wayAirline schedule databaseAccounting spreadsheetAutomating document productionDoctor writing discharge summariesProgrammer writing code documentation, logic description etc.In many contexts, human intervention is required to create texts 32Application of NLG with human intervention2NLG system is used to produce an initial draft of a document which can be further edited by human authorE.g.Weather Reporter, which helps meteorologists compose weather forecastsDRAFTER, which helps technical authors write software manualsAlethGen, which helps customer-service representatives write response letters to customers33Application of NLG without human intervention3Some NLG systems have been developed with the aim of operating as standalone systems.E.g.Model Explainer, which generates textual descriptions of classes in an object-oriented software systemLFS, which summarizes statistical data for the general publicPIGLET, which gives hospital patients explanations of information in the patient records.34Weather Reporter4Provide retrospective reports of the weather over periods whose duration is one monthTakes large set of numerical dataProduces short textsE.g. text produced by Weather reporterThe month was cooler and drier than average, with the average number of rain days. The total rain for the year so far is well below average. There was rain on every day for eight days from the 11th to the 18th 3535Weather Reporter536Weather Reporter6Data shown is real data collected automatically by meteorological data gathering equipmentWeather Reporter design is based on real input data and a real corpus of human-written texts 37Weather ReporterExample, using the historical data for 1-July-2005, the software producesGrass pollen levels for Friday have increased from the moderate to high levels of yesterday with values of around 6 to 7 across most parts of the country. However, in Northern areas, pollen levels will be moderate with values of 4.In contrast, the actual forecast (written by a human meteorologist) from this data wasPollen counts are expected to remain high at level 6 over most of Scotland, and even level 7 in the south east. The only relief is in the Northern Isles and far northeast of mainland Scotland with medium levels of pollen count.38Model Explainer7Generates textual description of information in models of object-oriented software.39Model Explainer8O-O models are usually depicted graphicallyModel Explainer is useful as certain kind of information is better communicated textuallyE.g. Via Model Explainer it is clear that a section must be taught by exactly one professorClear data especially for people who are not familiar with the notation used in the graphical depiction40Model Explainer94141Model Explainer10It also express relations from the object model in a variety of linguistic contextsE.g. teachesA professor teaches a courseA section must be taught by a professorProfessor smith does not teach any sections42Task-Based EvaluationTask-based evaluations measure the impact of generated texts on end users and typically involve techniques from an application domain such as medicine.For example, a system which generates summaries of medical data can be evaluated by giving these summaries to doctors, and assessing whether the summaries helps doctors make better decisions.43Evaluations Based on Human Ratings and JudgmentsAnother way of evaluating an NLG system is to ask human subjects to rate generated texts on an n-point rating scale44Unigram PrecisionCandidate: the the the the the the the.Reference 1: The cat is on the mat.Reference 2: There is a cat on the mat.Unigram Precision of above candidate is 7 as 7 candidate words(the) occur in reference1.But candidate is not appropriate.45Modified Unigram precisionCount the maximum number of times a word occurs in any single reference translation.Clip the total count of each candidate word by its maximum reference countAdd these clipped counts Divide by the total number of candidate words.46Modified unigram precisionCandidate: the the the the the the the.Reference 1: The cat is on the mat.Reference 2: There is a cat on the mat.Max count of the in ref1 is 2.Total # of candidate words is 7So, Modified Unigram Precision = 2/747Modified n-gram precisionAll candidate n-gram counts and their corresponding maximum reference counts are collected.The candidate counts are clipped by their corresponding reference maximum value.Add them.Divide by the total number of candidate n-grams48Modified n-gram precisionA translation using the same words (1-grams) as in the references tends to satisfy adequacy. The longer n-gram matches account for fluency.49Modified n-gram precisionCandidate 1: It is a guide to action which ensures that the military always obey the commands the party.Candidate 2: It is to insure the troops forever hearing the activity guidebook that party direct.Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party.Reference 3: It is the practical guide for the army always to heed directions of the partyModified Bigram Precision of candidate1 = 8/17Modified Bigram Precision of candidate2 = 1/1350Modified n-gram precisionon a multi-sentence51Modified N-gram Precision : Sentence LengthCandidate: of theReference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party.Reference 3: It is the practical guide for the army always to heed directions of the party.Modified Unigram Precision = 2/2Modified Bigram Precision = 1/152Brevity PenalityCandidate translations longer than their references are already penalized by the modified n-gram precision measure: there is no need to penalize them again.Brevity Penality = 1 if candidate matches a reference. Else it is < 1.53Effective Reference Lengthbest match lengthsWe call the closest reference sentence length to candidate length the best match length.Effective Reference LengthSum of all the best match lengths54 55ConclusionAlthough Natural Language Generation techniques do generate text from the underlying computer based representation,The output text of the existing NLG systems is not of fairly high quality this necessitates human intervention when a high quality text is desired[Sripada et. al, 2003]In NLG, as opposed to Machine Translation, it is better to automatic evaluation metrics only as a supplement to human evaluations and not as a replacement.[Reiter et. al, 2009] 56References1Dale, Robert; Reiter, Ehud (2000). Building natural language generation systems. Cambridge, UK: Cambridge University PressReiter E, Sripada S, Hunter J, Yu J, Davy I (2005). "Choosing Words in Computer-Generated Weather ForecastsB. Daz-Agudo, P. Gervas, and F. Peinado. A case based reasoning approach to story plot generation. In ECCBR04, Springer-Verlag LNCS/LNAI, Madrid, Spain, 2004Reiter E, Anja Belz (2009). An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems, Association for Computational Linguisticshttp://code.google.com/p/simplenlg/wiki/Section1M Lapata (2003). Probabilistic Text Structuring: Experiments with Sentence Ordering. Proceedings of ACL-2003http://web.science.mq.edu.au/~rdale/teaching/esslli/index.html http://www.wikipedia.org 57References2E Krahmer, S van Erk, A Verleg (2003). Graph-Based Generation of Referring Expressions. Computational LinguisticsM Poesio, R Stevenson, B di Eugenio, J Hitzeman (2004). Centering: A Parametric Theory and Its Instantiations. Computational LinguisticsR Turner, Y Sripada, E Reiter (2009) Generating Approximate Geographic Descriptions. Proceedings of ENLG-2009Kathleen R McKeown(1985). Discourse Strategies for Generating Natural-Language Text, Elsevier Science Publishers B.V. (North-Holland)S. Sripada, E. Reiter, I. Davy (2003), SumTime-Mousam: Congurable marine weather forecast generator, Expert Update 6 (3)Karin Harbush, Gerard Kempen (2009), Generating clausal coordinate ellipsis multilingually: A uniform approach based on postediting; Proceedings of the 12th European Workshop on Natural Language GenerationKishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu(2003) Bleu: a Method for Automatic Evaluation of Machine Translation, IBM Research Division58Thank You!59