natural language processing artificial intelligence

8/9/2019 natural language processing artificial intelligence

1/81

Module 5


2/81

Natural language Processing


3/81

Two of the most difficult tasks that facing AI researchers are

- developing programs that understand Natural language &

-comprehend Visual scenes

eveloping Programs that can understand Natural

!anguage is ver" difficult# $h"%

Natural languages are large

The" contain a numer of different sentences#

New sentences can alwa"s e produced#

There is amiguit" in a natural languageMan" words have several meanings and sentences can

have several meanings in different conte'ts


4/81

English sentences are incomplete descriptions of the information.

-some dogs are outside.

The same expression means different things in different contexts:

Where is the water ?

Advt: communicate about an infinite world using a finite number ofsymbols.

o natural language pgm can be complete because new words!

expressions and meanings can be generated "uite freely.There are lots of ways to say the same thing.

#ary was born on $ctober%%.

#ary&s birthday is on $ctober %%.


5/81

(verview of !inguistics

'inguistics ( study of language

'evels of )nowledge used in atural language understanding

%. *honological )nowledge

-)nowledge which relates sounds to the words

Phoneme –smallest unit of sound

+. #orphological )nowledge

- lexical )nowledge related to word constructions from basic units called morphemes.

Morphemes- smallest unit of meaning

,.yntactic )nowledge

.emantic )nowledge

/.*ragmatic )nowledge

0.World )nowledge


6/81


7/81

)rammers and languages

!anguage * can e considered as a set of strings of finite or infinite

length+tring * constructed " concatenating s"mols, alphaets

Alphaets * s"mols of the language#

+entences are constructed using a set of rules called grammer# !anguage generated " grammer ) * !,)

)rammer ) can e defined as ) . ,Vn/ Vt/ s/ p

Terminal s"mols

* s"mols which cannot e decomposed further#

eg0 ad1ectives / nouns or vers in 2nglish

NonTerminals s"mols * can e decomposed further or e'panded " rules#

eg0 Noun phrases or Ver phrases


8/81

Most common wa" to represent grammers is as a set of production rules

+ NP VP

NP A3T N

NP N

VP V NP

N o" 4 popsicle 4 frog

V ate 4 kissed 4flew

A3T the 4 a

$ith this )/ following sentence can generated0

The o" ate a popsicle The frog kissed a o"

A o" ate the frog


9/81

+ NP VP A3T N VP

the N VP

the o" VP

the o" V NP

the o" ate NP

the o" ate A3T N

the o" ate a N

the o" ate a popsicle#

A grammer does not gurantee the generation of meaningful sentences/onl" that the" are structurall" correct#

The Popsicle flew a frog6


10/81

+tructural 3epresentations

+entences can e represented as a tree or graph to e'pose

the structure of the constituent parts#

+

NP VP

A3T N

the o"

V NP

A3T N

a popsiclePhase marker or s"ntactic tree

ate


11/81

7asic Parsing Techni8ues

The process of determining the s"ntactical structure of a

sentence is known as parsing#

The process of anal"9ing a sentence " taking it apart

word-" word and determining its structure from its

constituent parts and su parts#

The structure of a sentence can e represented with a

s"ntactic tree or a list


12/81

To parse a sentence/ it is necessar" to find a wa" in which that

sentence could have een generated from the start s"mol# This

can e done in two wa"s0

Top-own Parsing

* 7egin with the start s"mol and appl" the grammar rules forwarduntil the s"mols at the terminals of the tree correspond to the

components of the sentence eing parsed

7ottom-up Parsing

- 7egin with the sentence to e parsed and appl" the grammar rules

ackward until a single tree whose terminals are the words of the

sentence and whose top node is the start s"mol has een

produced#


13/81

Parsing an input to create an output

structure

Input string Parser (utput

representation

structure

!e'icon


14/81

:ath" 1umped the horse

* 8*

8*

9athy 8*

9athy 8 *

9athy umped *

9athy umped A;T

9athy umped the 9athy umped the house

9athy umped the horse

umped the horse

8 the horse

8 A;T horse

* 8 A;T

* 8 * * 8*

Top down Parsing 7ottom up Parsing


15/81

The !e'icon

A dictionary of words! where each word contains some syntactic!

semantic and possibly some pragmatic information


16/81

T"pical entries in a le'icon

$ord T"pe ;eatures

a eterminer

eVer Trans0intransitive

7o" Noun

?an Noun

?arried ver form0 past/ past participle

0

0

(range ad1ective

Noun

To preposition


17/81

Bnderstanding written te't is easier than understanding

speech#

)eneral approaches to natural language Bnderstanding

The use of ke"word and pattern matching#

+"ntactic and semantic directed anal"sis#?omparing and matching the input to real world situations#

(f these second approach is the most popular one#


18/81

Transformational )rammars

Provide a mechanism to produce single representations for

sentences having the same meanings through a series

of transformations

)enerative )rammers

-produce different structures for sentences having

different s"ntactical forms even though the" ma" havethe same semantic content#

?onsider the following sentences


19/81

printed

NP VP

V NP

A3T

+

+usan N

the file

+

NP VP

N A3T

fileThe

printedwas

V PP

by susa


20/81

?ase )rammars

)rammer rules are written to descrie s"ntactic rather thansemantic regularities#

,printed ,agent +usan

,o1ect ;ile

Mother aked for three hours

,aked ,agent Mother

,timeperiod =-hours

,aked ,(1ect Pie

,timeperiod =-hours


21/81

6ifferent 3ases are used by 3ase grammer are

=A> Agent ( 5nstigator of the action=animate>

5nstrument - 3ause of the event or obect used in causing theevent=inanimate>

=6> 6ative- Entity affected by the action.=animate>

='> 'ocative- *lace of the event

=> ource ( *lace from which something moves

=> oal ( *lace to which something moves

=T> Time ( Time at which the event occurred.

=$> $bect ( Entity that is acted upon or that changes!

6escribe relationships between verbs and their arguments.


22/81

The process of parsing into a case representation is heavily directed

by the lexical entries associated with each verb

open @ $ =5> =A>B

The door opened

7ohn opened the door

7ohn open the door with a chisel.6ie @ 6B

7ohn died

9ill @ 6 =5> AB

1ill )illed 7ohn

1ill )illed 7ohn with a )nife.


23/81

*arsing using a case grammer is expectation-driven


24/81

Transition networks

Another popular method used to represent formal and

natural language structures

7ased on the application of directed graphs,digraphs and

finite state automata#

?onsists of a numer of nodes and laeled arcs#


25/81

+emantic Anal"sis and

3epresentation structures+emantic interpretation is the most difficult stage in the transformation

process#

The domain refers to the knowledge that is part of the world model the

s"stem knows aout#

-includes o1ect descriptions/ relationships and other relevant

concepts#

The conte't relates to previous e'pressions/ the setting and time of the

utterances / and the eliefs/ desires and intentions of the speakers#

The task is part of the service the s"stem offers/ such as retrieving

information from a data ase/ providing e'pert advice/ or performing

a language translation#


26/81

'exical semantics Approaches

%. based on emantic grammars

+.uses conceptual dependency theory.

emantic rammar

- a context free grammar in which the choice of nonterm inals and

production rules is governed by semantics as well as syntactic

function.

- there is usually a semantic action associated with each grammar rule.

Eg: *rimitive action 5ET with unfilled slots A3T$;!$17E3T and

TEE

=5ET =A3T$; nil>

=$17E3T nil>

=TEE past>


27/81

,IN)2+T ,A?T(3 nil

,(7C2?T nil

,T2N+2 past

The o" drank a soda

,IN)2+T ,A?T(3 ,PP NAM2 o",?!A++ PDE-(7C

,TEP2 ANIMAT2,32; 2;

,(7C2?T ,PP,NAM2 soda,?!A++ PDE-(7C

,TEP2 INANIMAT2,32; IN2;

,T2N+2 past


28/81

?ompositional semantics Approaches

The meaning of an e'pression is derived from the meanings of the parts

of the e'pression#

- The target knowledge structures constructed in this approach are

t"picall" logic e'pressions such as the formulas of ;(P!#

2g0 N! statement - Sample24 contains silicon

3esult of parsing

(S DCL

(NP (N Sample 24)))

(AU (!"NS"(P#"S"N!)))

(VP (V contain))

(NP (N (silicon))))Bsing this structure/ the semantic interpreter would produce the following

predicate clause

(C$N!A%N sample24 silicon)


29/81

Natural language )eneration

2'act inverse of language undestanding#

More difficult than understanding/ecause the s"stem must

decide

- what to sa"/ andhow the utterances should e stated

which form is etter,active or passive

which words and structures est e'press the intent

when to sa" what#


30/81

The stud" of language generation falls naturall" into three areas0

@ the determination of content

formulating and developing a te't utterance plan/ and

= achieving a reali9ation of desired utterances#

?ontent determination

?oncerned with what details to include in an e'planation/ a re8uest/ a

8uestion or argument in order to conve" the meanings set forth "

the goals of the speaker#

Te't planning

Process of organi9ing the content to e communicated so as to

achieve the goals of the speaker#

3eali9ation

* the process of mapping the organi9ed content to actual te't#


31/81


32/81

Pattern 3ecognition

3omputer pattern recognition

- a process whereby computer programs are used to recogniCe

various forms of input stimuli such as visual or acoustic=speech>

patterns.

*attern recognition ystems are used to identify or classify

obects on the basis of their attribute and attribute-relation

values.

;ecognition is the process of establishing a close match between

some new stimulus and previously stored stimulus patterns.


33/81

D $bect classification is closely related to recognition.

D The ability to classify or group obects according to

some commonly shared features is a form of class

recognition.D 3lassification is

- essential for decision ma)ing! learning! and many

other cognitive acts.- 6epends on the ability to discover common patterns

among obects.


34/81

!&e reco'nition and classiication process

tep %

- stimuli produced by obects are perceived by sensory devices.The more prominent attributes= such as siCe! shape! color! andtexture> produce the strongest stimuli. The values of these attributesand their relations are used to characteriCe an obect in the form of a

pattern vector

- The range of characteristic attribute values is )nown as themeasurement space #

tep +

A subset of attributes whose values provide obect grouping or

clustering are selected. The range of the subset of attribute values is )nown as the feature

space F.


35/81

tep ,


36/81

The pattern recognition process

;eature

selection Matching

?lassification rules

+ensors

!earning

?lassification

+timuli


37/81

F There are two asic approaches to the

recognition prolem

@The decision theoretic approach

The s"ntactic approach


38/81

ecision Theoretic classification

F 7ased on the use of decision functions to classif" o1ects#

F A decision function maps pattern vectors G into decision

regions of #

+"ntactic ?lassification

-The s"ntactic recognition approach is ased on the

uni8ueness of s"ntactic structure6 among the o1ect

classes#

- a kind of grammar is defined for o1ect descriptions# - vocaula" is ased or shape primitives#


39/81

!earning ?lassification Patterns

F 7efore a s"stem can recogni9e o1ects/ it must posses

knowledge of the characteristics features for those

o1ects

F !earning decision functions/ grammars or other rules can

e performed in either of the two wa"s/ through

F +upervised learning

F Bnsupervised learning


40/81

+upervised !earning

- accomplished " presenting training e'amples to alearning unit#

The e'amples are laelled eforehand with their correct

identities or class# The attriute values and o1ect laelsare used " the learning component to e'tract and

determine pattern criteria for each class# This knowledge

is used to ad1ust parameters in decision functions or

grammer rewrite rules#


41/81

Bnsupervised !earning

- !aled training e'amples are not availale and little is

known eforehand regarding the o1ect population#In

such cases/ the s"stem must e ale to perceive and

e'tract relevant properties from the unknown o1ects/find common pattern among them/ and formulate

descriptions or discrimination criteria consistent with the

goals of the recognition process#


42/81

!earning through ?lustering

- 3lustering is the process of grouping or classifying obects on the basis of

a close association or shared characteristics.

- a discovery learning process in which similar patterns are found among a

group of obects.

The clustering problem gives rise to several subproblems

%. What set of attributes and relations are most relevant! and what weights

should be given to each?

+. What representation formalism should be used to characteriCe the obects?

,. What representation scheme should be used to describe the cluster

groupings or classifications?. What clustering criteria is most consistent with and effective in achieving

the obectives relative to the context or domain?

/. What clustering algorithms can best meet the criteria


43/81


44/81

F To covert speech to on- screen te't / a computer has to go through severalcomple' steps#

$hen we speak we create virations in the air# The analog-to-digitalconverter,A? translates this analog wave into digital data that thecomputer can understand#

To do this / it digiti9es the sound " taking precise measurements ofthe wave at fre8uent intervals# The s"stem filters the digiti9ed sound toremove unwanted noise/ and sometimes to separate it into different andsof fre8uenc"#

Ne't the signal is divided into small segments and the program thenmatches these segments to known phonemes in appropriate language# Aphoneme is the smallest element of a language- a representation of thesounds we make and put together to form meaningful e'pressions# The

program e'amines phonemes in the conte't of other phonemes aroundthem# The software language model compares the phonemes to wordsin its uilt-in dictionar"# The program then determines what the user wasproal" sa"ing and either outputs it as te't or issues a computercommand

Dow speech recognition works %


45/81


46/81

2'pert +"stem Architectures


47/81

"pert Systems

* a recent product o A%

* a +ind o +nowled'e based systems

* &a,e pro,en to be eecti,e in a number o problem

domains w&ic& re-uire t&e +ind o intelli'ence

possessed by a &uman epert.

Application Domains

Law aerospace

C&emistry military operations

/iolo'y inance

"n'ineerin' ban+in'

0edicine 'eolo'y

manuacturin'


48/81

efinition

A set o pro'rams desi'ned to act as anepert in a particular domain.

ot meant for replacing experts in that domain!but to assistthem.

?h t i ti f t f 2 t


49/81

?haracteristic features of 2'pert

s"stems Use +nowled'e rat&er t&an data

1nowled'e is encoded and maintained separately.

Capable o eplainin' &ow a particular conclusion

was reac&ed

Use symbolic representations or +nowled'e

Can reason wit& meta +nowled'e

%mportance o "pert Systems


50/81

2'pert +"stem Architectures

@ #ule based System or Production Systems

-use +nowled'e encoded in t&e orm o production rulesie . % .....t&en... rules.

"ac& rule represents a small piece o +nowled'erelatin' to t&e 'i,en domain o epertise.

? t f 2 t


51/81

?omponents of an 2'pert

+"stem

"planation0odule

%$%nterace

"ditor

%nerence"n'ine

1nowled'e base

Case &istory

ile

3or+in'memory

Learnin' 0odule

%NPU!

$U!PU!


52/81

1nowled'e /ase

Contains acts and rules about some domain.

"'

%5 !&e patient &as a c&ronic disorder6and t&e a'e o t&e patient is less t&an 786 and

t&e patient s&ows condition A6 and

test / re,eals bioc&emistry condition C

!9"N Conclude t&e patient:s dia'nosis is autoimmune*

c&ronic*&epatitis


53/81


54/81

The inferring process is carried out

recursivel" in three stages0

@ match

select

= e'ecute

The Production s"stem Inference c"cle


55/81

1nowled'e base 3or+in' 0emory

matc&

Conlict Set

Select

eecute

The Production s"stem Inference c"cle


56/81

7uilding a :nowledge ase

An editor is used by de,elopers to create new rulesor addition to t&e +nowled'e base6 to delete

outdated rules 6or to modiy eistin' rules in some

way.

0ost diicult tas+ in creatin' and maintainin'production systems is

*buildin' and maintainin' o a consistent but complete

set o rules. !&is s&ould be done wit&out addin'

redundant or unnecessary rules. "' o an intelli'ent editor < !"%#"S%AS (de,eloped to

wor+ wit& systems li+e 0=C%N)


57/81

IH( Interface

Permits the user to communicate with the s"stem in a

more natural wa"#

The s"stem must have special prompts or a speciali9ed

vocaular" which encompasses the terminolog" of the

given domain of e'pertise#

2g0 ME?IN has a vocaular" of some words#


58/81

!earning module and histor" file

- Not common components of e'pert

s"stems

- Bsed to assist in uilding and refining the

knowledge ase

Non Production s"stem


59/81

Non Production s"stem

Architectures- less common e'pert s"stem architecture#

- Instead of rules/ these s"stems emplo" more structured

representation schemes like

Associative or semantic networks

;rame structures

ecision trees

+peciali9ed networks like neural networks#

Associative or +emantic Network


60/81

Associative or +emantic Network

Architectures- useful in representing hierarchical knowledge structures/

where propert" inheritance is common#

- Not a popular form of representation for standard e'pert

s"stems#

- can e used in natural language s"stems or computer

vision s"stems also#

2g0 2'pert s"stem ased on the use of an associative

network representation * ?A+N2T

?A+N2T * ?ausal Associational Network

-used to diagnose and recommend treatment for

)laucoma


61/81

bird

ly

tweetyyellow

win's

CAN

A*1%ND*$5

C$L$#

9AS PA#!S

5ra'ment o an associati,e networ+


62/81

/ob Proessor

/i+e9ouseSandy

0A##%"D!$ $3NS

%SA

D#%V"S


63/81

5rame Arc&itectures

2g0 for a frame ased e'pert s"stem - PIP s"stem

PIP * Present Illness Program

Medical knowledge in PIP is organi9ed in frame structures#

:nowledge for e'pert s"stems ma" e stored in the form

of a decision tree when the knowledge can e structured

in a top-to-ottom manner#

:nowledge ase can e constructed with a special tree-uilding editor or with a learning module#

Decision !ree Arc&itectures

A se'ment o decision tree structure


64/81

A se'ment o decision tree structure

attribute>

oran'ered blue

yes no yes yesno no

/urn test

Solubility test

Compound*7? Compound*7@

J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J


65/81


66/81

7lackoard +"stem Architecture

- a special type o +nowled'e*based system w&ic& uses aorm o opportunistic reasonin'.

- Diers rom pure orward or pure bac+ward c&ainin'

- "it&er direction may be c&osen dynamically at eac&

sta'e in t&e problem solution process.

- /lac+board systems are composed o

a number o +nowled'e sources

a 'lobally accessible database structure6called ablac+board

Control %normation


67/81

1igsaw pu99le

A pule consistin' o a mass o irre'ularly s&aped pieceso cardboard6 plastic6 or wood t&at orm a picture w&enitted to'et&er. Also called picture puzzle.

?omponents of lackoard s"stems


68/81

?omponents of lackoard s"stems

1nowled'e sources

Control %normation

/lac+board

:nowledge sources


69/81

- separate and independent sets of coded knowledge

- ma" contain knowledge in the form of procedures/ rules/ or

other schemes#

2ach knowledge source ma" e thought of as a specialist in

some limited area needed to solve a given suset of

prolems

7lackoard

- ?ontain current prolem state and information needed " the

knowledge sources such as input data/ partial solutions/control data/ alternatives/ final solutions

- :nowledge sources make changes to the lackoard data#

- ?ommunication and interaction etween the knowledge

sources takes place solel" through the lack oard#

? t l I f ti


70/81

?ontrol Information

- Ma" e contained within the sources/ on the lack oard/

or possil" in a separate module#

- Monitors the changes to the lackoard and determines

what the immediate focus of attention should e in solving

the prolem#

- (ne of the application of 7lackoard +"stem Architecture

was in the D2A3+AE famil" of pro1ects,speech

understanding s"stems

Analogical 3easoning


71/81

Analogical 3easoning

Architectures- sol,e new problems li+e &umans6 by indin' a similar

problem solution t&at is +nown and applyin' t&e +nown

solution to t&e new problem6 possibly wit& some

modiications.

- 3ill re-uire a lar'e +nowled'e base &a,in' numeroussolutions and ot&er pre,iously encountered situations

or episodes.

- !&e inerence mec&anism must be able to etend

+nown situations or solutions to it t&e current problemand ,eriy t&at t&e etended solution is reasonable.


72/81

Neural Network Architectures


73/81

Artificial Neural networks


74/81

Artificial Neural networks

ANN are mathematical inventions inspired by observations

made in the study of biological system.

Loosely based on the actual Biology

Can be described as mapping an input space to an output

space.

Consists of artificial neurons composed of eights and

connections.


75/81


76/81

Modeling Neurons

A simplified model of the neuron

%

NP

U

!

S

$U!PU!

Articial neuron can e thought of as a small computing engine that takes in

input/ process them and then transmit an output#

K f L$ G7


77/81

K.f L$i Gi

7

2

>

8

3732

3>

38

B

i 8


78/81


79/81

Neural Network Architecture

Neural networks'arge networ)s of simple processing elements or nodes which process

information dynamically in response to external inputs

The nodes are simplified models of neurons.

The )nowledge in a neural networ) is distributed throughout the networ) inthe form of internode connections and weighted lin)s which form the

inputs to the nodes.

The lin) weights serve to enhance or inhibit the input stimuli values which are

then added together at the nodes.5f the sum of all the inputs to a node

exceeds some threshold value T! the node executes and produces an outputwhich is passed on to other nodes or is used to produce some output

response.

o output is produced if the total input is less than T

:nowledge +"stem 7uilding


80/81

:nowledge +"stem 7uilding

Tools- t&ese tools ran'e rom &i'& le,el pro'rammin'

lan'ua'es to intelli'ent editors.

3&en e,aluatin' buildin' tools or epert system

de,elopment6 t&e de,eloper s&ould consider t&e

ollowin' eatures and capabilities

>. 1nowled'e representation met&ods a,ailable.

2. %nerence and control met&ods a,ailable.

7. User interace c&aracteristics.

4. eneral system c&aracteristics and support a,ailable.


81/81

Personal ?onsultant Plus

3adian 3ule master

:22,:nowledge 2ngineering 2nvironment

(P+5 +"stem

natural language processing artificial intelligence

Documents