helbredte

1

Chapters 11 & 12, 13: Knowledge Acquisition,

Representation and Validation Opening Vignette: American Express

Improves Approval Selection with Machine Learning

The Problem: Loan Approval85 to 90 % Predicted Accurately10 to 15 % in Gray AreaAccuracy of Loan Officer’s Gray Area Decisions were at most 50 %

2

The Solution

ES with Knowledge Acquisition Method of Machine LearningRule Induction Method Gray Area: Induced Decision Tree Correctly Predicted 70 %Induced Rules Explain Why Rejected

3

Knowledge Engineering Deals with knowledge acquisition, representation, validation, inferencing, explanation and maintenance

It is the art of building complex computer programs that represent and reason with knowledge of the world

– (Feigenbaum and McCorduck [1983])

4

Knowledge Engineering Process

ActivitiesKnowledge AcquisitionKnowledge RepresentationKnowledge ValidationInferenceExplanation and Justification

5

Knowledge Acquisition Knowledge acquisition is the extraction of knowledge from sources of expertise and its transfer to the knowledge base and sometimes to the inference engine– Documented (books, manuals, etc.)– Undocumented (in people's minds)

• From people, from machines

– Knowledge Acquisition from Databases– Knowledge Acquisition Via the Internet

6

Difficulties in Knowledge Acquisition

Expressing the Knowledge– expert’s “rules” are compiled and often difficult to

articulate– not available to conscious introspection– knowledge engineer’s domain “knowledge”Transfer to a MachineNumber of ParticipantsStructuring the Knowledge

7

Experts may lack time or not cooperateTesting and refining knowledge is complicatedPoorly defined methods for knowledge elicitationSystem builders may collect knowledge from one source, but the relevant knowledge may be scattered across several sourcesCollect documented knowledge rather than use expertsThe knowledge collected may be incompleteDifficult to recognize specific knowledge when mixed with irrelevant dataExperts may change their behavior when observed and/or interviewedProblematic interpersonal communication between the knowledge engineer and the expert

Other Reasons

8

Overcoming the Difficulties

Knowledge acquisition tools with ways to decrease the representation mismatch between the human expert and the program (“learning by being told”)– Simplified rule syntax – Natural language processor to translate knowledge to a

specific representation

Critical– The ability and personality of the knowledge engineer – Must develop a positive relationship with the expert– The knowledge engineer must create the right impression

Computer-aided knowledge acquisition tools

9

Required Skills and Characteristics of Knowledge

Engineers Computer skillsTolerance and ambivalenceEffective communication abilitiesBroad educational backgroundAdvanced, socially sophisticated verbal skillsFast-learning capabilities (of different domains)Must understanding organizations and individualsWide experience in knowledge engineeringIntelligenceEmpathy and patiencePersistenceLogical thinkingVersatility and inventivenessSelf-confidence

10

Methods of Knowledge Acquisition: An Overview

Manual – Interviewing (Structured, Semistructured, Unstructured)– Tracking the Reasoning Process – Observing

Manual methods: slow, expensive and sometimes inaccurate

Semiautomatic– Support Experts Directly

Automatic (Computer Aided)– Expert’s and/or the knowledge engineer’s roles are

minimized (or eliminated) – Induction Method

11

Recommendation

Before a knowledge engineer interviews the expert(s)1. Interview a less knowledgeable (minor) expert

– Helps the knowledge engineer • Learn about the problem• Learn its significance• Learn about the expert(s)• Learn who the users will be• Understand the basic terminology• Identify readable sources

2. Next read about the problem3. Then, interview the expert(s) (much more

effectively)

12

Tracking Methods

Techniques that attempt to track the reasoning process of an expertFrom cognitive psychologyMost common formal method: Protocol Analysis

13

Protocol Analysis

Protocol: a record or documentation of the expert's step-by-step information processing and decision-making behaviorThe expert performs a real task and verbalizes his or her thought process (think aloud)

14

TABLE 13.3 Advantages and Limitations of Protocol Analysis

Advantages Limitations

Expert consciously considers decision-making heuristics

Requires that the expert be aware ofwhy he or she makes a decision

Expert consciously considers decisionalternatives attributes, values

Requires that the expert be able tocategorize major decision alternatives

Knowledge engineer can observe andanalyze decision-making behavior

Requires that the expert be able toverbalize the attributes and values of adecision alternative

Knowledge engineer can record, andlater analyze with the expert, keydecision points

Requires that the expert be able toreason about the selection of a givenalternative

Subjective view of decision makingExplanations may not track withreasoning

Source: K. L. McGraw and B. K. Harbison-Briggs, Knowledge Acquisition, Principles andGuidelines, Englewood Cliffs, NJ: Prentice-Hall, 1989, p. 217.

15

Observations and Other Manual Methods

Observe the Expert Workcard sortinginformation display boardsCase analysisCritical incident analysisDiscussions with the usersCommentariesConceptual graphs and modelsBrainstormingMultidimensional scaling

16

Expert-driven Methods Knowledge Engineers Typically – Lack Knowledge About the Domain– Are Expensive– May Have Problems Communicating With Experts

Knowledge Acquisition May be Slow, Expensive and UnreliableCan Experts Be Their Own Knowledge Engineers?Two approaches– Manual – Computer-Aided (Semiautomatic)

• To reduce or eliminate the potential problems – E.g. REFINER+ - case-based system – TIGON - to detect and diagnose faults in a gas turbine engine

17

Manual Method:Expert's Self-reports

Problems with Experts’ Reports and Questionnaires

1. Requires the expert to act as knowledge engineer

2. Reports are biased3. Experts often describe new and untested

ideas and strategies4. Experts lose interest rapidly5. Experts must be proficient in flowcharting6. Experts may forget certain knowledge7. Experts are likely to be vague

18

Machine Learning

Manual and semiautomatic elicitation methods: slow and expensiveOther Deficiencies– Frequently weak correlation between verbal reports

and mental behavior– Sometimes experts cannot describe their decision

making process– System quality depends too much on the quality of the

expert and the knowledge engineer– The expert does not understand ES technology– The knowledge engineer may not understand the

business problem – Can be difficult to validate acquired knowledge

19

Computer-aided Knowledge Acquisition, or Automated

Knowledge Acquisition Objectives

Increase the productivity of knowledge engineeringReduce the required knowledge engineer’s skill levelEliminate (mostly) the need for an expertEliminate (mostly) the need for a knowledge engineerIncrease the quality of the acquired knowledge

20

Automated Knowledge Acquisition (Machine

Learning)

Rule InductionCase-based ReasoningNeural Computing Intelligent Agents

21

Machine LearningKnowledge Discovery and Data Mining Include Methods for Reading Documents and Inducing Knowledge (Rules)Other Knowledge Sources (Databases)Tools– KATE-Induction – CN-2

22

Automated Rule Induction

Induction: Process of Reasoning from Specific to GeneralIn ES: Rules Generated by a Computer Program from CasesInteractive Induction

23

TABLE 13.6 Case for Induction - A Knowledge Map

(Induction Table)

Attributes

AnnualApplicant Income ($) Assets ($) Age Dependents Decision

Mr. White 50,000 100,000 30 3 Yes

Ms. Green 70,000 None 35 1 Yes

Mr. Smith 40,000 None 33 2 No

Ms. Rich 30,000 250,000 42 0 Yes

24

TABLE 13.12 Admission Cases

Case # GMAT GPA Decision

1 510 3.5 Yes

2 620 3.0 Yes

3 580 3.0 No

4 450 3.5 No

5 655 2.5 Yes

25

Case-based Reasoning (CBR) Adapt solutions used to solve old

problems for new problems CBR

– Finds cases that solved problems similar to the current one, and

– Adapts the previous solution or solutions to fit the current problem, while considering any difference between the two situations

Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ

26

Finding Relevant Cases Involves

– Characterizing the input problem, by assigning appropriate features to it

– Retrieving the cases with those features– Picking the case(s) that best match the

input best

– Extremely effective in complex cases– Justification - Human thinking does not

use logic (or reasoning from first principle)

– Process the right information retrieved at the right time


27

CBR Construction - Special Tools - Examples

ART*Enterprise and CBR Express (Inference Corporation)

KATE (Acknosoft)

ReMind (Cognitive Systems Inc.)


28

Example: College Major Advisor

• who are the experts?• what do they do?• how do they do it?• how do we know?• how can we represent their knowledge and

reasoning process?• how can the system mimic their reasoning

process?

29

APTITUDE INTERESTS FINANCIALNEED

RECOMMENDATIONOF A MAJOR

College Major Advisor: Initial Block Diagram

30

APTITUDE INTERESTS FINANCIALNEED

RECOMMENDATIONOF A MAJOR

College Major Advisor: Expanded Block Diagram

mathskills

prog.skills

manualdexterity

computers problemsolving

repairthings

desk vs.field

job atgraduation

31

math? (yes, no)

programming (yes, no)

manual dexterity (yes, no)

computers (yes, no)

problem solving (yes, no)

place (desk, field)

repair (yes, no)

finance (yes, no)

APTITUDE

INTERESTS

SUGGESTEDMAJOR

College Major Advisor: Dependency Diagram

32

Induction Table Example

Induction tables (knowledge maps) focus the knowledge acquisition process

Choosing a site for a hospital clinic facility

33

TABLE 13.9: Induction Table (Knowledge Map) Example

PopulationDensity

Densityover HowMany Sq.mi

Number of Near(within 2 miles)Competitors

AverageFamilyIncome

Near PublicTransportation?

Decision(Choices)

People /Square Mile

Numeric,Region Size

0, 1, 2, 3, ... Numeric,$ / Year

Yes, No Yes, No

>= 2000 >=4 0 Yes

>=3500 >=4 1 Yes

>=2 No

<30,000 No

34

Row 1: Factors Row 2: Valid Factor Values and Choices (last column)

Table leads to the prototype ESEach row becomes a potential ruleInduction tables can be used to encode chains of knowledge

35

Knowledge Representation

a variety of alternative representations have been developed: rules, frames, semantic nets

efficiency and effectiveness criteria incorporation into ES development

software

36

Major Advantages of Rules

Easy to understand (natural form of knowledge)

Easy to derive inference and explanations

Easy to modify and maintain Easy to combine with

uncertainty Rules are frequently

independent

37

Production Rules: Drawbacks variables

– unwieldy for large body of knowledge– only stand-alone, unrelated facts

production rules– have to embed all context into antecedent– not appropriate for all types of problem

solving or expertise

38

Frames an abstraction of a single object or a

concept– frame representing a car

slots are used to define properties and attributes of a frame

efficient for hierarchical knowledge structures: supports inheritance

inference can be done with production rules based on value expectations

KR With FramesANIMALS

MAMMALS REPTILES BIRDS FISH

DOG HUMAN WHALEPENGUIN

LONG JOHNSILVER

Reproduce: YesLife-Form: Yes

Scales: YesWarmblooded: No

Warmblooded: YesSpinal Cord: YesLegs: 4

Legs: 2

Legs: 1

Legs: 0Fins: Yes

Legs: 2Wings: YesFeathers: Yes

Feathers: No

Scales: YesFins: yes

40

Semantic Networks

network of nodes and links– nodes = concepts– links = relationships

supports certain types of inference– “is-a” = inheritance (generalization/specialization)– “has-a” = property aggregation– “part-of” = physical component aggregation

41

KR With Semantic Networks

LIGHT

WORKINGHEADLIGHT

FUNCTIONALLIGHT BULB

FUNCTIONALCIRCUIT

IS-A

HAS-A HAS-A

42

Semantic Networks advantages

– supports limited inferencing– good model of long-term memory

organization?– easy to understand

disadvantages– quickly becomes unwieldy– doesn’t distinguish between objects,

attributes, or values

KR with Multiple Schemes

IF Battery-Age < 5 and Battery_terminals are Uncorroded THEN Battery can work

If Battery can work and Fuse is Functional and Wiring is Functional THEN Electric Circuit can work

ELECTRIC SUBSYSTEM

IGNITION HEADLIGHT

LIGHT BULB ELECTRIC CIRCUITWorking

Head Light

FunctionalLight Bulb

FunctionalCircuit

WorkingBattery

FunctionalFuse

FunctionalWiring

5-YearsOld

UncorrodedTerminals

HAS-A HAS-A

HAS-A

HAS-A

NEEDS-A HAS-A

NEEDS-A

44

Choosing a Representation Scheme should support acquisition, retrieval,

and reasoning naturalness, uniformity, and

comprehensibility degree of explicitness Modularity and flexibility of the

knowledge base efficiency of knowledge retrieval and

heuristic power

45

Validation & Verification of the Knowledge Base

Evaluation– Assess an expert system's overall value– Analyze whether the system would be

usable, efficient and cost-effective

Validation – Deals with the performance of the system

(compared to the expert's)– Was the “right” system built (acceptable

level of accuracy?)

Verification– Was the system built "right"?– Was the system correctly implemented to

specifications?

46

TABLE 13.8 Measures of Validation

Measure (Criteria) Description

Accuracy How well the system reflects reality; how correct the knowledge is in theknowledge base

Adaptability Possibilities for future development, changes

Adequacy (or completeness) Portion of the necessary knowledge that is included in the knowledge base

Appeal How well the knowledge base matches intuition and stimulates thought andpracticability

Breadth How well the domain is covered

Depth Degree of the detailed knowledge

Face validity Credibility of knowledge

Generality Capability of a knowledge base to be used with a broad range of similarproblems

Precision Capability of the system to replicate particular system parameters;consistency of advice; coverage of variables in knowledge base

Realism Accounting for relevant variables and relations; similarity to reality

Reliability Fraction of the ES predictions that are empirically correct

Robustness Sensitivity of conclusions to model structure

Sensitivity Impact of changes in the knowledge base on quality of outputs

Technical andoperational validity

Quality of the assumed assumptions, context, constraints and conditions, andtheir impact on other measures

Turing Test Ability of a human evaluator to identify if a given conclusion is made by anES or by a human expert

Usefulness How adequate the knowledge is (in terms of parameters and relationships)for solving correctly

Validity Knowledge base's capability of producing empirically correct predictions

Source: Adapted from B. Marcot, "Testing Your Knowledge Base," AI Expert, August 1987.

47

Method for Validating ES

Test1. The extent to which the system

and the expert decisions agree2. The inputs and processes used by

an expert compared to the machine

3. The difference between expert and novice decisions

(Sturman and Milkovich [1995])

48

ES Elements to be Validated output, recommendations

– internal validity: completeness and consistency– external validity: test cases, Turing test

reasoning system/user interface efficiency

– hardware, software– KR and response time

cost-effectiveness

49

Inferencing Methods objective

– find logical reasoning path between known data and conclusions

backward chaining– goal-directed search– start with where you want to end up and see if

data will get you there

forward chaining– data-directed search– start with what you know and see where it takes

you

50

ExampleVariables: A = have $10,000

B = younger than 30C = education at college levelD = annual income > = $40,000E = invest in securitiesF = invest in growth stocksG = invest in IBM stocks

Rules: R1: if A and C then ER2: if D and C then FR3: if B and E then FR4: if B then CR5: if F then G

Initial Facts: A and B

51

Inferencing Methods

which way should you go?– forward chaining tends to collect more data– forward chaining is good for design problems– backward chaining tends to examine more rules – backward chaining is good for diagnosis problems

vp-expert is backward chaining other expert system shells are forward

chaining, backward chaining, or both

52

Inferencing with Uncertainty Uncertainty in AI - Three-step Process

1. An expert provides inexact knowledge in terms of rules with likelihood values. Also, users may be uncertain with respect to inputs.

2. The inexact knowledge of the basic set of events can be directly used to draw inferences in simple cases (Step 3)

3. Working with the inference engine, experts can adjust the Step 1 input after viewing the results in Steps 2 and 3.

– In Step 2: Often the various events are interrelated. – Necessary to combine the information provided in Step 1

into a global value for the system

Major integration methods: Bayesian probabilities, theory of evidence, certainty factors and fuzzy sets


53

Theory of Certainty (Certainty

Factors)

Uncertainty is represented as a Degree of Belief or Certainty FactorCertainty Factors (CF) express belief in an event (or fact or hypothesis) based on evidence (or the expert's assessment)CFs are NOT probabilitiesCFs need not sum to 100


54

Rule 9if wine_color = red and body = full and sweetness = drythen wine = cabernet_sauvignon cnf 80because "This is a good wine~";

Rule 1if main_component = meatthen wine_color = red; Rule 3if has-sauce = yes and sauce_type = creamythen body = full cnf 90because "Creamy sauce needs full bodied wine"; Rule 8if likes = drythen sweetness = dry;

INPUTS: Meat cnf=100; has_sauce=95; creamy=70; likes dry=.60INTERMEDIATE: wine-color=red 100; body=full 63 [min(95,70)*90]; sweetness=dry 60FINAL: wine=cabernet sauvignon 48 [(min 100, 63, 60)*80]


55

Calculating Confidence Factors

For conjunction (AND): – take min CNF of conditions

For disjunction (OR):– when one condition only is true take the CNF of true condition

– when both conditions are true:– CNF antecedent=max CNF of conditions

– CNF Rule: Antecedent CNF*Consequent CNF

56

AND

– IF inflation is high, CF = 50 percent, (A), AND– IF unemployment rate is above 7 percent, CF =

70 percent, (B), AND– IF bond prices decline, CF = 100 percent, (C)– THEN stock prices decline

CF(A, B, and C) = Minimum[CF(A), CF(B), CF(C)] The CF for “stock prices to decline” = 50 percentThe chain is as strong as its weakest link

57

IF inflation is low, CF = 70 percent; OR

IF bond prices are high, CF = 85 percent;

THEN stock prices will be high

Only one IF need be true Conclusion has a CF with the maximum of the two

– CF (A or B) = Maximum [CF (A), CF (B)]

CF = 85 percent for stock prices to be highDecision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ

OR

58

Combining Two or More Rules

Example:– R1: IF the inflation rate is less than 5 percent,

THEN stock market prices go up (CF = 0.7)

– R2: IF unemployment level is less than 7 percent,THEN stock market prices go up (CF =

0.6)

Inflation rate = 4 percent and the unemployment level = 6.5 percent Combined Effect– CF(R1,R2) = CF(R1) + CF(R2)[1 - CF(R1)]; or – CF(R1,R2) = CF(R1) + CF(R2) - CF(R1) CF(R2)


59

Assume an independent relationship between the

rules Example: Given CF(R1) = 0.7 AND CF(R2) = 0.6, then: CF(R1,R2) = 0.7 + 0.6(1 - 0.7) = 0.7 + 0.6(0.3) = 0.88

ES tells us that there is an 88 percent chance that stock prices will increase


helbredte

Documents

experts knowledge acquisition

specific knowledge

documented knowledge

extraction of knowledge

refining knowledge

knowledge base

relevant knowledge

knowledge experts rules