To Tell the Whole Truth and Nothing but the Truth The Strategic Regulation of Memory Accuracy and Informativeness Morris Goldsmith University of Haifa.

To Tell the Whole Truth and Nothing but the Truth

The Strategic Regulation of Memory Accuracy

and Informativeness

Morris Goldsmith

University of Haifa

BPS Cognitive Section 25th Anniversary Conference – Broadbent Lecture

Department of PsychologyInstitute of Information Processing and Decision Making



and Informativeness

Morris Goldsmith

University of Haifa





and Informativeness

Morris Goldsmith

University of Haifa



Trying ...

Control over memory reporting: A “real-life” example

Q: Please tell us what you saw as you were getting out of your car.

A: I had just opened the door when I heard someone scream. As I looked

up, a man in a dark sweatsuit burst through the gate of the yard and ran

full speed down the alley. I think he was carrying a bag or something.

Q: This bag – what color was it?

A: Umm… I'm not sure.

Q: Could you take a guess?

A: No.

Q: Do you remember what time it was?

A: Around 6 o'clock, maybe 6:30.

Q: Can you be more specific?

A: Umm… I’d say between 6:15 and 6:30.

Introduction

Personal Control vs. Experimenter Control

"Ironically, although the self-directed processes are not explicitly

acknowledged in most theories of memory, there is an implicit

acknowledgment on the part of investigators concerning the

importance of such processes. The evidence for this is that

investigators go to such great lengths to design experiments that

eliminate or hold those self-directed processes constant via

experimental control!"

Nelson & Narens (1994, p. 8)

Introduction


Approaches to personal control: Eliminate it Correct for it Ignore it

Introduction


Approaches to personal control: Eliminate it Correct for it Ignore it

STUDY IT!

Introduction

Grain size:

PRECISE OR

COARSE

Report option:

VOLUNTEER OR

WITHHOLD

Introduction

Nelson and Narens (1990)

Two Types of Report Control

Report Option – Withholding particular items of information

(responding “don’t know” or “don’t remember”) in order

to screen out wrong answers.

Grain Size – choosing a level of coarseness or generality at

which the answer is unlikely to be wrong.

BOTH INVOLVE:

Metacognitive monitoring and control processes.

Substantial effects on memory performance.

Accuracy – Informativeness (Quantity) trade-off.

Introduction

Reported Answer(or Omission)

Goldsmith et al. (2002)

Criterion

Koriat & Goldsmith (1996, Psychological Review)

What was the defendant holding when he threatened the deceased?

A walking stickpa= .60

prc= .85 “Don’t know”

“to tell the whole truth (quantity), and nothing but the truth (accuracy)”

BUT: QUANTITY – ACCURACY TRADE-OFF !!!

Report Option

Retrieval Monitoring Control Performance

Retention ("memory"): the amount and quality of the information that can be retrieved.

Monitoring effectiveness: (confidence correctness)

the extent to which the assessed probabilities successfully differentiate correct from incorrect candidate answers.

Report criterion setting: (confidence volunteering)

the confidence threshold set in accordance with competing demands for quantity and accuracy.

Control sensitivity: (confidence volunteering)

the extent to which the volunteering or withholding of answers is in fact based on the monitoring output.

Koriat and Goldsmith (1996)

Report Option

Koriat and Goldsmith (1996, Experiment 1)

60 general-knowledge questions

Test Format: Recall or Recognition (between subjects)

Report Option: Free and Forced (within subjects)

Two phases: Phase 1 — Forced report + confidence judgments Phase 2 — Free report + Accuracy Incentive

Accuracy Incentive (between subjects): Moderate +1 each right answer, –1 each wrong

answer High +1 each right answer, –10 each wrong

answer

Report Option

Retrieval Monitoring Control Performance

TAPS:

RETRIEVAL (forced-report % correct)

MONITORING (confidence correctness)

CONTROL (confidence volunteering)

PERFORMANCE (free-report) Quantity (input-bound) - % of questions answered correctly. Accuracy (output-bound) - % of answers that are correct.


Report Option

Results

MONITORING: (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition


Report Option

Results

MONITORING: (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition

CONTROL SENSITIVITY (confidence volunteering) : Gamma averaged .97 for recall, .93 for recognition (!)


Report Option

Results

MONITORING (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition

CONTROL SENSITIVITY (confidence volunteering) Gamma averaged .97 for recall, .93 for recognition (!)

REPORT CRITERION (control policy)

Effect of Accuracy Incentive (High vs. Moderate): Mean number of volunteered answers: 26.9 vs. 30.9 Mean confidence for volunteered answers: .93 vs. .84 Mean estimated criterion level: .84 vs. .61 Criterion accounted for 92% of report decisions.


Report Option

+31

+10

+24

+5

-6

-4

-12

-7

Koriat and Goldsmith (1996, Experiment 1). Free-report means are adjusted for chance initial differences in retention between the two incentive conditions.

Report Option

The Contribution of Control Policy …

Report Option

Quantity-Accuracy Profile Plots (K&G, 1996, Exp. 1)

MODERATEINCENTIVE

HIGHINCENTIVE

Report Criterion Level

Contribution of Control

Quantity-Accuracy Payoff Profile Plots (K&G, 1996, Exp. 1)

MODERATEINCENTIVE

HIGHINCENTIVE



Quantity-Accuracy Payoff Profile Plots (K&G, 1996, Exp. 1)

MODERATEINCENTIVE

HIGHINCENTIVE



The Contribution of Monitoring …

Report Option

“Prototypical” – uniform confidence distribution and perfect calibration (moderate relationship between confidence and correctness).

.

Monitoring simulations (K&G, 1996)

“Perfect discrimination” – polarized confidence distribution and perfect calibration (perfect relationship between confidence and correctness).


“No discrimination” – uniform confidence distribution and flat calibration curve (no relationship between confidence and correctness).


As monitoring improves: Larger accuracy gains are achieved at lower quantity costs.



Test Format: Recall only.

Report Option (counterbalanced order)

Forced report + confidence judgments Free report + moderate accuracy incentive

Monitoring Manipulation

Two sets of items (mixed within-subject):

Standard items – good monitoring. Deceptive items – poor monitoring.

Example: What is the capital of Australia? (many non-British participants are confident that it is Sydney).

Contribution of Monitoring


Test Format: Recall only.

Report Option (counterbalanced order)

Forced report + confidence judgments Free report + moderate accuracy incentive

Monitoring Manipulation

Two sets of items (mixed within-subject):

Standard items – good monitoring. Deceptive items – poor monitoring.

Example: What is the capital of Australia? (many non-British participants are confident that it is Sydney).


Koriat and Goldsmith (1996, Experiment 2). Calibration curve for standard items.


Koriat and Goldsmith (1996, Experiment 2). Calibration curve for deceptive items.


+47

-4

+9

-6

+52

-3

Koriat and Goldsmith (1996, Experiment 2).


+47

-4

+9

-6

+52

-3

Koriat and Goldsmith (1996, Experiment 2).


Scanned Images\SDOC2240R_Page_2.jpg




Scanned Images\SDOC2240R_Page_2.jpg




Interim Conclusions

Free report memory performance depends on both

memory and metacognitive monitoring and control.

Effects on memory performance (manipulations or

population differences) may be mediated by any or all of

these components.

It is therefore important to isolate and assess the

contribution of these components whenever “real-life”

(free-report) memory is of concern.

The “free-forced” paradigm and QAP methodology allows

one to do so (see also Higham’s, 2002, 2007, Type-2 SDT approach).

Report Option

QAP or Type-2 SDT

QAP MeasureTypeDescriptionPhaseType-2SDT

Retention (retrieval or ecphory)

MemoryProportion or percentage of forced-report answers that are correct.

Forced SAME

Monitoring resolution (discrimination)

MonitoringWithin-individual gamma correlation between confidence (assessed probability correct) in each answer and the correctness of each answer, or alternative measures such as ANDI (Yaniv et al., 1991).

ForcedA’

Monitoring calibration over/underconfidence

MonitoringDifference between mean assessed probability correct and proportion correct (positive values reflect overconfidence.

ForcedB’’D

Monitoring calibrationsquared or absolute-value deviations

MonitoringMean squared or absolute-value difference between the mean assessed probability correct and proportion correct of each confidence category used in plotting a calibration curve.

Forced–

Control sensitivityControlWithin-individual gamma correlation between confidence (assessed probability correct) in each answer and whether or not it was volunteered.

Forced + Free

–

Report criterion (Prc) estimateControl Estimate of each participant’s report criterion

(assessed probability level) that yields the maximum fit (fit rate) with his or her actual report decisions.

Forced + Free

B’’D

Report Option

QAP or Type-2 SDT

QAP MeasureTypeDescriptionPhaseType-2SDT

Prc fit rateControlThe proportion of each participant’s actual volunteering decisions that are compatible with the derived Prc estimate, and which is maximized by this

estimate. Can also be used as an index of control sensitivity.

Forced + Free

–

Control effectivenessControlAbsolute value of the difference between the estimated Prc for each participant and the optimal Prc, identified

as the Prc level that would maximize the participants’

payoff.

Forced + Free

Bias Profile

Free-report quantity(input-bound)

PerformanceProportion of correct reported answers out of the total number of questions (or studied items).

FreeSAME

Free-report accuracy(output-bound)

PerformanceProportion of correct reported answers out of the number of answers that were volunteered.

FreeSAME

Report Option

Applications

Children’s memory (Koriat et al., 2001; Roebers & Schneider, 2002, 2005)

Aging (Kelley & Sahakyan, 2003; Pansky et al., in press; Rhodes & Kelley, 2005)

Clinical populations (Danion et al., 2001; Koren et al., 2004, 2006)

Psychometric testing (Higham, 2007; Higham & Arnold, 2007; Notea-Koren,

2005)

Social cognition (Payne et al., 2001)

Changes in accuracy over time (Koriat & Goldsmith, in progress)

Encoding specificity (Higham, 2002; Higham & Tam, 2005)

Report Option

Expanding the Framework – Grain Size …

The “Problem” of Grain Size . . .

Neisser (1988): Recall subjects seem to choose "a level of

generality at which they are not mistaken."

Fisher (1996): Memory accuracy remained constant over a

40-day retention interval.

Explanation: The later information was more "coarse."

List learning: Reporting BIRD instead of ROBIN.

Expanding the Framework – Grain Size …

A METHODOLOGICAL PROBLEM,

but more importantly . . .

A FURTHER MEANS OF STRATEGIC REGULATION.

What was the defendant holding when he threatened the deceased?Control over Grain Size

What time did the incident occur?

6:20? best guess

6:15 – 6:30?

6:00 – 6:30?

probably

highly likely

“Sometime in the early evening …”

definitely

ACCURACY - INFORMATIVENESS TRADE-OFF !!!

Yaniv & Foster, 1995, 1997

Experimental Design

PHASE 1 -- Forced grain size at two grain levels

EXAMPLE: When did Neil Armstrong walk on the moon?A) Specify a 3-year interval: From _____ - _____B) Specify a 10-year interval: From _____ - _____

EXAMPLE: How many chromosomes are there in the nucleus of a human cell?

A) Give a specific number: _____B) Specify a 20-chromosome interval: _____ -

_____

PHASE 2 -- Free choice of grain size For each item, choose the answer that you would prefer to provide if

you were "an expert witness testifying before a government committee."

Goldsmith, Koriat, & Weinberg-Eliezer (2002, JEP:General)

Grain Size

Results

Exps. 1, 2, & 3 Chose fine 40%; chose coarse 60%

Achieved accuracy = .60

[ p(fine correct) = .32; p(coarse correct) = .75 ]

Control: fine confidence grain choice: Gamma = .82

Criterion estimates sensitive to informativeness

incentive: .58 (high incentive) vs. .74 (low incentive)

Report criterion accounts for 88% of actual choices

Goldsmith et al. (2002)

A “satisficing model” (cf. Simon, 1956)

Grain Size

Strategic Regulation of Memory Grain Size over Time

Goldsmith, Koriat, & Pansky (2005, JML special issue on Metamemory)

Grain Size

Control of Grain Size and Report Option

Both involve an accuracy – informativeness trade-off.

Both involve monitoring the correctness of candidate answers.

Both involve setting a report criterion (accuracy satisficing)

per competing incentives for accuracy and informativeness.

A single integrated model?

Current Directions


start

Confident in FINE answer? Provide FINE answer

no

Confident in COARSE answer? Provide COARSE

answer

no

WITHHOLD the answer

yes

yes

Prc = .71

Prc = .71

Goldsmith et al. (in progress)

.83

.83

Accounts for 90% of grain choices

Current Directions

Morre Goldsmith

Note: The fact that the withhold option was utilized here does not have any relevance to the "pragmatic" considerations for why people would prefer to withhold answers, rather, it can be explained by subjective utility (because they were limited to grain sizes set by the experimenter).


start

Confident in FINE answer? Provide FINE answer

no

Confident in COARSE answer? Provide COARSE

answer

no

WITHHOLD the answer

yes

yes

Sufficiently informative?

yes

no

Prc = .71.83

Prc = .71.83

- Pragmatics (Grice, 1965)- Social/situational norms

Goldsmith et al. (in progress)

Current Directions

Morre Goldsmith

Note: The fact that the withhold option was utilized here does not have any relevance to the "pragmatic" considerations for why people would prefer to withhold answers, rather, it can be explained by subjective utility (because they were limited to grain sizes set by the experimenter).

Conclusions

Report option and grain size are both important means of

regulating accuracy and informativeness of memory reports.

We must understand such regulation in order to understand

the factors underlying memory performance, particularly in

real-life settings.

Expanded conception of “retrieval”: Includes cognitive,

metacognitive, neurocognitive, personality, and social-pragmatic

contributions to memory performance.

Much more work remains to be done, both on these, and on

other types of strategic memory regulation.

http://www.eyewitmem.org/index.php?option=com_content&task=section&id=1&Itemid=2

+31

+10

+24

+5

-6

-4

-12

-7


Recall – Recognition Paradox (K&G, 1994)

ConfoundACC: Free recall > Forced Choice QTY: Free recall < Forced Choice

Report Option

+31

+10

+24

+5

-6

-4

-12

-7


Recall – Recognition Paradox (K&G, 1994)

ConfoundACC: Free recall > Forced Choice QTY: Free recall < Forced Choice

Free ReportACC: Free recall < Free Choice QTY: Free recall < Free Choice

Report Option

To Tell the Whole Truth and Nothing but the Truth The Strategic Regulation of Memory Accuracy and Informativeness Morris Goldsmith University of Haifa.

Documents

control processes

truth accuracy

experimental control

memory performance

memory reporting

theories of memory

truth quantity

quantity accuracy tradeoff