To Tell the Whole Truth and Nothing but the Truth The Strategic Regulation of Memory Accuracy and Informativeness Morris Goldsmith University of Haifa.
Post on 29-Dec-2015
217 Views
Preview:
Transcript
To Tell the Whole Truth and Nothing but the Truth
The Strategic Regulation of Memory Accuracy
and Informativeness
Morris Goldsmith
University of Haifa
BPS Cognitive Section 25th Anniversary Conference – Broadbent Lecture
Department of PsychologyInstitute of Information Processing and Decision Making
To Tell the Whole Truth and Nothing but the Truth
The Strategic Regulation of Memory Accuracy
and Informativeness
Morris Goldsmith
University of Haifa
BPS Cognitive Section 25th Anniversary Conference – Broadbent Lecture
Department of PsychologyInstitute of Information Processing and Decision Making
To Tell the Whole Truth and Nothing but the Truth
The Strategic Regulation of Memory Accuracy
and Informativeness
Morris Goldsmith
University of Haifa
BPS Cognitive Section 25th Anniversary Conference – Broadbent Lecture
Department of PsychologyInstitute of Information Processing and Decision Making
Trying ...
Control over memory reporting: A “real-life” example
Q: Please tell us what you saw as you were getting out of your car.
A: I had just opened the door when I heard someone scream. As I looked
up, a man in a dark sweatsuit burst through the gate of the yard and ran
full speed down the alley. I think he was carrying a bag or something.
Q: This bag – what color was it?
A: Umm… I'm not sure.
Q: Could you take a guess?
A: No.
Q: Do you remember what time it was?
A: Around 6 o'clock, maybe 6:30.
Q: Can you be more specific?
A: Umm… I’d say between 6:15 and 6:30.
Introduction
Personal Control vs. Experimenter Control
"Ironically, although the self-directed processes are not explicitly
acknowledged in most theories of memory, there is an implicit
acknowledgment on the part of investigators concerning the
importance of such processes. The evidence for this is that
investigators go to such great lengths to design experiments that
eliminate or hold those self-directed processes constant via
experimental control!"
Nelson & Narens (1994, p. 8)
Introduction
Personal Control vs. Experimenter Control
Approaches to personal control: Eliminate it Correct for it Ignore it
Introduction
Personal Control vs. Experimenter Control
Approaches to personal control: Eliminate it Correct for it Ignore it
STUDY IT!
Introduction
Two Types of Report Control
Report Option – Withholding particular items of information
(responding “don’t know” or “don’t remember”) in order
to screen out wrong answers.
Grain Size – choosing a level of coarseness or generality at
which the answer is unlikely to be wrong.
BOTH INVOLVE:
Metacognitive monitoring and control processes.
Substantial effects on memory performance.
Accuracy – Informativeness (Quantity) trade-off.
Introduction
Reported Answer(or Omission)
Goldsmith et al. (2002)
Criterion
Koriat & Goldsmith (1996, Psychological Review)
What was the defendant holding when he threatened the deceased?
A walking stickpa= .60
prc= .85 “Don’t know”
“to tell the whole truth (quantity), and nothing but the truth (accuracy)”
BUT: QUANTITY – ACCURACY TRADE-OFF !!!
Report Option
Retrieval Monitoring Control Performance
Retention ("memory"): the amount and quality of the information that can be retrieved.
Monitoring effectiveness: (confidence correctness)
the extent to which the assessed probabilities successfully differentiate correct from incorrect candidate answers.
Report criterion setting: (confidence volunteering)
the confidence threshold set in accordance with competing demands for quantity and accuracy.
Control sensitivity: (confidence volunteering)
the extent to which the volunteering or withholding of answers is in fact based on the monitoring output.
Koriat and Goldsmith (1996)
Report Option
Koriat and Goldsmith (1996, Experiment 1)
60 general-knowledge questions
Test Format: Recall or Recognition (between subjects)
Report Option: Free and Forced (within subjects)
Two phases: Phase 1 — Forced report + confidence judgments Phase 2 — Free report + Accuracy Incentive
Accuracy Incentive (between subjects): Moderate +1 each right answer, –1 each wrong
answer High +1 each right answer, –10 each wrong
answer
Report Option
Retrieval Monitoring Control Performance
TAPS:
RETRIEVAL (forced-report % correct)
MONITORING (confidence correctness)
CONTROL (confidence volunteering)
PERFORMANCE (free-report) Quantity (input-bound) - % of questions answered correctly. Accuracy (output-bound) - % of answers that are correct.
Koriat and Goldsmith (1996, Experiment 1)
Report Option
Results
MONITORING: (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition
Koriat and Goldsmith (1996, Experiment 1)
Report Option
Results
MONITORING: (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition
CONTROL SENSITIVITY (confidence volunteering) : Gamma averaged .97 for recall, .93 for recognition (!)
Koriat and Goldsmith (1996, Experiment 1)
Report Option
Results
MONITORING (confidence correctness) Mean Gamma: .87 for recall, .68 for recognition Overconfidence: .03 for recall, .03 for recognition
CONTROL SENSITIVITY (confidence volunteering) Gamma averaged .97 for recall, .93 for recognition (!)
REPORT CRITERION (control policy)
Effect of Accuracy Incentive (High vs. Moderate): Mean number of volunteered answers: 26.9 vs. 30.9 Mean confidence for volunteered answers: .93 vs. .84 Mean estimated criterion level: .84 vs. .61 Criterion accounted for 92% of report decisions.
Koriat and Goldsmith (1996, Experiment 1)
Report Option
+31
+10
+24
+5
-6
-4
-12
-7
Koriat and Goldsmith (1996, Experiment 1). Free-report means are adjusted for chance initial differences in retention between the two incentive conditions.
Report Option
Quantity-Accuracy Profile Plots (K&G, 1996, Exp. 1)
MODERATEINCENTIVE
HIGHINCENTIVE
Report Criterion Level
Contribution of Control
Quantity-Accuracy Payoff Profile Plots (K&G, 1996, Exp. 1)
MODERATEINCENTIVE
HIGHINCENTIVE
Report Criterion Level
Contribution of Control
Quantity-Accuracy Payoff Profile Plots (K&G, 1996, Exp. 1)
MODERATEINCENTIVE
HIGHINCENTIVE
Report Criterion Level
Contribution of Control
“Prototypical” – uniform confidence distribution and perfect calibration (moderate relationship between confidence and correctness).
.
Monitoring simulations (K&G, 1996)
“Perfect discrimination” – polarized confidence distribution and perfect calibration (perfect relationship between confidence and correctness).
Monitoring simulations (K&G, 1996)
“No discrimination” – uniform confidence distribution and flat calibration curve (no relationship between confidence and correctness).
Monitoring simulations (K&G, 1996)
As monitoring improves: Larger accuracy gains are achieved at lower quantity costs.
Monitoring simulations (K&G, 1996)
Koriat and Goldsmith (1996, Experiment 2)
Test Format: Recall only.
Report Option (counterbalanced order)
Forced report + confidence judgments Free report + moderate accuracy incentive
Monitoring Manipulation
Two sets of items (mixed within-subject):
Standard items – good monitoring. Deceptive items – poor monitoring.
Example: What is the capital of Australia? (many non-British participants are confident that it is Sydney).
Contribution of Monitoring
Koriat and Goldsmith (1996, Experiment 2)
Test Format: Recall only.
Report Option (counterbalanced order)
Forced report + confidence judgments Free report + moderate accuracy incentive
Monitoring Manipulation
Two sets of items (mixed within-subject):
Standard items – good monitoring. Deceptive items – poor monitoring.
Example: What is the capital of Australia? (many non-British participants are confident that it is Sydney).
Contribution of Monitoring
Koriat and Goldsmith (1996, Experiment 2). Calibration curve for standard items.
Contribution of Monitoring
Koriat and Goldsmith (1996, Experiment 2). Calibration curve for deceptive items.
Contribution of Monitoring
Scanned Images\SDOC2240R_Page_2.jpg
Report Criterion Level
Quantity-Accuracy Profile Plots (K&G, 1996, Exp. 2)
Contribution of Monitoring
Scanned Images\SDOC2240R_Page_2.jpg
Report Criterion Level
Quantity-Accuracy Profile Plots (K&G, 1996, Exp. 2)
Contribution of Monitoring
Interim Conclusions
Free report memory performance depends on both
memory and metacognitive monitoring and control.
Effects on memory performance (manipulations or
population differences) may be mediated by any or all of
these components.
It is therefore important to isolate and assess the
contribution of these components whenever “real-life”
(free-report) memory is of concern.
The “free-forced” paradigm and QAP methodology allows
one to do so (see also Higham’s, 2002, 2007, Type-2 SDT approach).
Report Option
QAP or Type-2 SDT
QAP MeasureTypeDescriptionPhaseType-2SDT
Retention (retrieval or ecphory)
MemoryProportion or percentage of forced-report answers that are correct.
Forced SAME
Monitoring resolution (discrimination)
MonitoringWithin-individual gamma correlation between confidence (assessed probability correct) in each answer and the correctness of each answer, or alternative measures such as ANDI (Yaniv et al., 1991).
ForcedA’
Monitoring calibration over/underconfidence
MonitoringDifference between mean assessed probability correct and proportion correct (positive values reflect overconfidence.
ForcedB’’D
Monitoring calibrationsquared or absolute-value deviations
MonitoringMean squared or absolute-value difference between the mean assessed probability correct and proportion correct of each confidence category used in plotting a calibration curve.
Forced–
Control sensitivityControlWithin-individual gamma correlation between confidence (assessed probability correct) in each answer and whether or not it was volunteered.
Forced + Free
–
Report criterion (Prc) estimateControl Estimate of each participant’s report criterion
(assessed probability level) that yields the maximum fit (fit rate) with his or her actual report decisions.
Forced + Free
B’’D
Report Option
QAP or Type-2 SDT
QAP MeasureTypeDescriptionPhaseType-2SDT
Prc fit rateControlThe proportion of each participant’s actual volunteering decisions that are compatible with the derived Prc estimate, and which is maximized by this
estimate. Can also be used as an index of control sensitivity.
Forced + Free
–
Control effectivenessControlAbsolute value of the difference between the estimated Prc for each participant and the optimal Prc, identified
as the Prc level that would maximize the participants’
payoff.
Forced + Free
Bias Profile
Free-report quantity(input-bound)
PerformanceProportion of correct reported answers out of the total number of questions (or studied items).
FreeSAME
Free-report accuracy(output-bound)
PerformanceProportion of correct reported answers out of the number of answers that were volunteered.
FreeSAME
Report Option
Applications
Children’s memory (Koriat et al., 2001; Roebers & Schneider, 2002, 2005)
Aging (Kelley & Sahakyan, 2003; Pansky et al., in press; Rhodes & Kelley, 2005)
Clinical populations (Danion et al., 2001; Koren et al., 2004, 2006)
Psychometric testing (Higham, 2007; Higham & Arnold, 2007; Notea-Koren,
2005)
Social cognition (Payne et al., 2001)
Changes in accuracy over time (Koriat & Goldsmith, in progress)
Encoding specificity (Higham, 2002; Higham & Tam, 2005)
Report Option
Expanding the Framework – Grain Size …
The “Problem” of Grain Size . . .
Neisser (1988): Recall subjects seem to choose "a level of
generality at which they are not mistaken."
Fisher (1996): Memory accuracy remained constant over a
40-day retention interval.
Explanation: The later information was more "coarse."
List learning: Reporting BIRD instead of ROBIN.
Expanding the Framework – Grain Size …
A METHODOLOGICAL PROBLEM,
but more importantly . . .
A FURTHER MEANS OF STRATEGIC REGULATION.
What was the defendant holding when he threatened the deceased?Control over Grain Size
What time did the incident occur?
6:20? best guess
6:15 – 6:30?
6:00 – 6:30?
probably
highly likely
“Sometime in the early evening …”
definitely
ACCURACY - INFORMATIVENESS TRADE-OFF !!!
Yaniv & Foster, 1995, 1997
Experimental Design
PHASE 1 -- Forced grain size at two grain levels
EXAMPLE: When did Neil Armstrong walk on the moon?A) Specify a 3-year interval: From _____ - _____B) Specify a 10-year interval: From _____ - _____
EXAMPLE: How many chromosomes are there in the nucleus of a human cell?
A) Give a specific number: _____B) Specify a 20-chromosome interval: _____ -
_____
PHASE 2 -- Free choice of grain size For each item, choose the answer that you would prefer to provide if
you were "an expert witness testifying before a government committee."
Goldsmith, Koriat, & Weinberg-Eliezer (2002, JEP:General)
Grain Size
Results
Exps. 1, 2, & 3 Chose fine 40%; chose coarse 60%
Achieved accuracy = .60
[ p(fine correct) = .32; p(coarse correct) = .75 ]
Control: fine confidence grain choice: Gamma = .82
Criterion estimates sensitive to informativeness
incentive: .58 (high incentive) vs. .74 (low incentive)
Report criterion accounts for 88% of actual choices
Goldsmith et al. (2002)
A “satisficing model” (cf. Simon, 1956)
Grain Size
Strategic Regulation of Memory Grain Size over Time
Goldsmith, Koriat, & Pansky (2005, JML special issue on Metamemory)
Grain Size
Control of Grain Size and Report Option
Both involve an accuracy – informativeness trade-off.
Both involve monitoring the correctness of candidate answers.
Both involve setting a report criterion (accuracy satisficing)
per competing incentives for accuracy and informativeness.
A single integrated model?
Current Directions
Control of Grain Size and Report Option
start
Confident in FINE answer? Provide FINE answer
no
Confident in COARSE answer? Provide COARSE
answer
no
WITHHOLD the answer
yes
yes
Prc = .71
Prc = .71
Goldsmith et al. (in progress)
.83
.83
Accounts for 90% of grain choices
Current Directions
Control of Grain Size and Report Option
start
Confident in FINE answer? Provide FINE answer
no
Confident in COARSE answer? Provide COARSE
answer
no
WITHHOLD the answer
yes
yes
Sufficiently informative?
yes
no
Prc = .71.83
Prc = .71.83
- Pragmatics (Grice, 1965)- Social/situational norms
Goldsmith et al. (in progress)
Current Directions
Conclusions
Report option and grain size are both important means of
regulating accuracy and informativeness of memory reports.
We must understand such regulation in order to understand
the factors underlying memory performance, particularly in
real-life settings.
Expanded conception of “retrieval”: Includes cognitive,
metacognitive, neurocognitive, personality, and social-pragmatic
contributions to memory performance.
Much more work remains to be done, both on these, and on
other types of strategic memory regulation.
+31
+10
+24
+5
-6
-4
-12
-7
Koriat and Goldsmith (1996, Experiment 1)
Recall – Recognition Paradox (K&G, 1994)
ConfoundACC: Free recall > Forced Choice QTY: Free recall < Forced Choice
Report Option
top related