DAVID DUNNING CORNELL UNIVERSITY Flawed Self-Assessment: On the failures of incompetence to recognize itself With thanks to: Dan Ames, Jeremy Cone, Clayton Critcher, Joyce Ehrlinger, Kerri Johnson, Justin Kruger, Fiona Lee, Oliver Sheldon, & Nora Williams
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
DAVID DUNNING CORNELL UNIVERSITY
Flawed Self-Assessment: On the failures of incompetence to recognize itself
With thanks to: Dan Ames, Jeremy Cone, Clayton Critcher, Joyce Ehrlinger, Kerri Johnson, Justin Kruger, Fiona Lee, Oliver Sheldon, & Nora Williams
Presenter
Presentation Notes
Report back on a topic covered 14 years ago…but really game show host.
When Thales was asked what was difficult, he said “To know one’s self,” and what was easy, “To advise another.” --Diogenes Laertius (c. 200 AD)
Correlation Between Self-, Peer, and Supervisor Ratings with ABSITE scores (Risucci et al., 1989)
0
0.1
0.2
0.3
0.4
0.5
0.6
Supervisor Peer Self
Cor
rela
tion
Rater
Overconfident Self-Views Regarding Competence in Medical Procedures (from Barnsley et al., 2004)
01020304050607080
Needs Supervision Can Teach Others
Perc
ent
Assigned Category
Self-RatingInstructor Rating
Where in the world is Ukraine? (Dropp, Kertzer, & Zeitzoff)
Presenter
Presentation Notes
But here’s the final and perhaps most important assertion I want to make about life in the shadow of inevitable ignorance. One of the most cruel and potentially most costly ironiesis that one of the things we are most ignorant about is that boundary between knowledge and ignorance… the complete shape and scope of our ignorance. Don’t know the boundary line, cross it and act on ignorance far more often than we realize. Not only do we suffer from inevitable ignorance, but we also suffer from meta-ignorance. (neology) in that we often do not know when our knowledge and expertise ends and our ignorance begins. We are a little bit like Wile E. Coyote chasing after our prey and often not noticing when we’ve run past the cliff of our expertise and into the thin air that comprises our ignorance. What I would like to argue to argue today is that one of the most important things we remain ignorant of is the contours of our own ignorance…and why that is inevitable. Most visible, most remarkable, is when ignorance goes to the most extreme…what about people who live in this…
The Dunning-Kruger Effect
1. Poor performers do not cannot recognize shortcomings in their expertise or knowledge.
Presenter
Presentation Notes
…the best acid test that we don’t know the boundaries and scope of our ignorance is to examine those who have the most reason to do so…the most incompetent, unknowledgeable…this leads to what has come to be known as the Dunning-Kruger effect.
The Dunning-Kruger Effect
That which makes one believe that only other people are subject to the Dunning-Kruger effect.
--astronautgo (Twitter)
Presenter
Presentation Notes
Now, people choose what they think is the most reasonable, so by definition think they are doing okay. Don’t see or recognize superior choices.
Judging Self-Performance on a Course Exam (Dunning et al., Current Directions, 2003)
0
10
20
30
40
50
60
70
80
90
100
Bottom Second Third Top
Perc
entil
e
Objective Performance Quartile
Actual
Mastery
Test Performance
Judging Self-Performance on a Course Exam (Dunning et al., Current Directions, 2003)
0
10
20
30
40
50
60
70
80
90
100
Bottom Second Third Top
Perc
entil
e
Objective Performance Quartile
ActualMasteryTest Performance
Dunning-Kruger Effect
Metacognition at the Gun Club
0
10
20
30
40
50
60
70
80
90
100
Bottom Second Third Top
Perc
entil
e
Actual Knowledge Performance
In the Hospital Lab (Haun et al., 2000)
No Incentive for Accuracy? (Ehrlinger et al., OBHDP, 2008, Study 1)
0
10
20
30
40
50
60
70
80
90
100
10 30 50 70 90
Perc
eive
d Pe
rcen
tile
Actual Percentile
Control
Incentive
Actual6
7
8
9
10
11
12
13
14
15
8 10 12 14
Perc
eive
d R
aw S
core
Actual Raw Score
Control
Incentive
Actual
Presenter
Presentation Notes
Not sloppy…
No Incentive for Accuracy? (Ehrlinger et al., OBHDP, 2008, Study 1)
0
10
20
30
40
50
60
70
80
90
100
10 30 50 70 90
Perc
eive
d Pe
rcen
tile
Actual Percentile
Control
Incentive
Actual6
7
8
9
10
11
12
13
14
15
8 10 12 14
Perc
eive
d R
aw S
core
Actual Raw Score
Control
Incentive
Actual
Presenter
Presentation Notes
Paying up to $100 for accurate assessments does nothing to improve…
A Boggle Puzzle Based on True Events Caputo & Dunning (JESP, 2005)
D N R L
E I A R
M S L T
A L E S
Presenter
Presentation Notes
In our own work, done something like this…have people play Boggle, and them guess how many solutions missed. Usually underestimate by an order of magnitude. In this puzzle are 614 solutions.
Intuitive Physics Williams et al., JPSP, 2013
Imagine you are looking straight down at a tube lying on a table. A ball is thrown into the tube where the arrow is. What path will the ball take once it exits the tube? To the right of the tube are depictions of 4 different paths the ball might take. Which is the correct path?
A B
C D
Presenter
Presentation Notes
Give participants task like this.
Estimated v. Actual Correct (Williams et al., 2013, JPSP, Study 2)
Presenter
Presentation Notes
Why people getting all wrong almost as confident as those with perfect scores? Because most ALL WRONG participants always giving the same answers; they have an algorithm/rule in their head that turns out to be wrong, but compelling
Some days it would be enough just to know which side of the Dunning-Kruger effect I'm operating under.
--naomib33 (Twitter)
Presenter
Presentation Notes
When you're arguing with a fool, make sure he isn't doing the same thing.
Mediocrity knows nothing higher than itself, but talent instantly recognizes genius. Sir Arthur Conan Doyle, Sr.
Presenter
Presentation Notes
Do the explanation here…
Cone & Dunning (ongoing) Logical Reasoning I Participants Grade 6 Tests Scores: 2, 9, 11, 14, 17, 20 n = 50
Logical Reasoning II Participants Grade 5 Tests Scores: 4, 8, 12, 16, 20 Roughly half given financial incentives for accuracy
(up to $50) n = 37
Logical Reasoning I
0
2
4
6
8
10
12
14
16
18
20
0 5 10 15 20
Esti
mat
ed S
core
Target Exam Score
Estimated
Actual
Logical Reasoning II
0
2
4
6
8
10
12
14
16
18
20
0 5 10 15 20
Esti
mat
ed S
core
Target Exam Score
No Incentive
Financial Incentive
Actual Performance
If Put in IQ Terms
40
60
80
100
120
140
160
Bottom Target Top Target
"LQ
" Sco
re
Actual
Perceived
Education
That which discloses to the wise and disguises
from the foolish their lack of understanding. --Ambrose Bierce (Devil’s Dictionary)
The Problem with Massed Training Leads to rapid learning and high initial performance Leads also to high satisfaction and feelings of competence But … the problem is retention (over time) and transfer
(to different situations).
Impact of Massed vs. Distributed Training (Simons & Bjork, 2002)
Impact of Massed vs. Distributed Training (Simons & Bjork, 2002)
Predicted Error
Actual Error Distributed
Training
Massed Training
Impact of Massed vs. Distributed Training (Simons & Bjork, 2002)
Predicted Error
Actual Error Distributed
Training
Massed Training
Correlation Between 6th Year Self-Assessments and Some Potential Predictors (Arnold et al., 1985)
0
0.1
0.2
0.3
0.4
0.5
0.6
Docent's Rating Exam Scores Self/ 3 years Earlier
Cor
rela
tion
Rater
Presenter
Presentation Notes
Note here nothing about psychological errors per se. Just talking about the task demands of solo self-assessment. But what about motivated reasoning? Don’t need it to get the Dunning-Kruger effect, but perhaps to stay within in… medical school student assessments of skill are notoriously “sticky.”
Emotional Intelligence Sheldon et al., 2014, J Applied, Study 2
0102030405060708090
100
Bottom Second Third Top
Perc
enti
le
Performance Quartile
Estimated EQ
Actual EQ
Presenter
Presentation Notes
This may result given how people respond to feedback. How will react to feedback; get on to the task of self-improvement, developing their skills. Self-assessments of business school students after a section on EI, take the Meyer, Salovey, Caruso test…
Purchases of Self-Improvement Book after Feedback Sheldon et al., 2014, J Applied, Study 2)
20
33
41
64
0
10
20
30
40
50
60
70
Bottom Second Third Top
Perc
ent B
uyin
g B
ook
Performance Quartile
Presenter
Presentation Notes
Paradoxical effect; asked if want book The emotionally intelligent manager, discount ½ off.
Benchmarking (Martin et al., 1998)
Benchmarking (Martin et al., 1998)
Benchmarking (Martin et al., 1998)
The only true wisdom is to know that you know nothing.
--Socrates
A man’s gotta know his limitations. --Inspector Harry Callahan (SFPD)
An education isn’t how much you’ve committed to memory, or even how much you know. It’s being able to differentiate between what you know and what you don’t.
--Anatole France
One habit is revealed by the following task: Schizophrenia patients who fall prey to their
delusions tend to reach their decision after only one or two balls are drawn
Problem Habits? Jumping to Conclusions
Two urns are placed in from the the participant. One contains a majority of white balls, the other a majority of black balls. The experimenter draws one ball at a time from the same urn and announces its color. The participant is told to stop the experimenter when he or she is ready to state which urn he or she thinks the balls are being drawn from.
JTC and Belief Bias in Logic
Example question: A rose needs water Living things need water ∴ Roses are living things (belief bias answer: yes; correct answer: we cannot conclude).
30
40
50
60
70
80
90
100
Non-Group JTC Group
Perc
ent C
orre
ct
Degree of JTC Bias
Actual
Perceived
t(104) = 4.24, p < .001
JTC and Avoiding System 1 Errors
Example question: A ball and a bat cost $1.10
together, and the bat costs $1 more than the ball. How much does the ball cost?
20
30
40
50
60
70
80
90
100
Non-Group JTC Group
Perc
ent C
orre
ct
Degree of JTC Bias
Actual
t(104) = 3.33, p < .001
JTC and Ratio Bias
Example question: Can bet to win $10. Prefer
to bet on an urn with 1 winning ball and 9 losers, or urn with xx winners and (100-xx) losers?
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
6
13 12 11 10 9 8 7
Pref
eren
ce fo
r x/1
00 U
rn
Degree of JTC Bias
Non Group
JTC Group
t(104) = 3.56, p < .001
The Dunning-Kruger Effect
Hey bloggers, knowing about the Dunning-Kruger effect should make you less sure about other peoples' ignorance...not more sure.
--rahkan (Twitter)
Presenter
Presentation Notes
Was heartened to see this comment flicker across twitter this past week, more gets it. We are just as likely to suffer as the person we are arguing with. What is this research about? Cranks, stupidity, incompetence, hubris, arrogance, .small-mindedness…may be about that, but it’s not the word I have tape above my desk at work, sums up what I want to remember as I think about these issues… that word is humility.
On the Failure of Ignorance to Recognize Itself
Poor performers fail to recognize depths of their shortcomings (Dunning-Kruger effect)
Confidence flows from misinformation much like it does from accurate information
We have a collective problem of recognizing the best people (and ideas) among us
Motivated resistance may lead to paradoxical effects due to feedback
Presenter
Presentation Notes
Further implications…not ashamed to say I don’t know what they are…
Your answer to the question is in error. But if it is any consolation, many of my academic colleagues have also been stumped by this problem.
Since you seem to enjoy coming straight to the point, I'll do the same. You blew it! Let me explain.
I have been a faithful reader of your column, and I have not, until now, had any reason to doubt you. However, in this matter (for which I do have expertise), your answer is clearly at odds with the truth.
May I suggest that you obtain and refer to a standard textbook on probability before you try to answer a question of this type again?
I am in shock that after being corrected by at least three mathematicians, you still do not see your mistake.
You made a mistake, but look at the positive side. If all those Ph.D.'s were wrong, the country would be in some very serious trouble.
You are the goat!
Presenter
Presentation Notes
10,000 letters; 92% disagreeing;
Regression to the Mean? (Ehrlinger et al., OBHDP, 2008, Study 1)
Health Literacy: Asthma Sufferers (Williams et al., 1998)
Asthma sufferers asked to operate an inhaler Checked to see if sufferers executed 6 steps
correctly (e.g., shake inhaler, exhale before use, wait 30 sec before additional puff).
Percentage of those performing 4 or more steps correctly:
Presenter
Presentation Notes
Study by Williams in 1998…group down at Emory…
Health Literacy: Asthma Sufferers (Williams et al., 1998)
Asthma sufferers asked to operate an inhaler Checked to see if sufferers executed 6 steps
correctly (e.g., shake inhaler, exhale before use, wait 30 sec before additional puff).
Percentage of those performing 4 or more steps correctly: Of those at high school reading level: 52% Of those at only 3rd grade reading level: 12%
Presenter
Presentation Notes
Study by Williams in 1998…group down at Emory…
Related Documents
