‘An objective evaluation of student selection’ Has the time yet come? Chris McManus Division of Medical Education UCL UCL Medical Education Conference 28 th June 2011
‘An objectiveevaluation of
student selection’Has the time yet come?
Chris McManusDivision of Medical Education
UCL
UCL Medical Education Conference28th June 2011
Alexander ToddBaron Todd of Trumpington
1907-1997
Nobel Prize in Chemistry, 1957“"for his work on nucleotidesand nucleotide co-enzymes"
First to isolate Vitamin B12
Alexander ToddBaron Todd of Trumpington
1907-1997
Nobel Prize in Chemistry, 1957“"for his work on nucleotidesand nucleotide co-enzymes"
First to isolate Vitamin B12
• “We hope that research will in thecourse of time lead to the establishmentof reasonably objective criteria ofprofessional performance, valid in themany different fields and kinds ofmedical practice …
• … and thereby make possible anobjective evaluation of studentselection.
• This is a long-term aspiration however.”
• “We hope that research will in thecourse of time lead to the establishmentof reasonably objective criteria ofprofessional performance, valid in themany different fields and kinds ofmedical practice …
• … and thereby make possible anobjective evaluation of studentselection.
• This is a long-term aspiration however.”
Medical school selection
• Academic factors– A-levels
– Aptitude tests (BMAT, UKCAT, GAMSAT, etc)
• Non-academic factors– Personality
– Motivation
– Stress / Resilience / Burnout / Satisfaction
• Widening access– Complex issue
Selection for medical schools
• “Aptitude” tests– BMAT (UK)
– UKCAT (UK)
– GAMSAT (Australia and UK)
– MSAT (Australia)
– MCAT (US)
Selection for medical schools
• What is meant by ‘aptitude’?
• Tests are of two fundamental types:
– Achievement: Knowledge/content– A-levels
– BMAT: Section 2: Scientific knowledge and applications;
– MCAT biological sciences / physical sciences subtests
– GAMSAT: III: Reasoning in Biological and Physical Science
– Ability/Aptitude: Essentially IQ tests, although that term islittle used:
– UKCAT: Verbal reasoning/ Quantitative reasoning / Abstract reasoning/ Decision analysis
– BMAT: Section 1: Aptitudes and skills
– MCAT: Verbal reasoning subtest
– GAMSAT: I - Reasoning in the Humanities and Social Sciences
(1973)
Commissioned byCommittee of Vice-Chancellorsand Principals
The ISPIUA project: Investigationinto Supplementary PredictiveInformation for University Admission
Test of Academic Aptitude (TAA):Mathematics (M) and Verbal (V)
October 1967: 27,315 students inthe 4th term of the Sixth Form
The situation today
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
8 10 12 14 16 18 20 22 24 26 28 30
I
II.i
II.ii
III
Left
A-level points (A=10, B=8, C=6 etc)
Per
cen
tage
A-levels and university achievement, 2002(1999 entrants)
www.hefce.ac.uk/pubs/hefce/2003/03_32.htm
Sutton Trust SAT Study
• 9000 students took SAT in Autumn 2005
• 8000 matched to A-levels in 2006
• 2754
– Entered university 2006
– Graduated 2009
• 3800 still studying within HE
Sutton Trust SAT Study
• Correlations with degree class (N=2754):
– Average A-level points score 0.38
– Average KS4 (GCSE) score 0.36
– Total A-level points score 0.34
– Mean SAT score 0.26
– SAT Writing score 0.26
– SAT Reading score 0.24
– SAT Maths score 0.18
ISPIUA
Sutton Trust SAT Study
• Correlations with degree class (N=2754):
– Average A-level points score 0.38
– Average KS4 (GCSE) score 0.36
– Total A-level points score 0.34
– Mean SAT score 0.26
– SAT Writing score 0.26
– SAT Reading score 0.24
– SAT Maths score 0.18
Other aptitude tests….
• UKCAT
– Published evidence ‘mixed’
• BMAT
– Published evidence sparse and equivocal
• GAMSAT
– Published evidence from Australia suggestsno predictive value
Other medical schools
• Cambridge
• High A-level grades require:
• Intellectual ability
•Motivation / drive / study habits /learning styles / personality(conscientiousness)
•Content knowledge (e.g. biology /chemistry)
A-levels as predictors
Westminster Medical School Follow-up
• Dr Peter Fleming– Clinical entrants to Westminster, 1975-1982
– Age, about 21 (born 1954-1961)
– Timed IQ test (AH5): Verbal & Spatial scores
– N=511
• Followed up 1989 (McManus & Tunnicliffe)
– Qualified 1978-1985 (4-11 years previously;aged 28-35))
• Followed up 2002 (McManus, Smithers & Partridge)
– Qualified for 17-24 years
– Age, about 41-48
ALEVELS
5.33
5.00
4.67
4.33
4.00
3.67
3.33
3.00
2.67
2.33
2.00
1.67
1.33
100
80
60
40
20
0
Std. Dev = .71
Mean = 4.00
N = 508.00
TOTALJT
70605040302010
AL
EV
EL
JT
5.00
4.67
4.33
4.00
3.67
3.33
3.00
2.67
2.33
2.00
1.67
1.33
1.00
Westminster Hospital Follow-up• IQ and A-levels
62.5
60.0
57.5
55.0
52.5
50.0
47.5
45.0
42.5
40.0
37.5
35.0
32.5
30.0
27.5
25.0
22.5
20.0
100
80
60
40
20
0
Std. Dev = 7.74
Mean = 40.4
N = 511.00
Mean A-level grade
TotalAH5score
1716151413121110987654321
100
90
80
70
60
50
40
30
20
10
0
1716151413121110987654321
100
90
80
70
60
50
40
30
20
10
0
PRHO PRHOSHO SHO
Registrar
SR
Consultant GP Principal
Trainee/SpR
GP AssistantOther
Years after qualification Years after qualification
Hospital Doctors General Practitioners
Career progression
1716151413121110987654321
100
80
60
40
20
0
1716151413121110987654321
100
80
60
40
20
0
Memberships
Diplomas
Academicdegrees
Years after qualification Years after qualification
Hospital Doctors General Practitioners
Memberships
Academicdegrees
Diplomas
Post-graduate qualifications
17161514131211109876543210
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Years after qualification
AAA, AAB
ABB, BBB, BBC
BCC, CCC or less
p<.001
A-levels, IQ and Memberships
Intelligence: p=.012 (after taking A-levels into account, p=.423)
The 1991 Cohort Study
EntrantsOct 1991
EntrantsOct 1992
N ~ 3300N ~ 590092%
ApplicantsOct-Dec 1990
Questionnaire
Leverhulme/DoH
N ~ 270060%
Final Year1996
Questionnaire
Final Year1997
Questionnaire
DoH
N ~ 270055%
PRHOsAug 1996-Jul 1997
Questionnaire
PRHOsAug 1997-Jul 1998
Questionnaire
NTPMDE
N ~ 240060%
SHO/SpRGPs
Dec 2002
Questionnaire
LondonDeanery
MRCP~1997-99
N = 92531%
March2009
N ~ 270061%
LondonDeanery
Questionnaire
Part 1mark at first attempt
80.075.0
70.065.0
60.055.0
50.045.0
40.035.0
30.025.0
20.015.0
10.0
100
80
60
40
20
0
Std. Dev = 11.34
Mean = 47.7
N = 925.00
MRCPmark
MRCP(UK) markat first attempt
5.55.04.54.03.53.02.52.01.51.0
Pa
rt1
ma
rka
tfirs
ta
tte
mp
t
75
70
65
60
55
50
45
40
35
30
25
20
MRCP(UK) markat first attempt
EEE DDD CCC BBB AAA
A-level grade
A-levels
BasicMed Sci
BSc Class
MRCPmark
BSc Taken
GCSE
AH5
MRCP(UK) markat first attempt
Non-cognitive aspects
Can selectors predict whowill be a happy
or an unhappy doctor?
The 1991 Cohort Study
EntrantsOct 1991
EntrantsOct 1992
N ~ 3300N ~ 590092%
ApplicantsOct-Dec 1990
Questionnaire
Leverhulme/DoH
N ~ 270060%
Final Year1996
Questionnaire
Final Year1997
Questionnaire
DoH
N ~ 270055%
PRHOsAug 1996-Jul 1997
Questionnaire
PRHOsAug 1997-Jul 1998
Questionnaire
NTPMDE
N ~ 240060%
SHO/SpRGPs
Dec 2002
Questionnaire
LondonDeanery
MRCP~1997-99
N = 92531%
Satisfaction with a medical career
0 1 2 3 4 5 6
0 1 2 3 4 5 60123456
181614121086420
60
50
40
30
20
10
0
Unhappy/Dissatisfied
Happy/Satisfied
The 1991 Cohort Study
EntrantsOct 1991
EntrantsOct 1992
N ~ 3300N ~ 590092%
ApplicantsOct-Dec 1990
Questionnaire
Leverhulme/DoH
N ~ 270060%
Final Year1996
Questionnaire
Final Year1997
Questionnaire
DoH
N ~ 270055%
PRHOsAug 1996-Jul 1997
Questionnaire
PRHOsAug 1997-Jul 1998
Questionnaire
NTPMDE
N ~ 240060%
SHO/SpRGPs
Dec 2002
Questionnaire
LondonDeanery
MRCP~1997-99
N = 92531%
The 1991 Cohort Study
EntrantsOct 1991
EntrantsOct 1992
N ~ 3300N ~ 590092%
ApplicantsOct-Dec 1990
Questionnaire
Leverhulme/DoH
N ~ 270060%
Final Year1996
Questionnaire
Final Year1997
Questionnaire
DoH
N ~ 270055%
PRHOsAug 1996-Jul 1997
Questionnaire
PRHOsAug 1997-Jul 1998
Questionnaire
NTPMDE
N ~ 240060%
SHO/SpRGPs
Dec 2002
Questionnaire
LondonDeanery
MRCP~1997-99
N = 92531%
March2009
N ~ 270061%
LondonDeanery
Questionnaire
A B
Set A: 20 pairsone happy/ one unhappy
Set B: 20 pairsone happy/ one unhappy
All pairs aged under 21 at entry
All pairs matched for sex
96
19
35
22
20
Total
10.03
10.37
9.60
10.27
10.20
Mean nocorrect/20
4848Total
910Doctors
1916Selectors
1012Medical Students
1010Psychology Students
Set BSet A
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Correct
0
5
10
15
20
25
0
12
24
36
48
60
72
84
96
Performance of individual judges
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Correct
0
5
10
15
20
25
0
12
24
36
48
60
72
84
96
Performance of individual judges
So are the judges perhaps justresponding at random?
• Maybe the task is simply too difficult?
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Correct
0
1
2
3
4
5
0
4
8
12
16
20
24
28
32
36
40
Performance on particular pairs of doctors
95% CI
Significantlybetter than
chance
Significantlyworse than
chance
Consensus of judges
• +1 if answer consensus question insame way as other judges
• -1 if answer consensus question inopposite way to other judges
Time (minutes)
2402101801501209060300
Co
nse
nsu
sm
ea
su
re
100
80
60
40
20
Correlation of consensus measure with time taken
What differentiates successfullydiscriminated pairs fromunsuccessfully discriminatedpairs?
• No relationship to personality of doctors
– Big Five
– Empathy
• Correlation with educationalachievement
– Particularly with predicted A-levels
• (r=.321, p=.011, N=62)
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Correct
0
1
2
3
4
5
0
4
8
12
16
20
24
28
32
36
40
95% CI
Significantlybetter than
chance
Significantlyworse than
chance
‘An objectiveevaluation of
student selection’Has the time yet come?