Postgraduate Medical Education and Training Board National Trainee Survey 2006 – key findings
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Postgraduate Medical Educationand Training Board
National Trainee Survey 2006 – key findings
Published by Postgraduate Medical Education and Training Board
Hercules HouseHercules RoadLondonSE1 7DU
Telephone: 020 7160 6100
www.pmetb.org.uk/traineesurvey
ISBN: 978-0-9555910-0-6
April 2007
Report written by Daniel Smith (PMETB), Patricia Le Rolland (PMETB) and ProfessorElisabeth Paice (London Deanery)
© PMETB 2007
National Trainee Survey 2006 – key findings 3
4 National Trainee Survey 2006 – key findings
Contents
Acknowledgments..................................................................................6
Summary.............................................................................................. 7
Main findings....................................................................................................7
Supervision – the key to satisfaction.................................................................7
Layout of this report..........................................................................................8
1. Introduction.......................................................................................9
Background...................................................................................................... 9
Testing the face validity of the survey...............................................................9
Deaneries’ previous work............................................................................... 10
2. Methodology and data preparation...................................................... 11
Data collection................................................................................................11
Inclusion criteria.............................................................................................12
Data preparation.............................................................................................13
Response rates for the included cases............................................................13
The response rate and outlying indicator scores............................................ 14
Derivation of the indicator scores................................................................... 15
Method variance.............................................................................................25
Time in post....................................................................................................25
Adjustment for method variance.................................................................... 26
3. Key findings..................................................................................... 27
The survey’s outcome variables......................................................................27
Overall Satisfaction.........................................................................................28
Factors associated with job satisfaction.......................................................... 28
Medical Errors................................................................................................31
Supervision Score........................................................................................... 34
Feeling bullied – overall prevalence...............................................................35
The source of the perceived bullying............................................................. 35
Reporting of bullying and subsequent action................................................. 36
Are certain types of trainee more likely to be bullied?...................................36
Perceived bullying and indicator scores.........................................................38
National training initiatives: Radiology Academies and Hospital at Night...................................................................................... 39
Deaneries, training provider and differences in Overall Satisfactionand Supervision – a multilevel model analysis ...............................................44
National Trainee Survey 2006 – key findings 5
4. The survey data as a quality management tool.......................................45
Data releases.................................................................................................. 45
The identification of outliers for follow up...................................................... 47
An example of identifying outliers – surgical specialty group........................ 47
Deanery quality improvement........................................................................ 48
5. The future of the National Trainee Survey..............................................50
Route of administration................................................................................... 50
The content of the survey................................................................................50
Adequate experience..................................................................................... 51
Overall Satisfaction Score............................................................................... 51
Follow up of concerns..................................................................................... 51
Personality variables...................................................................................... 51
Dissemination of the results............................................................................ 52
Population data...............................................................................................53
The survey as screening tool...........................................................................54
Making the survey mandatory.........................................................................55
Comparison with 2006 data.............................................................................55
National use of the data by Royal Colleges.....................................................55
The 2007 survey and other data sources.........................................................55
Deanery action plans......................................................................................56
Indicative plan for the 2007 survey................................................................. 56
Feedback form............................................................................................... 56
References.......................................................................................... 57
Appendices – available online from www.pmetb.org.uk/traineesurvey
6 National Trainee Survey 2006 – key findings
Acknowledgments
PMETB thanks the following groups of people for their help with this work:
● all the deanery survey contacts who administered the survey in their area;
● all the postgraduate medical education centre staff who assisted with the datacollection;
● the deans from across the UK who supported this work.
The Surveys Working Group members:
Professor Elisabeth PaiceChair of COPMeD and Chair of Surveys Working Group
Maryanne AitkenCOPMeD National Trainee Survey Manager and Quality Manager, LondonDeanery
Dr Gellisse BagnallEducational Development Manager, NHS Education For Scotland
Dr Elaine DolmanTrent Deanery
Professor Shelley HeardPostgraduate Dean, London Deanery
Dr Stephen KellyAssociate Dean,West Midlands Deanery
Dr Mark RickenbachAssociate Dean, NHS Education South Central - Wessex Deanery
Dr Heather PayneAssociate Dean,Wales Deanery
Surveys Working Group PMETB staff:
Mark DexterHead of Policy
The contractors who assisted with data collection:
Nathan Collins WebLabs
Ray Flux CivilEyes
Bhavna Patel Document Capture Company
Samir Sayed Document Capture Company
Jane Smith Athene Communications
National Association of Medical Education Management
PMETB is responsible for this report’s contents.
National Trainee Survey 2006 – key findings 7
Summary
Main findings
Overall the majority of trainees who took part responded positively to the itemsthey completed for the National Trainee Survey. This indicates that the majority oftrainees were broadly satisfied with their training posts but there were interestingvariations by provider and specialty. For example, it is possible to identifyspecialties and training providers that had low scores when compared to thenational figures and deaneries across the UK will use this information to target areasfor improvement. To ensure that the survey delivers improvements to trainees’experience of their training programmes, all UK deaneries have been using thedata from the survey to prepare action plans based on their results.
PMETB’s analysis has shown which facets of trainees’ experience of their posts wererelated to the survey’s outcome measures: these outcome measures were ‘OverallSatisfaction’ and ‘Medical Errors’. The analysis shows that better supervision isassociated with trainees reporting fewer perceived medical errors and greatersatisfaction with their posts.
Supervision – the key to satisfaction
The survey suggests that trainees’ perception of their satisfaction and supervisionvary by specialty grade group. GP trainees, for example, had the highest scores onboth these indicators, suggesting that they were both better supervised and moresatisfied than other trainees. Surgical group Senior House Officers (SHOs) had thelowest. Reported bullying also varied by specialty group and grade: it is interestingto note that the best supervised and most satisfied trainees – the GPs – had thelowest incidence of reported bullying.
The relationship between demographic variables and reported bullying varies byspecialty. As an example, female trainees were more likely to report being bulliedin anaesthetic, emergency medicine and surgical specialty groups than in others.
Survey data were also used to measure the impact of national initiatives on training.As examples, radiology trainees in academies reported higher Overall Satisfactionwith their posts than those not in academies and trainees working at sites in theHospital at Night initiative were more likely to report multidisciplinary handoversthan trainees who were not.
8 National Trainee Survey 2006 – key findings
Layout of this report
Chapter 1 of this report covers the background to the survey. Chapter 2 covers thedata collection methodology and the derivation of the indicator scores used in thescreening tools that PMETB has provided to deaneries. Chapter 3 presents some ofthe key national findings from the data as a means to demonstrate that the indicatorscores are effective measures of providers’ adherence to PMETB’s Genericstandards for trainingi. Both chapters 2 and 3 contain statistical tables for interestedreaders. Chapter 4 gives details of how the indicator scores were used in thescreening tool. To avoid using the data as a league table and maintain the emphasison quality improvement, data on individual educational providers are not given.Deaneries do have indicator score data for providers in the screening tools.Chapter 5 makes recommendations for the next trainee survey based on PMETB’sevaluation of the 2006 work and feedback received from those who worked on theproject.
Throughout this report indicator scores are in title case, for instance OverallSupervision. This refers specifically to the Overall Satisfaction Score derived fromitems on this survey, as opposed to overall satisfaction generally.
The appendices referred to throughout are available online atwww.pmetb.org.uk/traineesurvey
National Trainee Survey 2006 – key findings 9
1. Introduction
Background
In October 2005 PMETB inaugurated a Surveys Working Group chaired by ProfessorElisabeth Paice, charged with implementing a national survey of trainee doctors.This group reports to PMETB’s Statutory Training Committee. The stated purpose ofthe survey was to determine whether national training standards were being met.PMETB’s Generic standards for training were developed in tandem with items for theNational Trainee Survey and survey items were mapped to the standards (see Table5, Chapter 2). However, it was not possible to measure adherence to all of thestandards using a trainee survey instrument. Indeed, earlier work by Grant et alii
recognised that any trainee survey would need to be one part of a national data setused to quality assure postgraduate medical training. Grant et al proposed that thisnational data set could be used to inform quality assurance visits and to comparethe quality of training across providers and over time.
Grant et al noted that Professor Paice at London Deanery had developed a Point ofview survey to record trainees’ opinions about their trainingiii. They recommendedthat it would be appropriate to use London Deanery’s Point of view survey as thebasis for the proposed trainee instrument because:
● it addressed many of the issues included in the hospital visiting documentationused by Royal Colleges and deaneries;
● its core questions were developed and validated through face-to-face interviewswith over 300 SHOs;
● it had been successfully administered since 1996;
● the questions were suitable for every grade and specialty;
● it has also been used in Kent, Surrey Sussex (KSS) and Eastern deaneries.
PMETB followed this recommendation and the PMETB National Trainee Survey 2006used many items from the Point of view survey.
As part of the initial development of the survey, PMETB consulted with a number oftrainee groups, including the Academy of Medical Royal Colleges Trainee DoctorsGroup and the British Medical Association’s Junior Doctors Committee. Several ofthe Academy group’s suggestions were followed: adding an item on beingpressured to submit working hours that are compliant with the European WorkingTime Directive (item D2 on the survey) and an item on research opportunities, anddropping an item on trainees’ sexual orientation.
Testing the face validity of the survey
PMETB commissioned researchers at the University of Winchester to recruitvolunteer trainees, administer the survey to them and conduct interviews aftercompletion of the survey to ascertain that the face validity of items was acceptable.Data were obtained from 64 volunteers working in a range of specialties from fourdeaneries on:
● the clarity of the items and their understanding of the items;
● whether they had any problems understanding the item;
● whether the questionnaire covered all the relevant issues.
10 National Trainee Survey 2006 – key findings
Participants’ comments were synthesised by the Winchester researchers intorecommendations for the surveyiv. It was possible to incorporate one of theserecommendations in the 2006 survey, namely the inclusion of items on makingmedical errors. The items used were derived from a United States study by Baldwinand Daugherty, which looked at the effect of sleep deprivation on residents’workingv. The Winchester group’s remaining recommendations will bereconsidered for the next National Trainee Survey.
Deaneries’ previous work
The majority of deaneries (17 out of the 18 for which there are data) had alreadyconducted surveys of trainees of one sort or another. PMETB’s and COPMeD’sintention continues to be that the National Trainee Survey would replace localsurveys at the time of administration to avoid questionnaire fatigue among traineesand to adhere to the principles of better regulation laid out in the concordatvi.Where deanery surveys occur more frequently than annually, their surveys that donot take place when the national survey is running can continue to take place.
Differences in methods of administering the previous surveys across deanerieswere apparent. Some deaneries chose to survey at a particular point in time(snapshot), while others chose to survey at the end of the trainees’ posts. Owing tothe fact that the PMETB and COPMeD National Trainee Survey needed to obtaindata on all posts within a limited timeframe, a snapshot approach was used; thismeans that respondents had been in post for varying lengths of time at the pointthey completed the survey. The Surveys Working Group will review the merits ofsnapshot versus end-of-post surveys when planning future survey work. It is worthnoting that the two approaches may not be mutually exclusive; for example, sometrainees could be surveyed at end of post during a snapshot survey.
Queries regarding the content of the report should be sent to:
National Trainee Survey 2006 – key findings 11
2. Methodology and data preparation
Data collection
Data were collected between 15 May and 4 August 2006. Deaneries nominated asurvey contact for their deanery to work with the PMETB/COPMeD team. Deaneriesused one or more of the routes of administration detailed in Table 1 and werekindly supported by Postgraduate Medical Education Centres. Each route used thesame items, with minor variation in presentation where required by the format.
1 A summary of deaneries’ data collection is available here:http://www.pmetb.org.uk/fileadmin/user/QA/Trainee_Survey/PMETBCOPMEdTraineeSurveyBriefingNote3_1.pdf
2 Some emails did not get through the first time due to respondents having set their email accounts to haveenhanced junk mail filters. PMETB publicised this problem to ensure users checked their junk mail folders andresent them using an Outlook mail-merge rather than the database to get past the filter. Further consideration willbe given to this issue for future surveys.
Route Description Provider N (post-exclusions)
Portable electronicsurvey units(known as blackboxes)
PMETB –web by email
Respondents completedthe survey on theseunits by pressingnumeric keys. The unitswere placed inPostgraduate MedicalEducation Centres for alimited time-span.Postgraduate MedicalEducation Centre staffdirected the trainees tothe units.
PMETB set-up a website(www.traineesurvey.org.uk). Access to thesurvey was controlledby an individualpassword that wasemailed to the trainee.Reminder email wassent2.
Civil EyesResearchLimited
PMETB usingForms mastersoftware
6,463
Table 1 Routes of administration 1
PMETB –web by letter
PMETB set-up a website(www.traineesurvey.org.uk). Access to thesurvey was controlledby an individualpassword that was sentto the trainee by post asthe deanery wished touse the website but didnot have valid emailaddresses for thetrainees.
PMETB usingForms mastersoftware
7,488
12 National Trainee Survey 2006 – key findings
Route Description Provider N (post-exclusions)
Data obtained through these routes were merged into one SPSS data file; this was aslow process as not all providers complied with the data template issued by PMETB.Final variables (i.e. the merged versions) were checked against the various sourcesto ensure that the data were not corrupted.
The items from the survey are given in Appendix 1 (available fromwww.pmetb.org.uk/traineesurvey).
PMETB –scannable paper form
The survey wasformatted to fit onto foursides of A4 (i.e. onepiece of A3 paper) forease of scanning. Aseparate sheet withlisting specialties andtheir codes wasprovided sorespondents couldanswer A5 and A6 usingcodes. Returns werescanned using OpticalMark and OpticalCharacter Recognition.
DocumentCaptureCompany
8,634
Severn andWessexwebsite
A website provided bythe suppliers of theIntrepid database andallowing the deanery tolink response directly totheir Intrepid data.
Hicomm 2,295
Inclusion criteria
The focus of the survey was trainees in educationally approved posts, so thefollowing criteria were used:
● Included: SHOs in approved posts, Specialist Registrars (SpRs), LocumAppointment Training posts (LATS), Fixed Term Training Appointments (FTTAs)and GP Registrars (GPRs).
● Excluded: Foundation Year 1 and Year 2 (F1s and F2s) (unless the deaneryincluded these for local analysis only)3, people in non-approved posts such asclinical fellows or trust SHOs, trainees on maternity leave, trainees on out ofprogramme experience (OOPE), non-medical public health specialists anddentists (unless dual-registered).
3 Following a request from the General Medical Council to avoid confusion with their work.
National Trainee Survey 2006 – key findings 13
Grade N
Table 2 Grade exclusions
F1 1,669
F2 501
Trust doctor 551
Staff grade/other 739
Total 3,460
Data preparation
A total of 29,146 responses were received before the cut-off point for data entry. Thefollowing cases were excluded:
Grade exclusions
The questionnaire included response options for grade (item A1, see onlineAppendix 1) for groups of trainees who were excluded in case they received thesurvey by mistake and for foundation trainees included on a local basis only bysome deaneries. Table 2 gives the numbers of these respondents who were notincluded in PMETB reports.
Other exclusions
Cases were also excluded for the following reasons:
● data were missing on more than 14 items;
● there was a free text comment indicating that the respondent should not havebeen included, such as a reference to OOPE;
● there was a comment to say that the respondent was completing a second return.
This process left 24,880 cases available for analysis. The numbers available forindividual items on the survey may be below this due to “Not applicable” responseoptions and completion errors on the paper form.
Response rates for the included cases
As no master population file, listing all trainees by specialty and provider, wasavailable at the start of the survey, period data quality issues affected thecalculation of response rates. These were manifest in two ways: response ratesgreater than 100 per cent and locations with no denominator. In addition, recordingof SHO VTS (Vocational Training Scheme for GPs) was inconsistent on these returns.These problems were greater for finer aggregations, such as specialty groups,within a location. Given this, response rates are only reported by deanery here.Deaneries have received a response rate file that included the response rate foreach specialty group at each acute provider; 66 per cent (214 out of 326 acuteproviders with population data) had overall response rates of 50 per cent or more.
14 National Trainee Survey 2006 – key findings
The response rate and outlying indicator scores
The provider level response file was merged with the aggregated data set and foreach specialty group correlations were obtained between:
● SHO response rate and the total number of outlying indicator scores below themean and indicator scores above the mean;
● SpR response rate and the total number of outlying indicator scores below themean and indicator scores above the mean.
The Bonferroni correctionvii was applied for multiple testing; only one of thecorrelations was found to be statistically significant at P <0.05. The surgery group:the correlation between the Surgery Group SHO response rate and the total numberof indicator scores lower than national mean (r = 0.27, P = 0.04). This suggests thatthe higher the response rate for this group of trainees at a provider the morenegative the indicator scores for the provider, although the correlation was weak.Overall, it can be concluded that providers’ response rates are not correlated withtheir indicator scores.
Table 3 Response rate by deanery
Deanery Response rate %
Eastern 100**Kent, Surrey, Sussex 68Leicestershire, Northamptonshire & Rutland 64London 64Mersey 52North Western 57Northern 66Northern Ireland 63Oxford 63Peninsula 65Scotland (East) 57Scotland (North) 97Scotland (South East) 47Scotland (West) 49Severn and Wessex 74South Yorkshire & South Humber 41Trent 56Wales 33West Midlands 81Yorkshire 65
**The response rate was greater than 100 per cent; presumably either sometrainees recorded their grades incorrectly or the population data were inaccurate.
National Trainee Survey 2006 – key findings 15
Derivation of the indicator scores
Indicator scores were derived from the items by mapping to PMETB’s Genericstandards for training. This mapping is shown in Table 5 below, together with detailsof how the scores were calculated. For ordinal scores, the reliability coefficients4
measuring the score’s internal consistency are given, which, according toNunnally’sviii thresholds, are acceptable for the following scores: AdequateExperience, Feedback, Handover and Overall Satisfaction.
Items that were ordinal in nature (that is the responses that were on scales such as“Very Poor” to “Excellent”) were subjected to exploratory factor analysis5 to test forconstruct validityix. Items measuring a given construct should load6 on the samefactor and not on other factors. The results of the factor analysis are given in Table 4below.
Using Eigen values greater than 1 as a criterion, the factor analysis supports theconstruction of the following indicator scores, because the items used load on thesame factor and not on other factors, Overall Satisfaction, Supervision,Workload,Handover and Access to Resources.
However, other scales were more problematic; the Other Learning Opportunitiesand Feedback Score items did not load on their own factors and some of the itemsfrom these two scores did not load on any of the factors. The construction of both ofthese scores will be reviewed for the next survey.
The factor analysis failed to provide evidence of discriminant validity for theAdequate Experience Score; items D7 and D8 loaded on the same factor as theOverall Satisfaction Score items. This suggests that they are not measuring aseparate construct from Overall Satisfaction, perhaps because they are also overallrating items like the H2 to H6 items that are used to derive the Overall SatisfactionScore. PMETB plans to address this issue in 2007 using specialty specific items tolook at facets of experience. Rather than asking for a rating overall, the survey willask respondents about particular facets of their placement; for instance, there maybe items on theatre time for surgical trainees.
The results of the factor analysis were used to inform which items were used toderive the indicator scores but, due to the stated objective of the work, priority wasgiven to the mapping to the Generic standards for training when developing theindicator scores.
4 Cronbach’s alpha: the mean of all possible split-half reliability coefficients5 The goal of factor analysis is to summarise the pattern of correlations among the items to reduce the number of
items to a smaller number of factors.6 The loading is the correlation between the survey item and the factor, the size of the loading reflects the extent of
the relationship between the item and the factor.
16 National Trainee Survey 2006 – key findings
Var
ian
ce e
xpla
ine
dH
4H
ow w
oul
d y
ou
rate
th
e q
uali
ty o
f exp
eri
en
ce in
th
is p
ost
?H
5H
ow w
oul
d y
ou
de
scri
be
th
is p
ost
to
a fr
ien
d w
ho
was
th
ink
ing
of a
pp
lyin
g f
or
it?
D7
How
wo
uld
yo
u ra
te t
he
pra
ctic
al e
xpe
rie
nce
yo
u ar
e g
ett
ing
in t
his
po
st?
H6
How
use
ful d
o y
ou
fee
l th
is p
ost
wil
l be
fo
r yo
ur fu
ture
car
ee
r?D
8H
ow c
on
fid
en
t ar
e y
ou
that
yo
ur c
urre
nt
po
st w
ill h
elp
yo
u ac
qui
re t
he
co
mp
ete
nce
s yo
u n
ee
d a
t th
is s
tag
e o
f yo
ur t
rain
ing
?H
3H
ow w
oul
d y
ou
rate
th
e q
uali
ty o
f sup
erv
isio
n in
th
is p
ost
?H
2H
ow w
oul
d y
ou
rate
th
e q
uali
ty o
f te
ach
ing
in t
his
po
st?
G7
In t
his
po
st,h
ow w
oul
d y
ou
rate
th
e e
nco
urag
em
en
t yo
u h
ave
had
to
tak
e s
tud
y le
ave
?C
2H
ow o
fte
n,i
f eve
r,h
ave
yo
u b
ee
n s
upe
rvis
ed
by
som
eo
ne
wh
o y
ou
fee
l isn
’t co
mp
ete
nt
to d
o s
o?
C1
How
oft
en
hav
e y
ou
felt
fo
rce
d t
o c
op
e w
ith
pro
ble
ms
bey
on
d y
our
co
mp
ete
nce
or
exp
eri
en
ce?
C4
Do
yo
u al
way
s k
now
wh
o is
pro
vid
ing
yo
ur c
lin
ical
sup
erv
isio
n w
he
n y
ou
are
wo
rkin
g?
C3
How
oft
en
hav
e y
ou
be
en
exp
ect
ed
to
ob
tain
co
nse
nt
for
pro
ced
ure
s w
hic
h y
ou
do
no
t ca
rry
out
yo
urse
lf?
C5
Ple
ase
ind
icat
e y
our
pe
rce
pti
on
of t
he
way
in w
hic
h c
riti
cal e
ven
ts a
nd
ne
ar m
isse
s ar
e r
ep
ort
ed
in y
our
de
par
tme
nt.
D9
How
wo
uld
yo
u ra
te t
he
inte
nsi
ty o
f yo
ur w
ork
,by
day
?D
10H
ow w
oul
d y
ou
rate
th
e in
ten
sity
of y
our
wo
rk,b
y n
igh
t?D
4H
ow o
fte
n h
as y
our
cur
ren
t w
ork
ing
pat
tern
left
yo
u fe
eli
ng
sh
ort
of s
lee
p w
he
n a
t w
ork
?D
3H
ow o
fte
n d
o y
ou
wo
rk b
eyo
nd
yo
ur r
ost
ere
d h
our
s?G
5*H
ow o
fte
n d
o y
ou
hav
e t
he
op
po
rtun
ity
to le
arn
to
ge
the
r w
ith
oth
er
he
alth
care
pro
fess
ion
als?
G3*
Ho
urs
of r
ele
van
t,ti
me
tab
led
,org
anis
ed
ed
ucat
ion
al m
ee
tin
gs
or
oth
er
eve
nts
of e
duc
atio
nal
val
ue o
n a
vera
ge
eac
h w
ee
kE
1H
ow o
fte
n h
ave
yo
u h
ad in
form
al f
ee
db
ack
fro
m a
se
nio
r cl
inic
ian
on
how
yo
u ar
e d
oin
g in
th
is p
ost
?H
1H
ow w
oul
d y
ou
rate
th
e q
uali
ty o
f in
duc
tio
n in
th
is p
ost
?D
5W
hic
h o
f th
e f
oll
owin
g b
est
de
scri
be
s h
and
ove
r ar
ran
ge
me
nts
AFT
ER
nig
ht
dut
y in
yo
ur p
ost
?D
6W
hic
h o
f th
e f
oll
owin
g b
est
de
scri
be
s h
and
ove
r ar
ran
ge
me
nts
BE
FOR
E n
igh
t d
uty
in y
our
po
st?
G9
Do
yo
u h
ave
acc
ess
to
th
e I
nte
rne
t at
yo
ur p
lace
of w
ork
?G
10H
ow e
asy
is it
to
ge
t ac
cess
to
th
e li
bra
ry s
erv
ice
s yo
u n
ee
d?
*Fo
r p
rese
nta
tio
n in
th
is t
able
th
e t
ext
of t
he
se it
em
s h
ave
be
en
sli
gh
tly
ed
ite
d.
Ext
ract
ion
Me
tho
d:P
rin
cip
al C
om
po
ne
nt
An
alys
is.R
ota
tio
n M
eth
od
:Var
imax
wit
h K
aise
r N
orm
alis
atio
n.R
ota
tio
n c
onv
erg
ed
in s
ix it
era
tio
ns.
On
ly lo
adin
gs
gre
ate
r th
an 0
.32
are
dis
pla
yed
.Kai
ser-
Mey
er-
Olk
in M
eas
ure
of S
amp
lin
g A
de
qua
cy =
0.8
97.N
= 1
6,60
9
19%
0.88
0.84
0.82
0.81
0.78
0.65
0.63
0.37
0.35
8% -0.6
6-0
.63
0.49
-0.4
70.
47
-0.3
3-0
.35
8% 0.76
0.73
0.62
0.50
7% 0.39
0.66
0.61
0.46
0.43
7% 0.89
0.88
6% 0.76
0.74
Q
Item
s
Overall Satisfaction
Supervision
Workload
Mixture -learning
Handover
Access toResources
Ta
ble
4
Fact
or a
na
lysi
s of
ord
ina
l su
rvey
ite
ms
National Trainee Survey 2006 – key findings 17
Ta
ble
5
Th
e in
dic
ato
r sc
ores
Ind
ica
tor
scor
e
Acc
ess
to
Ed
ucat
ion
alR
eso
urce
s
Ad
eq
uate
Exp
eri
en
ce
Fee
db
ack
Do
mai
n 8
.Ed
ucat
ion
alre
sour
ces
and
cap
acit
y
8.2
Th
ere
mus
t b
e a
cce
ss t
oe
duc
atio
nal
fac
ilit
ies
(in
clud
ing
a li
bra
ry),
and
reso
urce
s (i
ncl
udin
g a
cce
ssto
th
e I
nte
rne
t in
all
wo
rkp
lace
s) o
f a s
tan
dar
d t
oe
nab
le t
rain
ee
s to
ach
ieve
th
eo
utco
me
s o
f th
e p
rog
ram
me
as s
pe
cifi
ed
in t
he
curr
icul
um.
G9.
Do
yo
u h
ave
acc
ess
to
the
In
tern
et
at y
our
pla
ce o
fw
ork
?
G10
.Hav
e y
ou
had
th
eo
pp
ort
unit
y to
par
tici
pat
e in
rese
arch
in t
his
po
st?
D7.
How
wo
uld
yo
u ra
te t
he
pra
ctic
al e
xpe
rie
nce
yo
u ar
eg
ett
ing
in t
his
po
st?
D8.
How
con
fid
en
t ar
e y
ou
that
yo
urcu
rre
nt
po
st w
ill h
elp
yo
uac
qui
re t
he
co
mp
ete
nce
syo
u n
ee
d a
t th
is s
tag
e o
fyo
ur t
rain
ing
?
E1.
How
oft
en
hav
e y
ou
had
info
rmal
fe
ed
bac
k fr
om
ase
nio
r cl
inic
ian
on
how
yo
uar
e d
oin
g in
th
is p
ost
? E
2.H
ave
yo
u h
ad a
fo
rmal
me
eti
ng
wit
h y
our
sup
erv
iso
r to
tal
k a
bo
ut y
our
pro
gre
ss in
th
is p
ost
? E
3.H
ave
yo
u h
ad f
orm
alas
sess
me
nt
of y
our
pe
rfo
rman
ce in
th
ew
ork
pla
ce?
Th
e it
em
s in
clud
ed
he
reun
de
r e
duc
atio
nal
re
sour
ces
are
th
ose
th
at r
ela
te t
oin
div
idua
l po
sts.
Lib
rary
serv
ice
s re
late
to
loca
tio
ns
as a
wh
ole
an
d a
re d
eal
tw
ith
els
ewh
ere
.
Pra
ctic
al e
xpe
rie
nce
is t
he
sine
qua
non
of p
ost
gra
dua
tetr
ain
ing
.Wh
ile
D7
and
D8
mig
ht
in t
he
ory
hav
e e
lici
ted
dif
fere
nt
resp
on
ses,
resp
on
ses
are
hig
hly
corr
ela
ted
.
Fee
db
ack
is a
n im
po
rtan
tfa
cto
r in
lear
nin
g.T
his
sco
reis
bas
ed
on
th
e a
vail
abil
ity
of d
ay-t
o-d
ay f
ee
db
ack
,ap
pra
isal
an
d a
sse
ssm
en
t.
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d o
r if
on
lyo
ne
ite
m is
pre
sen
t it
ssc
ore
is u
sed
.
0.41
0.80
0.72
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d,o
r if
on
lyo
ne
ite
m is
pre
sen
tit
s sc
ore
is u
sed
.
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d if
at
leas
ttw
o o
f th
e t
hre
e it
em
sar
e p
rese
nt.
5.1
Suff
icie
nt
pra
ctic
alex
pe
rie
nce
mus
t b
e a
vail
able
wit
hin
th
e p
rog
ram
me
to
sup
po
rt a
cqui
siti
on
of
com
pe
ten
ce a
s se
t o
ut in
th
ecu
rric
ulum
.
6.6
Tra
ine
es
mus
t h
ave
furt
he
r m
ee
tin
gs
wit
h t
he
ire
duc
atio
nal
sup
erv
iso
r (o
rre
pre
sen
tati
ve)
at le
ast
thre
e-
mo
nth
ly,t
o d
iscu
ss t
he
irp
rog
ress
,out
stan
din
gle
arn
ing
ne
ed
s an
d h
ow t
om
ee
t th
em
.
Do
mai
n 5
.De
live
ry o
fcu
rric
ulum
incl
udin
gas
sess
me
nt
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
18 National Trainee Survey 2006 – key findings
Ind
ica
tor
Scor
e
Han
dov
er
Oth
erLe
arni
ngO
pp
ortu
nitie
s
Do
mai
n 1
.Pat
ien
tSa
fety
1.6
Tra
ine
es
in h
osp
ital
po
sts
mus
t h
ave
we
llo
rgan
ise
d h
and
ove
rar
ran
ge
me
nts
en
suri
ng
con
tinu
ity
of p
atie
nt
care
at t
he
sta
rt a
nd
en
d o
f day
or
nig
ht
dut
ies.
D5.
Wh
ich
of t
he
fo
llow
ing
be
st d
esc
rib
es
han
dov
er
arra
ng
em
en
ts B
EFO
RE
nig
ht
dut
y in
yo
ur p
ost
? D
6.W
hic
ho
f th
e f
oll
owin
g b
est
de
scri
be
s h
and
ove
rar
ran
ge
me
nts
AFT
ER
nig
ht
dut
y in
yo
ur p
ost
?
G1.
To w
hat
ext
en
t ar
e y
ou
invo
lve
d in
cli
nic
al a
udit
inth
is p
ost
? G
2.D
o y
ou
curr
en
tly
hav
e a
cce
ss t
o e
-le
arn
ing
mat
eri
al r
ele
van
t to
your
tra
inin
g?
G5.
How
oft
en
do
yo
u h
ave
th
e o
pp
ort
unit
yto
lear
n t
og
eth
er
wit
h o
the
rh
eal
thca
re p
rofe
ssio
nal
s(e
.g.n
urse
s,p
hysi
oth
era
pis
ts e
tc.)
? G
6.H
ave
yo
u ap
pli
ed
fo
r st
udy
leav
e in
th
is p
ost
? G
7 In
th
isp
ost
,how
wo
uld
yo
u ra
te t
he
en
cour
age
me
nt
you
hav
eh
ad t
o t
ake
stu
dy
leav
e?
G10
Hav
e y
ou
had
th
eo
pp
ort
unit
y to
par
tici
pat
e in
rese
arch
in t
his
po
st?
Hig
he
r sc
ore
s in
dic
ate
th
ath
and
ove
r is
mo
re f
orm
ally
org
anis
ed
an
d m
ore
like
ly t
ob
e in
clus
ive
of t
he
full
mul
ti-
pro
fess
ion
al t
eam
.
Th
is in
dic
ato
r co
mb
ine
s a
ran
ge
of u
nre
late
dad
dit
ion
al o
pp
ort
unit
ies.
Alo
w s
core
wo
uld
ind
icat
e t
he
ne
ed
to
exp
lore
wh
ich
of
the
se w
as p
rob
lem
atic
.
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
,so
alo
w s
core
ind
icat
es
ale
ss f
orm
al h
and
ove
r.
A m
ean
is t
he
nca
lcul
ate
d,o
r if
on
lyo
ne
ite
m is
pre
sen
t it
ssc
ore
is u
sed
.
0.78
0.36
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d if
at
leas
tfi
ve o
f th
e s
ix it
em
sar
e p
rese
nt.
6.13
Tra
ine
es
mus
tre
gul
arly
be
invo
lve
d in
th
ecl
inic
al a
udit
pro
cess
,in
clud
ing
pe
rso
nal
lyp
arti
cip
atin
g in
pla
nn
ing
,d
ata
coll
ect
ion
an
dan
alys
is.6
.17
Tra
ine
es
mus
th
ave
th
e o
pp
ort
unit
y to
lear
n w
ith
oth
er
he
alth
care
pro
fess
ion
als.
6.20
Tra
ine
es
mus
t b
e a
ble
to
tak
e s
tud
yle
ave
up
to
th
e m
axim
ump
erm
itte
d in
th
eir
te
rms
and
co
nd
itio
ns.
6.25
Tra
ine
es
sho
uld
be
exp
ose
d d
urin
g t
he
irtr
ain
ing
to
th
e a
cad
em
ico
pp
ort
unit
ies
avai
lab
le in
the
ir s
pe
cial
ty.
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
National Trainee Survey 2006 – key findings 19
Ind
ica
tor
scor
e
Ove
rall
Sati
sfac
tio
nSc
ore
Sup
erv
isio
n
N/A
inte
rnal
out
com
em
eas
ure
N/A
inte
rnal
out
com
em
eas
ure
H2.
How
wo
uld
yo
u ra
te t
he
qua
lity
of t
eac
hin
g in
th
isp
ost
? H
3.H
ow w
oul
d y
ou
rate
th
e q
uali
ty o
fsu
pe
rvis
ion
in t
his
po
st?
H4.
How
wo
uld
yo
u ra
te t
he
qua
lity
of e
xpe
rie
nce
in t
his
po
st?
H5.
How
wo
uld
yo
ud
esc
rib
e t
his
po
st t
o a
frie
nd
wh
o w
as t
hin
kin
g o
fap
ply
ing
fo
r it
? H
6.H
owus
efu
l do
yo
u fe
el t
his
po
stw
ill b
e f
or
your
futu
reca
ree
r?
C1.
How
oft
en
hav
e y
ou
felt
forc
ed
to
co
pe
wit
hp
rob
lem
s b
eyo
nd
yo
urco
mp
ete
nce
or
exp
eri
en
ce?
C2.
How
oft
en
,if e
ver,
hav
eyo
u b
ee
n s
upe
rvis
ed
by
som
eo
ne
wh
o y
ou
fee
l isn
'tco
mp
ete
nt
to d
o s
o?
C3.
How
oft
en
hav
e y
ou
be
en
exp
ect
ed
to
ob
tain
con
sen
t fo
r p
roce
dur
es
wh
ich
yo
u d
o n
ot
carr
y o
utyo
urse
lf?
C4.
Do
yo
u al
way
s k
now
wh
ois
pro
vid
ing
yo
ur c
lin
ical
sup
erv
isio
n w
he
n y
ou
are
wo
rkin
g?
C5.
Ple
ase
ind
icat
e y
our
pe
rce
pti
on
of t
he
way
inw
hic
h c
riti
cal e
ven
ts a
nd
ne
ar m
isse
s ar
e r
ep
ort
ed
inyo
ur d
ep
artm
en
t.
Th
is in
dic
ato
r co
mb
ine
ssa
tisf
acti
on
wit
h e
ach
of t
he
key
ele
me
nts
of a
tra
inin
gp
ost
an
d p
rovi
de
s a
glo
bal
sati
sfac
tio
n s
core
.
Wh
ile
th
e q
uali
ty o
f pra
ctic
alex
pe
rie
nce
in a
po
st is
th
efa
cto
r m
ost
clo
sely
re
late
d t
oO
vera
ll S
atis
fact
ion
,th
eq
uali
ty o
f sup
erv
isio
n is
mo
st c
lose
ly r
ela
ted
to
th
ere
po
rtin
g o
f me
dic
al e
rro
rs.
Go
od
tra
inin
g r
eq
uire
s g
oo
dp
ract
ical
exp
eri
en
ce u
nd
er
safe
sup
erv
isio
n.
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d if
at
leas
tfo
ur o
f th
e f
ive
ite
ms
are
pre
sen
t.
0.89
0.54
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
A m
ean
is t
he
nca
lcul
ate
d if
at
leas
tfo
ur o
f th
e f
ive
ite
ms
are
pre
sen
t.
It c
ove
rs t
he
fo
llow
ing
man
dat
ory
sta
nd
ard
s:1.
1T
rain
ee
s m
ust
mak
e t
he
ne
ed
s o
f pat
ien
ts t
he
ir f
irst
con
cern
.1.2
Tra
ine
es
mus
t b
eap
pro
pri
ate
ly s
upe
rvis
ed
acco
rdin
g t
o t
he
ir e
xpe
rie
nce
and
co
mp
ete
nce
.1.3
Th
ose
sup
erv
isin
g t
he
cli
nic
al c
are
pro
vid
ed
by
trai
ne
es
mus
t b
ecl
ear
ly id
en
tifi
ed
,co
mp
ete
nt
to d
o s
o,ac
cess
ible
an
dap
pro
ach
able
by
day
an
d b
yn
igh
t,w
ith
tim
e f
or
the
sere
spo
nsi
bil
itie
s cl
ear
lyid
en
tifi
ed
wit
hin
th
eir
job
pla
n.1
.4 T
rain
ee
s m
ust
be
exp
ect
ed
to
ob
tain
co
nse
nt
on
ly f
or
pro
ced
ure
s w
hic
hth
ey a
re c
om
pe
ten
t to
pe
rfo
rm.
Do
mai
n 1
:Pat
ien
tSa
fety
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
20 National Trainee Survey 2006 – key findings
Ind
ica
tor
scor
e
Wo
rkIn
ten
sity
Wo
rklo
ad
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
6.9
Wo
rkin
g p
atte
rns
and
inte
nsi
ty o
f wo
rk b
y d
ay a
nd
by
nig
ht
mus
t b
e a
pp
rop
riat
efo
r le
arn
ing
(n
eit
he
r to
o li
gh
tn
or
too
he
avy)
.
D9.
How
wo
uld
yo
u ra
te t
he
inte
nsi
ty o
f yo
ur w
ork
,by
day
? D
10.H
ow w
oul
d y
ou
rate
th
e in
ten
sity
of y
our
wo
rk,b
y n
igh
t?
D3.
How
oft
en
do
yo
u w
ork
bey
on
d y
our
ro
ste
red
ho
urs?
D4.
How
oft
en
has
yo
urcu
rre
nt
wo
rkin
g p
atte
rn le
ftyo
u fe
eli
ng
sh
ort
of s
lee
pw
he
n a
t w
ork
? D
9.H
oww
oul
d y
ou
rate
th
e in
ten
sity
of y
our
wo
rk,b
y d
ay?
D10
.H
ow w
oul
d y
ou
rate
th
ein
ten
sity
of y
our
wo
rk,b
yn
igh
t?
Th
is in
dic
ato
r m
ust
be
tre
ate
d w
ith
cau
tio
n:a
hig
hre
sult
is g
oo
d,b
ut a
low
resu
lt w
ill n
ot
dis
tin
gui
shb
etw
ee
n e
xce
ssiv
e o
rin
ade
qua
te w
ork
load
s.
Low
sco
res
are
an
ind
icat
or
of a
po
st w
he
re w
ork
inte
nsi
ty a
nd
/or
lon
g h
our
sm
ay le
ad t
o s
lee
pd
ep
riva
tio
n o
r ex
hau
stio
n.
Ite
ms
reco
de
d a
sfo
llow
s:0
'Ve
ryin
app
rop
riat
e w
ork
inte
nsi
ty -
to
o h
eav
yo
r to
o li
gh
t'
50 'S
lig
htl
yin
app
rop
riat
e w
ork
inte
nsi
ty -
sli
gh
tly
too
he
avy
or
slig
htl
y to
oli
gh
t'
100
'wo
rk in
ten
sity
abo
ut r
igh
t'
Th
en
th
e m
ean
of t
he
two
ite
ms
isca
lcul
ate
d,o
r if
on
lyo
ne
ite
m is
pre
sen
t it
ssc
ore
is u
sed
.
0.47
0.63
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
,so
alo
w s
core
is a
he
avy
wo
rklo
ad.
A m
ean
is t
he
nca
lcul
ate
d if
at
leas
tth
ree
of t
he
fo
ur it
em
sar
e p
rese
nt.
1.5
Shif
t an
d o
n-c
all r
ota
pat
tern
s m
ust
be
de
sig
ne
d s
oas
to
min
imis
e t
he
ad
vers
ee
ffe
cts
of s
lee
p d
ep
riva
tio
n.
Do
mai
n 1
:Pat
ien
tSa
fety
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
National Trainee Survey 2006 – key findings 21
Ind
ica
tor
scor
e
Car
ee
rA
dvi
ce
Ho
urs
of
Ed
ucat
ion
Ed
ucat
ion
Sup
erv
isio
n
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
6.8
Th
ere
mus
t b
e r
ead
yac
cess
to
car
ee
r ad
vice
.E
4.H
ave
yo
u h
ad a
dis
cuss
ion
wit
h a
se
nio
rco
lle
ague
ab
out
yo
ur c
are
er
pla
ns?
G3.
How
man
y h
our
s o
fre
leva
nt,
tim
eta
ble
d,
org
anis
ed
ed
ucat
ion
alm
ee
tin
gs
or
oth
er
eve
nts
of
ed
ucat
ion
al v
alue
do
yo
uta
ke p
art
in o
n a
vera
ge
eac
hw
ee
k?
F1.D
o y
ou
hav
e a
de
sig
nat
ed
ed
ucat
ion
alsu
pe
rvis
or?
F2.
Do
yo
u h
ave
a tr
ain
ing
/le
arn
ing
agre
em
en
t w
ith
yo
ursu
pe
rvis
or,
sett
ing
out
yo
urre
spe
ctiv
e r
esp
on
sib
ilit
ies?
F3.A
re y
ou
usin
g a
lear
nin
gp
ort
foli
o in
th
is p
ost
? F4
.Are
you
usin
g a
log
bo
ok
in t
his
po
st?
F5.H
ave
yo
u b
ee
n t
old
wh
om
to
tal
k t
o in
con
fid
en
ce if
yo
u h
ave
con
cern
s,p
ers
on
al o
re
duc
atio
nal
?
Car
ee
r ad
vice
is p
arti
cula
rly
imp
ort
ant
for
the
SH
Og
rad
e.T
his
ind
icat
or
is n
ot
com
par
ed
to
th
e n
atio
nal
me
an o
r in
clud
ed
in t
he
tota
ls b
elo
w a
nd
ab
ove
th
en
atio
nal
me
an,b
eca
use
con
fid
en
ce in
terv
als
can
no
tb
e c
alcu
late
d a
s fo
r so
me
loca
tio
ns
the
sta
nd
ard
dev
iati
on
was
0.
Th
is in
dic
ato
r lo
ok
s at
th
eh
our
s o
f we
ek
ly e
duc
atio
n.
Tra
ine
es
are
un
like
ly t
o t
ake
into
acc
oun
t m
on
thly
or
less
fre
que
nt
reg
ion
al t
rain
ing
day
s,e
tc.
Th
is in
dic
ato
r is
ab
out
th
ee
duc
atio
nal
fram
ewo
rkun
de
rpin
nin
g t
he
po
st.
Eve
ry e
lem
en
t o
f th
efr
amew
ork
is a
sso
ciat
ed
wit
hg
oo
d t
rain
ing
.Low
sco
res
sug
ge
st a
tte
nti
on
sh
oul
d b
ep
aid
to
pro
gra
mm
em
anag
em
en
t an
d t
he
ro
le o
fth
e d
ire
cto
r o
f me
dic
ale
duc
atio
n in
en
suri
ng
stru
ctur
es
and
sys
tem
s ar
ein
pla
ce.
Ite
ms
reco
de
d t
o 0
to
100
scal
e,w
he
re 1
00is
a g
oo
d s
core
.
n/a
n/a
n/a
Me
an n
umb
er
of
ho
urs.
Th
ere
is a
ceil
ing
of e
igh
t,w
hic
his
lab
ell
ed
8 o
r m
ore
.
Sum
of "
Yes"
resp
on
ses
acro
ss t
he
five
ite
ms.
Sco
re c
anb
e fr
om
0 t
o 5
.If a
nit
em
has
no
re
spo
nse
,n
o s
core
isca
lcul
ate
d.
5.3
Tra
ine
es
mus
t b
e a
ble
to
acce
ss a
nd
be
fre
e t
o a
tte
nd
trai
nin
g d
ays,
cour
ses
and
oth
er
mat
eri
al t
hat
fo
rms
anin
trin
sic
par
t o
f th
e t
rain
ing
pro
gra
mm
e.
6.3
Tra
ine
es
mus
t h
ave
ad
esi
gn
ate
d e
duc
atio
nal
sup
erv
iso
r.6.
4 T
rain
ee
s m
ust
sig
n a
tra
inin
g/l
ear
nin
gag
ree
me
nt
at t
he
sta
rt o
f eac
hp
ost
.6.5
Tra
ine
es
mus
t h
ave
alo
gb
oo
k a
nd
/or
a le
arn
ing
po
rtfo
lio
re
leva
nt
to t
he
ircu
rre
nt
pro
gra
mm
e,w
hic
hth
ey d
iscu
ss w
ith
th
eir
ed
ucat
ion
al s
upe
rvis
or
(or
rep
rese
nta
tive
).6.
7 T
rain
ee
sm
ust
hav
e a
me
ans
of f
ee
din
gb
ack
in c
on
fid
en
ce t
he
irco
nce
rns
and
vie
ws
abo
utth
eir
tra
inin
g a
nd
ed
ucat
ion
exp
eri
en
ce t
o a
n a
pp
rop
riat
em
em
be
r o
f lo
cal f
acul
ty.
Do
mai
n 5
.De
live
ry o
fcu
rric
ulum
incl
udin
gas
sess
me
nt
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
22 National Trainee Survey 2006 – key findings
Ind
ica
tor
scor
e
Ind
ucti
on
Bul
lyin
g b
yC
on
sult
ants
6.1
Eve
ry t
rain
ee
sta
rtin
g a
po
st o
r p
rog
ram
me
mus
tat
ten
d a
de
par
tme
nta
lin
duc
tio
n t
o e
nsu
re t
hey
und
ers
tan
d t
he
cur
ricu
lum
,h
ow t
he
ir p
ost
fit
s w
ith
in t
he
pro
gra
mm
e,th
eir
dut
ies
and
rep
ort
ing
arr
ang
em
en
ts,t
oe
nsu
re t
hey
are
to
ld a
bo
utd
ep
artm
en
tal p
oli
cie
s an
d t
om
ee
t ke
y st
aff.
6.2
At
the
sta
rt o
f eve
ry p
ost
wit
hin
a p
rog
ram
me,
the
ed
ucat
ion
al s
upe
rvis
or
(or
rep
rese
nta
tive
) m
ust
dis
cuss
wit
h t
he
tra
ine
e t
he
ed
ucat
ion
al fr
amew
ork
an
dsu
pp
ort
sys
tem
s in
th
e p
ost
and
th
e r
esp
ect
ive
resp
on
sib
ilit
ies
of t
rain
ee
an
dtr
ain
er
for
lear
nin
g.T
his
dis
cuss
ion
sh
oul
d in
clud
e t
he
sett
ing
of a
ims
and
ob
ject
ive
sfo
r th
e t
rain
ee
to
ach
ieve
inth
e p
ost
.
B1
Did
so
me
on
e e
xpla
inyo
ur r
ole
an
dre
spo
nsi
bil
itie
s in
yo
ur u
nit
or
de
par
tme
nt
at t
he
sta
rt o
fth
is p
ost
? B
2 D
id y
ou
ge
t al
lth
e in
form
atio
n y
ou
ne
ed
ed
abo
ut y
our
wo
rkp
lace
wh
en
you
star
ted
wo
rkin
g t
he
re?
B3.
Did
yo
u si
t d
own
wit
hyo
ur s
upe
rvis
or
and
dis
cuss
your
ed
ucat
ion
al o
bje
ctiv
es
for
your
cur
ren
t p
ost
?
A g
oo
d in
duc
tio
n s
ets
th
eto
ne
fo
r th
e w
ho
le p
ost
.
n/a
n/a
Sum
of "
Yes"
resp
on
ses
acro
ss t
he
five
ite
ms.
Sco
re c
anb
e fr
om
0 t
o 3
.If a
nit
em
has
no
re
spo
nse
,n
o s
core
isca
lcul
ate
d.
6.11
Tra
ine
es
mus
t n
ot
be
sub
ject
ed
to,
or
sub
ject
oth
ers
to,
be
hav
iour
th
atun
de
rmin
es
the
irp
rofe
ssio
nal
co
nfi
de
nce
or
self
-est
ee
m.
J1.H
ave
yo
u b
ee
nsu
bje
cte
d t
o p
ers
iste
nt
be
hav
iour
in t
his
po
st t
hat
has
un
de
rmin
ed
yo
urp
rofe
ssio
nal
co
nfi
de
nce
and
se
lf-e
ste
em
?
J3.W
hic
h o
ne
of t
he
foll
owin
g is
th
e m
ain
sour
ce o
f th
is b
eh
avio
ur?
Perc
en
tag
e o
f re
spo
nd
en
tsw
ho
re
po
rt b
ein
gsu
bje
cte
d t
o b
eh
avio
urd
efi
ne
d in
J1
by
con
sult
ants
- t
he
irre
spo
nse
to
J3.
Bul
lyin
g m
ay a
lso
be
pe
rpe
trat
ed
by
oth
er
trai
ne
es,
nurs
es,
man
age
rsan
d e
ven
pat
ien
ts.
Th
is in
dic
ato
r o
nly
loo
ks
at c
on
sult
ant
bul
lyin
g,b
eca
use
this
so
urce
isas
soci
ate
d w
ith
th
em
ost
str
ess
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
Do
mai
n 6
.Sup
po
rtan
d d
eve
lop
me
nt
of
trai
ne
es,
trai
ne
rs a
nd
loca
l fac
ulty
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
National Trainee Survey 2006 – key findings 23
Ind
ica
tor
scor
e
Me
dic
alE
rro
r
Eur
op
ean
Wo
rkin
gT
ime
Dir
ect
ive
pe
rce
nta
ge
Do
mai
n 2
.Qua
lity
Ass
uran
ce,R
evie
wan
d E
valu
atio
n
2.1
Pro
gra
mm
es,
po
sts,
asso
ciat
ed
man
age
me
nt,
and
dat
a co
lle
ctio
n c
on
cern
ing
trai
ne
es
and
loca
l fac
ulty
mus
t co
mp
ly w
ith
th
eE
uro
pe
an W
ork
ing
Tim
eD
ire
ctiv
e,D
ata
Pro
tect
ion
Act
and
Fre
ed
om
of I
nfo
rmat
ion
Act
.
N/A
inte
rnal
out
com
em
eas
ure
to
ass
ess
th
ep
roce
ss s
tan
dar
ds
rela
tin
g t
op
atie
nt
safe
ty in
do
mai
n 1
of
the
Gen
eric
sta
ndar
ds
for
trai
ning
.
D1.
Are
yo
ur r
ost
ere
dw
ork
ing
ho
urs
com
pli
ant
wit
h t
he
Eur
op
ean
Wo
rkin
gT
ime
Dir
ect
ive
?
D2.
Hav
e y
ou
be
en
ask
ed
to
sub
mit
ho
urs
that
are
com
pli
ant
wit
h t
he
Eur
op
ean
Wo
rkin
g T
ime
Dir
ect
ive,
wh
en
th
e h
our
s yo
u ac
tual
lyw
ork
are
no
t co
mp
lian
t
H7
In t
he
last
mo
nth
hav
eyo
u m
ade
a s
eri
ous
me
dic
ale
rro
r?
H8
In t
he
last
mo
nth
,hav
eyo
u m
ade
a p
ote
nti
ally
seri
ous
me
dic
al e
rro
r?
Th
is s
core
is d
iffi
cult
to
inte
rpre
t,as
it is
no
tm
eas
urin
g a
ctua
l err
ors
but
wh
eth
er
the
tra
ine
e w
ill
rep
ort
hav
ing
mad
e t
he
m o
nth
is s
urve
y in
stru
me
nt.
Perc
en
tag
e o
fre
spo
nd
en
ts w
ho
answ
ere
d "
Yes"
to
D1
and
"N
o"
to D
2.If
an
ite
m h
as n
o r
esp
on
se,
no
sco
re is
calc
ulat
ed
.
n/a
n/a
Sco
red
1 if
th
ere
spo
nd
en
t in
dic
ate
dth
ey h
ad m
ade
on
e o
rm
ore
th
an o
ne
seri
ous
an
d/o
rp
ote
nti
ally
se
rio
usm
ed
ical
err
or
in t
he
last
mo
nth
.Oth
erw
ise
sco
red
as
0.R
esp
on
de
nts
wit
hm
issi
ng
dat
a o
r w
ho
ind
icat
ed
th
ey d
id n
ot
wis
h t
o a
nsw
er
we
rese
t to
mis
sin
g a
nd
excl
ude
d fr
om
th
ean
alys
is.
N/A
inte
rnal
out
com
em
eas
ure
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
24 National Trainee Survey 2006 – key findings
Ind
ica
tor
scor
e
Oth
er
Form
alTe
ach
ing
6.16
Tra
ine
es
mus
t b
e a
ble
to
acce
ss t
rain
ing
in g
en
eri
cp
rofe
ssio
nal
sk
ills
at
all
stag
es
in t
he
ir d
eve
lop
me
nt.
G4.
Hav
e y
ou
had
fo
rmal
teac
hin
g s
ince
leav
ing
me
dic
al s
cho
ol i
n a
ny o
f th
efo
llow
ing
?
Co
mm
unic
atio
n,a
pp
rais
alsk
ills
,te
ach
ing
sk
ills
,le
ade
rsh
ip,t
eam
wo
rkin
g,
pat
ien
t sa
fety
,tim
em
anag
em
en
t,m
ed
ical
eth
ics.
As
the
wo
rdin
g o
f th
ese
ite
ms
did
no
t p
ert
ain
to
th
ecu
rre
nt
po
st,t
his
ind
icat
or
can
on
ly b
e a
pp
lie
d a
td
ean
ery
leve
l an
d n
ot
the
leve
l of p
rovi
de
r.
Sco
red
1 f
or
eac
hg
en
eri
c sk
ill t
he
trai
ne
e h
as r
ece
ive
dtr
ain
ing
in.P
oss
ible
ran
ge
0 t
o 8
.
n/a
PM
ET
B G
ener
icst
and
ard
s fo
r tr
ain
ing
Dom
ain
PM
ET
B G
ener
icst
an
da
rds
cove
red
Inte
rpre
tati
onIt
ems
incl
ud
edC
alc
ula
tion
Inte
rna
l re
lia
bil
ity
:C
ron
ba
ch’s
Alp
ha
(a)
National Trainee Survey 2006 – key findings 25
Method variance
Route of administration
The data were analysed to test for any differences by route of administration (seeTable 1 above). Differences did emerge. Respondents who submitted a return usingthe portable electronic survey units seemed more likely to give more negativeresponses. Some examples of this effect are given below in Table 6 for items D2 andH7. It seems likely that this effect is due to trainees perceiving the survey units asmore anonymous than the web (trainees were contacted by email) and paper (barcodes were present on the paper forms), so respondents using the survey unitswere less inclined to give more socially desirable responses as they were confidentthey could not be identified.
Time in post
As the survey was a snapshot, respondents had been in post for variable lengths oftime. Comparing the indicator scores against the time in post item (A2), showed thattrainees who had been in post longer were slightly more likely to give morepositive answers.
All other routes
In the last month, haveyou made a seriousmedical error? -percentage yes to onceor more than once
Have you been asked tosubmit hours that arecompliant with theEuropean Working TimeDirective, when thehours you actually workare NOT compliant? -percentage yes
Portableelectronic surveyunits
Portableelectronic surveyunits
All other routes
2.0
5.5
11.9
33.5
18,102
6,365
16,563
201.98
1,332.77
<0.001
<0.001
5,342
N x2 p
Table 6 Examples of method variance
26 National Trainee Survey 2006 – key findings
Adjustment for method variance
This method variance was particularly problematic because route of administrationwas confounded by provider and deanery. In order to make valid comparisonsacross providers or deaneries and be certain that differences were not due tovariation in the route or the length of time in post, all the indicator scores8 wereadjusted as follows:
Each indicator score was regressed on to the survey route recoded (1 for portableelectronic survey units and 0 for all other routes) and the length of time in post (asper A2).
An adjusted indicator score was calculated by applying the regression coefficientsobtained in step 1: adjusted score = score + (survey route recoded * B1) + (lengthof time in post * B2).
There were then no differences in mean adjusted indicator scores by route or timein post.
It was not possible to adjust the indicator scores based on categorical items at therespondent level using this method. So these should be interpreted in isolationrather than comparing across providers.
8 Access to Educational Resources, Adequate Experience, Feedback, Handover, Other Learning Opportunities,Overall Satisfaction, Supervision,Work Intensity and Workload
National Trainee Survey 2006 – key findings 27
3. Key findings
The objective of the analysis presented in this chapter is to demonstrate that theindicator scores derived from the survey have both concurrent and constructvalidityix and that the scores can show differences across providers in trainees’perceptions of the quality of their training. The successful demonstration of thesefeatures of the scores shows that the survey can be appropriately used as ascreening tool of quality assurance and quality management work. To demonstrateconcurrent validity, analysis was undertaken using the surveys outcome variables.To demonstrate construct validity, the survey was used to assess national traininginitiatives. Finally, analysis was undertaken to look for differences across trainingproviders on the indicator scores that earlier analysis had shown to be the mostvalid and reliable of the scores.
The survey’s outcome variables
The survey collected a number of items that may be viewed as outcome variables:items used in the Overall Satisfaction Score and the Medical Error Score; andwhether the trainee experienced behaviour that undermined their self-confidence(colloquially this might be termed bullying). Trainees’ responses to these outcomevariables can be predicted using the other scores from the survey to answerquestions such as which aspects of trainees’ perceptions of their posts are moststrongly related to trainees’ Overall Satisfaction Scores. If relationships betweenthese measures of aspects of the trainees’ experience and these outcomes variablesare found, they provide evidence that the indicator scores have concurrent validity;i.e. trainees’ perceptions of facets of their training relate to outcomes measures thatwere obtained at the same time as the facet measures, such as the OverallSatisfaction Score. One would expect these relationships to exist, so the object is notto demonstrate that the relationships exist (as one might do in research), but toprovide evidence that the survey items are measuring what they purport to. Forexample, one would expect the supervision of trainees to be related to whetherthey report having made perceived medical errors; therefore the Supervision Scoreshould be associated with the Medical Error Score.
90
GP (N =
1,856)
Patholo
gy SHO (N =
96)
Psych
iatry
SpR (N =
773)
Opthalm
ology SpR (N
= 267)
Anaesthetic
s SHO (N
= 1,1
86)
Opthalm
ology SHO (N
= 194)
Emerg
ency M
edicin
e SHO (N =
902)
Radiolo
gy SpR (N =
818)
Psych
iatry
SHO (N =
1,674)
Surgery
SpR (N =
2,101)
Patholo
gy SpR (N =
467)
Paediatri
cs and C
hild H
ealth SHO (N
= 1,2
14)
Medic
ine SpR (N
= 2,9
83)
Anaesthetic
s SpR (N
= 1,4
19)
Paediatri
cs and C
hild H
ealth SpR (N
= 1,1
13)
Emerg
ency M
edicin
e SpR (N =
350)
Obstetri
cs and G
ynaecolo
gy SpR (N =
882)
Obstetri
cs and G
ynaecolo
gy SHO (N =
906)
Medic
ine SHO (N
= 2, 9
05)
Surgery
SHO (N =
1,905)
Overall Satisfaction
Ove
rall
Sat
isfa
ctio
n
Sco
re A
dju
ste
d
85
80
75
70
65
60
55
50
28 National Trainee Survey 2006 – key findings
On this measure, surgical SHOs are the least satisfied with their training. Looking atthe items included in the Overall Satisfaction Score, 3.9 per cent of surgical SHOs (N= 1,904) indicated that they would describe their post as very poor to a friend,compared with only 0.4 per cent of GPRs (N = 1,853). Similarly, 1.6 per cent (N =1,907) of SHO surgeons rated their supervision as very poor compared with only0.2 per cent (N =1,855) of the GPs. In terms of numbers of trainees, there are 1,237trainees in the specialties and grades on Chart 1 (5.2 per cent) who indicated thatthey would describe their post as poor or very poor to a friend.
Overall Satisfaction
Trainees’ Overall Satisfaction with their training posts varies by specialty andgrade; this is illustrated in Chart 19.
Chart 1 Trainees’ Overall Satisfaction Score by specialty/grade groups
Factors associated with job satisfaction
The two scores most closely associated with trainees’ Overall Satisfaction Score areSupervision and Adequate Experience; good posts are those where trainees areexposed to useful experience under good supervision. However, the AdequateExperience Score was excluded from the analysis that follows because the factoranalysis (see Table 4, Chapter 2) suggested that it is not measuring a separateconstruct from the Overall Satisfaction Score. The two scores are highly correlated(r = 0.759, N = 24,843). This measurement problem may be partly due toperceptions of the adequacy of the experience gained being so inextricably boundup in trainees’ overall satisfaction with post; it is not possible to be satisfied with apost that provides inadequate experience. Another plausible explanation,mentioned in Chapter 2, is that the Adequate Experience items are also overallrating type items like those used in the Overall Satisfaction Score. Plans to tacklethis measurement issue are outlined in Chapter 5.
9 The chart excludes Public Health and Occupational Medicine due to wide confidence intervals. GPs includesGPRs and SHO based in GP practices, but not SHO VTS trainees based in acute sites who are included in theappropriate acute specialty group.
National Trainee Survey 2006 – key findings 29
Sequential multiple regression was used to examine which facets of the trainees’job, as measured by the survey’s indicator scores, were most strongly related totheir Overall Satisfaction Score. Independent variables were entered into the modelin the order given in Table 7 below. At each step, the addition of the new variablessignificantly improved the prediction. A considerable amount of the variance (39per cent) in the Overall Satisfaction Score is explained by the survey indicatorscores entered in step three of the multiple regression. This is over and above thevariance accounted for by the variables entered in steps one and two. So, forinstance, one could not argue that difference by specialty group on the indicatorscores are the explanation for differences in the Overall Satisfaction Score, becausethe model has already accounted for differences by specialty group.
Table 7 Overall model predicting the trainees’ Overall Satisfaction with theirtraining
10 Ethnicity was recoded into white or non-white for the purposes of this analysis.11 Derived from A5 and A6, e.g. respondent working in medicine and intending to practise as consultant in surgery.
Demographicvariables
Time in current postSurvey route
0.008 0.008 86.465 <0.001
0.074 0.073 81.489 <0.001
0.396 0.395 1,439.848 <0.001
Method variance1
2
3 Survey indicatorscores
N = 21,660. All specialties included
GradeSpecialty group (dummy coded)Ethnicity10
SexDifferent group from intended11
Forces Year qualifiedWhere qualified
Access to EducationalResources Bullying by ConsultantsEducational SupervisionHours of Education Induction Other Learning Opportunities Supervision Workload
R2Step Type of variables Variables R2adj F change P
30 National Trainee Survey 2006 – key findings
The following indicator scores were excluded from this model:
● Career Advice, Feedback, Handover and European Working Time scores. Thesewere excluded because the number of not applicable/not sure/missing responsesreduced the number of cases available for the analysis (all had over 3,650respondents with missing data).
● The Work Intensity Score was removed because it contains items (D9 and D10)that were also included in the Workload Score (though coded differently) and istherefore not independent.
Table 8 Survey variables and Overall Satisfaction
2.015
6.838
-0.632
-0.004
-0.392
-2.962
-3.403
-0.320
-5.325
-9.673
-5.936
-7.374
-4.302
-4.752
-3.241
-4.366
-5.548
-1.029
-1.468
3.325
-7.957
-7.762
9.697
-14.426
15.583
21.923
23.718
33.856
49.129
-1.321
0.044
0.000
0.527
0.996
0.695
0.003
0.001
0.749
0.000
0.000
0.000
0.000
0.000
0.000
0.001
0.000
0.000
0.303
0.142
0.001
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.187
0.001
0.000
0.000
0.000
0.000
0.000
0.000
0.001
0.003
0.001
0.002
0.001
0.001
0.000
0.001
0.001
0.000
0.000
0.000
0.002
0.002
0.003
0.006
0.007
0.013
0.016
0.032
0.067
0.000
Constant
Survey route
Time in post
Grade (SpR/GPR = 1)
Forces (Forces =1)
Anaesthetics
Emergency medicine
General practice
Medicine
Obstetrics and gynaecology
Occupational medicine
Paediatrics and child health
Pathology
Psychiatry
Public health
Radiology
Surgery
Different group from intended
Year qualified
Sex (female = 1)
Ethnicity (minority ethnic group = 1)
Where qualified (outside UK = 1)
Access to Educational Resources Score
Bullying by Consultants Score (1 = reported)
Educational Supervision Score
Hours of Education Score
Induction Score
Other Learning Opportunities Score
Supervision Score
Workload Score
Shared variance = 0.238, Unique variance = 0.158
Variables that are statistically significant at P < 0.05 are shaded.
tPredictor
0.038
-0.004
-0.000
-0.002
-0.038
-0.034
-0.004
-0.091
-0.109
-0.034
-0.092
-0.033
-0.059
-0.020
-0.039
-0.083
-0.006
-0.010
0.019
-0.051
-0.055
0.055
-0.078
0.095
0.128
0.145
0.209
0.310
-0.008
B Standardised
Sig. sr2
97.448
1.283
-0.074
-0.001
-0.357
-1.866
-2.357
-0.208
-3.194
-6.329
-9.222
-4.687
-3.402
-3.009
-3.962
-3.134
-3.378
-0.257
-0.035
0.571
-1.551
-1.673
0.051
-5.817
1.149
1.103
2.656
0.177
0.342
-0.007
B Unstandardised
National Trainee Survey 2006 – key findings 31
Predictors that are statistically significant at P < 0.05 are shaded. The direction ofrelationship is shown by whether a coefficient is positive or negative. All specialtiesare dummy coded with a 1 indicating the trainee is in that specialty and 0indicating they are notx.
As ethnicity is coded 1 for minority ethnic groups and the regression coefficient isnegative, trainees from ethnic minorities are less likely to be satisfied with theirtraining posts. This relationship was tested after accounting for whether the traineequalified outside of the UK (where qualified in Table 8).
With the exception of the Workload Score the indicator scores relate to the OverallSatisfaction Score (see Table 8). This shows that as one would hope and expect theyare measuring facets of the trainees’ experience that relates to their satisfactionwith the post. The finding that the Workload Score is not associated with the OverallSatisfaction Score, suggests that trainees do not object to working hard.
Based on the unique contribution made by the given score (sr2 in Table 8), the mostimportant predictor12 is the Supervision Score, which accounts for 6.7 per cent of thevariance in the Overall Satisfaction Score after accounting for the other measures inTable 8. This finding reflects the importance of the clinical supervisors in ensuringthat the trainees have a positive training experience. Trainees who perceive theirsupervision to be good are more likely to report being satisfied with their post. Ofcourse, if the Adequate Experience Score is included, it is by far the most importantpredictor of the Overall Satisfaction Score (sr2 = 0.264, with a R2 = 0.660, withSupervision Score the second most important. The Adequate Experience Score isonly excluded from this analysis due to measurement issues noted above.
When the scores that were excluded due to the number of cases with missing dataare included in the model, together with the Adequate Experience Score (reducingthe N to 10,416), the pattern remains the same: Adequate Experience is the mostimportant followed by Supervision. The Handover Score was not related to OverallSatisfaction Score; while Career Advice, Feedback, and European Working TimeScores were all related to the Overall Satisfaction Score.
Medical Errors
The Supervision Score is also the most important predictor of a trainee reportingthat they perceived that they had made a serious or potentially serious medicalerror (combined into the Medical Error Score – see Table 5). Logistic regressionwas used to analyse which of the survey’s indicator scores predicted the MedicalError Score (a binary outcome variables scored 0 or 1, see Table 5), the analysisstatistically controlled for the same method and demographic variables as themultiple regression used to predict the Overall Satisfaction Score. Adding theindicator scores into the model in step 3 in Table 9 significantly improves theprediction.
12 Although Adequate Experience was removed
32 National Trainee Survey 2006 – key findings
Table 9 Overall model predicting the trainees’ reporting of medical errors
Table 10 Survey variables and reporting making a medical error
13 Ethnicity was recoded into white or non-white for the purposes of this analysis.14 Derived from A5 and A6, e.g. respondent working in medicine and intending to practise as consultant in surgery.
Demographicvariables
Time in current postSurvey route
29.33 <0.001 29.33 <0.001
465.850 <0.001 495.18 <0.001
334.790
<0.001
829.97 <0.001
Method variance1
2
3 Survey Indicatorscores
N = 21,252. All specialties included
GradeSpecialty group (dummy coded)Ethnicity13
SexDifferent group from intended14
ForcesYear qualifiedWhere qualified
Access to Educational Resources Adequate ExperienceBullying by ConsultantsEducational SupervisionHours of Education Induction Other Learning Opportunities Supervision Workload
xx2-stepStep Type of variables Variables P xx2 -model P
0.491
3.623
167.984
6.074
0.108
0.468
7.310
1.915
25.062
18.919
5.182
14.775
4.792
0.001
0.174
3.998
1.250
125.957
67.402
7.239
0.483
0.057
0.000
0.014
0.742
0.494
0.007
0.166
0.000
0.000
0.023
0.000
0.029
0.981
0.676
0.046
0.264
0.000
0.000
0.007
1.039
0.938
1.187
0.978
0.846
1.022
0.932
0.748
0.747
0.997
1.007
1.256
0.999
1.007
0.938
1.002
0.978
0.988
0.000
Survey route
Time in post
Specialty group
Different group from intended
Grade (SpR/GPR = 1)
Forces (Forces =1)
Year qualified
Sex (female 1)
Ethnicity (minority ethnic group = 1)
Where qualified (outside UK = 1)
Access to Educational Resources Score
Adequate Experience Score
Bullying by Consultants Score
Educational Supervision Score
Hours of Education Score
Induction Score
Other Learning Opportunities Score
Supervision Score
Workload Score
Constant
Variables that are statistically significant at P < 0.05 are shaded.
WaldPredictor
0.038
-0.064
0.171
-0.022
-0.167
0.022
-0.070
-0.291
-0.292
-0.003
0.007
0.228
-0.001
0.007
-0.063
0.002
-0.022
-0.012
-43.991
B Sig. Odds Ratio
National Trainee Survey 2006 – key findings 33
In interpreting these findings, it should be noted that the variable being predictedis the trainees’ reporting of perceived medical errors in the last month on thissurvey instrument (and not necessarily their reporting of any incident through theappropriate local procedures for reporting incidents), i.e. not actual medical errors.It is likely that the actual rate of perceived medical errors is higher than reportedon the survey forms, given the method variance finding (see Table 6, Chapter 2).
Predictors that are statistically significant at P <0.05 are shaded on Table 10. Thedirection of relationship is shown by whether a coefficient is positive or negative.There are a number of findings of note. Trainees from a minority ethnic group andtrainees who qualified outside the UK are less likely to indicate they made amedical error in the last month. Trainees who report being bullied are more likelyto report making an error.
It is likely that the effect for specialty group is related to the type of work. Forinstance, as one might expect, emergency medicine trainees report having mademore errors (16.8 per cent, N = 1,217) than trainees in other specialties (9.3 percent, N = 22,829).
Higher Induction, Supervision and Workload Scores are associated with traineesbeing less likely to report making medical errors. This means that a clearer andmore complete induction, better clinical supervision and not being overloaded withwork15 are associated with trainees being less likely to report making medicalerrors. A higher Adequate Experience Score is associated with trainees more likelyto report errors: a job offering plenty of experience is related to trainees beinglikely to report making errors.
Given the importance of the Supervision Score and the Workload Score inpredicting the reporting of medical errors, the individual items were examined;some of these are presented below in Tables 11 and 12. Trainees who reportmaking medical errors are more likely to report saying they are forced to cope withproblems beyond their competence and that they feel that they were supervised bysomeone not competent to do so. They are also more likely to report workingbeyond their rostered hours and feeling sleep deprived.
Table 11 Medical errors and Supervision Score items
15 All Scores were calculated so that a high score was good
10.1 7.1 1.3 21,937No medical errors reported
One or more serious or potentiallyserious medical errors reported
No medical errors reported
How often, if ever, have you beensupervised by someone who youfeel isn’t competent to do so?
Monthly%
How often have you felt forced tocope with problems beyond yourcompetence or experience?
Weekly%
Daily%
60.2
Rarely%
21.4
16.2 14.2 3.1 2,33955.211.2
3.9 2.4 0.8 21,90336.656.4
6.8 4.7 1.0 2,33843.643.9
Never%
N
One or more serious or potentiallyserious medical errors reported
34 National Trainee Survey 2006 – key findings
PMETB in conjunction with members of the Survey Working Group will publish amore detailed paper analysing these data.
Table 12 Medical Errors and Workload items
The Supervision Score has been shown out of all the indicator scores to be the moststrongly related to Overall Satisfaction Score, Medical Errors Score and ConsultantBullying Score. Like the Overall Satisfaction Score, it varies by grade and specialty.
Supervision Score
To understand what the differences in the Supervision Score mean, one can look atsome of the individual items included in its derivation: 9.7 per cent (184/1,902) ofsurgical SHOs, who as a group are the least satisfied with their supervision,indicated that they were supervised by someone they felt was not competent to doso, monthly or more frequently, whereas only 3.0 per cent (23/772) of the psychiatrySpRs felt that this was the case. Of the surgical SHOs, 26.6 per cent (507/1,909)reported feeling forced to cope with problems beyond their competence orexperience monthly or more frequently, compared with 8.4 per cent (65/775) of thepsychiatry SpRs.
Chart 2 Supervision Score by specialty/grade groups
12.7 31.6 13.5 21,904No medical errors reported
One or more serious or potentiallyserious medical errors reported
No medical errors reported
How often, if ever, have you beensupervised by someone who youfeel isn’t competent to do so?
Monthly%
How often do you work beyondyour rostered hours?
Weekly%
Daily %
33.8
Rarely%
8.4
13.4 37.9 21.3 2,33822.25.1
18.9 15.0 3.4 21,83941.121.6
23.7 23.6 7.1 2,33233.612.0
Never%
N
One or more serious or potentiallyserious medical errors reported
100
Patholo
gy SHO (N =
96)
Psych
iatry
SpR (N =
772)
Patholo
gy SpR (N =
459)
GP (N =
1,851)
Anaesthetic
s SHO (N
= 1,1
85)
Psych
iatry
SHO (N =
1,663)
Paediatri
cs and C
hild H
ealth (N
= 110)
Paediatri
cs and C
hild H
ealth (N
= 1,2
11)
Anaesthetic
s SpR (N
= 1,4
15)
Emerg
ency M
edicin
e SpR (N =
349)
Obstetri
cs and G
ynaecolo
gy SpR (N =
882)
Opthalm
ology SpR (N
= 267)
Surgery
SpR (N =
2,102)
Medic
ine SpR (N
= 2,9
82)
Radiolo
gy (N =
818)
Emerg
ency M
edicin
e SHO (N =
900)
Opthalm
ology SHO (N
= 194)
Medic
ine SHO (N
= 2,9
02)
Obstetri
cs and G
ynaecolo
gy SHO (N =
907)
Surgery
SHO (N =
1,904)
Supervision
Sup
erv
isio
n
Sco
re A
dju
ste
d
95
90
85
80
75
70
65
60
National Trainee Survey 2006 – key findings 35
Feeling bullied – overall prevalence
To ascertain whether trainees felt bullied, the questionnaire included an itempreviously usedxiii, “J1. Have you been subjected to persistent behaviour in this postthat undermined your professional confidence and/or self-esteem?”
Overall, 10.5 per cent of trainees (N = 23,198) reported being subjected topersistent behaviour in their current post that undermined their professionalconfidence and/or self-esteem. A further 1,460 trainees did not wish to answer andanother 222 left the item blank16, suggesting that the rate may be as high as 16.5 percent, were these responses taken as yes.
The rate is lower than that reported by Paice et alxiv who also used the Hicks item.On their survey, 18.1 per cent of hospital based trainees reported experiencingbullying in their current post, compared with 10.9 per cent of hospital basedtrainees on this survey (N = 20,947, GPRs excluded).
The source of the perceived bullying
Respondents who indicated they had been subjected to this type of behaviour wereasked to indicate its main source. Among hospital trainees (including VTS traineeson hospital placements) this varied by grade. SpR trainees were more likely toreport that a consultant was the source, while SHO grade trainees were more likelyto report that other trainees or nursing staff were the source (see Chart 3):x2 = 307.4, P < 0.001.
16 This was possible on the paper route.
Chart 3 The source of the perceived bullying
1 SHO (Spec and VTS) N = 1,091
2 Spr N = 1,029
The main source of behaviour
Source
70%
60%
50%
40%
30%
20%
10%
0%Consultants Other
traineesNurses or midwives
Managers Patients or relatives
The wholeculture at work
Other
36 National Trainee Survey 2006 – key findings
Reporting of bullying and subsequent action
Of the trainees who reported bullying, 35.1 per cent (N = 2,268 – missing data in167 cases) indicated that they reported it to their employer/someone in authority; ofthese, 41.7 per cent (N = 575 – missing data in 221 cases) indicated that successfulaction was taken to stop it. Of all the reported incidents (all incidents includingthose for which there are missing responses to the reporting and successful actionitems) only 9.9 per cent (N = 2,435) were reported and subjected to remedialaction.
Whether a trainee who experienced bullying reported it varied according to thesource of the bullying. Trainees were more likely to report it if it came frommanagers or patients/relatives than if it came from other clinical staff (x2 = 21.41, P < 0.001, N = 1,757 cases of bullying with a specific source only). 34.9per cent of bullying involving trainees, consultant or nursing staff was reported,compared with 54.0 per cent of bullying involving managers or patients/relatives.
Have you been subjected to persistent behaviour in this post that has underminedyour professional confidence and/or self-esteem?
20%
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%
Obstetri
cs and G
ynaecolo
gy SpR (N =
793)
Obstetri
cs and G
ynaecolo
gy SHO (N =
844)
Emerg
ency M
edicin
e SHO (N =
814)
Surgery
SHO (N =
1,745)
Anaesthetic
s SHO (N
= 1,0
92)
Anaesthetic
s Spr (
N = 1,3
43)
Psych
iatry
SHO (N =
1,583)
Radiolo
gy SpR (N =
765)
Patholo
gy SHO (N =
765)
Psych
iatry
SpR N (N
= 747)
Opthalm
ology SHO (N
= 178)
GP (N =
1,782)
Patholo
gy SpR (N =
428)
Emerg
ency M
edicin
e SpR (N =
335)
Medic
ine SHO (N
= 2,7
10)
Medic
ine SpR (N
= 2,8
19)
Paediatri
cs and C
hild H
ealth SHO (N
= 1,1
29)
Paediatri
cs and C
hild H
ealth SpR (N
= 1,0
26)
Surgery
SpR (N =
1,983)
Opthalm
ology SpR (N
= 245)
Chart 4 Bullying by specialty/grade group
Are certain types of trainee more likely to be bullied?
Trainees’ reporting of bullying also varies by specialty and grade, as illustrated inChart 4 below (xx2 = 175.37, P < 0.001. N = 22,450)17. The obstetrics and gynaecologytrainees report the most bullying, while GPs report the least.
Confidence interval calculated following Altman et alxv (2000)
These findings concur with existing literature that has shown an increasedprevalence of “harassing or discriminatory behaviours” in surgery and obstetricsand gynaecologyxvi.
Logistic regression was used to look at the relationships between feeling bulliedand demographic variables. These relationships varied by specialty group (seeTable 13).
17 For clarity on the chart, very small or non-existent groups were excluded: public health, occupational medicine,and radiology SHOs.
Table 13 The relationship between demographic variables and reportingbullying for each specialty group and overall
Table 13 gives the odds ratios for each variable that is related to trainee reportingbullying at P < 0.05.Where no odds ratio is given, the relationship is not statisticallysignificant. For instance, for surgical trainees the following variables are associatedwith an increased probability of reporting bullying on this survey: being in postlonger, qualifying outside the UK and being female. None of the other variables areassociated with reporting bullying.
With the exception of time in post and year qualified, all variables are binary,where they are coded 1 for the presence of the description given in Table 13 (e.g.female = 1). The Odds ratio for time in post and year qualified indicate that thelonger the trainee has been in post, the more likely they are to report being bulliedin that post and trainees who qualified more recently (i.e. left medical schoolrecently) are more likely to report being bullied.
For binary variables, the odds ratios show the increased probability of reportingbullying if the factor is present. For all specialties, trainees from a minority ethnicgroup are 1.3 times more likely to report being bullied than white trainees.
In logistic regression, when all the variables are entered at the same stage, eachvariable is tested after accounting for all the other variables in the model, soqualifying outside the UK and being a non-white trainee increases the probability ofbeing bullied. This is illustrated in Table 14.
National Trainee Survey 2006 – key findings 37
Anaesthetics
Traineeis locum
Traineein HMforces
Yearqualified
Timein post
2,284
N QualifiedoutsideUK
1.51
Emergency medicine(model not significantat P <0.05)
1,099 1.56
Female Traineenot white
4.11General Practice 1,754 1.82
Medicine 1.235,287 1.34
1.99 1.04Obstetrics andgynaecology 1,535 1.71
11.59Ophthalmology 401
Paediatrics andchild health
1.412,063
Pathology (model notsignificant at P <0.05) 477
1.59Psychiatry (model notsignificant at P <0.05) 2,176
Radiology 1.98744 1.92
1.33Surgery 1.203,489 1.56
1.01 1.15All specialtiespredicting anybullying.
1.2121,758 1.11 1.30
1.42 0.95 0.74All specialties -predicting consultantbullying
1.5021,758
Variables in model. If the variable is related (P <0.05) to reporting bullying theOdds ratio is reported, if it is not related the cell is blank.
38 National Trainee Survey 2006 – key findings
Looking at reports of feeling bullied by consultants only, in the final row of Table 13,gives a different pattern; neither the trainees’ sex nor their ethnicity (white versusnon-white) is associated with the trainees reporting bullying by a consultant.Furthermore, trainees from outside the UK are less likely to feel bullied byconsultants (odds ratio = 0.74) and older trainees are more likely to report feelingbullied by consultants (this fits with the finding that SpR are more likely to feelbullied by consultants – see Chart 3).
This pattern of results reflects trainees’ own perceptions. Of those who reportedbeing bullied and answered whether this was related to their ethnicity, sex, sexualorientation or religious beliefs, 36.1 per cent of trainees who felt bullied by anynon-consultant source felt it was related to one of these, against 28.3 per cent ofthose who reported the source as being a consultant (N = 1,888, x2 = 12.8,P < 0.001).
The model varies across specialties; the trainee’s gender is only related toperceived bullying within certain specialty groups - anaesthetics, emergencymedicine and surgery. GP trainees working in the armed forces are more likely toreport perceived bullying than civilian GPs (seven of the 22 GPs in the armedforces answered yes to item J1).
Perceived bullying and indicator scores
Respondents who reported being bullied had lower scores on all the indicatorscores, except the Handover Score (all mean comparisons were statisticallysignificant at P 0.05 after the Bonferroni correction was applied), suggesting thatexperiencing bullying is associated with a generally less positive training post. Thelargest association was with the Supervision Score, again indicating the importanceof this indicator.
PMETB in conjunction with members of the Survey Working Group will publish amore detailed paper analysing these data. It is worth noting here that, as Paice et alnote, some of the behaviours that erode trainees’ professional confidence or self-esteem may be attempts to improve trainees’ performance, so an educational ratherthan punitive approach is required to tackle the problem.
Table 14 Trainees’ ethnicity and place of qualification and bullying
9,966White qualifying within the UK
White qualifying outside the UK
Minority ethnic group qualifying within the UK
Minority ethnic group qualifying outside the UK
% reporting bullying
8.6
N
1,5699.6
3,40910.8
7,19511.9
National training initiatives
The survey data were used to assess the impact of two national initiatives that havesought to improve the training experience of junior doctors:
● Radiology academiesxvii
● Hospital at Nightxviii
In both cases, some of the predicted differences emerged, providing furtherevidence of the construct validity of the scores derived from the survey data andhighlighting their potential for evaluating other initiatives that are designed toimprove junior doctors’ training posts.
Radiology academies
Three academies were delivered as part of the ‘Radiology - Integrated TrainingInitiative (R-ITI)’, a national programme to provide an increased number of highquality radiologists.
Numbers of respondents
Only trainees in clinical radiology (also known as diagnostic radiology andformerly known as radiology) were included. The four clinical radiology traineeswhose grade was recorded as something other than SpR were excluded. This left 68trainees in the academies (as detailed in Table 15) and 536 trainees not in theacademies. Trainees at the academy sites but not in the 2005 or 2006 cohorts oftrainees have also been classified as training at an academy.
Approach to analysis and results
The mean indicator scores (derived from survey items – see Table 5 in Chapter 2)for the two groups were compared and the ANOVA (F-test) used to test fordifferences; the Bonferroni correction for multiple testing was applied. A number ofthe mean scores were significantly different at P < 0.05; they are shaded in Table 16below. All the differences are in the expected direction. Trainees based within anacademy reported a more positive training experience. The survey was notspecifically designed to evaluate the radiology academies and it is likely that asurvey with items concerned specifically with the training of radiologists wouldhave found more marked differences.
National Trainee Survey 2006 – key findings 39
Table 15 Respondents by radiology academy
24Leeds and West YorkshireRadiology Academy
HospitalsAcademy
Leeds Teaching Hospitals NHS Trust
16Norfolk & NorwichRadiology Academy
Norfolk & Norwich University Hospital NHSTrust
N
2215
Peninsula RadiologyAcademy
Plymouth Hospitals NHS TrustRoyal Devon & Exeter NHS Foundation TrustSouth Devon Healthcare NHS Trust
40 National Trainee Survey 2006 – key findings
Ta
ble
16
Com
pa
riso
n b
etw
een
aca
dem
y a
nd
non
-aca
dem
y t
rain
ees
on t
he
Na
tion
al T
rain
ee S
urv
ey I
nd
ica
tors
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.00
1.00
1.00
0.04
0.03
1.00
0.02
Acc
ess
to
Ed
ucat
ion
al R
eso
urce
s Sc
ore
Ad
eq
uate
Exp
eri
en
ce S
core
Sco
re
Car
ee
r A
dvi
ce S
core
Bul
lyin
g %
re
po
rtin
g
Ed
ucat
ion
Sup
erv
isio
n S
core
Eur
op
ean
Wo
rkin
g T
ime
Dir
ect
ive
% "
Yes"
Fee
db
ack
Sco
re (
Ad
just
ed
)
Han
dov
er
Sco
re (
Ad
just
ed
)
Ho
ur E
duc
atio
n S
core
Ind
ucti
on
Sco
re
Oth
er
Lear
nin
g O
pp
ort
unit
ies
Sco
re (
Ad
just
ed
)
Ove
rall
Sat
isfa
ctio
n S
core
(A
dju
ste
d)
Sup
erv
isio
n S
core
(A
dju
ste
d)
Wo
rk I
nte
nsi
ty S
core
(A
dju
ste
d)
Wo
rklo
ad S
core
(A
dju
ste
d)
me
an s
core
at
loca
tio
n
P (
wit
hB
onfe
rron
i)
2.90
1.47
0.50
0.28
0.04
0.02
0.76
1.11
30.6
6
0.21
0.01
9.14
9.76
0.11
10.9
2
F
12.3
9
11.7
5
33.7
8
17%
1.11
37%
20.6
1
14.9
3
2.70
0.80
16.3
9
9.90
12.2
0
22.0
9
17.5
2
Sta
n D
ev:
Aca
dem
y
68 68 58 68 68 56 57 23 68 68 67 68 68 68 68N:
Aca
dem
y
83.0
2
74.9
0
85.3
4
3% 3.96
84%
53.7
8
33.0
1
4.75
2.51
51.6
6
82.3
5
88.9
9
88.7
8
64.7
8
Mea
n:
Aca
dem
y
14.1
8
13.0
1
36.4
3
20%
1.15
38%
23.7
6
18.4
1
1.87
0.76
15.2
1
12.6
7
13.5
8
19.4
6
14.4
4
Sta
n D
ev:
non
Aca
dem
y
536
536
403
536
535
444
436
276
535
534
521
536
536
535
534
N:n
onA
cad
emy
79.9
5
76.9
1
81.7
6
4% 3.99
83%
50.9
0
28.8
6
3.34
2.56
51.4
3
77.5
2
83.5
8
89.6
4
58.4
7
Mea
n:
non
Aca
dem
y
Sig
nif
ican
t d
iffe
ren
ces
in m
ean
sco
res,
P <
0.0
5 ar
e s
had
ed
ab
ove.
The Hospital at Night
The Hospital at Night initiative aims to reduce dependency on training gradedoctors for providing cover at night, in order to reduce their working hours andensure that these are compliant with the European Working Time Directive18, whileensuring that there is no negative impact on their training. Hospital at Nightadvocates supervised multi-disciplinary handover in the eveningsxix.
The impact of the Hospital at Night programme was assessed by comparingtrainees working in hospitals with Hospital at Night teams with trainees in otheracute hospitals that do not have these teams. Locations were classified asparticipating in Hospital at Night on the basis of data supplied by the Hospital atNight team19. On the basis of Hospital at Night’s stated objectives, particularindicator scores and items were compared across these two groups of trainees.
National Trainee Survey 2006 – key findings 41
18 http://www.dh.gov.uk/en/Policyandguidance/Humanresourcesandtraining/Workingdifferently/Europeanworkingtimedirective/index.htm.
19 Data kindly supplied by Gerry Bolger, Project Director - National Hospital at Night Team
Table 17 Handover
Hospital at Night respondents should report having multi-disciplinary handovers.These data suggest that this is the case (Table 17 below): 23.7 per cent reportedthat nurses are involved in handovers at Hospital at Night locations compared with11.7 per cent at non-Hospital at Night providers. The difference is more marked ifthe comparison is restricted to medical trainees.
All traineesat acutesites
Medical trainees at acute sites
Surgical trainees at acute sites
P(Bonferroniapplied)
xx2Anorganisedmeeting ofdoctors andnurses %
Anorganisedmeeting ofdoctors %
A phone oremailcommunication%
Informal%
None %
Not implemented N = 11,388
ImplementedN = 4,028
Not implementedN = 3,393
ImplementedN = 1,228
Not implemented N = 2,266
ImplementedN = 725
1.9
1.9
1.2
2.0
1.6
2.2
38.5
28.0
44.4
20.0
35.5
30.1
4.2
3.8
5.2
5.5
5.3
5.5
43.7
42.7
32.0
24.3
49.9
49.8
11.7
23.6
17.1
48.2
7.8
12.4
378.63
507.07
19.33
0.00
0.00
0.00
Implementation status
Which of the following best describes handover arrangements BEFOREnight duty in your post?
42 National Trainee Survey 2006 – key findings
Table 18 Impact on working times
The Hospital at Night programme aims to help trusts comply with the EuropeanWorking Time Directive. The data below do not suggest this, as the difference incompliance is in the opposite direction to that predicted and the differences,although small, are statistically significant for all trainees at acute sites and the sub-group of medical trainees at acute sites. Similarly, there are small statisticallysignificant differences on the second item for all trainees and the sub-group ofmedical trainees; trainees working at Hospital at Night sites are slightly more likelyto have been asked to submit hours that are compliant, when the hours actuallyworked were not compliant..
Obviously this finding is only an association. It may be the case that Hospital atNight providers knew there was a problem in complying with the EuropeanWorking Time Directive and, as a result, have started to tackle it, but the changeshave not yet bedded down sufficiently to impact upon working times. The analysiscan be repeated with the 2007 data to test this hypothesis and would bestrengthened by more detailed data on the implementations of the initiative atHospital at Night providers
The difference between night to day handover between trainees at Hospital at Nightsites compared with trainees at other acute sites was far less marked. The biggestdifference was for the medical trainees; for this group Hospital at Night sites had2.7 per cent more respondents saying their handover involved nurses. As the focusof Hospital at Night is on the day to night handover, these data would seem toreflect changes to handover process related to Hospital at Night implementation.
All trainees at acute sites
Medical trainees at acute sites
Surgical trainees at acute sites
P (Bonferroni)xx2N% YesP (Bonferroni )xx2N
Not implemented
Implemented
Not implemented
Implemented
Not implemented
Implemented
10,834
3,750
3,086
1,132
2,150
668
% Yes
89.0
87.4
88.4
84.9
82.0
81.4
7.160
9.145
0.090
0.045
0.015
1.000
18.9
21.8
19.9
23.6
24.5
24.8
11,716
4,010
3,371
1,210
2,371
725
16.064
7.080
0.023
0.000
0.047
1.000
Implementation status
Are your rostered working hourscompliant with the European WorkingTime Directive?
Have you been asked to submit hoursthat are compliant with the EuropeanWorking Time Directive, when thehours you actually work are NOTcompliant?
National Trainee Survey 2006 – key findings 43
Table 19 Work intensity
The work intensity items reflect a similar pattern, with trainees at Hospital at Nightsites reporting a heavier workload during the night. Again, the differences are smallbut, due to the sample sizes available, they are statistically significant. Thedifferences in workload during the day are less reliable; the difference is onlystatistically significant when all trainees are included in the analysis.
Table 20 Impact on training
The Hospital at Night initiative is not meant to impact on doctors’ training and thesurvey data suggest that, on balance, it does not affect trainees’ perceptions of theexperience they are getting, as there are no consistent statistically significantdifferences between trainees at sites that have or have not implemented Hospital atNight on the Adequate Experience Score (derivation of scores is outlined in Table5, Chapter 2).
All trainees at acute sites
Medical trainees at acute sites
Surgical trainees at acute sites
P (Bonferroniapplied)xx2N
%respondingheavy orvery heavy
P (Bonferroniapplied)xx2N
Not implemented
Implemented
Not implemented
Implemented
Not implemented
Implemented
13,290
4,651
3,829
1,380
2,700
853
% respondingheavy or very heavy
Day Night
37.6
41.9
41.0
44.8
34.4
34.1
28.08
7.99
2.70
0.00
0.55
1.00
39.1
44.8
47.0
52.7
25.1
31.2
12,132
4,163
3,460
1,219
2,597
804
49.05
15.65
15.37
0.00
0.02
0.02
How would you rate the intensity of your work by:
All traineesat acute sites
Medicaltrainees atacute sites
Surgicaltrainees atacute sites
P (Bonferroni applied)FNSD
Not implemented
Implemented
Not implemented
Implemented
Not implemented
Implemented
16.06
16.63
15.32
16.27
19.30
19.90
Mean
73.54
72.99
72.58
72.73
70.35
68.49
13,330
4,657
3,842
1,384
2,702
853
4.02
0.09
5.95
0.14
1
0.04
Adequate Experience Score
Chi-squared test run across all five response options. Presented as % heavy/very heavy in this table for ease interpretation only
44 National Trainee Survey 2006 – key findings
Deaneries, training provider and differences in OverallSatisfaction and Supervision – a multilevel model analysis
These data are inherently multilevel: trainees work within departments, which arewithin hospitals that fall within a given deanery’s responsibility. PMETBconsequently commissioned multilevel modellingx to analyse these data at differentlevelsxi. This analysis looked for differences between deaneries, training providersand specialty groups within training providers on two of the indicator scores:Overall Satisfaction and Supervision.
The model included a number of background variables to statistically control fortheir effects, so that differences between providers cannot be attributed todifferences between them on these background variables.
The model has four nested hierarchical levels that reflect the organisation ofpostgraduate medical education: trainees; within specialty groups within providers(e.g. Medicine at a given provider); within providers; within deaneries. Significantvariation was found on the Overall Satisfaction Score and the Supervision Score atthe level of specialties within providers (P < 0.001 and P < 0.001) and providers (P= 0.001 and P = 0.005). There is no difference on either measure at the level ofdeaneries. The differences do not reflect differences in trainee mix in terms of thebackground variables: specialty, sex, grade, type in post, years qualified, or route ofresponding to the questionnaire.
While these results show that there are differences in trainees’ perceptions of thequality of their training associated with training providers and specialties withintraining providers, it is not possible to use these data alone to reliably identifypoorly performing training providers. This suggests that, while the survey data mayhave utility as a screening tool as proposed by PMETB this year, it would beinappropriate to take action or identify poorly performing training providers on thebasis of these data alone.
National Trainee Survey 2006 – key findings 45
4. The survey data as a quality management tool
As the primary objective of the survey work has been quality improvement, PMETBhas focused on releasing data to deaneries, which are responsible for local qualitymanagement. Deaneries have been working with training providers within theirarea on action plans in response to the survey’s findings. Table 21 details the datareleased to deaneries that was used for this work.
Data releases
Table 21 Data that PMETB has released
Data Release data and format Audience
Data by specialty group andprovider
PMETB/COPMeD NationalScreening Tool with indicatorscores for each specialty groupfor all providers in the UK withmore than three respondentsderived from the survey data.
Specialty group data byprovider benchmarked tonational means and quartiles
The indicator scores (used on theCompare CD above) withcomparison with the nationalmeans and quartiles for the givenspecialty group; these means andquartiles use the data from allrespondents, including thosewhere there are less than threerespondents.
5 December 2006
Compare CD –
Compare software allows the user toexplore these data and producereports as Word documents for agiven location and specialty group.The user can choose which otherlocations/VTSs to compare theirchosen location/VTS with on thecharts.
7 December 2006
Screening Tool Access Databasethat allows the user to obtain areport for a group of indicatorswhere scores for all availablespecialty groups are displayedfor each location.
Deaneries
Deaneries
Specialty data
Indicator scores aggregated tospecialty and grade groups bydeanery.
This dataset contains indicatorscores for each specialty andgrade group in each deanery,where there are more than threerespondents available, e.g.scores for cardiology SpRs inLondon Deanery. This report willtherefore include specialtiessuch as public health andoccupational medicine that, dueto low numbers at individualproviders, have not beenincluded in data by specialtygroup and location.
11 January 2007
Compare CD
Royal Colleges
46 National Trainee Survey 2006 – key findings
Data Release data and format Audience
Specialty data – UK wide
Indicator scores aggregated tospecialty and grade for thewhole of the UK.
This dataset contains indicatorscores for each specialty andgrade group across the UK,where there are more than threerespondents available, e.g.scores for cardiology SpRs UKwide.
Provider profile report
A summary of each provideracross all their specialty groups.It identifies the providers withthe highest proportion of outlierindicator scores (both aboveand below the mean) across allthe specialty groups.
11 January 2007
Compare CD
1 February
Excel
Royal Colleges
Deaneries
Deanery level analysis oftraining since leavingmedical school
An analysis based on responseto item G4 by deanery.
21 December 2006
Excel
Deaneries
NHS Institute forInnovation andImprovement
Intended specialty analysis 18 December 2006
Excel spreadsheet available here:http://www.pmetb.org.uk/index.php?id=intendedspecialtyanalysis
Trainees
Deaneries have been using the data released to them for their own qualitymanagement activities. PMETB has not been prescriptive about how deaneriesshould go about this and generally deaneries have discussed and disseminated thedata locally as they deem appropriate. It is important for deaneries to undertakethis activity, as a national organisation such as PMETB would not be able to interpretthe data within the context of the NHS locally.When considering the reason for alow score on an Indicator Score derived from survey data, it is advisable to excludecauses such as particular local circumstances (for instance a change to theconfiguration of services that has been disruptive) before suggesting the score isindicative of a performance issue. Only deaneries would be able to take suchcauses into consideration when planning follow up on the survey data.
National Trainee Survey 2006 – key findings 47
MerseyEasternWest MidlandsNorth WesternOxfordYorkshireEasternScotland (West)Scotland (West)OxfordLondonScotland (West)
ClassificationUpper 95%Lower 95%N
123456789101112
192312121239607101675
MeanDeanery Provider
69.9067.3067.0069.0671.6574.4474.5068.3371.0570.8570.6266.01
60.2859.5356.0661.1665.7869.5569.9655.7857.3561.1960.2251.33
79.5375.0777.9376.9777.5379.3279.0480.8884.7680.5281.0280.70
Outlier
Below mean and 2nd
quartile Not outlier
Bottom quartile Not below mean Not outlier
Table 22 Surgical providers in the bottom quartile and/or below the nationalmean based on the confidence intervals
20 Adjusted for method variance – see Chapter 2
The identification of outliers for follow up
PMETB only intends the survey data to be used as an initial screening tool, toidentify areas that may require further investigation. For the purposes of identifyingoutliers, the following analysis was conducted.
The survey data were aggregated to the level of specialty groups within providers,so for example there were mean adjusted20 scale scores for surgical trainees at agiven provider. To ensure any given trainee’s responses remained anonymous aspromised, scores were only calculated where there were more than threerespondents working in the specialty group at the given provider. This is datarelease, data by specialty group and provider, in Table 21.
The mean scores for each provider’s specialty group of trainees were comparedwith the national mean and quartiles for that specialty group (as per specialtygroup data by provider benchmarked to national means and quartiles in Table 21).The national comparison group included all trainees within that specialty group,including those working at providers with less than three respondents. So, forinstance, the mean score for the given provider’s surgical trainees was comparedwith the national mean and quartiles for all surgical trainee respondents across theUK. A score was defined as outlying if it was in the bottom quartile and below thenational mean (based on the confidence intervals not overlapping). The advantageof including data from the confidence intervals in the comparison is that, for asample from a particular provider, the confidences are narrower if there are morerespondents and if there is more agreement among those who have responded. Justusing the quartile information takes no account of sampling error and just using acomparison of the means meant that it was possible for providers not in the lowestquartile to be deemed outliers because they had very narrow confidence intervals.Therefore using both pieces of information was felt to be more appropriate.
An example of identifying outliers – surgical specialty group
For the all the surgical group trainees in the UK (N = 4,022), the mean SupervisionScore is 80.58 (95 per cent confidence intervals 80.11 to 81.06). The 25th percentileis 71.09. The 209 providers for whom there were four or more respondents werethen compared with these figures. Table 22 shows which providers were classifiedas outliers because they met both criteria, and the providers that only met one ofthe two criteria and were therefore not classified as outliers. Providers 1 to 4 areoutliers; providers 5 to 13 are not.
48 National Trainee Survey 2006 – key findings
The proportion of outlying indicators across all available specialties was calculatedfor each provider. Of the providers in this analysis, 34 per cent (120/355) had noindicator scores that were outliers (taken from the Provider profile report). Only 15per cent (52/355) had 10 per cent or more of indicator scores across all theirspecialties with respondents that were outliers.
Deanery quality improvement
All deaneries were asked to provide an action plan in response to the deaneryspecific data provided from the National Trainee Survey. The deaneries were givena proforma with headings that are standard in most action plans (see Appendix 2for a blank proforma at www.pmetb.org.uk/traineesurvey).
The deaneries were given three months to return an action plan to PMETB21. Alldeaneries achieved a response, many with a full and considered action plan. InScotland, NHS National Education for Scotland (NES) provided a Scotland wideresponse and then, in two cases, a deanery specific overview. The responses werevariable in style and content. This was expected at this early stage and with thechallenges of effective dissemination from PMETB to the deaneries, and from thedeaneries to the education providers (e.g. NHS trusts and boards).
The data received by deaneries were affected by the filter to keep trainees’identities anonymous, in that specialty groups at providers for which less than fourresponses were received were not included in the report. For some specialtygroups at some providers, deaneries have noted that more information is requiredto link the trainees’ responses to a specific clinical environment/department andthus write a detailed action plan. PMETB will explore whether some specialties (asopposed to specialty groups) can be reported for some larger providers instead ofjust across the deanery. However, it will not be possible to report on something suchas a department within a provider that is not defined by a provider listed by theNational Administrative Codes Service and a specialty or sub-specialty.
Equally, deaneries need to adapt and plan their responses. Several of the deaneriesdecided to document their actions in order to create a full action plan which theywill send to PMETB at a later point – an ‘action plan for action plans’.
All deaneries used the outlier identification detailed above as the rationale forinclusion of a provider within an action point. The action plans were very clear onthe levels for responsibility and generally the headings do not appear to havecaused problems. Some of the deaneries were notable for their clarity of bothresponsibility and timescales.Virtually all deaneries evidenced a clear grasp ofpriorities and risk management, and there was significant consistency in theidentification of high risk and appropriate timescales. One deanery used a threelevel approach, with urgent action within one month as the first level, the secondlevel for action within three months and the remainder as developmental changes.
Several of the deaneries documented their concerns about disseminating the datain a useful way to their local education providers and PMETB will explore thisfurther when consulting on reporting (see Chapter 5). Half of the deaneries felt ableto be specific in identifying the education providers concerned. Several of thedeaneries referred very helpfully to previous quality management activity and/orthat the data matched their experience of the quality of education for thoseeducation providers. Occasionally, the deaneries have questioned a mismatchbetween their own and the education provider’s experiences and the outcomes inspecific areas such as handover. In addition to this retrospective analysis, severaldeaneries noted future activity which could occur where there was evidence of
21 Proforma sent out 7 December 2006 with specialty group data by provider benchmarked to national means andquartiles (see table 21)
National Trainee Survey 2006 – key findings 49
continuing problems. One deanery noted that the next step could be initiation of atriggered PMETB visit for one education provider.
The utility of specialty group based data and identification of specialty groupsthemselves was also interesting and will be considered in future development ofthe surveys. Some (less than 50 per cent) of the action plans identified the specialtygroup clearly and specifically. Other action plans did not identify the providerspecialty group clearly, concentrating on the issue and the education provider(s).Several linked specialty group, the issue highlighted by the indicator score and theprovider; this approach makes the action plan both understandable and easier tofollow up. Once action planning is embedded in the deanery systems, theinvolvement of specialties and schools will become the norm. One deanery hadvery clear identification of the role of the schools and Specialist TrainingCommittee chairs within the action plan; this was one of the more successful inintegrating issue, specialty and provider.
Deaneries will use the action plans for several purposes, including deanery qualitymanagement, but also to provide the information and reassurance that PMETB willneed that Generic standards for training are being attained and maintained. Formneeds to follow function; it may be that action plans at a higher level and withbroader scope will meet regulatory and performance management issues, while amore detailed approach is needed at quality management (deanery) and qualitycontrol (provider) levels. A lack of detail in many of the action plans means thatexternal readers cannot assess accuracy, the relevance of issues or appropriatenessof actions taken; neither can they identify if the actions have been undertaken.
Many of the deaneries and NES confirmed their intention to improve both theresponse rate and the accuracy of their population data (an issue discussed inChapter 5).
50 National Trainee Survey 2006 – key findings
5. The future of the National Trainee Survey
The National Trainee Survey of 2006 was the first national survey conducted byPMETB and COPMeD and there were significant challenges in its delivery and anumber of important lessons learnt. The high response rate and the debate anddiscussion that followed the release of information so far indicates that there is asignificant role for this survey in the development and quality assurance ofpostgraduate medical education in the future.
A number of changes are planned for the 2007 survey; these are described belowand are based on PMETB’s experience of the 2006 survey and feedback from thedeanery staff that kindly helped with the work.
Route of administration
To eliminate the method variance problem outlined in Chapter 2, where trainees’responses to some items varied by the route by which the survey was administered,the 2007 survey will only be administered using one route: a national website.Webadministration of the survey has a number of benefits over the other methods usedfor the 2006 survey. These are:
1. It is more economical than scanning paper forms and the portable electronicsurvey units.
2. It allows the easy administration of specialty specific versions of thequestionnaire as items can be conditionally displayed based on the responses toearlier items.
3. The system can produce a reference number so that doctors can prove they havecompleted the survey, thereby providing deaneries with the data to enforce themandatory status of the future trainee surveys.
4. A web based survey will dramatically reduce local administrative burden. Thetask managed locally will be the collection of valid email addresses (which willhave considerable collateral benefit).
It is known that trainees may alter their responses when they feel these could beidentified (although assurances were given that the data were confidential andreporting would ensure that responses remained anonymous); this might reduce thetotal variance available for analysis from some items. However, this will notinvalidate comparison between training providers, as it may be assumed that anymeasure of it is not associated with any particular provider since all are using thesame method of administration and impression management22 is an individual-levelvariable. There may be inter-specialty differences in impression management but,as benchmarking is within specialty group, any such differences will not bias theanalysis.
The content of the survey
Changes to the content of the survey are constrained by two factors:
1. The data will be used to test for changes over time; therefore the items used toderive the indicator scores must remain the same.
2. Shorter questionnaires are known to elicit a higher response ratexx and thus thequestionnaire should not be substantially lengthened.
Within these constraints a number of changes are proposed for the 2007 work.
22 Impression management: the extent to which an individual answers items to give a more positive impression ofthemselves.
National Trainee Survey 2006 – key findings 51
Adequate experience
The sine qua non of any post is to provide the trainee with the experience s/heneeds to acquire the competences set out in the curriculum that s/he is following.Currently, this standard is addressed through two summative evaluation items onwhich trainees are asked to rate the quality of the experience overall. It has notbeen possible to demonstrate the construct validity of these items; it cannot beshown using factor analysis that the items capture something different from overallsatisfaction with the post – a failure of discriminant validity. This means that thescore is not adding substantial information to that which is already contained in theOverall Satisfaction Score. Furthermore, it does not provide diagnostic informationon which aspects of the experience offered by the post were found to beinadequate. As all trainees received the same items in 2006, it would not have beenpossible to have items that related to facets of training experience, as these arespecialty specific. However, in 2007, it will be possible to display itemsconditionally, based on the respondent’s earlier answers. It will therefore bepossible to ask about particular facets of experience in relation to both the trainee’sspecialty and stage of training. Item development for these speciality-specificadequate experience items will be done in conjunction with lead deans and theRoyal Colleges and faculties. Items will be related to the PMETB approvedcurricula. The challenge will be to devise items that will provide sufficient datavariability to distinguish between posts providing experience that traineesperceive as ‘good’ and posts that are not perceived as ‘good’. Posts are part of aprogramme that is designed to provide the necessary experience overall, and thusit may be necessary to ask whether the post provided the experience the traineeexpected, given the design of the overall programme.
Overall Satisfaction Score
There is a ceiling effect on the Overall Satisfaction Score, which means it is notpossible to distinguish between posts at the top end of the score. All doctors whoare satisfied with their posts are alike as far as the survey data is concerned, 12.4per cent (N = 24,848) of respondents having the maximum possible score on thisscale. This measurement issue will be addressed in the 2007 survey with theaddition of items written with a view to making distinctions possible at the positiveend of the overall satisfaction score.
Follow up of concerns
The survey currently promises confidentiality for respondents, but this promiseconflicts with some respondents’ expressed desire for action to be taken inresponse to concerns they have raised in free text comments. Therefore, the 2007survey will allow respondents to indicate that they would be happy to be identifiedin the course of any follow-up action in response to their concerns. This approach istaken by the Training and Development Agency for Schools in their newly qualifiedteacher survey, which also examines the perceived quality of training receivedxxi.
Personality variables
From the occupational psychology literature it is known that a respondent’spersonality accounts for some of the variance in their self-report of constructs suchas job satisfactionxxii. Incorporating a personality measure in the model can improvethe prediction of turnoverxxiii; it might therefore be expected that incorporating apersonality measure would improve the trainee survey’s predictive validity (as the
52 National Trainee Survey 2006 – key findings
effects of the personality measure can be statistically adjusted). Negative ratingsfrom trainees who are generally positive but negative about their current trainingpost are more likely to relate to problems with the posts than negative ratings fromtrainees who are generally negative. The 2007 survey will therefore include ashortened measure of positive and negative affect to allow for statistical adjustmentof personality factors when using the Overall Satisfaction Score to benchmarktraining providersxxiv.
Responses varied by route of administration. For some items, more positiveresponses were more likely with less anonymous routes of survey administrationand thus it would seem appropriate to include some items to measure the extent towhich trainees are considering the impression of themselves that they arepresenting when answering. This measure of impression management can then betested for associations with items used in the indicator scores and, if necessary,statistically controlled for. Merrill et alxxv have developed a scale designed tomeasure impression management that is domain specific for medicine, which couldbe used by the 2007 survey.
Dissemination of the results
The time from data collection to reporting of the 2006 data was longer than it shouldhave been. Reporting was impeded by the following factors, all of which will not bepresent for the 2007 work:
1. Four routes of administration, each with variance from the master data template,had to be merged to create one data set.
2. Indicator scores had to be adjusted for method variance to allow comparisonacross training providers because the provider was confounded by the route ofadministration.
3. Some respondents’ data had to be remapped to an alternative location. Forinstance, in one English Postgraduate Medical Education Centre, GPRs hadreceived paper forms with the acute trust identifier.
It should therefore be possible to report the findings back to the deaneries andother stakeholders in a more timely fashion. A variety of data aggregations areappropriate, given the local systems in place for quality managing postgraduatemedical education; these are described below in Table 23. Each aggregationreports on the same indicator scores. Aggregations are filtered, in that onlyproviders or rotations with more than three respondents are reported to ensure thatindividual trainees’ responses remain anonymous.
Table 23 Aggregations for reporting
Audience Aggregation
Deans and Directors of Medical Education Training provider and specialty group, for example anaesthetics trainees at St Elsewhere NHS Trust
Deans and Specialist Training Committees Deanery Specialty Grade, for example London Cardiology SpR trainees
National Trainee Survey 2006 – key findings 53
The Deanery Specialty aggregation ensures that the responses from trainees in thesmaller specialties are reported on, as frequently there would not be enoughresponses to include this in the Training Provider and Specialty Group aggregation.For example, with 2006 data, occupational medicine was reported by DeanerySpecialty Grade aggregation only and not by location, as there are not enoughtrainees at a given location to get above the reporting filter.
As the focus of the work is on quality management, reporting back to those withaccountability for the training and seeking their responses to the data shouldalways be the priority. However, other parties, not least the trainees who completedthe survey, have a right to see the data for the locations of interest to them. Tofacilitate this, PMETB plans to report local findings through a web based reportingservice. This will supplement or replace (depending on whether all the requiredfunctionality can be obtained) the existing method which uses a CD-ROM and theCompare software provided by the Healthcare Commission. PMETB will consult onthe reporting options for the 2007 survey as part of the quality assurance (QA)framework consultation to establish:
● that the aggregations are at a level of granularity that allows data to be linkedback to a clinical area for which an action plan can be written; constraints includethe identification of the entity to aggregate to and the need to ensure that no onetrainee’s responses can be identified in the report;
● that the reporting tool allows deaneries to disseminate the findings to theirproviders with their local contextual information included;
● which aggregations are most important and should be reported on first;
● other desirable aggregations such as hospital based VTS GP trainees only;
● benchmarking groups – for each aggregation the units can be grouped indifferent ways for comparison purposes, for instance one could comparecardiology trainees’ experience with the experience of all other medicalspecialties, or just the trainees in the acute medical specialties.
When considering proposals for reporting it is important to remember that PMETBhas made the following guarantees:
● all survey data are confidential and it will not be possible for deanery staff toidentify the responses of an individual trainee doctor;
● data will be reported in such a way that responses remain anonymous and cannotbe identified due to small numbers in any particular group.
Population data
The 2006 survey’s population comprised all trainees in approved training posts.PMETB was dependent on deaneries to identify these doctors, as there is presentlyno single, centrally maintained UK database of trainees in approved posts.Headcount of doctors in approved posts by specialty, provider and grade was notavailable at the start of the data collection period due to time pressure, whichmeant that response rates could not be calculated immediately. In addition, therewere issues around how posts are defined as approved, since there are posts thatare deanery funded and educationally approved, posts that are locally funded andeducationally approved, and posts that are service only. Information on these postswas not generally held in one place; for instance, deaneries hold SpR data sincethey issue National Training Numbers, but deaneries only hold whole timeequivalent data for SHO posts that are deanery funded. Data quality issues wereapparent; for instance, over 200 of the same SpRs were listed on two differentdeaneries’ databases.
54 National Trainee Survey 2006 – key findings
Both PMETB and the deaneries are committed to improving these data for the 2007survey. PMETB has been consulting with deaneries’ business managers over howbest to obtain data on the population of trainees in approved posts, and will beissuing their data request well in advance of the 2007 survey. At a national and UKlevel, PMETB is working with representative groups to discuss the development of aminimum data set, of which these data will be part. These proposals will beincluded in the consultation.
Deaneries will know how many specialty trainee posts there are, as these werecounted as part of the preparation for the implementation of Modernising MedicalCareers (MMC)23. However, the national counting of fixed-term specialty trainingappointment (FTSTA) posts may be an issue as, according to the forthcoming Aguide to specialty training: the gold guidexxvi, there is no mandatory requirement fordeaneries to record these posts and currently no specified data template on whichto do so. PMETB will consult with deaneries to establish an agreed template forrecording details of these posts as part of the QA framework.
In England, the Department of Health conducts an annual census of medical staffxxvii.It uses pay scales to classify doctors by grade and does not collect data on whetherthe doctor is in an educationally approved post. Trainees are currently classified as:
● registrar group, which refers to the combined grouping of specialist registrars,senior registrars and registrars and other staff working at equivalent grades thatare not in an educationally approved post;
● doctors in training and equivalents (previously known as junior doctors): registrargroup, senior house officer, foundation doctors, house officers and other staffworking at equivalent grades that are not in an educationally approved post.
PMETB has requested that the Information Centre for Health and Social Care inEngland include a field on the census to indicate whether the doctor is in aneducationally approved post or not. PMETB will also establish whether the otherthree nations’ departments conduct similar censuses of medical staff.
The survey as screening tool
The utility of the survey as a screening tool is determined by its predictive validity -whether it can identify posts or programmes that may require remedial action orhighlight areas of good practice. The forthcoming consultation on the QAframework will identify a criterion score that could be used to measure the survey’spredictive validity. A measuring instrument may have to be developed; scores fromthis can be used as the dependent variable, with the indicator scores from thesurvey as the independent variables. This will establish whether trainees’perceptions of a provider correlate with the criterion score. To avoid rangerestriction, the criterion score will have to be collected from providers at all levelsof the trainees’ ratings, some from the bottom quartile and some in the middle, andso on for the duration of the exercise. For the purposes of the validation exercise,anyone involved in obtaining the criterion score will need to be blinded to thesurvey data.
The exercise could establish which indicator scores are most effective and adjustthe identification of outliers to ensure that the survey is more sensitive than specific,as it may be better to have false positives than for PMETB or the deaneries to fail tofollow up a problem.
23 See: http://www.mmc.nhs.uk/pages/home
National Trainee Survey 2006 – key findings 55
Making the survey mandatory
Feedback from deanery and postgraduate centre staff indicates that making thesurvey mandatory could make administration far easier. Due to the different routesof administration employed, it would not have been possible to enforce this for the2006 survey. The forthcoming A guide to specialty training: the gold guide and theGeneral Medical Council’s Good medical practicexxviii both indicate that doctors arerequired to take part in systems of quality assurance and quality improvement. Thesurvey forms a key part of the quality assurance of doctors’ training programmesand they will therefore be expected to complete a return for future surveys. As thesurvey will be exclusively administered through a website, which will providereference numbers for successfully completed submissions, doctors will be able toshow that they have completed the survey. In addition, PMETB will be able to listwho has not completed a return and supply this information to the deaneries andcentres for follow-up action. Together, this will provide a mechanism for deaneriesto encourage a high response rate.
Recognising that PMETB is committed to the principles of better regulation and sohas a duty to minimise the burden of its data collection activities, PMETB will ensurethat doctors’ participation in the work is as easy as possible and will thereforeprovide support throughout the survey period to ensure that doctors are able tocomplete a return. In addition, a full pilot of the process will be conducted in onedeanery to spot any process issues prior to national roll-out.
Comparison with 2006 data
Once data from two years are available it will be possible to perform analysislooking at the relationship between the 2006 and the 2007 indicator scores forparticular aggregations; for instance, for a given specialty group, do the samelocations come up as low scoring? In addition, the 2006 data provide a baselineprior to the implementation of MMC.
National use of the data by Royal Colleges
For the 2006 survey, PMETB supplied Royal Colleges with national levelaggregations for each specialty. These data will be made available using the 2007data too, but for 2007 these data will be of more interest as they will include thespecialty specific items that the colleges will have helped to develop.
The 2007 survey and other data sources
The survey data can be used to test other hypotheses. For example, if it is knownthat some trusts are participating in a quality improvement activity and otherswithin the deanery are not, a comparison of the relevant indicator score can bemade between the two groups. Analyses such as these can be performed, providingdeaneries submit the grouping data available and it is possible to specify whichindicator scores should vary by group to avoid multiple testing.
During the 2007 trainee survey, a survey of trainers will be conducted concurrently.The aggregated data from these two surveys will be linked using specialty andlocation. This will allow comparison of the trainees’ and trainers’ perspectives andthe potential development of discrepancy indicators; the greater the differencebetween the two perspectives, the greater the potential problem. It is likely thattrainees’ perceptions will not be as positive as those of trainers. Baker andSpracklingxxix compared the two perspectives (consultants and their VTS SHOs) and
56 National Trainee Survey 2006 – key findings
found that consultants reported more positive perceptions. For instance, 32 per centof the trainees in their sample indicated that teaching took place in protected time,whereas the figure for consultants was 67 per cent.
Deanery action plans
To ensure that survey data are always used for quality improvement work, actionplans should be written for an external audience so that trainees can see that theirparticipation in the survey is a worthwhile activity, and that patients can bereassured that deaneries are effective in ensuring the safe training of doctors.PMETB will also be seeking evidence of trainee involvement in the agreement andmeasurement of the action plans.
Indicative plan for the 2007 survey
Table 24 shows key external dates for the trainee in the 2007 survey plan. PMETBrecognises that deaneries need sufficient notice that data will be released to allowthem to schedule resources for onward dissemination of the data to their providersand the preparation of action plans in conjunction with the providers. PMETB willprovide more detailed plans nearer the time.
Feedback form
If you have comments on the 2006 work or the plans for the 2007 work, please go towww.pmetb.org.uk/traineesurvey and complete the feedback form.
Milestone
Development of specialty specific items in conjunctionwith lead deans and Royal Colleges/faculties.
Population request to deaneries for completion. This willconsist of list of approved posts detailing the specialty,grade and location of each.
Population request returned to deaneries for them tosupply the email addresses of the posts’ currentincumbents. A draft version of this request is available atwww.pmetb.org.uk/traineesurvey.
Full pilot of trainee questionnaire in one deanery.
Staged launch of National Trainee Survey.
National Trainee Survey reports (all aggregations, seeTable 22) made available to deaneries, Royal Colleges,faculties and specialist associations.
Date
April – July 2007
June 2007
August 2007
October 2007
December 2007– January 2008
March 2008
Table 24 2007 survey plan
National Trainee Survey 2006 – key findings 57
References
i PMETB. Generic standards for training (2006). Available at:http://www.pmetb.org.uk/fileadmin/user/Policy/Policy_Statements/Generic_standards_for_training_April_06.pdf
ii Grant, J, Maxted M, Owen H,Wooding K, Black C, Neville E, Gale R.Development Of A National Generic Data Collection System To SupportInspection Of Hospital Visiting (2004). Available at:http://www.pmetb.org.uk/index.php?id=backgroundinfo#c977
iii Paice E, Aitken M, Cowan G and Heard S. (2000). Trainee satisfaction beforeand after the Calman reforms of specialist training: questionnaire survey.British Medical Journal;320: 832-836.
iv Leach C and Goodman J. (2006). PMETB Questionnaire Assessment Final Report.Available at: http://www.pmetb.org.uk/index.php?id=backgroundinfo#c977
v Baldwin D C Jr and Daugherty S R. Sleep deprivation and fatigue in residencytraining: results of a national survey of first- and second-year residents. Sleep.2004 Mar 15;27(2):217-23.
vi Healthcare Commission (2006). Working in partnership: Getting the best frominspection, audit, review and regulation of health and social care.Available here: http://www.concordat.org.uk/_db/_documents/Concordat_-_version_(May_06).pdf
vii Bland M J, Altman D. (1995). Multiple significance tests: the Bonferroni Method.British Medical Journal. 1995;310;170.
viii Nunally J C. (1978). Psychometric theory (2nd edn). New York: McGraw-Hill.
ix Cronbach L J and Meehl P E. (1955). Construct Validity In Psychological Tests.Psychological Bulletin, 52, 281-302.
x Cohen J and Cohen P. (1983). Applied Multiple Regression/Correlation Analysisfor the Behavioral Sciences. (2nd ed.). Lawrence Erlbaum Assoc.
xi Goldstein H. (1995). Multilevel statistical models. 2 ed. London: Arnold.
xii McManus C. (2007). Multilevel modelling of PMETB data on trainee satisfactionand supervision. Report produced for PMETB available at:http://www.pmetb.org.uk/index.php?id=2006results#c978
xiii Hicks B. (2000). Time to stop bullying and intimidation. HospitalMedicine;61:428-31.
xiv Paice E, Aitken M, Houghton A, Firth-Cozens J. (2004). Bullying among doctorsin training: cross sectional questionnaire survey. British MedicalJournal;329(7467):658-9.
xv Altman D, Machin D, Bryant T N and Gardner M J. (2000). Statistics WithConfidence: Confidence Intervals and Statistical Guidelines. London: BMJ Books
xvi Frank E, Carrera J S, Stratton T, Bickel J and Nora L M. Experiences ofbelittlement and harassment and their correlates among medical students inthe United States: longitudinal survey. British Medical Journal;333;682-8.
xvii For details see: http://www.riti.org.uk/index5.html
58 National Trainee Survey 2006 – key findings
xviii For details see:http://www.healthcareworkforce.nhs.uk/index.php?option=com_d4j_ezine&Itemid=332
xix Mahon A, Harris C, Tyrer J, Carr S, Lowson K, Carr L, Chaplin S and Wright D.(2005). The implementation and impact of Hospital at Night pilot projects: Anevaluation report. Department of Health. Available at:http://www.wise.nhs.uk/sites/workforce/usingstaffskillseffectively/Hospital%20at%20Night%20Document%20Library/1/Hosp%20At%20Night%20Final%20Report.pdf
xx Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S,Wentz R, Kwan I.(2002). Increasing response rates to postal questionnaires: systematic review.British Medical Journal;324;1183-1191.
xxi Training and Development Agency for Schools (2006). Results of the newlyqualified teacher survey 2006. Available here:http://www.tda.gov.uk/upload/resources/pdf/n/nqt%20survey%202006%20-%20sector%20level%20report.pdf
xxii Spector P E. (1994). Using self-report questionnaires in OB research: acomment on the use of a controversial method. Journal of OrganizationalBehaviour, 15, 385-392.
xxiii Judge T A. (1993). Does Affective Disposition Moderate the RelationshipBetween Job Satisfaction and Voluntary Turnover. Journal of AppliedPsychology, 78, 3, 395-401.
xxiv Mackinnon A, Jorm A F, Christensen H, Korten A E, Jacomb P A and Rodgers N.(1999). A short form of the Positive and Negative Affect Schedule: evaluation offactorial validity and invariance across demographic variables in acommunity sample. Personality and Individual Differences, 27, 405-416.
xxv Merrill J M, Laux L F, Lorimor R J, Thornby J I,Vallbona C. (1995). Measuringsocial desirability among senior medical students. Psychological Reports. 1995Dec;77(3 Pt 1):859-64.
xxvi Department of Health (In press 2007). A guide to specialty training: the goldguide.
xxvii The Information Centre (April 2006). Hospital and Community Health ServicesMedical and Dental staff in England: 1995-2005. Available here:http://www.ic.nhs.uk/pubs/nhsstaff/medicaldentalintro/file
xxviii General Medical Council (2006). Good Medical Practice. Available here:http://www.gmc-uk.org/guidance/good_medical_practice/GMC_GMP.pdf
xxix Baker M and Sprackling P D. (1994). The educational component of seniorhouse officer posts: differences in the perceptions of consultants and juniordoctors. Postgraduate Medical Journal. 1994 Mar;70(821):198-202.
Postgraduate Medical Educationand Training Board
Hercules HouseHercules RoadLondon SE1 7DU
Tel +44 (0)20 7160 6100
Fax +44 (0)20 7160 6102
www.pmetb.org.uk
© PMETB 2007
ISBN: 978-0-9555910-0-6
Related Documents
