Roland - Evaluating Education

Evaluating

Education

This talk may save

Your life

This talk may save

you strifeYour life

Evaluating

Educationbit.ly/evalref

Education is rarely described as an intervention

Reaction

Reaction

Learning

Reaction

Learning

Behaviour

Reaction

Learning

Behaviour

Results

bit.ly/evalref

Did a specific outcome occur?

What were the outcomes of this intervention?

Statistical Conclusion Validity

Internal Validity

Construct Validity

External Validity

The four criterion of methodological quality according to Shadish, Cook and Campbell

bit.ly/mededvalidity

Time

Change)

Intervention

Time

Change

Intervention

Population

Population

Population

Population(e.g Under or

Postgraduates)

• Sample Size• Power Calculation• Criterion for

Statistical Significance Used

Tim

Change)

Intervention

Statistical Conclusion Validity is the extent to whether the statistical analysis can correctly identify a change if it is present

Population• Was there a risk of a

type II error?

Population• Were all outcomes

correctly powered?

Population

Time (Which can be hours e,g pre and post test or up to years)

Change (Which can be measured in a variety of ways)

Intervention

Internal Validity is the extent to which the intervention can reliably be ascribed to have effected the change

Population

Population

Population

bit.ly/mededvalidity

Threats to Internal Validity are best analysed by considering an intervention

and control group

• HistoryAnother event happens at

the same time as the intervention

• SelectionThere are pre-existing differences

between the groups• Regression to the Mean

Extreme pre-test baseline performance may mean change happens by chance (e.g. unusually good or bad at outset)

• MaturationThere is already a pre-existing trend for

change

• InstrumentationThere are changes in the methods of measurement

(e.g data collectors become more experienced)

• TestingPre-testing changes the performance of

participants (e.g. promotes information seeking)

• Differential AttritionDrop out from the study is not equal

between groups

Intervention GroupControl Group

Time (Which can be hours e,g pre and post test or up to years)

Change Y (Which can be measured in a variety of ways)

Intervention A Population

Construct Validity relates to the association between the concept you are investigating and the measures you use to test it

Time (Which can be hours e,g pre and post test or up to years)

Change (Which can be measured in a variety of ways)

Intervention Population A

External Validity is the extent to which the intervention is reliable in different populations (this may in terms of participants, settings, times etc)

Time (Which can be hours e,g pre and post test or up to years)

Change (Which can be measured in a variety of ways)

Intervention

Population B

Statistical Conclusion Validity

Internal Validity

Construct Validity

External Validity

InteractionInteraction

InteractionInteraction

Inte

rfac

eIn

terf

ace

InteractionInteraction

Instruction

(The educational

Intervention)Inte

rfac

eIn

terf

ace

InteractionInteraction

Instruction

(The educational

Intervention)

Ideation

Ideation

Inte

rfac

eIn

terf

ace

InteractionInteraction

Instruction

(The educational

Intervention)

Ideation

Integration

i) Knowledgeii) Behaviour

Ideation

Integration

i) Knowledgeii) Behaviour

Inte

rfac

eIn

terf

ace

InteractionInteraction

Instruction

(The educational

Intervention)

Ideation

Integration

i) Knowledgeii) Behaviour

Ideation

Integration

i) Knowledgeii) Behaviour

Impl

emen

tatio

nIm

plem

enta

tion

Inte

rfac

eIn

terf

ace

InteractionInteraction

Instruction

(The educational

Intervention)

Ideation

Integration

i) Knowledgeii) Behaviour

Ideation

Integration

i) Knowledgeii) Behaviour

Impl

emen

tatio

nIm

plem

enta

tion

Inte

rfac

eIn

terf

ace

Impr

ovem

ent

Impr

ovem

ent

InteractionInteraction

Instruction

(The educational

Intervention)

Ideation

Integration

i) Knowledgeii) Behaviour

Ideation

Integration

i) Knowledgeii) Behaviour

Impl

emen

tatio

nIm

plem

enta

tion

Inte

rfac

eIn

terf

ace

Impr

ovem

ent

Impr

ovem

ent

The 7I’s Framework

Thanks

SMACCProf. Tim Coats + Dr. David MathesonDr. Ffion DaviesStaff/Patients at Leicester Royal Infirmary

@damian_roland

Education is rarely described as an intervention