A Taxonomy of Research Design Peter T. Donnan Professor of Epidemiology and Biostatistics Statistics for Health Research.

A Taxonomy of Research Design

Peter T. Donnan

Professor of Epidemiology and Biostatistics

Statistics for Health Statistics for Health ResearchResearch

Objectives of sessionObjectives of session

• Realise importance of designRealise importance of design• Understand difference between Understand difference between

experimental and observational experimental and observational designdesign

• Choose appropriate designChoose appropriate design• Understand main forms of Understand main forms of

experimental designexperimental design• Understand main forms of Understand main forms of

observational designobservational design

• Realise importance of designRealise importance of design• Understand difference between Understand difference between

experimental and observational experimental and observational designdesign

• Choose appropriate designChoose appropriate design• Understand main forms of Understand main forms of

experimental designexperimental design• Understand main forms of Understand main forms of

observational designobservational design

Why is design Why is design important?important?

• Poor design may not answer Poor design may not answer question question

• Choice of design determines the Choice of design determines the type of analysistype of analysis

• Incorrect design leads to waste of Incorrect design leads to waste of resourcesresources

• Poor design is unethical if patient Poor design is unethical if patient exposed to danger for little returnexposed to danger for little return

• Poor design may not answer Poor design may not answer question question

• Choice of design determines the Choice of design determines the type of analysistype of analysis

• Incorrect design leads to waste of Incorrect design leads to waste of resourcesresources

• Poor design is unethical if patient Poor design is unethical if patient exposed to danger for little returnexposed to danger for little return

A Taxonomy of Research A Taxonomy of Research DesignDesign

• ExperimentalExperimentalRandomised Controlled TrialsRandomised Controlled TrialsFactorial designFactorial designCrossoverCrossoverCluster randomisationCluster randomisation

n.b. All are prospective since following experimental units n.b. All are prospective since following experimental units forward in timeforward in time

• ExperimentalExperimentalRandomised Controlled TrialsRandomised Controlled TrialsFactorial designFactorial designCrossoverCrossoverCluster randomisationCluster randomisation

n.b. All are prospective since following experimental units n.b. All are prospective since following experimental units forward in timeforward in time

A Taxonomy of Research A Taxonomy of Research DesignDesign

• Non-randomised - ObservationalNon-randomised - Observational1) Cohort (Prospective 1) Cohort (Prospective

and and Retrospective) Retrospective)2) Case-Control2) Case-Control3) Cross-sectional3) Cross-sectional4) Time series4) Time series

• Non-randomised - ObservationalNon-randomised - Observational1) Cohort (Prospective 1) Cohort (Prospective

and and Retrospective) Retrospective)2) Case-Control2) Case-Control3) Cross-sectional3) Cross-sectional4) Time series4) Time series

RANDOMISED CONTROLLED RANDOMISED CONTROLLED TRIAL (RCT)TRIAL (RCT)

•Gold standard method to Gold standard method to assess assess EfficacyEfficacy of treatment of treatment or demonstrate causal or demonstrate causal relationshiprelationship

•First proper clinical trials First proper clinical trials from 1940s in tuberculosisfrom 1940s in tuberculosis

•Ethical requirements set Ethical requirements set out in Declaration of out in Declaration of Helsinki, 1960Helsinki, 1960


Random allocation to Random allocation to intervention or control so intervention or control so likely balance of all likely balance of all factors affecting outcomefactors affecting outcome

Hence any difference in Hence any difference in outcome ‘caused’ by the outcome ‘caused’ by the interventionintervention

Randomised Controlled TrialRandomised Controlled Trial

RANDOMISED

Eligible subjects

Intervention

Control


•RCT necessary to RCT necessary to demonstrate efficacydemonstrate efficacy

•Cost-effectiveness Cost-effectiveness (ICER)(ICER)

•NICE and SMC NICE and SMC assessment for assessment for recommendation to NHSrecommendation to NHS

Examples of RCTExamples of RCT

•Trials of tamoxifen for treatment of Trials of tamoxifen for treatment of early breast cancerearly breast cancer11

•Reduction in recurrence of 29% Reduction in recurrence of 29%

•Reduction in mortality by 14% in Reduction in mortality by 14% in women with oestrogen receptor women with oestrogen receptor positive cancerspositive cancers

•Trial published recently in Lancet Trial published recently in Lancet on herceptin (trastuzumab)on herceptin (trastuzumab)

1. Lancet 1998; 351: 1451-67.

RANDOMISED FACTORIAL RANDOMISED FACTORIAL DESIGNDESIGN

Patients are randomised Patients are randomised more than oncemore than once

All possible combinations All possible combinations of interventions are of interventions are studiedstudied

Simple 2-way factorial Simple 2-way factorial has 4 combinationshas 4 combinations

Randomised Factorial TrialRandomised Factorial Trial

RANDOMISED

Eligible subjects

Intervention 1

Control 1

RANDOMISED

Intervention 2

Control 2

RANDOMISED

Intervention 2

Control 2

Example factorial designExample factorial design

• Scottish Trial of steroids or Scottish Trial of steroids or acyclovir in Bell’s Palsyacyclovir in Bell’s Palsy

• Recruited 500 patientsRecruited 500 patients

1.1. Steroids / PlaceboSteroids / Placebo

2.2. Steroids / AcyclovirSteroids / Acyclovir

3.3. Acyclovir / PlaceboAcyclovir / Placebo

4.4. Placebo / PlaceboPlacebo / Placebo

Strengths and Weaknesses Strengths and Weaknesses of factorial designof factorial design

• Efficient Efficient – two trials for the price of – two trials for the price of oneone

• Relies heavily on assumption of no Relies heavily on assumption of no interaction between treatments i.e. interaction between treatments i.e. effects of treatments are additiveeffects of treatments are additive

• Results – Sullivan, Swan, Donnan Results – Sullivan, Swan, Donnan et et alal, , Early treatment with Early treatment with prednisolone or acyclovir and prednisolone or acyclovir and recovery in Bell’s palsy. recovery in Bell’s palsy. NEJMNEJM 2007; 2007; 357: 1598-607 357: 1598-607

Crossover designsCrossover designs

•In this design each patient In this design each patient receives ALL treatmentsreceives ALL treatments

•Order of receipt is randomisedOrder of receipt is randomised

•Requires a wash-out period Requires a wash-out period between treatmentsbetween treatments

•Only applicable to transient Only applicable to transient effects in chronic conditions e.g. effects in chronic conditions e.g. asthma, painasthma, pain

Crossover TrialCrossover Trial

RANDOMISED

Eligible subjects

Intervention

Control

Control Intervention

Wash-outWash-out Wash-outWash-out

Example of Crossover TrialExample of Crossover Trial

•2-period 2-treatment 2-period 2-treatment trial of acarbose vs. trial of acarbose vs. placebo in type 2 placebo in type 2 diabetesdiabetes

•Mean difference in Mean difference in HbA1HbA1cc was 0.3% with SD was 0.3% with SD of 0.5%of 0.5%

•No significant effectNo significant effect

Strengths and Weaknesses Strengths and Weaknesses of crossover designof crossover design

• Within patient characteristics Within patient characteristics remain same as matched analysisremain same as matched analysis

• Smaller sample size needed Smaller sample size needed compared with parallel designcompared with parallel design

• Not suitable for intervention that Not suitable for intervention that ‘cures’, used in chronic pain, ‘cures’, used in chronic pain, asthma, diabetes.asthma, diabetes.

• Carry-over effect needs wash-out Carry-over effect needs wash-out periodperiodSenn S. (1993) Crossover trials in clinical research. Chichester,

John Wiley

CLUSTER RANDOMISATIONCLUSTER RANDOMISATION

Man is a unit of the Man is a unit of the greater beasts, the greater beasts, the phalanx. The individual phalanx. The individual relates to the large unit relates to the large unit or phalanxor phalanx

John John SteinbeckSteinbeck

WHY USE CLUSTER WHY USE CLUSTER RANDOMISATION?RANDOMISATION?

•Intervention at the group level Intervention at the group level e.g.new appointment system in e.g.new appointment system in general practice general practice

•Sometimes only practical designSometimes only practical design

•Can reduce contamination Can reduce contamination

•Easier to implement intervention Easier to implement intervention (e.g counselling to reduce (e.g counselling to reduce smoking)smoking)

Intervention at Intervention at practice level & effects practice level & effects on patientson patients

Practice level

IMPLICATIONS OF USING IMPLICATIONS OF USING RANDOMISATION BY RANDOMISATION BY

PRACTICE?PRACTICE?

Sample size needs to be Sample size needs to be inflated:inflated:

Subjects within practice more Subjects within practice more alike alike than subjects in different than subjects in different practices practices so independence so independence assumption of assumption of statistical tests is statistical tests is

incorrectincorrect


PRACTICE?PRACTICE?Sample size obtained with no clustering is inflated by factor:

Where is the intra-cluster correlation and is the mean cluster (practice) size

m

ρ1)-m(+1=IF

Inflation or DESIGN EFFECT Inflation or DESIGN EFFECT depends on sizedepends on size of intra-cluster of intra-cluster correlation correlation Assume 40% reach lipid target on Assume 40% reach lipid target on new statin and 30% on old statin new statin and 30% on old statin

•Total(ind)Total(ind) InflationInflationTotal(Cl)Total(Cl)

• 944944 0.0000.000 1 1 944 944•944 0.0011.029•944 0.011.29

1218

971

•944 0.052.45

2312

How do you know degree of How do you know degree of intra-cluster correlation?intra-cluster correlation?

•Obtain from pilot workObtain from pilot work

•Previous published Previous published values values

•HSRU, AberdeenHSRU, Aberdeen

Outcomes Outcomes 0.05 0.05

Process Process 0.05 – 0.15 0.05 – 0.15


PRACTICE?PRACTICE?

Ethical issues:Ethical issues:

Cluster-cluster – guardian Cluster-cluster – guardian decides decides on behalf of all on behalf of all patientspatients

Cluster-individual – both Cluster-individual – both patient patient and GP decide on and GP decide on consentconsent


PRACTICE?PRACTICE?

Analysis:Analysis:

Need to take clustering Need to take clustering into into account e.g. Multi-level account e.g. Multi-level

modelling modelling

Study Design - Study Design - ObservationalObservational

•CohortCohort

•Case-ControlCase-Control

•Cross-sectional Cross-sectional surveysurvey

•Time seriesTime series

Walker & StampferWalker & Stampfer

...”...”should not denigrate the should not denigrate the observational nature of the observational nature of the data. Most of what we learn, data. Most of what we learn, and will continue to learn, and will continue to learn, about adverse drug effects are about adverse drug effects are from observational studiesfrom observational studies””

Lancet Lancet 1996;348:4891996;348:489

Cohort DesignCohort Design

A cohort consists of a group A cohort consists of a group of individuals from a of individuals from a well-well-defined defined population (population (exposure exposure and characteristics knownand characteristics known) ) followed up over time to followed up over time to observe what happens to observe what happens to which groups and whenwhich groups and whenn.b.sociologists call these panelsn.b.sociologists call these panels

How many smokers and non-How many smokers and non-smokers died?smokers died?

EventsEventsSmokers Smokers

Non-smokers Non-smokers

Simple Cohort studySimple Cohort study

Smokers vs. non in Tayside population

TIME1/1/2006

Non-smoker

Framingham cohort Framingham cohort studystudy

Followed up 5,573 people Followed up 5,573 people - white, initially free of - white, initially free of CVD, but including people CVD, but including people with hypertension and with hypertension and diabetes from 1968 diabetes from 1968 onwards onwards Developed predictive Developed predictive algorithms for CHDalgorithms for CHD

Summary Statistics in Summary Statistics in Cohort StudyCohort Study

•Drug safety study of NSAIDS Drug safety study of NSAIDS and hospitalisation for GI and hospitalisation for GI bleedbleed

•Rate of hospitalisations for Rate of hospitalisations for GI bleed in NSAID and non-GI bleed in NSAID and non-NSAID users over 1 year NSAID users over 1 year

•Relative risk (RR) is ratio of Relative risk (RR) is ratio of ratesrates

Relative RiskRelative Risk

Yes No

Exposed NSAIDS

Unexposed

a b

c d

RR = a/(a+b) / c/(c+d)

Hosp GI bleed

Example Relative RiskExample Relative Risk

Yes No

Exposed NSAIDS

Unexposed

136 4,390

2,104 126,060

RR = a/(a+b) / c/(c+d) =

Hosp GI bleed

1.83

Example Relative RiskExample Relative Risk

Risk in exposed = 136/(136+4390) = 0.0300 or 30 per 1000 people

Risk in unexposed = 2104/(2104+126060) = 0.0164 or 16 per 1000 people

Hence RR = 0.0300/0.0164 = 1.83

InterpretationInterpretation

83% higher risk of 83% higher risk of hospitalisation for GI bleed in hospitalisation for GI bleed in those exposed to NSAIDS those exposed to NSAIDS compared with those not compared with those not exposedexposed

RR = 1.83 and 95% CI (1.19, RR = 1.83 and 95% CI (1.19, 2.82) which is highly 2.82) which is highly statistically significant (p < statistically significant (p < 0.001)0.001)

LimitationsLimitations

In reality subjects have different In reality subjects have different length of follow-up – so need length of follow-up – so need event rates per person-years event rates per person-years follow-upfollow-up

e.g. 30 events per 1000 person e.g. 30 events per 1000 person yearsyearsRaw unadjusted rates and Raw unadjusted rates and Relative Risk – Need to take Relative Risk – Need to take account of confounding account of confounding through regressionthrough regression

Retrospective Cohort Retrospective Cohort DesignDesign

•In this design group of subjects In this design group of subjects or cohort is identified in the pastor cohort is identified in the past

•Follow-up is then to the presentFollow-up is then to the present

•Advantage that most data Advantage that most data already collected and events have already collected and events have occurredoccurred

•Cheaper to perform as do not Cheaper to perform as do not have to wait long period before have to wait long period before analysisanalysis

Log-rank test Log-rank test 221 1 = 10.6, p = 0.001 = 10.6, p = 0.001

generalised Wilcoxon generalised Wilcoxon 221 1 = 7.5, = 7.5,

p = 0.006p = 0.006Donnan et al Prognosis following first acute myocardial infarction in type 2 diabetes: A comparative population study. Diabetic Med 2002; 19: 448-455.

Case-Control DesignCase-Control Design

•In this design group of subjects In this design group of subjects with disease or condition are with disease or condition are identified (Cases)identified (Cases)

•Suitable Control group identified Suitable Control group identified without the conditionwithout the condition

•Frequency of Exposure or risk Frequency of Exposure or risk factor compared in cases and factor compared in cases and controlscontrols


CASESUnexposed

Exposed

Exposed

Unexposed CONTROL

S

Past Present

Example of Case-Control Example of Case-Control StudyStudy

•Early studies of Early studies of leukaemia around leukaemia around nuclear power stationsnuclear power stations

•Consumption of red Consumption of red meat and risk of meat and risk of colorectal cancercolorectal cancer

•Use of mobile phones Use of mobile phones and RTAsand RTAs


•May be matched or unmatchedMay be matched or unmatched

•Selection of cases and controls prone Selection of cases and controls prone to biasto bias

•Sometimes have both hospital and Sometimes have both hospital and population controlspopulation controls

•Ascertainment of exposure prone to Ascertainment of exposure prone to bias (e.g. recall bias)bias (e.g. recall bias)

•Relatively cheap to carry out butRelatively cheap to carry out but difficulties with confoundingdifficulties with confounding

Cross-sectional surveyCross-sectional survey

Events

UnexposedExposed

Exposed Unexposed

No Events

Present

Cross-sectional DesignCross-sectional Design

•Both exposure and events measured Both exposure and events measured at same time at same time

•Often questionnaire–based surveysOften questionnaire–based surveys

•Prone to volunteer bias and poor Prone to volunteer bias and poor responseresponse

•Relatively cheap to performRelatively cheap to perform

•Main difficulty in ascertaining Main difficulty in ascertaining temporal directiontemporal direction

•Examples are postal surveys such as Examples are postal surveys such as censuscensus

Time seriesTime series

•Can be considered a set of regular Can be considered a set of regular cross-sectional surveys over timecross-sectional surveys over time

•Examples are rainfall on daily basis, Examples are rainfall on daily basis, performance of stocks and shares, no. performance of stocks and shares, no. of hospitalisations on daily basisof hospitalisations on daily basis

•Requires sophisticated analyses which Requires sophisticated analyses which account for autocorrelation – ARIMA account for autocorrelation – ARIMA modelsmodels

•Interrupted time series can be Interrupted time series can be powerful method of assessing change powerful method of assessing change following policy or organisational following policy or organisational changeschanges

Example Time seriesExample Time series

Change in mean HbA1c following introduction (vertical line) of web-based Managed Clinical Network for diabetes in Tayside

1 3 5 7 9 11131517192123252729313335373941434547495153555759616365676971737577798183858789

Month (1=Jan 1998)

5.00

6.00

7.00

8.00

9.00M

ea

n H

bA

1c

(t2

)

Study DesignStudy Design

What design could be utilised to What design could be utilised to answer following questions?answer following questions?•Mobile phone use and brain cancerMobile phone use and brain cancer

•Effect of Herceptin on later stage breast Effect of Herceptin on later stage breast cancercancer

•Methadone and deaths in drug usersMethadone and deaths in drug users

•Intensive management of high risk patients Intensive management of high risk patients to prevent emergency hospitalisationto prevent emergency hospitalisation

•Effect of MMR vaccination and development Effect of MMR vaccination and development of autismof autism

Study DesignStudy Design

•Design is critical to successDesign is critical to success

•Design determines type of Design determines type of analysisanalysis

•Remember Remember

‘‘Chance favours the prepared Chance favours the prepared mind’mind’

A Taxonomy of Research Design Peter T. Donnan Professor of Epidemiology and Biostatistics Statistics for Health Research.

Documents

appropriate design

importance of design

question poor design

question choice of design

waste of resources poor

little return poor design

intervention slide

time slide