Capturing the benefits to individuals Stephen Platt RUHBC, University of Edinburgh Big Lottery Fund Evaluation Conference: Tidy Findings in an Untidy World.

Capturing the benefits to Capturing the benefits to individualsindividuals

Stephen PlattStephen PlattRUHBC, University of EdinburghRUHBC, University of Edinburgh

Big Lottery Fund Evaluation Big Lottery Fund Evaluation Conference: Conference:

Tidy Findings in an Untidy World Tidy Findings in an Untidy World

London 1 November 2007London 1 November 2007

Measuring health Measuring health improvement: key issuesimprovement: key issues

Specifying the research question(s)Specifying the research question(s) Deciding on the level of analysis Deciding on the level of analysis

(individual, ecological or mixed)(individual, ecological or mixed) Adopting an appropriate research designAdopting an appropriate research design Attributing (change in) health outcomes Attributing (change in) health outcomes

to specific interventions: perils and to specific interventions: perils and pitfallspitfalls

Conceptualising and measuring health Conceptualising and measuring health outcomesoutcomes

Research designsResearch designs

Classical experimental Classical experimental (before/after with controls), with (before/after with controls), with randomisationrandomisation

Quasi-experimental (before/after Quasi-experimental (before/after with controls), without with controls), without randomisationrandomisation

Panel/longitudinal (no control)Panel/longitudinal (no control) Cross-sectional/correlationalCross-sectional/correlational Case studyCase study

Experimental and quasi-Experimental and quasi-experimental designsexperimental designs

Intervention and control groupsIntervention and control groups Pre-and post measuresPre-and post measures

Control group: Control group: O1 O1 O2O2 Intervention group:Intervention group: O1 X O1 X O2O2

Random (experimental) or non-random Random (experimental) or non-random (quasi-experimental) assignment to groups(quasi-experimental) assignment to groups

Where non-random assignment, every Where non-random assignment, every effort made to maximise equivalence of effort made to maximise equivalence of groups on baseline outcome measuresgroups on baseline outcome measures

Limitations of experimental Limitations of experimental and quasi-experimental and quasi-experimental designsdesigns Randomised designs often difficult to Randomised designs often difficult to

implement … especially for complex implement … especially for complex health interventionshealth interventions

The underlying mechanisms by which The underlying mechanisms by which the programme affects outcomes are the programme affects outcomes are unclearunclear

Sometimes experimental designs are Sometimes experimental designs are inappropriate or unethicalinappropriate or unethical

Narrow specification of experimental Narrow specification of experimental design can threaten external validitydesign can threaten external validity

Quasi-experimental design: difficult to Quasi-experimental design: difficult to deal with threats to internal validitydeal with threats to internal validity

The problem of establishing The problem of establishing causalitycausality

How do we know that a How do we know that a programme/policy programme/policy causescauses an effect? an effect?

How do we quantify the How do we quantify the causal causal effecteffect??

What are mechanisms by which the What are mechanisms by which the programme causes the effect?programme causes the effect?

Have we ruled out other possible Have we ruled out other possible explanations for the observed explanations for the observed changes?changes?

Dimensions of diseaseDimensions of disease

Cancer(s)Cancer(s) Psychiatric conditions and Psychiatric conditions and

psychological morbiditypsychological morbidity Respiratory conditionsRespiratory conditions Neurological conditionsNeurological conditions Rheumatological conditionsRheumatological conditions Cardiovascular diseasesCardiovascular diseases OtherOther

Dimensions of healthDimensions of health

Health-related quality of lifeHealth-related quality of life Functional ability Functional ability Social functioning Social functioning General health status General health status Psychological well-beingPsychological well-being Positive mental health and well-Positive mental health and well-

beingbeing Self-esteemSelf-esteem

Subjective health Subjective health measurement: advantagesmeasurement: advantages

Opportunity to capture qualitative Opportunity to capture qualitative dimensions of healthdimensions of health

Means to explore aspects of health that Means to explore aspects of health that are difficult to capture via are difficult to capture via physical/laboratory measuresphysical/laboratory measures

Avoidance of invasive procedures and Avoidance of invasive procedures and expensive equipmentexpensive equipment

Potential for understanding (under-Potential for understanding (under-utilisation of health servicesutilisation of health services

Key issues in the use and Key issues in the use and

administration of scalesadministration of scales (1)(1)

The quantification of subjective The quantification of subjective experienceexperience

ReliabilityReliability ValidityValidity Sensitivity to changeSensitivity to change

Key issues in the use and Key issues in the use and

administration of scalesadministration of scales (2)(2)

Methods of administering an Methods of administering an outcome measure outcome measure

Types of measureTypes of measure Ethical issuesEthical issues Cross-cultural validation and Cross-cultural validation and

scale translationscale translation Criteria for scale selectionCriteria for scale selection

Bridge Consortium Bridge Consortium evaluation: aim of Focus 1evaluation: aim of Focus 1

To develop and implement a health To develop and implement a health information framework (‘Health information framework (‘Health Monitoring System’ [HMS]) which Monitoring System’ [HMS]) which permits tracking of changes in permits tracking of changes in health and well-being over time health and well-being over time – among HLC project users/recipients among HLC project users/recipients – in communities where HLC projects in communities where HLC projects

are locatedare located

HMS: necessary featuresHMS: necessary features

Should be easy to use and acceptableShould be easy to use and acceptable Should capture broader (as well as Should capture broader (as well as

narrower) aspects of health and well-narrower) aspects of health and well-beingbeing

Should collect information about Should collect information about users/recipients of project facilities/ users/recipients of project facilities/ services and the wider community in services and the wider community in which projects are locatedwhich projects are located

Developing the HMSDeveloping the HMS

The HMS was developed through The HMS was developed through a partnership of members of the a partnership of members of the research team, representatives research team, representatives from HLCs and an expert group from HLCs and an expert group

Extensive consultations were Extensive consultations were undertaken with HLCsundertaken with HLCs

HMS datasetsHMS datasets

The HMS consists of two datasets:The HMS consists of two datasets: a user dataseta user dataset a community dataseta community dataset

Health Monitoring System: Health Monitoring System: community-focused elementcommunity-focused element

Annual collection and collation of Annual collection and collation of information on health status and information on health status and other characteristics of the other characteristics of the populations living in areas where populations living in areas where HLC projects are locatedHLC projects are located– to identify the social, environmental to identify the social, environmental

and health-related context in which and health-related context in which each HLC project operateseach HLC project operates

Health Monitoring System: Health Monitoring System: user-focused elementuser-focused element

Main aimsMain aims– to assess the impact of HLC attendance on a range to assess the impact of HLC attendance on a range

of outcomes, including physical and mental health, of outcomes, including physical and mental health, smoking, regularity of alcohol consumption, smoking, regularity of alcohol consumption, regularity of consumption of fruit and vegetables regularity of consumption of fruit and vegetables and involvement in physical activityand involvement in physical activity

– to assessto assess perceived impact of the HLC on health & perceived impact of the HLC on health & QoL QoL

Longitudinal survey of HLC usersLongitudinal survey of HLC users– Baseline survey using self-report questionnaire, Baseline survey using self-report questionnaire,

followed by repeated contact at 6 and 18 monthsfollowed by repeated contact at 6 and 18 months Resources insufficient for recruitment and Resources insufficient for recruitment and

follow-up of control samplefollow-up of control sample Study aim to be addressed via comparison of Study aim to be addressed via comparison of

outcomes between regular (weekly) and non-outcomes between regular (weekly) and non-regular usersregular users

Health Monitoring System: Health Monitoring System:

participationparticipation (1)(1)

155 (44%) HLCs agreed to participate155 (44%) HLCs agreed to participate 4502 returned baseline 4502 returned baseline

questionnairesquestionnaires– Mean HLC response = 29 questionnairesMean HLC response = 29 questionnaires– Range: 1-132Range: 1-132

Of these, 2663 (59%) provided Of these, 2663 (59%) provided contact details for follow-upcontact details for follow-up

1400 (31% of baseline; 53% of those 1400 (31% of baseline; 53% of those giving contact details) returned giving contact details) returned questionnaires at 18 monthsquestionnaires at 18 months

Health Monitoring System: Health Monitoring System: non-participation by HLCsnon-participation by HLCs

Did not respond (n=35)Did not respond (n=35) Refused outright (n=26)Refused outright (n=26)

– Insufficient resources to administer surveyInsufficient resources to administer survey– Too burdensome to usersToo burdensome to users– Approach/measures irrelevant or Approach/measures irrelevant or

potentially harmful to userspotentially harmful to users Did not commit (n=61)Did not commit (n=61) Accepted packs, then refused (n=10)Accepted packs, then refused (n=10) Did not return packs (n=63)Did not return packs (n=63)

Health Monitoring System: Health Monitoring System: outcomesoutcomes

Deterioration in physical and mental health Deterioration in physical and mental health experienced by non-regular users is not found experienced by non-regular users is not found among regular usersamong regular users

A higher percentage of regular users reported a A higher percentage of regular users reported a change over time towards a healthier lifestyle change over time towards a healthier lifestyle (less smoking, more physical activity and (less smoking, more physical activity and fruit/vegetable consumption) compared to non-fruit/vegetable consumption) compared to non-regular usersregular users

Exception to this trend was alcohol consumption: Exception to this trend was alcohol consumption: differences between regular and non-regular users differences between regular and non-regular users at follow-up were not statistically significantat follow-up were not statistically significant

Regular HLC attenders were more likely to give a Regular HLC attenders were more likely to give a positive assessment of the impact of HLC on own positive assessment of the impact of HLC on own health and quality of life than non-regular health and quality of life than non-regular attendersattenders

Health Monitoring System: Health Monitoring System: interpreting findings interpreting findings (1)(1)

Findings take into account baseline Findings take into account baseline values (outcomes) and characteristics values (outcomes) and characteristics known to be associated with study known to be associated with study outcomes (age, sex, housing tenure, outcomes (age, sex, housing tenure, financial status, life events)financial status, life events)

Nevertheless, Nevertheless, we have to consider the we have to consider the possibility that the findings reflect self-possibility that the findings reflect self-selection processes rather than a causal selection processes rather than a causal impact of HLC attendanceimpact of HLC attendance– those who enjoyed better health and led those who enjoyed better health and led

healthier lifestyles (at baseline) would be healthier lifestyles (at baseline) would be more likely to note and take advantage of the more likely to note and take advantage of the opportunities offered by the HLC, as reflected opportunities offered by the HLC, as reflected in their (more) regular attendance at HLC in their (more) regular attendance at HLC activities activities

Health Monitoring System: Health Monitoring System: interpreting findings interpreting findings (2)(2)

Self-selection hypothesis can be tested Self-selection hypothesis can be tested in in part by comparing regular and non-regular part by comparing regular and non-regular users at baselineusers at baseline

Compared to non-regular attenders, regular Compared to non-regular attenders, regular attenders had better mental health and were attenders had better mental health and were more likely to have healthier lifestyles (less more likely to have healthier lifestyles (less smoking and drinking, more physical activity smoking and drinking, more physical activity and fruit and vegetable consumption), but and fruit and vegetable consumption), but had worse physical healthhad worse physical health

While the evidence of a protective effect of While the evidence of a protective effect of HLC attendance is fairly persuasive, some HLC attendance is fairly persuasive, some selection effect cannot be entirely ruled outselection effect cannot be entirely ruled out

Key lessonsKey lessons

Assess balance of costs and benefits of Assess balance of costs and benefits of longitudinal surveylongitudinal survey– Costs may outweigh benefitsCosts may outweigh benefits

Address causal attribution issue at the outsetAddress causal attribution issue at the outset Consider use of more ‘customised’ approach Consider use of more ‘customised’ approach

to outcomes (linked to aims/purpose of to outcomes (linked to aims/purpose of intervention)intervention)

Build cooperation with survey into contractual Build cooperation with survey into contractual obligations/expectationsobligations/expectations

Work harder and more imaginatively to bring Work harder and more imaginatively to bring gatekeepers on boardgatekeepers on board

Avoid over-ambitious follow-up timescaleAvoid over-ambitious follow-up timescale