Improving measurement through Operations Research Peter Winch [email protected] Department of International Health, Johns Hopkins Bloomberg School of Public Health and Technical Advisor on Operations Research, MCHIP Project
Dec 14, 2014
Improving measurement through Operations
Research
Peter [email protected]
Department of International Health, Johns Hopkins Bloomberg School of Public Health
andTechnical Advisor on Operations Research, MCHIP Project
Brief history of measurement in NGO child survival projects
(1) Early projects
– Wide variation in quality of evaluation
– Lack of standardized outcome measures, sampling techniques and questionnaires
– Hard to compare results between baseline and final, or between projects
Brief history of measurement in NGO child survival projects
(2) The response: Focus on indicators
– Definition of standard indicators, measurement tools and sampling techniques (30-cluster sampling, LQAS)
– Training/capacity building on evaluation methodology
– NGOs have applied this expertise in measurement of indicators as they have become engaged in projects funded by: Global Fund, PEPFAR, PMI etc.
Measurement in NGOOperations Research projects
Measurement itself has not been the major focus of innovation in NGO operations research projects
The measurement tradition has been “indicatorcentric”:– Main form of measurement is cross-sectional
household surveys– Main purpose of surveys is to measure
indicators– Indicators are pre-defined, and calculated in
standard ways Not much experimentation with measurement,
beyond measurement of standard indicators
Strengths of focus on indicators
Results of projects easily communicated to various audiences, including Congress
Can summarize results across projects Can standardize:
– Definition of indicators– Training and procedures for data
collection– Training and procedures for data
analysis
Weaknesses of focus on indicators
Less effort invested in developing innovative approaches to measurement
Many fundamental questions have not been addressed:– Why do some interventions work well in
one site, but not in another?– How can we select the most appropriate
intervention for a certain type of community?
– Through what mechanism do community-based interventions achieve their impact?
Example: Odek et al. 2009
Odek et al. Effects of Micro-Enterprise Services on HIV Risk Behaviour Among Female Sex Workers in Kenya’s Urban Slums
Promotion of micro-enterprise as alternative to commercial sex work
Figure 2 from Odek et al. 2009
We invest great effort in measurement of indicators on the far right of this model
We invest relatively little effort in measurement of the variables in this box
NGO scorecard on measurement
Doing better Practices in the home Coverage, access to
services, use of services
Quality of care
Room for improvement
Characteristics of communities
Steps on the way to improved practices, coverage and quality
Steps on the way to improved practices, coverage and quality
Community mobilization Empowerment Social capital Strengthened household to hospital
continuum of care
Usual measurement approach to these steps
Assume intervention area has it, and comparison area doesn’t have it– Intervention area is mobilized– Comparison area is not mobilized
This prevents us from answering important questions:– Why do some communities respond better
than others to interventions?– What interventions are most appropriate
for a community with characteristics X and Y?
Community mobilization, empowerment and social
capital are constructs
What is a construct?
Also called “latent variable” Abstract theoretical variable invented
to represent a phenomenon of interest to scientists
Not directly measurable
What is a construct?
Measurement of constructs in psychology
Most developed in psychology Psychology as a science depends on
valid and reliable measurement of constructs
Some psychologists spend their entire careers working on measurement of one construct
Example: Depression
How do you measure a construct?
Different from an indicator– Generally have one single question– Think carefully about the
numerator and the denominator Usually put together a set of
questions (items) related to the construct
Ways to measure constructs
Single item on a questionnaire– Frequently misleading e.g. “what is
your monthly salary?” to measure wealth
Additive index: One point for each question where respondent answers YES
Complex scales
Example of additive indices:Measuring community
mobilization
Community Action Cycle in community mobilization
Measuring community mobilization and the action
cycle1. Exposure to the process, engagement in the Community Action Cycle
2. Changes in individual awareness of problems, self-efficacy, commitment to change
3. Actions taken after being mobilized
1. Exposure to the process, engagement in the Community
Action Cycle1. Have you attended a meeting with your
neighbors organized by X project? Y/N2. Did you discuss/list the different health
problems here in this village? Y/N3. Did you discuss which problems are most
important? Y/N4. Did you discuss the causes of these
problems? Y/N5. Did you discuss what can be done about
these problems? Y/N
2. Changes in individual awareness of problems, self-efficacy, commitment to
change
1. “I feel confident that I could provide feedback to the committee on the quality of services in the clinic”– Strongly disagree, disagree, agree,
strongly agree2. “I intend to work with the committee to
improve the quality of services in the clinic”– Strongly disagree, disagree, agree,
strongly agree
3. Actions taken after being mobilized
1. I have provided feedback to the committee on the quality of services in the clinic Y/N
2. I have attended committee meetings to work on improving health services here Y/N
3. I am working with the committee to improve the quality of services in the clinic Y/N
What I am recommending
Keep measuring indicators, but don’t just measure indicators
Branch out into measurement of constructs
Be aware that time and effort is needed to develop good measures of constructs
Steps in developing a scale to measure a construct
Qualitative research to understand what is happening
Draft set of questions to measure the construct
Administer questions to a small sample Statistical analysis to select the questions
(items) that belong in the scale Revision of the questions to measure the
construct Include questions in questionnaire
administered to larger sample
Systems
“Systems” is a core theme throughout many maternal and child health projects
Manifests itself in different ways:– District health systems– Household-to-hospital continuum of
care– Referral networks– Supervision systems
Continuum of care for health services (Kerber et al. Lancet 2007)
Sociometric questionnaire
Questionnaire to elicit the structure of a social network
The questions are generally about relations with other people, rather than questions about individual knowledge, behavior etc.
Valuable tool for examining systems
Sociometric questionnaire applied to complete network:
Steps Define the population, and its boundaries List all actors in the population/network
– All health workers– All CHWs and TBAs
Select one or more questions to be asked regarding each of the other actors in the network
Create sociometric questionnaire with list of actors and questions– Often in table format
Example of sociometric questionnaire
ID Name How many times a week to you talk to
____?
How many times per month do you seek advice from
____?
001 Person #1
002 Person #2
003 Person #3
004 Person #4
Questions for a sociometric questionnaire
You can’t have very many! Examples:
– Whom do you go to for advice?– Who provides you with support for
X?– Who have you talked to about how
to treat sick children?– Who has referred a patient to you?– Who have you talked to on the
mobile phone?
Sociometric questionnaire:Full matrix
P1 P2 P3 P4
P1 5 4 10
P2 5 0 1
P3 2 0 2
P4 10 1 3
Sociometric questionnaire:Symmetrized lower-half matrix
P1 P2 P3 P4
P1
P2 5
P3 3 0
P4 10 1 2.5
Analysis of the symmetrized lower-half matrix
Calculation of measures for actors in network: Degree centrality, betweenness, closeness centrality, etc.
Calculation of measures of network as a whole: Density, cohesion etc.
Multi-dimensional scaling Sociogram
Examples of what could be done in maternal and child
health projects
1. Referrals between facilities2. Communication between health
workers
Referrals between facilities
Take existing data, then enter into a lower-half matrix
From the lower-half matrix:– Manual analysis: Take map, draw lines
between facilities, width indicates number of completed referrals
– Multidimensional scaling on matrix– Calculate network density before and
after the intervention
Health facility referral matrix
Facility 1
Facility 2
Facility 3
Facility 4
Facility 1
Facility 2
20
Facility 3
5 3
Facility 4
0 0 5
Density: Method 1
“Linkage density”
Number of facility pairs with any linksTotal number of facility pairs
=4/6
Density: Method 2
“Referral density”:Total number of referrals
Total number of facility pairs
=33/6
Might also calculate referrals in relation to target population, e.g. maternal referrals per women of reproductive age
Comparisons of density
Comparisons can be made for trends in same referral system over time
Comparisons with other systems likely won’t be meaningful, due to differences in:– Number of facilities– Total population size– Population per facility
Multidimensional scaling
1
3
2
4
20
53
5