Collecting High Quality Outcome Data: Part 1 Collecting High Quality Outcome Data Copyright © 2012 by JBS International, Inc. Developed by JBS International for the Corporation for National & Community Service
Collecting High Quality Outcome Data: Part 1
Collecting High Quality Outcome Data
Copyright © 2012 by JBS International, Inc.
Developed by JBS International for the Corporation for National & Community Service
Collecting High Quality Outcome Data: Part 1
Learning Objectives
By the end of this module, you will be able to:
•Recognize the benefits of collecting high-quality data
•Use theory of change to think about measurement
•Identify and evaluate merits of data sources and instruments
•Describe some uses of data collection methods, and evaluate their merits
•Describe steps to implement data collection
•Recognize data quality
2
Collecting High Quality Outcome Data: Part 1
What Do We Mean By Data?
• Data: Information collected to answer a measurement question, also known as evidence
• Data collection occurs as a planned process that involves recording information in a consistent way
• Instruments aid in collecting consistent data
3
Collecting High Quality Outcome Data: Part 1
Ensuring Data Quality: Reliability, Validity, Bias
• Reliability is the ability of a method or instrument to yield consistent results under the same conditions.
• Validity is the ability of a method or instrument to measure accurately.
• Bias involves systematic distortion of results stemming from how data are collected and how instruments are designed.
4
Collecting High Quality Outcome Data: Part 1
Benefits of Collecting High-quality Data
• Sound basis for decision making
• Improve service quality and service outcomes
• Increase accountability
• Tell story of program achievements
5
Collecting High Quality Outcome Data: Part 1
Measurement Question Implied by Theory of Change
6
Intended Outcome
Students improve attitudes towardsschool.
Intended Outcome
Students improve attitudes towardsschool.
Community Problem/Need
Students with poor attitudes
towards school at risk of failing academically.
Community Problem/Need
Students with poor attitudes
towards school at risk of failing academically.
Specific Intervention
Individualized mentoring to
promote positive attitudes
towards school.
Specific Intervention
Individualized mentoring to
promote positive attitudes
towards school.
"Did students in the mentoring program improve their attitudes towards school?"
Collecting High Quality Outcome Data: Part 1
Identifying a Data Source
• Data source: The person, group or organization that has information to answer the measurement question
• Identify possible data sources; list pros and cons of each
• Identify a preferred data source; consider its accessibility
• Alternative data sources: consider if they can give you same or comparable data
7
Collecting High Quality Outcome Data: Part 1
Data source and type of outcome
Depends partly on the type of change you want to measure - attitude, knowledge, behavior, or conditions.
•Data on changes in attitudes or knowledge usually come directly from persons experiencing these changes.
•Data on changes in behavior or conditions can come from either persons experiencing these changes or from other observers.
8
Collecting High Quality Outcome Data: Part 1
“How did mentored students’ feelings towards teachers change over time?”
Pros Cons
Students• In best position to
describe how they feel about their teachers
• May not be open about their feelings towards teachers
Teachers• May know how
students feel towards them
• May not know how students feel about other teachers
• May only spend one class period with students
Mentors • May know how students feel about a wide range of issues, including teachers
• Depends on students’ willingness to share feelings with mentors
• Students and mentors may not discuss this issue much
Comparing Data Sources
9
Collecting High Quality Outcome Data: Part 1
Method: Process or Steps Taken to Systematically Collect Data
Survey Written questionnaire completed by respondent
InterviewInterviewer poses questions and records responses; face-to-face or via telephone
ObservationObserver records behavior or conditions using via checklist or other form
Standardized Test
Used to assess knowledge of academic subjects (reading, math, etc.)
Next, Consider Choice of Methods
10
Collecting High Quality Outcome Data: Part 1
Method: Process or Steps Taken to Systematically Collect Data
Tracking Sheet
Used to document service delivery; used primarily to track outputs
Focus GroupFacilitator leads small group through discussion in-depth discussion of topic or issue
Diaries, Journals
Respondent periodically (daily) records information about his/her activities or experiences
Secondary Data
Using data gathered by other agencies that can be used to assess program performance
Consider Choice of Methods (continued)
11
Collecting High Quality Outcome Data: Part 1
Method and Outcomes Type—Attitude and Knowledge
Attitude/Belief Knowledge/Skill
Definition Thoughts, feelingsUnderstanding,
know-how
ExamplesAttachment to school
(academic engagement)
Becoming a better reader
Generally Preferred Data Source/Method
Student:Survey or interview
Learner: Standardized test*
* Use of standardized tests is mandated for certain performance measures in the Education Focus Area. Other types of knowledge (e.g., financial literacy) can be measured using other types methods.
12
Collecting High Quality Outcome Data: Part 1
Method and outcome type—behavior and condition
Behavior Condition/Status
Definition Action, conduct, habitsSituation or
circumstances
ExamplesExercising more
frequentlyImproving stream
banks
Generally Preferred Data Source/Method
Beneficiary:Exercise log
Land manager: Observation checklist
or rubric
13
Collecting High Quality Outcome Data: Part 1
Where to Find Instruments
• For CNCS priorities and performance measures, look for instruments by goal and focus area
• Go to http://www.nationalservice.gov/resources/npm/home
• Programs and projects can look anywhere they like to find instruments:
• Use Internet search engines
• Talk to others within you professional network to find out what they are using
• Look at evidence for intervention – how measured before?
14
Collecting High Quality Outcome Data: Part 1
Evaluating Instruments
• Pre-post measurement is preferable to post-only
• Can the instrument measure the outcome?
• Appropriate for your intervention?
• Appropriate for your beneficiaries?
• How many questions measure the outcome?
• Single question low-quality data
• Series of questions: Too long or complex?
• Instrument should not exceed 2 pages
• Do questions cover all relevant aspects of your intervention? Can questions not specific to your intervention be removed?
15
Collecting High Quality Outcome Data: Part 1
Define Outcome Dimensions
Outcome Dimensions: The main aspects, features, or characteristics that define an outcome and that should be taken into account for measurement to be valid
Example: Increased attachment to school:•Feelings about being in school•Feelings about doing school work•Feelings towards teachers•Feelings towards students
16
Collecting High Quality Outcome Data: Part 1
Example: Dimensions of Attachment to School
a
a
b
b
c
c
17
d
d
a
b
c
d
Collecting High Quality Outcome Data: Part 1
Summary: Identifying Outcome Dimensions
• National performance measures: look at performance measurement instructions
• Look at your theory of change
• Talk to stakeholders and program staff
• Build up a list of dimensions; look for repeated themes
18
Evidence•Guides choice of intervention•Supports cause-effect relationship
Evidence•Guides choice of intervention•Supports cause-effect relationship
Community Problem/Need
Community Problem/Need
Specific Intervention
Specific Intervention
Intended OutcomeIntended Outcome
Collecting High Quality Outcome Data: Part 1
Instrument Design Issues
• Crowded layout
• Double-barreled questions
• Biased or “leading” questions
• Questions that are too abstract
• Questions that use unstructured responses inappropriately
• Response options that overlap or contain gaps
• Unbalanced scales
19
Collecting High Quality Outcome Data: Part 1
Crowded Layout
20
Problem: Crowded layout
Most of the time, how do you feel about doing homework?
☐ I usually hate doing homework I usually don’t like doing ☐homework I usually like doing homework I usually love doing ☐ ☐homework
Solution: Don’t use crowded layouts
Most of the time, how do you feel about doing homework?
☐ I usually hate doing homework
☐ I usually don’t like doing homework
☐ I usually like doing homework
☐ I usually love doing homework
Collecting High Quality Outcome Data: Part 1
Double-barreled Question
21
They strongly like it☐
Theylike it☐
They are undecided
☐
They dislike it☐
They stronglydislike it
☐
Problem: Asking two questions in one
How do teachers and students at your school feel about the mentoring program?
Solution: Break out questions separately
How do teachers at your school feel about the mentoring program?
How do students at your school feel about the mentoring program?
They strongly like it☐
Theylike it☐
They are undecided
☐
They dislike it☐
They stronglydislike it
☐
They strongly like it☐
Theylike it☐
They are undecided
☐
They dislike it☐
They stronglydislike it
☐
Collecting High Quality Outcome Data: Part 1
Biased or “Leading” Question
22
Problem: Biased or “leading” questions
Has the mentoring program improved how you feel about going to school?
☐ Yes
☐ No
☐ No opinion
Solution: Use neutral questions
How has the mentoring program affected how you feel about going to school?
☐ I feel better about going to school.
☐ I feel worse about going to school.
☐ I feel about the same about going to school.
☐ No opinion
Collecting High Quality Outcome Data: Part 1
Abstract or Broad Question
23
Problem: Questions are too abstract or broad.
Did you enjoy the mentoring program?
Yes No Not Sure
Solution: Make questions more concrete and specific.
Would you recommend the mentoring program to other students?
Yes No Not Sure
Collecting High Quality Outcome Data: Part 1
Not Using Structured Responses
24
Problem: Using unstructured responses when structured responses are appropriate
How much do your grades matter to you?
Solution: Provide structured responses when appropriate
How much do your grades matter to you?
☐ Not at all
☐ A little
☐ Somewhat
☐ A lot
Collecting High Quality Outcome Data: Part 1
Response Options with Overlaps or Gaps
25
Problem: Response options that overlap or contain gaps
Approximately how many hours a day to you typically spend doing homework?
☐ Less than 1 hour
☐ 0 to 2 hours
☐ 4 to 5 hours
☐ More than 5 hours
Solution: Scale with no overlaps or gaps
Approximately how many hours a day to you typically spend doing homework?
☐ Less than 1 hour
☐ About 1 hour
☐ About 2 hours
☐ About 3 hours
☐ About 4 hours
☐ More than 4 hours
Collecting High Quality Outcome Data: Part 1
Unbalanced scales
26
Problem: Using unbalanced scales
Solution: Use balanced scales
Poor☐
Average☐
Good☐
Very Good
☐
Excellent☐
VeryPoor☐
Poor☐
Average☐
Good☐
Very Good
☐
Collecting High Quality Outcome Data: Part 1
What else to look for in selecting an instrument
• Can the instrument work in your context?
• Does the instrument use simple and clear language?
• Is the instrument appropriate for the age, education, literacy, and language preferences of respondents?
27
Collecting High Quality Outcome Data: Part 1
What else to look for in selecting an instrument, continued
• Does the instrument rely mostly on multiple choice questions?
• Is the ready for use, or does it need to be modified?
• How will you extract information from the instrument to address performance measurement targets?
28
Collecting High Quality Outcome Data: Part 1
Implementing Data Collection
After identifying a data source, method and instrument:
1.Identify data collection participants
2.Set a schedule for collecting data
3.Train data collectors
4.Pilot test the data collection process
5.Make changes
6.Implement data collection
29
FOR BEST RESULTS make key decisions about how to implement data collection BEFORE program startup!
Collecting High Quality Outcome Data: Part 1
Step 1: Identifying data collection participants
30
• Brainstorm a list of all the relevant players in the data collection process. This includes:
Collecting High Quality Outcome Data: Part 1
Step 2: Creating A Data Collection Schedule
31
• Identifies who will collect data, using which instrument, and when
• Share with team to keep everyone informed• Include stakeholders in planning• Include dates for collecting, analyzing, and reporting data• Select a format
Collecting High Quality Outcome Data: Part 1
Step 3: Training Data Collectors
• Determine best person(s) to collect data
• Provide written instructions for collecting data
• Explain importance and value of data for program
• Walk data collectors through instrument
• Practice or role play data collection
• Review data collection schedule
• Explain how to return completed instruments
32
Collecting High Quality Outcome Data: Part 1
Step 4: Pilot Testing for Feasibility and Data Quality
1. Try out instruments with a small group similar to program participants
2. Discuss instrument with respondents
3. Analyze pilot test data toensure the instrument yields the right information
33
Questions for Debrief
How long did it take to complete?
What did you think the questions were asking you about?
Were any questions unclear, confusing, or difficult to answer?
Were response options adequate?
Did questions allow you to say everything you wanted to say?
Collecting High Quality Outcome Data: Part 1
Steps 5 & 6: Make Changes & Implement Your Plan
34
Make Changes•Based on pilot test analysis:
• Improve instrument
• Strengthen process
Implement Your Plan•Perform periodic quality control checks
Collecting High Quality Outcome Data: Part 1
Ensuring Data Quality:Key Criteria
• Criteria for collecting high-quality, useful outcome data:
• Reliability
• Validity
• Minimizing Bias
35
Collecting High Quality Outcome Data: Part 1
Reliability
• Reliability: The ability of a method or instrument to yield consistent results under the same conditions.
• Requires that instruments be administer the same way every time:o Written instructions for
respondents
o Written instructions for data collectors
o Train and monitor data collectors
36
Collecting High Quality Outcome Data: Part 1
Reliability
• Design instruments to improve reliability
o Use clear and unambiguous language so question meaning is clear.
37
Unclear language
“How has the availability companionship services altered your capacity with respect to attending visits with medical practitioners in a timely manner?”
Clear language
“How has use of companionship services affected your ability to get to medical appointments on time?”
Collecting High Quality Outcome Data: Part 1
Reliability
• Design instruments to improve reliability
o Use attractive, uncluttered layouts that are easy to follow.
38
Cluttered layout
“What grade are you in?”
☐6th grade ☐7th grade ☐8th grade
Uncluttered layout
“What grade are you in?”
☐ 6th grade
☐ 7th grade
☐ 8th grade
Collecting High Quality Outcome Data: Part 1
Validity
39
Collecting High Quality Outcome Data: Part 1
Minimizing Sources of Bias
• Bias involves systematic distortion of results stemming from how data are collected and how instruments are designed.
• Who: Non-responders = hidden bias
• How: Wording that encourages or discourages particular responses
• When and Where: Timing and location can influence responses
• Bias can lead to over- or under-estimation of program results
40
Collecting High Quality Outcome Data: Part 1
7 Ways of Minimizing Bias
1. Get data from as many respondents as possible
2. Follow up with non-responders
3. Take steps to reduce participant attrition
4. Work with program sites to maximize data collection
41
Collecting High Quality Outcome Data: Part 1
7 Ways of Minimizing Bias(continued)
5. Pilot test instruments and data collection procedures
6. Mind your language
7. Time data collection to avoid circumstances that may distort responses
42
Collecting High Quality Outcome Data: Part 1
Did students in the mentoring program increase their attachment to school?
OutputNumber of disadvantaged youth/mentor matches that were sustained by the CNCS-supported program for at least the required time period (ED4A)
OutcomeNumber of students in grades K-12 that participated in the mentoring or tutoring or other education program who demonstrated improved academic engagement (ED27)
How Measured Pre/post survey of students to gauge attachment to school
Outcome Target90 (of 100) students in grades 6-8 that participate in the after-school program for 9 months will improve academic engagement, defined as feelings of attachment to school.
Academic Engagement
43
Collecting High Quality Outcome Data: Part 1
Did students in the mentoring program increase their attachment to school?
OutcomeNumber of students in grades K-12 that participated in the mentoring or tutoring or other education program who demonstrated improved academic engagement (ED27)
How Measured Pre/post survey of students to gauge attachment to school
ReliabilityDo survey responses reflect students’ stable and established beliefs about school or just fleeting and changeable feelings?
ValidityDoes the survey get at the dimensions of school attachment that are relevant to the intervention? Are students telling us how the really feel or what they think we want to hear?
BiasDoes the survey ask the questions in a neutral way? Have we timed the survey to avoid unrelated factors like “exam stress” that could contaminate our results?
Academic Engagement—Reliability, Validity, Bias
44
Collecting High Quality Outcome Data: Part 1
Summary of key points
Steps to implement data collection include identifying the
players involved in data collection, creating a data collection
schedule, training data collectors, pilot testing instruments,
and revising instruments as needed.
• A data collection schedule identifies who will
collect data, using which instrument, and when.
• Training data collectors by walking them through
the instrument and role playing the process.
• Pilot testing involves having a small group of
people complete an instrument and asking them
about the experience.
45
Collecting High Quality Outcome Data: Part 1
Summary of key points
Reliability, validity, and bias are key criteria for data quality.
• Reliability is the ability of a method or instrument to
yield consistent results under the same conditions.
• Validity is the ability of a method or instrument to
measure accurately.
• Bias involves systematic distortion of results due to
over- or under-representation of particular groups,
question wording that encourages or discourages
particular responses, and by poorly timed data
collection.
46
Collecting High Quality Outcome Data: Part 1
Summary of key points
• The benefits of collecting high-quality data include
providing a sound basis for decision making, improving
service quality and outcomes, increasing accountability,
and telling your story in a more compelling way.
• Your theory of change, and the key measurement
question embedded in it, is a useful a guide to
measurement.
• The type of outcome to be measured influences
decisions about data sources, methods, and
instruments.
47
Collecting High Quality Outcome Data: Part 1
Additional resources
CNCS Performance Measuremento http://nationalservice.gov/resources/npm/home
Instrument Formatting Checklisto https://www.nationalserviceresources.org/npm/practicum-
collecting-data-part-2
Practicum Materialso http://www.nationalservice.gov/resources/npm/core-
curriculum
48