Data Collection and Aggregation 2/8/2014 Data Collection and Aggregation 1 Presented by AmeriCorps Program Staff and JBS International.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 1

Data Collection and Aggregation

Presented byAmeriCorps Program Staff

and JBS International

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 2

Learning Outcomes

As a result of this session, participants will:• Understand core concepts associated with data

collection and aggregation;• Better understand how to assess the quality of your

current data collection tools and systems; and• Identify upgrades you need to make to insure

rigorous data collection and meaningful reporting.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 3

Agenda

• What are You Trying to Measure?• Are You Measuring it the Best Way?• Measuring Outcomes

– Data Collection Method Considerations – Instrument Considerations

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 4

Food for Thought

• Why are you doing the intervention?• What change do you want to create?• Can you measure the change/outcome?• If you can’t measure the outcome, are you sure you

are doing the right thing?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 5

Now What?

You have identified the desired change (outcome) through your program design and theory of change….

Are you clear about WHAT to measure?AND

Are you measuring it the “BEST” way?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 6

What are Your Desired Outcomes?

• What question are you trying to answer? What change are you expecting?– Did you select one of the CNCS priority measures?– Did you create your own program-specific outcome?

Either way…the data collection and aggregation basics are the same.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 7

Driving School Example

• Intervention: ABS driving school gives a 10 week course that meets twice a week for 60 minutes that include classroom-based and on road lessons on driving skills

• Desired Outcome: Students have basic driving proficiency.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 8

Real Life Example: Driving Test

Question: Do you like driving?Answer: I LOVE driving!!

“Do you like driving?” gets information about an attitude – not skill level.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 9

Real Life Example: Driving Test

Question: Do you think you are a skilled driver?

Answer: I think I am a GREAT driver!

“Are you a skilled driver” gets information about self perception, a thought – not actual skill level.Self-ratings are subjective NOT objective.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 10

Real Life Example: Driving Test

Question: Do you know the state driving laws ?

Answer: I got 100% correct on my written driver’s test!

Knowing state driving laws reflects knowledge – not actual skill level even though it is objective.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 11

Real Life Example: Driving Test

Question: Did you pass your road test?Answer: YES!

An on-road driving test DOES measure skill level or proficiency and is objective.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 12

What kind of information do you want?

• Subjective - includes an element of opinion or personal feeling.– Example: how someone feels about driving, confidence

• Objective - is not dependent upon opinions or personal feelings. It is based on facts that are observable and measurable. – Example: knowledge of driving laws and skill driving

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 13

How Do You Choose Your Method?

• Choice depends on: – what you want to measure; and.– the situation (i.e. resources for data collection/aggregation,

site/partner agreements/restrictions, etc.)• Each method is more appropriate in some situations than

others (e.g., age, language, content sensitivity, etc.)• Will it get you high quality data?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 14

Does Your Method Measure Your Outcome?

• Commonly used data collection methods– Surveys– Pre/post tests– Observations– Standardized tests– Interviews– Focus Groups– Diaries, Journals, Self-reported Checklists – Available secondary data

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 15

How Do You Choose Your Instrument?• Whichever method you select, what instrument will

you use?• “Borrow” vs. develop• Does it ask the “right” questions to get at your desired

outcome?• Does it have all the necessary components?• What information will each question yield? • How will you use information, if not related to outcome?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 16

What is “Qualitative” Data?

• Describes or characterizes through words• Focuses on meaning, experience or attitudes • Collected through focus groups, interviews, opened

ended questionnaire items, and other less structured situations.

• Not the same as anecdotal Information

Qualitative → Quality

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 17

What is “Quantitative” data?

• Focus on numbers and frequencies.• Data which can be measured• Length, height, area, volume, weight, speed, time,

temperature, humidity, sound levels, cost, ages, scores, etc.

Quantitative → Quantity

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 18

Example: How Do You Feel?

Qualitative Data Quantitative DataHave you felt sad or depressed at all lately, or have you generally been in good spirits?

Thinking about the past week, how depressed would you say you have been on a scale from 0 to 10, where 0 means "not at

all" and 10 means "the most possible

I’m not at all depressed. I feel great! I love my new job. I’ve lost 20 pounds and feel healthier than I have in years.

0

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 19

Are You Asking the Question You Mean to Ask?

Does your instrument or data collection method help you measure your desired outcome?•Example:

– If desired outcome = improved academic performance• DON’T measure attendance or attitude toward school• DO measure improved proficiency in a subject

VALIDITY

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 20

Validity Example• Desired Outcome (O11): Economically disadvantaged

individuals transitioned into safe, healthy, affordable housing

• Data to collect (from NOFO):– An inspection report and certificate of occupancy, – proof of residence such as lease or mortgage, or– other verification from an external agency that the work

was completed and is being occupied might be used.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 21

Validity Example (cont.)• Which questions are most likely to yield the desired

information?– Do you feel safe in your new home?– Can you afford this house?– Do you like this house?– Do you have a lease or mortgage for this house? Who

holds the lease/mortgage? Please share a copy of the signed paperwork.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 22

What is a Standardized Test?• A test that:

– is administered and scored in a consistent or “standard” manner.

– has been validated externally on a randomly-selected population

• Need not be “high-stakes” tests, time-limited tests, or multiple choice tests– e.g., state administered proficiency tests generally should

NOT be used

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 23

Can It Be Repeated?

1. Does your instrument measure the same thing, the same way every time it is used?

2. Does every person collecting data use the instrument the same way? Have they been trained?

3. Are your instrument instructions clear so respondents have a similar frame to answer?

RELIABILITY

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 24

Reliability Example

• Intervention: Preschool children attend early childhood education programs providing school readiness activities in three areas (social/emotional development, literacy and numeracy skills) 4 days a week for 4 hours/day

• Desired Outcome (ED23): Children demonstrate gains in social and/or emotional development.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 25

Reliability Example (cont.)

• Observations:– Open-ended Question: Does the child seem well adjusted

and ready to attend kindergarten?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 26

Instrument Pilot Testing

• Test before using - with small number of people similar to those who will respond

• Pilot test analysis should look at: – Were the questions clear enough? Did people understand

what you were asking?– Do the answers seem appropriate given what you were

asking?

• Make revisions based on results of pilot

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 27

Is Something Systematically Off?Bias:• Problems with WHO you ask

– sampling bias– response rates, etc.

• Problems with HOW you ask– method inappropriate – construction of your instrument

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 28

What is Bias?• Measurement bias occurs when information

collected for use is inaccurate.• Bias may be introduced by poor measurement design

or poor data collection.• Bias cannot be “controlled for” at the analysis stage.• Bias risks readers drawing conclusions that are

systematically different from the truth.• Bias can lead to an over or underestimation of an

effect.

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 29

Revisit: What to Look For?

• Are you measuring what counts/matters? • Is your measurement approach credible?• Are your instruments valid?• Are your instruments reliable?• Are your measurements precise?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 30

Instrument Mapping• Look at each question on your data collection tool and ask:

– Does this help us measure the desired outcome?• Is there one question? More than one?

– What kind of data will we get? • Subjective? Objective?• Quantitative? Qualitative?

– If it doesn’t measure the outcome, do we really need to ask it? How will we use the answer?

• Nice to know but won’t use it? Internal use? – How will we analyze this?– What is our target? How much change is “enough”?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 31

Target Example: Pre/Post test

“Mentored children will enhance developmental assets in the areas of social competence and positive identity.”

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 32

Do You Have a Summary Sheet w/Target?

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 33

After You Get the Data, then What?

• Do you have a plan for who and how the data collected will be aggregated and summarized?– Given the types of questions on your tool(s), is this

realistic?• Which data DIRECTLY relate to the desired outcome?

NOTE: Just because you asked it, doesn’t mean it helps you report on your outcome…don’t confuse people!

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 34

Data Collection and Aggregation

04/10/23 Data Collection and Aggregation 35