Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.

Evaluation DesignDesigning & Refining Questions

Ben SillimanYouth Development Specialist

NC 4-H

Who Knows What?

• Youth (over age 3) can report their– Feelings, interests, experiences, goals, opinions/

attitudes on knowledge and skills– (reliable observers may give opinions on these)

• General Rules for who answers what:– You can’t know what you don’t know (thus

retrospective or observer ratings usually better)– No one knows your feelings, experiences, and

attitudes/aspirations better than you(let respondents speak for themselves)

Who Knows What?

• Youth (over age 3) can complete valid, developmentally appropriate tests of– Knowledge, Skill, Practice(reliable observers may also give opinions on

these)

• How to know what others know– Demonstration, description, or defense

(authentic assessment) gives best evidence…esp. if judged by trained, reliable observers using the same criteria given to youth

Thinking About Questions

Adjust for Differences in Self-Administered vs. Interviewer-Administered Questionnaires

• Self: make it simple (Eighth Grade), straightforward, and logical. Q1-> Q2 (complex skip patterns, crowded pages, complex instructions delay and discourage respondents)

• Interviewer: accommodate skip patterns, allow for diverse lines of questioning, interaction about ideas, objects, events (note interviewer skills)

Thinking About Questions

Order questions for smooth flow– Be forthcoming about purpose and

sponsor– Group topic or type of questions– Move from general to specific questions– Ask the easy questions first-build rapport– Ask more complex or thought-provoking

questions near the middle—use rapport– Ask sensitive questions near the end-ditto

Thinking About Questions

Phrase questions carefully to avoid bias

• “Leading” the respondent– What do you like about… vs

– What, if anything, do you like…(not assuming respondents like

something)

Thinking About Questions

Keep audience focused• Avoid multiple-option items

– Did public speaking help you gain confidence and skill?

• Avoid complex phrases(Answer Never-Seldom-Often-Always)– This 4-H program did not have an

appropriate level of discipline

Thinking About Questions

Adjusting questionnaire length, remembering that– A relevant, purposeful study can

motivate cooperation and thought– A clear and smooth-flowing routine can

focus and energize respondents– -A respectful and gracious approach can

affirm respondents, making them feel valued

Which Questions to Use

• Open-ended: no set responseIs there anything else you would like to

tell about your 4-H camping experience?– Communicates listening– Allows evaluator to see how

respondents prioritize and organize information

Which Questions to Use

• Close-ended: set response choicesFor the 4-H project areas listed, rank (1st,

2nd, 3rd) each according to your interest:___ Arts & Communication___ Citizenship & Leadership___ Family & Consumer Science___ Health & Nutrition___ Science & Technology

Styles of Close-ended Qs

Likert (esp. for feelings, attitudes)How important are cash prizes in your

decision to participate in 4-H events?Not at all VeryImportant

Important1 2 3 4

5

Styles of Close-ended Qs

Multiple Choice (best or all relevant options)

What is the best thing to do when you plan not to use a gun for a while? (choose one)

a) Cover it with a clothb) Place it on top of a cabinetc) Remove ammunitiond) Call law enforcement

Styles of Close-ended Qs

Ordinal (ranking of options)Rank these topics in order of importance

(1st, 2nd, 3rd, etc) for volunteer training___ Managing a 4-H club___ Teaching using experiential method___ Planning service activities___ Conducting parties___ Mastering program evaluation

Styles of Close-ended Qs

Categorical (identifying by group)Tell us about yourself. I am a a) Girl b) Boy

Numerical (identifying by number)Tell us about yourself. My age is ___ years