swords-to-plowshares.org VETERAN EMPLOYMENT COLLABORATION Program Evaluation: The Structured Interview
swords-to-plowshares.org VETERAN EMPLOYMENT
COLLABORATION
Program Evaluation:The Structured Interview
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
Introduction
GOALS: PROVIDE
• Introductory to intermediate level content• Easy-to-use guide • Lots of practical tips• Stretch your thinking• Opportunity for your input• Time for Q & A
TOPICS
• Structured interviews in context of survey tools• Best practices• Protocol design• Question types• Tips for interviewers• Dealing with data
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
Introduction
SURVEY TOOLS
• Open-ended interviews (rare)• Semi-structured interviews• Structured interviews• Self-completed questionnaire forms (on line or on paper)
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
Interviews vs. Questionnaires
INTERVIEWS QUESTIONNAIRES More labor intensive Less labor intensive
Higher skill level Lower skill level
Can follow-up for clarity Not possible to follow up
Richer data Thinner data
Questions & activities Questions only
Less anonymous (courtesy bias) Anonymous
Using Structured InterviewsTake advantage of opportunity to discuss/exchangeEmployment placementExit interviews for staff and clientsExploring potential partnershipsConsider addressing issues related to outputs (activities) as well as outcomes (resulting improvements to quality of life)
Using Structured Interviews
→ Take advantage of opportunity to discuss/exchange
→ Employment placement→ Exit interviews for staff and clients→ Exploring potential partnerships→ Consider addressing issues related to outputs (activities) as well as outcomes (resulting improvements to quality of life)
Best Practices – Research Planning
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
GATHER INPUT FROM STAKEHOLDERS o Funder requirementso In-house development departmento Those responsible for data collection and management
CONSENSUS ON PROJECT GOALSo Reason for researcho Research questionso Application of findings
Best Practices - Research
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
METHODOLOGYo Match the tool to the research goals and contexto Recognize strengths and limits of toolo Pilot test all phases of protocol (recruitment, data collection, data management, analysis)
o Sample sizeWORKING BILINGUALLY
o Appropriate for regiono Back translation to test instrumento Allow for translation of datao Native speakers preferred
Best Practices - Respondents
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
HSS human subject research guidelines (well-being, respect, protect)
Culturally responsible approach (more than just linguistics)
Accessible content
Don’t assume literacy
Structured Interview
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
PROTOCOL DESIGN - GENERIC
Similar rules of thumb for questionnaires. Brief explanation of the purpose. Start easy…
• Break the ice• Move toward more complex or personal content• Explore the positives of X before turning to
examine the negatives “Areas of Inquiry”
• Begin open-ended… then hone in with pre-meditated prompts.
Wrap-up – it can be useful to get direct re: research goal.
Structured Interview
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
PROTOCOL DESIGN - INTAKE
The respondent is likely to be seeking you out, rather than having been recruited for the interview.
The nature of this interaction, the intake is to collect/share lots of personal information. You might start off with demographics and service history, etc.
Caseworkers are comfortable working from unscripted forms.
Poll Questions
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
HOW WELL DO YOU THINK THE WEBINAR IS GOING?
1. Extremely well.
2. Very well.
3. Somewhat well.
4. It's okay.
5. Wait, there's something wrong with this question.
Structured Interview
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
QUESTION DESIGN Similar rules of thumb for questionnaires.
NO nice-to-know questions.
Clear, concise wording (don’t make respondent struggle to understand).
NO double-barreled questions.
Examine draft for bias, leading or judgmental verbiage.
Give permission to respondent to disagree or be candid.
Prepare appropriate prompts and follow-up questions in advance.
Keep fixed response choices simple, consider using visual aid.
Use fixed response questions as springboards for conversation.
Question Type:
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
OPEN-ENDED WITH PRE-MEDITATED PROMPTS
Example: What can you tell me about today’s workshop? Was it useful?
Useful prompts when the respondent needs encouragement: Did you get any new ideas? Can you describe that/one for me? Did you pick up any information that you think will be helpful? What information?
Did you get your questions answered? What questions did/do you have? Were there any highlights? Did anything jump out at you? Were you able to _________________________________?
Question Type:
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
OPEN-ENDED WITH PRE-MEDITATED PROMPTS
Example: What can you tell me about today’s workshop? Was it useful?
Possible follow-ups or probes: Was that new information for you? How do you think you might be able to use/follow up… Is there anything that could have made it more useful…
Question Type:
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
FIXED RESPONSE
Multiple Choice & Ratings Scales Keep simple Consider providing a printout of the scale graphic Use as a springboard for discussion: Why did you score X the way you did?
What are 3 words that describe the program you attended today?
You can let this go (great on questionnaire forms too) or follow up on one of the words they offer.
Question Type:
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
USING IMAGES
Reflect on images to encourage discussion Be careful of getting too concrete, respondents can fixate on the example.
Card Sorts Print a set of images or phrases on cards which respondents can sort into categories.
Preferences, likelihood of behaviors or choices, level of interest or familiarity, etc.
Tips for the Interviewer
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
USEFUL ALL-PURPOSE FOLLOW-UPS
Remember, context and tone of voice make all the difference. Can you tell me more about that… about X? What do you mean when you say X? Can you think of an example? Was that what you had expected? Why do you think it was memorable? What made you think that? Why do you think that was/happened? How do you feel about that?
Tips for the Interviewer
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
TECHNIQUE Consider yourself as a neutral but friendly extension of the survey instrument. Your professional, friendly manner helps the respondent feel confident.
After delivering your concise question, be willing to wait for a response, but be ready with prompts if needed.
Remember what you’ve learned about active listening and put it to use.
When probing for clarity, use respondents’ words back to them.
Once you get started, avoid narrating the process.
Tips for the Interviewer
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
TECHNIQUE
Avoid “leading” the respondent by suggesting ideas or vocabulary. They can feel influenced to agree with your suggestion and/or struggle to make it fit their experience.
Natural, conversational prompts are helpful, e.g., “Mhm.” Avoid comments which include suggestions of judgement, e.g., “That’s a great observation!” You can say, “I’m interested to hear more about that.”
Sometimes starting a sentence and purposely leaving it open is a good way to prompt a respondent. E.g.: So, you would say that it….? So, does that mean you would…?
Recording
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
Audio recording
Handwritten notes
Entering into database at keyboard (intake)
Data Management & Analysis
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
Custom Databases
SurveyMonkey
Microsoft Excel
Coding open-ended data = categorizing
Poll Questions
War causes wounds and suffering that last beyond the battlefield. Swords to Plowshares’ mission is to heal the wounds of war, to restore dignity, hope, and self-sufficiency to all veterans in need, and to prevent and end homelessness and poverty among veterans.
HOW WOULD YOU RATE THE USEFULNESS OF THIS WEBINAR?
1. Very useful.2. Somewhat useful.3. Not very useful.4. About as useful as this question is without opportunity
to follow-up about the rating I give.5. More useful than answer No. 4 suggests.
Thank Your For Your Attendance
Copyright © 2017 by Swords to PlowsharesAll rights reserved, including the right of reproduction in whole or in part in any form.
Swords to PlowsharesThe California Veterans Employment and Training Collaborative 1060 Howard StreetSan Francisco, CA 94103Amy [email protected]
Wendy MeluchPrincipal & Founder [email protected](415) 897.4051