Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions is a critical phase of the evaluation.

Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions

is a critical phase of the evaluation process The essence of evaluation is generating

credible answers to questions about the performance if a social program.

Questions must be answerable with the resources available

Evaluation revolves around the hub of carefully crafted evaluation questions Questions need to be meaningful for

stakeholders and program decision makers Who will use the the evaluation results? What info do they need? How do they expect to use it? The evaluator’s own analysis of the

program is also important

Communication with decision makers and stakeholders Questions are formulated through

discourse and negotiation Engaging these groups increases their

understanding and appreciation, and makes effective use of the findings when available

Evaluators can’t depend solely on their perspectives – they are not experts on evaluation

Role of the Evaluator Framing the questions – the evaluator knows how

to analyze a program and focus an evaluation. Raise issues that might be overlooked Identify aspects of program’s operation and

outcomes that warrant inquiry Draw out concerns from stakeholders that can be

translated into questions that can be answered by evaluation research

Written summary of questionsHelps evaluator guide the evaluation design.

This is a useful reference for design and selecting research procedures.

Can be shared with stakeholders to ensure their concerns are addressed.

Safeguards against later misunderstanding of what the evaluation was suppose to accomplish.

The 2 most important topics related to evaluation questions: 1. How to formulate evaluation question

that can be addressed using research procedures available to the evaluator

2. How to determine specific questions that the evaluation should focus.

What makes a good evaluation question?

Focus the evaluation on the areas of program performance at issue for key decision makers and stakeholders

Facilitate design of data collection to provide meaningful info about performance

Identify a distinct dimension of program performance that can be credibly assessed

What it means to evaluate something Establishing criteria of merit – on what

dimension must the evaluand do well? Constructing standards – how well should should

the evaluand perform? Measuring performance and comparing with

standards – How well did the evaluand perform? Synthesizing and integrating data into a judgment

of merit or worth. What is the merit or worth of the evaluand

……a thought To evaluate anything means to assess the

merit or worth of something against criteria and standards.

Dimensions of Program Performance Good evaluation questions must first be

reasonable and appropriate The questions must be answerable –

specific, concrete, practical, and measurable

Reasonable and Appropriate Questions Work with stakeholders to scale down and

focus the questions – Lit Review – social science and social

service literature Get acquainted with the program – its

structure, activities, roles and tasks of personnel, nature of participants, and assumptions.

Questions must be Answerable Easy to formulate unanswerable questions

without realizing it….Does this program enhance family values?

Must be possible to identify some evidence or “observables” that can be obtained and will be credible as the basis for an answer.

Have measurable performance dimensions stated in unambiguous terms with non-controversial definitions.

Criteria for Program Performance What distinguishes evaluation questions is they

have to do with performance and are associated with criteria by which performance can be judged.

Performance Criterion (or standard) that apply, as well as the performance dimension at issue.

An evaluation that only describes performance, but doesn’t assess it, is not truly evaluation.

Standards by which program performance may be evaluated: Needs/wants of target population Program goals and objectives Professional standards Customary practice, norms Legal requirements Ethical/moral values: social justice, equity

Standards…continued Past performance; historical data Targets set by program managers Expert opinion Pre-intervention baseline levels for target

population Conditions expected in the absence of the

program Cost or relative cost

Places to look for performance criteria: Professional standards – esp in medical and

health programs Prior experience Evaluation and program literature Judgement ratings from stakeholders to

establish criterion levels or ranges (low to high performance)

Typical evaluation Questions

1. Need for program services (needs assessment)2. Program’s conceptulization or design

(assessment of program theory)3. Program operations and service delivery

(assessment of program process)4. Program outcomes (impact assessment)5. Program cost and efficiency (efficiency

assessment)

Order of the five categories or questions The questions draw meaning from the answers to the

prior questions. Implementation failure – poor outcomes due to

program activities needed to bring about desired improvements did not actually occur. Nutritional status of homeless people did not improve because soup kitchen was rarely open.

Theory failure- Program conceptualization and design can’t generate desired outcomes no matter how well implemented – soup kitchen was far away.

Evaluation Hierarchy – Assessment of: 1. Need for the Program 2. Program Design and Theory 3. Program Process and Implementation 4. Program Outcome?Impact 5. Assessment of Program Cost and

Efficiency

When developing questions Start at the bottom of the evaluation

hierarchy and consider what is known and what needs to be known about the issues

Then move up the ladder Logical interdependencies between the

levels Premature attention to higher-order

evaluation questions can be avoided

Specific Questions Evaluators Should Ask Don’t rely only on input from sponsors or

stakeholders – they are familiar with the program and can overlook critical, routine aspects or performance.

Evaluator should make an independent analysis of program performance pertinent for investigation

continued Multiple stakeholders = multiple views Set priorities with them and integrate as

many relevant concerns as possible If they have constraints, the evaluator must

decide whether or not to conduct the evaluation anyway or negotiate to broaden additional perspectives

Essential Strive to ensure stakeholders understand

and accept the nature of the eval process, the type of info it will produce, what it might mean if results come out one way or another, and what ambiguities or unanswered questions remain.

Snowball approach Identify and contact a person or group,

they then nominate others, who in turn nominate others.

The process ends when no new nominations are given.

Topics to discuss with Stakeholders Why the evaluation is needed The program goals and objectives What are the most important questions for

the evaluation to answer?

Backward mappingStart with a specification of the desired

endpoint and work backward to determine what must be done to get there.

Analysis of Program Assumptions and Theory Construct a conceptual model of how the

program is expected to work and the connections presumed between activities and functions and social benefits it is intended to produce.

Program theory is the set of assumptions about relationships between strategy and tactics and the social benefits it expects to produce.

Program theory is also called program conceptulaization, program plan, blueprint or design.

Logic Model Lays out expected sequence of steps going

from program services to client outcomes.

Collating Eval Questions and Setting Priorities Questions tend to cluster around program

functions – ex services or outcomes Also around issues – ex need, design,

impact When the evaluator prioritizes the

questions for a selected program, the next phase is design.

Summary Assess the need for a program Program theory or plan for addressing the need Implementation of the program and associated

process Impact or outcome of the program

implementation on the social need Efficiency with which the program attains its

outcomes