Innovative Synthesis Methods

Innovative Synthesis Methodologies

Donna Smith-Moncrieffe Canadian Evaluation Society Conference

Fairmont Royal York, TorontoJune 9-12, 2013

Innovative Synthesis Methodologies

Donna Smith-Moncrieffe Canadian Evaluation Society Conference

Fairmont Royal York, TorontoJune 9-12, 2013

2

Summary of the PresentationSummary of the Presentation

NCPC Mandate

Government Context Evaluation considerations at the corporate level (Performance) Evaluation considerations at the project’s level (Knowledge products)

Overview of 3 Types of synthesis methods Key points Challenges

Case Study (Step by Step Process) Collating data from projects with different target groups, interventions and outcomes Synthesis tables Providing reporting options for summarizing various statistical results: percentages,

Effect Sizes, T tests and F coefficients

Conclusions

3

Government Context: Evaluation Approaches Government Context: Evaluation Approaches

4

Why did we need to Synthesize the Data?Why did we need to Synthesize the Data?

● We needed to respond to the questions about, “what works in crime prevention”

● We needed to provide findings and conclusive statements about 13 programs (total of 36 project sites) with different:

Target groups;

Interventions;

Outcomes; and

Evaluation designs

● Single site repeated design

● Quasi-experiment design (sometimes with a matched comparison group)

5

Types of Synthesis MethodsTypes of Synthesis Methods

6

Synthesis Method Requirements for quality and RigorSynthesis Method Requirements for quality and Rigor

7

What approach did we use to Synthesize 13 Evaluations?

What approach did we use to Synthesize 13 Evaluations?

8

Step 1: Identify Key QuestionsStep 1: Identify Key Questions

What behaviour related changes were made (i.e. changes in violent offending, police contact and victimization?

What percentage of projects made favorable changes in relation to the criminal justice system (i.e. reduced violent offending, police contacts etc..)?

How much change was made?

What contextual and program related factors contributed to these changes (i.e. geographical location, partnerships, resources etc..)?

9

Step 2: What are the inclusion criteria?Step 2: What are the inclusion criteria?

● In the Government context, we need to use as many evaluation studies as possible in the synthesis; even when they do not meet rigid evaluation standards.

● Evaluation studies that met the following minimum requirements were included in the synthesis:

Study uses quantitative data (descriptive or inferential statistics);

Measures that correspond with key crime related outcomes (i.e. association with anti-social peers, emotional regulation, substance abuse, offending, education, employment etc..);

Study used at least the basic evaluation design (single group pre-post test repeated measures design); and

At least pre and post test measures had been available

10

Step 3: Enter the Data (Partial Sample)Step 3: Enter the Data (Partial Sample)

Name of Study

Description of Variable

Sample Size

Result P Level Type of Change

Level of Study

Project Intervention Toronto

Arrests E=76C=43

F=0.291 P=.823 No Change

Exemplary

Multi-Systemic Therapy

Youth not arrested for an offence

E=28C=12

E=89.3%C=66.7%

N/A Favorable Basic

Towards No Drugs

Weapon Carrying

E=847C=54

F=0.63 P<0.01 Favorable Notable

Velocity Police contacts

E=87 69%(reduction)

N/A Favorable Notable

Interpretation of Results

11

Step 4: Analysis and InterpretationStep 4: Analysis and Interpretation

Synthesis for 9 evaluation studies measuring changes in criminal behaviours

Overall interpretation

Background Information•Number of evaluation studies: 9•Number of sites: 11•Common measures include: 1) Arrests; 2) Non-Violent Offending; 3) Violent offending; 4) Criminal Victimization; and 5)Weapon Carrying

Quantitative results•% of change by sub-measure: 50%•% of projects demonstrating favorable change (showing at least one stat. significant result at the project level: 75%•% of projects demonstrating unfavorable change:0%•% of projects demonstrating no change: 25%

Qualitative results•Provide qualitative results by model, site or project level to qualify the quantitative findings 12

Step 5 : Reporting Aggregate DataStep 5 : Reporting Aggregate Data

13

Step 5: Reporting Aggregate Data (Sample of Approach # 1)Step 5: Reporting Aggregate Data (Sample of Approach # 1)

14

Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change ) Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change )

15

ConclusionsConclusions

● What synthesis method works best when the evaluation studies have a variety of interventions, outcomes and statistical measures?

Elements of the Realist Synthesis review and Multiple-case Study Synthesis are recommended; and

NCPC utilized elements of both of these synthesis methods but could benefit from responding to the Why questions incorporated in the Multiple- Case Study Synthesis

● What reporting method works best to summarize non-standard data?

Approach #1 and # 2 are both useful; and

Assess your audience to determine the best reporting method

16

Conclusions cont’dConclusions cont’d

● What planning activities would be required to utilize both Realist and Multiple Case study Methods?

Develop strategic direction (strategic plan to help focus the direction of the evaluation);

Prioritize key areas to minimize complexities;

Ensure that the information management system is designed to:

● Clarify the unit of analysis (by model, by project, evaluation questions, etc..);

● Respond to How and Why questions such as: How does the program produce these effects or How does the intervention contribute to change?; and

● Collect data about the context for each model, site or project.

17

References for Recommended Synthesis MethodsReferences for Recommended Synthesis Methods

18

19

Thank YouThank You

Contact:

Donna Smith-Moncrieffe, BSc. Crim Dip, MSc.

Senior Evaluation Advisor

Public Safety Canada, National Crime Prevention Centre

Policy, Research and Evaluation

Tel: 416-952-0423

Fax: 416-952-0483

Email: [email protected]

Web : www.PublicSafety.gc.ca | www.SecuritePublique.gc.ca