Evidence Based Programs

EVIDENCE BASED PROGRAMSDr. Carol Albrecht Utah State ExtensionAssessment Specialist [email protected]

Evidence Based Programs Relatively new approach Program decisions are based primarily

on scientific method and scientific evidence

Use best evidence available to make decisions about program

Recognizes that knowledge is provisional and subject to refutation

Evidence Based Programs combines (1) Program Manager’s Expertise, (2) Client Attributes, and (3) Research Findings

Research Findings

Client

Attributes

Program Manager’

s Expertise

EBP

Best Research Evidence Available Can Be Found in Current Literature

Step 1 – Review The Literature “Track down” the evidence- ongoing

process Note Inconsistencies Note research suggesting some

interventions are ineffective

Step 2 – Critique the Literature You Find Examine methods used*

Client Attributes Take values, concerns, expectations of

client into consideration Involve clients in decision making

process Determine if this program will “fits” your

client. Not “one size fits all” situation Consider race, gender, idiosyncratic

characteristics

Program Manager Expertise Use Common Sense Avoid “fallacies” which include

Overgeneralization - Stereotyping Selective Perception/Observation Ex-post Facto Hypothesizing

Provide alternative explanation after the “study”

Accepting “new” as better Accepting “traditional” as better

How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies

Step 4 – Apply the Literature to Your Question and/or Program

Step 5 – Conduct Your own Research

Step 1 – Formulate a Question What objectives do you want to

accomplish? What intervention is necessary to reach

objectives? Does Intervention A work? Is Intervention A cost effective? Does Intervention A work better than

Intervention B? Does Intervention A work for this type of client? Does intervention A work in this kind of

situation?

Useful Acronym for Formulating Question

Use acronym CIAO Client characteristics Intervention being considered Alternative intervention being considered (if

any) Outcome of intervention (does it work?)

Example Client Characteristics

If children involved in 4-H Intervention

…..personally interact with a mentor for several years…..

Alternative Intervention…..or communicate with a mentor over the internet…..

Outcome…..are they more likely to seek post-high school education?

Step 2 – Search for the Evidence

Helpful “hints” to finding articles Use Google or Google Scholar and type in correct

search terms Combine general terms (e.g., volunteers, evaluation etc.) With more specific terms (i.e., grades in high school, crop

yield etc.)

Top Down and Bottom Up Searches (Mullen 2006) Top Down – Find Articles/Books that have Compiled the

Literature (i.e., meta-analysis on your topic) Bottom Up – Find a very recently published article and

look at references.

Examples of Resources for “Top Down” Searches

*Campbell collaboration: http://www.campbellcollaboration.org/index.html

**Cochrane Collaboration: http://www.cochrane.org Substance Abuse:http://nrepp.samhsa.gov/htpp:/www.modelprograms.samhsa.gov/index.htm Crisis Intervention and Domestic Violence National Resource: http://www.crisisinterventionnetwork.com Expert Consensus Guidelines series: http://www.psychguides.com National Guidelines Clearinghouse http://www.guidelines.gov National Institute on Drug Abuse: http://www.nida.nih.gov/ Substance Abuse and Mental Health Services Administration:http://www.samhsa.govindex.aspx*Focuses on Social Welfare, Education and Criminal Justice**Focuses on Health Care Interventions

Step 3 – Critically Appraise Relevant Articles

Emphasis Should be on Methods Used to Collect Empirical Information

How to Complete a Literature Review

Step 1 – Formulate a Question

Step 2 – Search for Evidence

Step 3 – Critically Appraise Relevant Studies

Step 4 – Apply the Literature to Your Question and/or Program

Step 5 – Conduct Your own Research

Step 4 – Apply the Literature to Your Research Question

Which intervention currently has the best empirical support?

Is the situation same/similar to yours? Are the participants in the study the

same or similar to the ones in your program/study?

Step 5 – Conduct Your Own Research

Use Literature Review to help you develop valid and reliable indicators.

Use Literature Review to help you determine your sampling technique and sample size.

Use the Literature Review to help you determine the most appropriate type of data collection.

Use the Literature Review to help you determine ways you can further contribute to the research (i.e., what is missing?)

Evaluate impact of program on your participants

Concerns About Evidence Based Programs

Merely Cost Cutting Have to use the methods “in stock”

Evidence Based May not be Applicable Leads to rigid implementation Your participants may be different

Evidence is in Short Supply – not enough research Real World Obstacles Hinder Implementation of

Programs Time and money are scarce

Devalues Qualitative Research Argument that “Empirical Evidence” is not obtainable

Suggestions for Appraising Journal Articles in the Literature

Step 1 – Identify Purpose Stated purpose of article

This should appear in first paragraph(s).

Step 2 – Literature Review Is it adequate?

They should have at least 15 references.

Appraising Literature Step 3 - Identify objectives/outcomes

Do they identify and define their outcomes/objecties It must be clear and succinct. They must be clearly related to the

implementation of program. How do they measure their

concepts/outcomes? Are measurements valid? (face validity) Are measurement reliable (consistent)

Could you repeat study?

Appraising Literature Step 4 – Appraising Sampling Technique

It must be clearly specified? What is their unit of analysis? Do you know exactly how these units will be selected?

It must be appropriate?

Does the sampling technique “fit” with the data collection technique?

Do they draw valid conclusions based on sampling technique? Note – if they generalize they must have random

sample

Remember the Sampling Techniques

Random Sample – can generalize Simple or Systematic Random Multi-stage Cluster Stratified Random

NonRandom Sample – cannot generalize Theoretical or Purposive Snowball Convenient

Appraising Literature Step 5 – Research Design/Data Collection

Technique It must be clearly specified?

Do you know exactly how they will collect their data?

It must be appropriate Keep in mind the strengths and weaknesses of

the different research designs/data collection techniques

Remember Research Designs or Data Collection Techniques

Data Collection/Research Design Survey or Questionnaire Secondary Experimental

Lab Experiments Quasi – Field Experiment/Trial

Field Research (Qualitative) Participant Observation Content Analysis Intensive Interviews Case Studies

Appraising Literature Step 6 - Critique Tables/Graphs/Charts

Their analysis must be correct? Did they include descriptive statistics of the sample? Did they include inferential statistics and discuss level

of significance? Are their tables constructed correctly?

They must be interpreted accurately. Does the text accurately describe the data?

They should look professional Do they provide sufficient information?

Do they tell you how they measured the indicators used in the tables?

Appraising Literature Step 7 – Summary and Conclusions

Do they accurately summarize their research?

Do they identify strengths and weaknesses?

Do they acknowledge limitations? Do they make suggestions for future

research?

Easy Steps

1. Identify Purpose 2. Critique Literature Review 3. Identify Success 4. Sample Technique 5. Data Collection Technique/Research

Design 6. Tables and Charts 7. Summary and Conclusions

Contact Information Dr. Carol Albrecht Assessment Specialist Utah State University 979-777-2421 [email protected]