How to Increase Extension Program Outcomes and Impacts (Slides for Educators)

How to Improve Extension Program Outcomes using a Monitoring and Evaluation

Strategy

Dr. Ayanava MajumdarExtension Entomologist, SARE Coord. (AU)

111-A Duncan Hall, Auburn [email protected]

251-331-8416

Focus of this presentation• Structure & implementation of Alabama IPM

Program• Provide some basics of evaluation – Extension

perspective!• Encourage you to be consistent in your

evaluation efforts• Develop capacity building within organization

About the Author (Ayanava)Ext. evaluations:

• Memberships: American Eval. Assoc., Southeast Eval. Assoc.

• AEA365 Blog Curator (2010-2011), Organization Capacity Building TIG Grant Panel, Ext. Eval. TIG member

• Lead evaluator: USDA-NIFA, AL DoA, commodity grants, School IPM, AACAAS, NACAA

Organizational capacity building:

• Initiated ACES Program Evaluation Resource Committee, 2010… created the Evaluation Toolkit [online]

• Workshops, webinars, publications: total ~7 hr, trained 161 Agents, County Coordinators, Specialists, Ext. Administrators

Alabama Small Farms• Produce fruits & vegetables that

are consumed locally

• Area = 6,000+ acres

• Crop value = $48 million

• Growth rate for industry = 14%

• Vegetables: Tomatoes, sweet corn, watermelon, crucifers

• Farmer markets: 135 (1100 farms)

• Direct sales from AL farmers to consumers account for 0.2% of farm sales (Meter 2012)

Alabama Cooperative Extension System

Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension.

Regional Extension Agents (REAs)

Regional or statewide IPM events

Outputs change annuallyOutcomes do not change!Impacts now underway.

+ CECs

ACES Commercial Horticulture Team – Vegetable IPM Team Members

ACES Home Grounds Team – Vegetable IPM Team Members

Chris Becker, REA Willie Datcher, REA Mike McQueen, REA

Alfred Jackson, Tuskegee Extension

Awards & Recognition• ‘Friends of IPM - Pulling Together Award’ from the Southern

Region IPM Center, Raleigh, NC (2014) for IPM newsletter.• Friends of IPM: Future Leader Award (2012)

By Southern Region IPM Center, Raleigh, NCRecognizes excellence in IPM program directly

• Search for Excellence – Crops (2012)By National Association of County Agricultural AgentsRecognizes a high impact IPM program

• Achievement Award (2012)By the National Association of County Agricultural Agents

• Communication Awards (6) By National Association of County Agricultural Agents

Structure of Alabama IPM Program:A Model Program for Study

Fund acquisition (external)• Funds needed to support REA travel & IPM

demonstration plots

• USDA/NIFA Collaborative Grants: $124,500 (part of two large grants worth $2.1 million) = 6%

• SARE support: $10,000 + personnel for website

• Alabama Department of Agriculture: $25,000

• Industry support: $45,000

Theory of Planned Behavior (TpB)

Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes. 50: 179-211

Theory of Change: Ext. Logic Model

IPM Theory of Change

Communities

ImpactsModified from Kokate et al. 2009

Farmer-to-farmerTrain-the-trainer

Sustain change

Key farmers

Technology acceptance

ExtensionResearchProject Management

Capacity Building

Need identified(context)

Ext. IPM Project Implementation

Transformational

learning

(full tech.

adoption)

Information

transfer

(tech.

awareness)

Service

(technology

integration &

refinement)

Facilitation

(demonstration/

tech. transfer)

Keep informed(state ag dept)

Cooperate closely(key farmers) Network closely

(commodity group)

Collaborate(industry)

Continuous needs & outcomes assessments

Transformational education model Stakeholder expectation

1

2

3

4

1

2

3

4

Model by Dr. A. Majumdar 2011Based on Rockwell et al. 2003

Co

mm

un

icat

ion

Str

ateg

y (I

PM

-CO

RE

)

No cost tools

Technology identified & assessed

Story developed

Press releaseExtension Websites Social sharing

(Slideshare, YouTube)Social networking (Facebook)

Medium cost tools

IPM Communicator Newsletter

Extension regional meetings, IPM

Web Conferences

IPM Project flyers,

bookmarksHigh cost tools

Multiple partnerships (~20)

IPM Exhibitions,Newspaper

advertisements

IPM on-farm demonstrations/communications

Small Farm IPM Field

Guide

IPM App?

Level 1: Regional IPM Meetings• Participants: Vegetable producers, certified

organic farms, gardeners & homeowners, crop advisors, nonprofit organizations (partners)

• Teaching techniques: Presentation, pest specimens

• Typical length of talks: 46 min• Participants total: 1,535• Number of events completed: 58• Evaluation: Events are monitored for quality

and impacts via multimodal surveys

Regional IPM Training Meetings (Level 1 training)

Level 2: IPM Workshops

• Participants: Small vegetable producers, low resource farmers, Master Gardeners, nonprofit organizations, Extension Agents (in-service)

• Teaching techniques: Case studies, simulations, specimens

• Typical length of talks: 140 min• Participants total: 570 • Number of events completed: 14• Evaluation: Qualitative & quantitative feedback,

impact evaluations

Hands-on IPM Workshops (Level 2 training)

Level 3: IPM Field Days (Demonstration Sites)

• Participants: Producers, low resource farmers, gardeners

• Teaching techniques: Scouting, pest identification, decision-making system

• Typical training time: 3-4 hours• Participants total: 308• Number of events completed: 12• Evaluation: Qualitative & quantitative feedback,

impact evaluations, farm visits

IPM Field Days (Level 3 training)

Farmer-to-farmer training model is very influential for IPM project

Farmer-to-Farmer training during IPM Field Days

Causes high information retention & impact on communities

Basics of Program Monitoring & Evaluation (PME)

Evaluation• Evaluation is the systematic collection of information

about the merit, worth or significance of a program (Scriven 1999).

• Evaluation is broadly defines as the systematic collection of data – both quantitative and qualitative – to aid users in developing knowledge about & managing a targeted set of activities (Scheirer 2012).

• Ext. evaluation practice aims to find plausible conclusions, not cause & effect!

Rate your IPM program evaluation experience(useful exercise for audience)

Common myths!

• My clients like me, so I do not need to evaluate.

• Evaluation is hard & outside my job description.

• Public does not respond to my surveys.

• Nobody is going to look at the evaluations once they are done.

• Only ‘experts’ can do evaluation. Why bother!

PERCEPTION REALITY• Evaluation helps document your

success.

• 20% effort generates 80% information.

• Provide time to respond, part of agenda, show improvements!

• Share the results with all stakeholders.

• Evaluation can be done by any trained individual committed to the standards.

Who can do the evaluation?Internal evaluators

• Project leadership team directly guides evaluation (cost-effective)

• Indicates organizational commitment /capacity building

• Project monitoring possible (utilization-focus)

• Fast assessment of reactive programs

• Good participation

External evaluators

• Neutral third-party observers (expensive)

• No internal capacity building

• Good for impacts

• Slow assessments & less useful

• Poor participation (trust)

Based on Patton (1997)

CDC Framework for Program Evaluation Standards

Thomas Chapel, Chief Eval. Officer, CDC

Purpose of Evaluation

Formative evaluation• Done when the program

is active• Project monitoring• Continuous use• Flexible (mixed) designs

Summative evaluation• Done at the end of a

program• Measure impacts• One time use• Rigid design, expensive

Patton (1997), Scriven (1972)

Utilization-focused Evaluations• Evaluation technique emphasizing USE (Patton 1993)

• Improvement-oriented evaluation: includes formative evaluations, quality enhancement, Total Quality Management (TQM)

• TQM includes using information systems to monitor program efforts & outcomes continuously

• Use feedback as a monitoring system (like insect traps!).

+ Trouble= = Do something about it!

Evaluation techniquesReactive assessment:• Participants are aware of the

assessment• Examples: surveys, interviews,

tests

Non-reactive assessment:• Participants are unaware of the

assessment• Examples: observation

Haas (2005)

Choose outcome ‘indicators’ carefully

• Indicators are critical questions• Based on program objectives• Used for monitoring progress• Usually expressed as rates, percentages,

efficiencies, etc.• There should be few trackable indicators

(QUAL and QUAN)

Taylor-Powell & Henert 2008

Needsassessments

Processevaluations(formative)

OutcomeEvaluations(formative)

Impactevaluations

(summative)

Project monitoring

Project improvement

Modified strategy

Modified outputs

Evaluation Strategy in Extension IPM Program

Graphic by Dr. Ayanava Majumdar, ACES

What do we measure?Hierarchy of Outcome Effects

• Level 1: REACTION • Level 2: LEARNING

• Level 3: BEHAVIOR• Level 4: RESULT

Increasing complexity of

evaluation

Evaluate during program

Kirkpatrick Four Levels, 1959, 1994

Post program evaluation

QUAN

QUAL

Hierarchy of Effects in IPM Program (In Pictures)

• Level 1: REACTION

• Level 2: LEARNING

Hierarchy of Effects in IPM Program (In Pictures)

• Level 3: BEHAVIOR

• Level 4: RESULTS

AL IPM ProjectOUTCOMES

(Usefulness of Project Monitoring)

Project outcomes (2008-2012)• Developed a unique Theory of Change and

Transformational Extension Education Model based on long-term outcomes

• New producers reached 14% and rising (transitioning farms 7-10%).

• General IPM adoption rate: over 90%• Adoption rate for unique IPM tactics: 63% (up from

38%)• Publication in print & electronic have significantly

reduced barrier to IPM adoption • Funding for Small Farm IPM program has doubled

since 2012. Much focus on Ext. Agent training.

• Rising interest of producers/Pull on project:– Over 10 times rise in participation in IPM events– 10 times rise in number of IPM events

• Barriers to IPM adoption (crop advisors):– Lack of awareness reduced by 33% in 3 years– Difficulty in accessing information reduced by 33%

• Barriers to IPM adoption (producers):– Lack of awareness reduced 16% in 3 years– Difficulty in accessing IPM information reduced 13%

Project outcomes (2008-2012)

IPM Theory of Change

Communities

ImpactsModified from Kokate et al. 2009

Farmer-to-farmerTrain-the-trainer

Sustain change

Key farmers

Technology acceptance

ExtensionResearchProject Management

Capacity Building

Need identified(context)

2009 2010 2011 20120

100200300400500600700800900

1000

67

259

505

945

39 54 57 553 4 28 27

Total participants

Survey return rate (%)

Major IPM Events

GROWING INTEREST OF FARMERS IN IPM

2009 2010 20110

10

20

30

40

50

60

70

80

1.56 6.9

73

36

26

58 10.18

3429.1

117

20.9

Beginning farmers (%)Conventional producers (%)LRFs (%)Transitioning farms (%)Gardeners (%)

NATURE OF AUDIENCE AT IPM EVENTS

2009 2010 20110

5

10

15

20

25

30 28

13.6

9.3

Average land holding (acres)

CHANGING NATURE OF EXTENSION CLIENTELE(reason for splitting IPM program into two campaigns to

increase impacts)

Organic/Small Farms IPM

Campaign

Conventional farmers(with 15 to 50 acres)

2012

Farmers(<10 acres)

Gardeners(<2 acres)

Conventional/Large Farms IPM

Campaign

Conventional vegetable farmers = 15-50 acres

Transitioning farmers = 10-14 acres

Beginning farmers = <10 acresLow resource farmers = 2 to 4 acres

CHANGING NATURE OF EXTENSION CLIENTELE(reason for splitting IPM program into two campaigns to

increase impacts)

Two IPM Campaigns launched in 2012 with separate strategies and outcomes. Rapid increase in small

farm IPM funding and number of participants due to rising interest in local food systems.

2008 20110

5

10

15

20

25

30

35

40

45

5045

29

10

23.2

30

2.2

20

11.4

2.55

Lack of awareness (%)

High cost (%)

Difficult to access information (%)

Low availability of insecticides (%)

Lack of time (%)

BARRIERS TO IPM ADOPTION & CORRECTIONS

Major concerns at the time

PROJECT IMPACTS(Watch 2013 Impact Video at http://

youtu.be/aqrjQINLUdw)

Impact Evaluation, 2010-2011 (n=58):

• E-subscriptions growth rate = 12%

• Current subscription base ~1350 with much higher readership.

• 34% farmers, 28% company representatives, 38% others

• 53% read for 15 minutes and 22% for 30+ minutes

• 94% support continuation of the newsletter (2011 survey)

• 70% use the information (2013 survey)

• Six financial gains reports (2013): $3,550 in pesticide saving

• Short-term impact of publication: $591 per case of adoption

IPM Newsletter Impacts

Program Impacts: IPM AdoptionIndicator 2011 2010 2009 Increase

Adoption of insect monitoring/scouting practices

73.5% 41.0% +32%

Adoption of insecticide recommendations

79.6% 46.5% 33%

Use of biological insecticides

42.3% 26.5% 16%

Overall IPM adoption 62.7% 38.5% 24%

Economic & social impact assessments ongoing in 2012 and beyond.

Received IRB Approval for Surveys.

Economic impacts• Yield loss in the absence of IPM: 44% or

more• Vegetable producers gained $246 per acre by

using IPM recommendations• Profits occur by judicious use of pesticides,

adoption of appropriate control tactics after pest identification, and timely action.

• Weekly IPM newsletter resulted in saving of over $500 per adoptive farmer

• Increased use of biological insecticides (42%)

• Conservation of natural enemies (unknown)

• Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action.

Environmental impacts

• Three new partnerships with influential commodity & consumer organizations

• New partnerships with nonprofit agencies serving LRFs in Black Belt of Alabama

• Impact assessments continue in 2012 & beyond.

Social impacts

Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used the IPM recommendation from the handbook and received training from Dr. A. In the hoop house alone, we sold about $5,500 worth of tomatoes which is double the output from last year. Fruitworms and stinkbugs used to do about 50% yield loss but not this year. With abundant production, we are taking our produce to large farmer markets in Birmingham and Decatur, not limiting ourselves just to the farm stand.” (Surveyed on July 5, 2012)

Danny Dickie, vegetable producer (Oneonta, AL): “We use the insecticide recommendations from the SE Vegetable Handbook and consult Mel Wade before we make a treatment decision. Without insecticides, we can potentially lose over 80% of our crops to worms and stink bugs that may result in over $20,000 loss per acre.” (Surveyed on May 30, 2012)

Albert Riddle, vegetable producer (Titus, AL): “We use the integrated pest management recommendations provided by Chip East – our Regional Extension Agent. I can lose over 50% of my tomato crop if I did not follow IPM for insect and disease management. That is about $10,000 loss per acre.” (Surveyed on May 23, 2012)

AND it all comes down to writing good reports…

Questions for Dr. A?Thank you for your patience!