Clearwater County Sixth Grade Forestry Tour: Program Evaluation Methods and Tools

Clearwater County SGFT Program Evaluation Jillian Marotz Moroney University of Idaho Bioregional Planning and Community Design

Roadmap • Program History • 2012 Program Evaluation

• Methods/Findings • Recommendations • Tools for Assessment of Participants

• Knowledge • Behaviors/Attitudes • Perceptions

• Lessons learned and suggestions

Program History

• Established 1961 • Completely free to participants • Three days, two nights • Teacher continuing education

certification through University of Idaho • Covers topics including:

• Forestry • Fish and wildlife • Range • Water Quality • Geology • Search and Rescue • Fire

Program History

• 32 students • 15 boys/17 girls • 11-12 years old • Have parents

who work in the logging industry: 9

• Born outside of Idaho: 19

2012 Forestry Tour

Program Evaluation

• Evening questionnaire asking

participants to rate program:

• Logistics

• Staff

• Facilities

July 10, 2012 Event Assessment

Tour Event Mean Standard

Deviation Median Mode

Pierce Pool 3.97 0.16 4 4

Empire Lumber 3.74 0.44 4 4

Outdoor Skills 3.44 0.75 4 4

Resource Careers 3.28 0.76 3 3

Dwarshak Fisheries 3.21 0.73 3 3

Wetlands 3.18 0.88 3 4

Nez Perce Fisheries 3.13 0.70 3 3

Tree ID 3.05 0.94 3 4

Rodeo Grounds 3 0.83 3 4

(1=Poor, 2=Fair, 3=Good, 4=Excellent) N=32

Participant Assessment of Program : Day 1

Wetlands

NezPerce Fisheries

Tree ID

Rodeo Grounds

• Heat/Sun • Instructors • Activities

July 11, 2012 Event Assessment

Tour Event Mean Standard

Deviation Median Mode

Fire Control and Airplane Drop 3.95 0.22 4 4

Mining Demo 3.92 0.27 4 4

Search and Rescue 3.68 0.53 4 4

ID Dept. of Lands 3.51 0.61 4 4

Pierce Museum 3.42 0.67 3 3

Jaype Mill Pond 3.39 0.68 3.5 4

Potlatch Forestry Management 3.39 0.69 3.5 4

Forestry Consulting 3.32 0.70 3 4

Group Activity 3.22 0.76 3 3

NRCS 2.89 0.77 3 3

(1=Poor, 2=Fair, 3=Good, 4=Excellent) N=32

Participant Assessment of Program : Day 2

• 95% Said that they learned something this day

• Events with hands on

activities scored highest

July 12, 2012 Event Assessment

Tour Event Mean Standard Deviation Median Mode

Bertha Hill Lookout 3.92 0.28 4 4

Outdoor Challenge 3.57 0.61 4 4

Natural Resource Activity 3.51 0.61 4 4

Group Activity 3.4 0.65 3 4

Woody Biomass 3.2 0.80 3 3

Logging Operations 3.06 1.01 3 4

Tree ID 2.94 1.01 3 4

(1=Poor, 2=Fair, 3=Good, 4=Excellent) N=32

Participant Assessment of Program : Day 3

64% of participants said that the lookout was their favorite part of Thursday.

http://idahoptv.org/outdoors/shows/eyesoftheforest/berthaHillLookout.cfm

The Tree ID quiz scored the lowest out of any activity from the whole tour.

• Staff • Adults • Teens • Guest Speakers

Participant Assessment of Program : Day 3

Staff Assessment

Staff Member Mean Standard Deviation

Median Mode

Adults 3.75 0.82 4 4

Guest Speakers 3.74 0.61 4 4

Teens 3.57 0.65 4 4

(1=Poor, 2=Fair, 3=Good, 4=Excellent) N=32

Table 5: Facilities Assessment

Tour Event Mean Standard Deviation

Median Mode

Breakfast 3.97 0.17 4 4

Lunch 3.75 0.55 4 4

Dinner 3.89 0.40 4 4

Name Badges 3.41 0.82 4 4

Tarps 3.31 0.75 3 3

Bathrooms 2.75 0.97 3 2

Transportation 3.47 0.70 4 4

Campfire 3.72 0.45 4 4

Schedule 3.28 0.70 4 4

Group Activities 3.41 0.78 4 4

Games 3.74 0.56 4 4

Skits 3.46 0.82 4 4

(1=Poor, 2=Fair, 3=Good, 4=Excellent) N=32

Overall Favorite Meal: Pancakes and Spaghetti

Lowest Ranking Facility: Bathrooms

Comments: “More Hiking!” and “Less Hiking!”

• Keep time of day in mind when planning the schedule

• Hands on activities with lectures • More training/screening of teen leaders • More free time in the evenings

Recommendations

• Communicate program themes to the presenters • Connect students to Idaho natural resources • Increase student interest in natural resources • Communicate what impact humans have on

nature and natural resources • Increasing the comfort and connection students

have with the outdoors

Recommendations Cont.

• Knowledge gain and retention (20) • Multiple Choice • True/False

• Attitudes/Behaviors (10) • Likert Scale

• Perceptions (2) • Drawing • Knowledge rating

• Demographics (5) • Gender • Parental Occupation • Town/ County

Knowledge Assessment Instrument

Pre-Test Delayed Post-Test Post-Test

Administered spring 2012 to all sixth grade students in Clearwater County, Idaho.

Administered fall 2012 to all seventh grade students in Clearwater County, Idaho.

Administered at the end of the three day Forestry Tour to all participants.

• 20 Questions • Questions submitted by presenters, program

organizer, state and federal curriculum • Example:

Knowledge Gain and Retention

What are plants that annually lose their leaves called?

a. Deciduous b. Nocturnal c. Naked d. Coniferous

What is woody biomass used for? a. Generating electricity b. Heating homes and other buildings c. Making liquid fuel d. All of the above

Fish have a very good sense of smell.

a. True b. False

Pre and Post-Test Averages Pre-Test Post-Test

Control Average: 11.9 Participant Average: 12.2

Control Average: 11.8 Participant Average: 13.8

Control N= 32 Participant N= 25

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Tes

t S

core

Participants

Participants Pre, Post, and Delayed Post-test Scores

Pre-Test Post-Test Delayed Post-Test

Male Mean: 12.8 Female Mean: 11.7

Male Mean: 13.7 Female Mean: 13.9

Male Mean: 14.1 Female Mean: 13.5

N Males= 12 N Females= 13

Participant Averages by Gender

• 10 Statements • Statements from extensive literature review,

adapted to age group • Example:

Attitudes/Behaviors

STRONGLY DISAGREE

DISAGREE DON’T KNOW

AGREE STRONGLY

AGREE

I am interested in forestry and natural resources

1 2 3 4 5

I talk about natural resources with my friends

1 2 3 4 5

I feel good when I am outside in nature

1 2 3 4 5

I feel a strong connection to the forests in Idaho

1 2 3 4 5

0

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10

Rat

ing

Question Number

Change in Behaviors and Attitudes

012345

1 2 3 4 5 6 7 8 9 10

Rat

ing

Question Number

Post-Test

0

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10

Rat

ing

Question Number

Pre-Test C

ontr

ol v

Par

tici

pan

ts B

ehav

iors

an

d A

ttit

ud

es

• “Draw yourself in a forest.” • Looked for before and after differences in

drawings • Tree number/type • Natural features

Forest Perceptions

“Draw yourself in a forest.”

Pre-Test Post-Test Delayed Post-Test

• 18/25 (72%) students who attended the tour had more in depth drawings

• 13/32 (41%) who took only pre and delayed post test showed more in depth drawings

Pre-Test Post-Test Delayed Post-Test

Pre-Test Post-Test Delayed Post-Test

Cam

p P

arti

cip

ants

Dra

win

gs

Pre-Test Delayed Post-Test

Pre-Test Delayed Post-Test Con

trol

Gro

up

Dra

win

gs

Lessons and Suggestions • 2012 Program Evaluation

• Methods • Follow up questions to yes/no • Communicate with presenters program goals

and themes • Consider weather, age group, activities…

• Tools for Assessment of Participants • Many tools (knowledge, behaviors, attitudes,

perceptions, demographics….) • More tools means more data- pros and cons

Jillian Marotz Moroney

University of Idaho [email protected]

Thank you!

Thesis Committee

Dr. Tamara Laninga Dr. Randy Brooks

Dr. James Connors