Program evaluation 2013

Program Evaluation – Final Report

Program Evaluation- Final Report

Georgia Southern University

FRIT 8435 Program Evaluation

Lara Komanecky & Lisa Witteman

Dr. Carlson

Spring 2013


Table of Contents

I. Executive Summary………………………………………………………………………………………………………………….…4

Purpose and Scope …………………………………………………………………………………………………………………….4

Audience…………………………………………………………………………………………………………………………………….4

Data Collection and Analysis………………………………………………………………………………………………………4

Findings…………………………………………………………………………………………………………………………………..5-6

Recommendations………………………………………………………………………………………………………………………6

II. Introduction………………………………………………………………………………………………………………………………..6

Purpose of the Evaluation…………………………………………………………………………………………………………..7

Design of the Evaluation……………………………………………………………………………………………………………..7

III. Methodology

Methods of Data Collection………………………………………………………………………………………………………..7

On-Task Behavior Chart…………………………………………………………………………………………………………..7-8

Picture Survey………………………………………………………………………………………………………………………….8-9

Pre-and Post Modified KWL for Sixth Graders……………………………………………………………………………..9

Photo Interview Questions for Elementary Students…………………………………………………………………10

Interview Questions for Sixth Grade…………………………………………………………………………………………10

Limitations of Evaluation…………………………………………………………………………………………………………..11


IV. Results……………………………………………………………………………………………………………………………………..12

EvaluationFindings……………………………………………………………………………………………………………………12

Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings………………………………….12

SEN K-2 Visual Intrinsic and Extrinsic Survey Findings………………………………………………………………13

On-Task Behavior Findings………………………………………………………………………………………………………..14

Photo Conference Questions Findings……………………………………………………………………………………..15

V. Conclusions and Recommendations…………………………………………………………………………………………15

VI. References………………………………………………………………………………………………………………………………..16


I. Executive Summary

Purpose and Scope

Interactive white boards (IWBs) are used in almost every classroom in this school district. They

are used across content areas and have a variety of applications and uses. IWBs are currently

used to promote learning in K-6 reading programs. The scope and purpose of this evaluation

was to determine whether IWBs impact students’ attending behaviors and organizational

strategies in K-6 reading programs. Student behavior logs, student interviews, SEN surveys and

Photo “wh” questions were used to determine effectiveness of the IWBs.

Based on the percentage data results of this evaluation, we found that the use of interactive

white boards in K-6 reading programs does improve students attending behaviors and

organizational strategies; therefore, this program should be maintained.

Audience

This evaluation took place from February 22, 2013 to April 5, 2013 in the KW School System and

seeks to inform the stakeholders at KW School System on the impact of IWBs on K-6 reading

programs.

Data Collection and Analysis

Data were collected through behavior logs, interview sessions, self-reporting and surveys of 58

students participating in K-6 reading programs. Behavior logs were tracked during white board

segments, while daily observational data was collected by the evaluator. All of the evaluation

participants were observed for attentiveness during their academic period. In the study,

attentiveness was identified and operationally defined as active learning (i.e., looking at the

speaker, focusing on the whiteboard, utilizing books, clipboards/dry erase boards, manipulating

props and materials). Student’s off task behavior was identified and operationally defined as

disengaged (i.e., playing with clothes, shoes, distracted, looking up at the ceiling, etc.).

Attention to task was discussed at the beginning of each lesson and defined by the students as

well. Data was recorded on a 5 minute-interval tally chart during a thirty-minute lesson. Active

learners received a plus sign (+) and disengaged students received a minus sign (-). Weekly

conferences were held for general education students, during which students completed photo

interview questions assessing information retrieval. Participating sixth grade students

completed pre and post evaluations to determine if there were correlations between IWBs and

students’ utilization of organizational strategies. Participating AU students took part in a three

session visual survey to determine if there were correlations between IWBs and students’

utilization of organizational strategies. Data herein is compiled into several charts, displaying

results. These research methods were used with both an AU self-contained group of students

and an inclusion group.


Findings

The findings of the evaluation are listed below, categorized into four areas.

Student Behavior Logs *(IWB = Interactive white board, WWB=with white board & WOWB=without white board)

Daily 30 minute behavior logs were utilized by observers, at 5 minute intervals for six

weeks to determine students’ on-task behavior. Both Group A and B were assigned

numbers for student identification. Data on all eight students in Group A were recorded

daily. Data on the 48 students in Group B was randomly taken due to class size. The

qualitative data was tabulated weekly to measure on task behavior with an IWB and

without IWB with a baseline of zero and a goal of 85%. Time on task was reported

higher with the IWB in both the AU self-contained group (WWB=72%, WOWB=65%) and

the inclusion group (WWB=78%, WOWB 52%). Although the researchers did not make

their intended goal of 85%, there was active growth in each group. These data

fluctuations were attributed to outliers of age and development of students, class-size,

pre CRCT and post CRCT variables, and prior knowledge fluency.

Interviews

A modified version of the KWL format was employed before and after the study with the

sixth grade inclusion group to measure the impact of IWB lessons on students’

confidence and expression of detail and interest. The qualitative interview questions

were intrinsic and extrinsic in nature, and open-ended. The responses were rated on a

3-point rubric scale (1 – little detail, 2- some details, 3 – very detailed). Pre-Evaluation

results were; averaged intrinsic responses, (1.75) and averaged extrinsic responses,

(2.25). Post-Evaluation results were; averaged intrinsic responses, (2.5) and averaged

extrinsic responses, (2.66). The results were compared and a percentage growth rate

was tabulated. Intrinsic responses increased by .75 and extrinsic responses increased by

.16. These data validated that IWB lessons contributed to students’ increased

confidence and expression of detail and interest.

SEN Survey

A qualitative SEN survey was used with the AU self-contained group. The survey was

visually designed (the employment of visual answers in place of verbal or written

responses), to measure how the IWB influenced their intrinsic and extrinsic motivation

in correlation with their implementation of organizational and retrieval strategies. An

emotion survey rating scale (1-4) was employed for the intrinsic questions (1-2) and

numbers were assigned for the students’ responses for the extrinsic based questions (3-

6). These data detected the students’ underdeveloped expressive and receptive

language skills, which is common in their pervasive disorder, in their extrinsic based

responses. Not unexpectedly, the percentages for extrinsic based responses yielded < -


.5 regression. In their intrinsic based responses, a significant negative correlation was

discovered in the 1.75 percent growth rate. These data percentages displayed the

participants dislike for conventional means of work tasks and an enjoyment for work

tasks on or including the IWB. The data validates that the employment of the IWB

impacts students’ intrinsic motivation, which theoretically, enables them to implement

organizational and retrieval strategies.

Photo “wh” Questions

The study included utilizing photographs to assess students’ engagement and retrieval

process during IWB lessons. Once a week students were shown a picture of their class

during an IWB reading large group lesson. All students were asked a combination of

questions, some open-ended and story element questions. They were instruction to

give detailed responses. The responses were rated on a 3-point rubric scale (1 – little

detail without explanation, 2- some details without explanation, 3 – very detailed with

descriptive explanation). The results were averaged each week and those averages were

divided into six. The final results were Group A (2.25) and Group B (2.5). Interestingly,

both groups remained relatively consistent with neither displaying a significant increase

or significant regression. With Group A, the photo questions showed that, although the

students could recognize themselves in the pictures and familiar stories, they were

unable to express detailed accounts. With Group B, the students exhibited engagement,

maturity and enjoyed detailing their photos.

Recommendations

1. IWB lessons should be implemented in K-6 reading programs, if they are not already.

2. K-6 reading programs already incorporating IWB lessons should do so at an increased level.

3. KW School System should encourage increased use of IWBs in reading programs.

4. Research should be continued to determine the effectiveness of IWBs on other academic

programs.

II. Introduction

The majority of classrooms in KW School District contain Interactive White Boards (IWBs). These

white boards are often used in reading programs. This program evaluation sought to determine

the effectiveness of IWBs in K-6 reading programs. Two evaluation questions drove this study:

a. Can the incorporation of an interactive white board impact students attending

behaviors during guided reading?

b. Can the incorporation of an IWB increase students’ (general education and SWD)

utilization of organizational strategies?


Purpose of the Evaluation

Stakeholders agreed that the impact of IWBs should be evaluated in K-6 reading programs.

These findings will be shared with faculty and staff to determine future use of IWBs in K-6

reading programs, as well as the use of IWBs in other content areas.

Design of the Evaluation

Two student groups were included in this study. Group A consists of eight self-contained autistic

students. Group B consists of 48 students in inclusion classrooms. Of the 48 students in group B,

19 receive gifted services, 10 receive special education services and one receives both gifted and

special education services. Data collection methods were the same for both groups. Behavior

logs were tracked during white board segments, a modified version of traditional KWL (what you

know, what you want to know and what you have learned) was utilized for interview questions

(Group A was given super symbols to answer interview questions), and photo questions were

used with both groups.

III. Methodology

Methods of Data Collection

Data were collected from February 22, 2013 to April 5, 2013. Behavior logs were tracked for the

duration of the study, six interview sessions took place from March 1st to April 5th, self-reporting

for sixth grade students took place on February 22nd and April 5th, and SEN surveys were

distributed three times, March 1st, March 15th and March 29th.

Figure A. On-Task Behavior Chart

The chart below was used to record on-task behavior with and without an IWB on all

participants.

With IWB

Days of the Week Monday Tuesday Wednesday Thursday Friday

Totals

Average

Number __ +/- Number __ +/- Number __ +/- Number __ +/- Number __ +/-


Without IWB

Days of the Week Monday Tuesday Wednesday Thursday Friday

Figure B- Picture Survey

The picture survey below was used for elementary students.

Picture Survey Questions: Student Number____ Time______ Date________

When I use scissors to cut out story pictures then glue stick to glue them in a square, I feel…….

1 2 3 4

When I use the white board and pen to move story pictures to a square, I feel……..

1 2 3 4

I can find the title of a book and the author using a…………

1 2 3

I can see the bigger words, story characters, story settings, when I look at…

1 2 3

Totals

Weekly Average

Number __ +/- Number __ +/- Number __ +/- Number __ +/- Number __ +/-


When I watch stories on the white board and see the big words and pictures, I……

1 2 3

It is easier to use my story checklist at my seat looking at the story on the whiteboard or book

1 2

Figure C - Pre-and Post- Modified KWL for Sixth Graders

Prediction and What I Know- Bubble Head (Intrinsic) Prediction and What I Know-verbally respond (extrinsic)

* Will you learn anything using the interactive whiteboard (IWB)?

1 – little detail 2- some details 3 – very detailed

• What new reading skill will learn a using the IWB?


• Will you be patient taking turns at the board?


• Do you think the IWB will change the way you think about reading? How?


• Have you ever put your own work on the IWB?


•Who would you tell in another school: a teacher/child to use the

IWB?


• Do you think other children/teachers/parents learn with the IWB?


• Do you think using the IWB for reading will be a good idea?

Why?


• Do you think it won’t be fun to use the IWB? 1 – little detail

2- some details 3 – very detailed

• Would you feel embarrassed to show anyone your work on the IWB? Why? 1 – little detail 2- some details 3 – very

detailed

• How do you feel about working on the IWB? Why? 1 – little detail


• What grade would learn the best using the IWB?1 – little detail


*How did answering these questions make you feel?



Figure D- Photo Interview Questions for Elementary Students

Interview questions Elementary

Student Number:____ Time:_____ Date:_________

Candid pictures of the students during their reading segment were employed to ask the following

questions….

What are you thinking in this picture?

1 – little detail (Student did not remember specific details) 2- some details (Student remembers 1-4

specific details) 3 – very detailed (Student remembers and describes specific details)

Do you remember the name of the story?

1 – no title 2- half title 3 – full title

What was your favorite part of that story? Why?

1 – little detail without explanation 2- some details without explanation 3 – very detailed with

descriptive explanation Total_____ Average______

Figure E - Interview Questions- 6th grade

Interview questions-6th Student Number:____ Time:_____ Date:_________

What are you thinking about in this picture?


When you look at the picture: Do you remember the day off the week?


What lesson is it?


What is going on in the story?


Who was the main character?


Total_____ Average______


Limitations of the Evaluation

Although IWBs are used in various programs throughout the school, this study focuses on their

effectiveness in reading programs only. The number of student participants in this study is also

limited. Due to time constraints, a larger number of students could not be assessed. Data

analysis tables were not preformed due to the limited sample population and relevant past

research was given honorary mention in the appendix for inspiring and guiding the study, but

not directly cited in the evaluation.

IV. Results

Findings

Research showed that IWBs positively impact students attending behaviors and organizational

strategies, as displayed in Pre-and Post-Sixth Grade Modified KWL Chart, SEN K-2 Intrinsic and

Extrinsic Survey chart, on-task behavior chart and photo questions survey chart, found in the

appendices.

Figure F - Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings

0

0.5

1

1.5

2

2.5

3

1

2

3,2

,1 A

vera

ged

Ru

bri

c Sc

ore

s

1 2

Pre 1.75 2.5

Post 2.25 2.66

Pre and Post Sixth Grade KWL Results

First and Last Week


The Pre and Post KWL Interview data were utilized to determine IWBs’ impact on students’

personal and academic (intrinsic and extrinsic) organizational strategies. These data illustrated a

positive correlation between students’ intrinsic and extrinsic organizational strategies with the

use of IWBs in reading programs.

Figure G - SEN K-2 Visual Intrinsic and Extrinsic Survey Findings

Elementary students’ intrinsic scores significantly improved with the utilization of IWBs in their

reading programs. These data can signify that the students are personally invested in their

academic tasks.

0

0.5

1

1.5

2

2.5

3

3.5

1

2

3

3,2

,1 A

vera

ged

Ru

bri

c Sc

ore

s

1 2 3

Group A SEN Intrinsic 3.25 3 3.5

Group A SEN Extrinsic 1.5 2 1.5

SEN K-2 Visual intrinsic and Extrinsic Survey

Number of Survey Weeks in


Figure H - On-Task Behavior Findings

All groups showed an increase in on-task behavior with IWB lessons.

0

0.5

1

1.5

2

2.5

3

1 2

3 4

5 6

1 2 3 4 5 6

Group A 2 2 2 2 2.5 3

Group B 2.25 2.5 2.25 2.75 2.75 2.5

Photo Conference Questions

Group A

Group B

Week Number

3,2

,1 A

vera

ged

Ru

bri

c Sc

ore

s


Figure I - Photo Conference Questions Findings

V. Conclusions and Recommendations

Because research showed improved on-task behavior and improved organizational strategies

with the incorporation of IWBs into K-6 reading programs, the following recommendations are

made:

1. IWB lessons should be implemented in K-6 reading programs, if they are not already.

2. K-6 reading programs already incorporating IWB lessons should do so at an increased

level.

3. KW School System should encourage increased use of IWBs in reading programs.

4. Research should be continued to determine the effectiveness of IWBs on other

academic programs.

5. Research should further explore the correlation between IWBs utilization in the

classroom and its impact on students’ beliefs about learning and metacognition.

Successful implementation of this study, in conjunction with past research and other school

improvement efforts, promises positive correlation with IWB’s and the students’ initiation and

continued self-regulation of attending (active learning) during lesson time, which research

states, improves students’ academic achievement and self-efficacy.

0

0.5

1

1 2 3

4 5

6

1 2 3 4 5 6

Group A NWB 0.56 0.63 0.59 0.65 0.71 0.74

Group A WB 0.62 0.67 0.73 0.77 0.75 0.81

Time on Task: Group B without IWB

0.4 0.33 0.55 0.65 0.7 0.5

Time on Task: Group B with IWB

0.75 0.7 0.7 0.85 0.9 0.7

Time On Task - with and without IWB

Week Number Ave

rage

d P

erc

en

tage

s -

Tas

k O

n T

ask


Reference Page

Honorary Mention

The following are articles that supported and gave perspective to the Evaluation Study.

Carnahan, C., Williamson, P., Hollingshead, A., & Israel, M. (2012). Using technology to support

balanced literacy for students with significant disabilities. Teaching Exceptional Children, 45(1), 20-29

Frauenberger, C., Good, J., & Keay-Bright, W. (2011). Designing technology for children with special

needs: bridging perspectives through participatory design. CoDesign, 7(1), 1-28. doi: 10.1080/15710882.2011.587013

McKenna, M., Labbo, L. & Reinking, D. (2003) Effective use of technology in literacy Instruction, in: L.

Morrow, L. Gambrell & M. Pressley (eds), Best Practices in Literacy Education, 2nd edn. (New York, The Guilford Press).

Solvie, P. (2007). Leaping out of our skins: postmodern considerations in use of an electronic

whiteboard to foster critical engagement in early literacy lessons. Educational Philosophy and Theory, Vol. 39, No. 7, 2007. doi: 10.1111/j.1469-5812.2007.00312.x

Program evaluation 2013

Documents