Evaluation Report of Big Brainz' Timez Attack Prepared for Big Brainz, Educational Gaming Company Prepared By The IP&T 661: Intro to Evaluation in Education Class, Spring 2010 Taught by Dr. Richard E. West Instructional Psychology and Technology Department David O. McKay School of Education Brigham Young University, Provo June 15, 2010
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Evaluation Report of Big Brainz' Timez
Attack
Prepared for
Big Brainz, Educational Gaming Company
Prepared By
The IP&T 661: Intro to Evaluation in Education Class, Spring 2010
Taught by Dr. Richard E. West
Instructional Psychology and Technology Department
International Student Assessment, 2005; Rosas, 2003; Sedighian & Sedighian, 1996; Shaffer et
al., 2005). This will, in turn, increase students‘ self-efficacy, thereby reducing their math anxiety
(Berk, 2009; Jackson, 1999; Jackson & Leffingwell, 1999; Meece et al, 1990; NRC, 1989;
Sarason, 1980; Scarpello, 2007; Swetman, 1994; Tankersley, 1993). They will also gain greater
mathematical automaticity enabling them to learn more complex math with less effort (NMAP,
2008; Shaffer et al., 2005). However, how they will benefit from these positive effects depends,
in part, on the real-world aspect of the games they use (Carter & Norwood, 1997; Federation of
American Scientists, 2006; Jackson, 1999; Meece et al., 1990; NMAP, 2008; NRC, 1989, 2001;
Scarpello, 2007; Shaffer et al., 2005).
Methodology
Research Design
Our evaluation team collected data using two studies that addressed TA's potential influence on
mastery of the 12s time tables and self-efficacy toward learning multiplication facts.
Multiplication with 12s was chosen for observation and analysis because the teachers had
already taught times tables 1-11 to all of the participants. We employed a mixed methods
approach in both studies in order to collect, code, and analyze the data. We collected the data
using a pre/post 12s times test, pre/post Multiplications Attitude Survey (MAS) and field notes
including observations and informal interviews.
Study 1 included 3rd-grade participants whose teacher had also previously taught the 12s
multiplication facts. Due to limited computer access participants were randomized into two
separate groups. However, since both groups had the same experience this study had no control
group. Pre/post-test instruments were used to measure changes in ability and self-efficacy of
participants after a 30-minute intervention with Timez Attack. As there was no control group we
employed a single group pre/post-test research design. This design is illustrated in Figure 1
below where PRE=pre-test, POST=post-test, and X= Observed Intervention (30-minutes of
playing Timez Attack).
Figure 1 -- Research Design for Study 1
All Participants
PRE
X
POST
Study 2 included 2nd and 3rd-grade participants whose teacher had not yet taught them the 12s
multiplication facts. Students were randomly assigned to an experimental or control group by
generating a randomized table. Study 2 used an experimental pre/post-test research design, as
illustrated in Figure 2, where R=Randomization, E=Experimental Group, C=Control Group,
PRE=pre-test, POST=post-test, Y=Traditional Teacher-Led Instruction, and X=Observed
Intervention (30-minutes of playing Timez Attack).
Figure 2 -- Research Design for Study 2
Group 1 (R,E)
PRE
X
POST
Y
Group 2 (R,C)
PRE
Y
POST
X
POST2
Participants
Participants in Study 1 included 26 3rd-grade students. Participants in Study 2 included 12 2nd-
grade and 10 3rd-grade students all from the same elementary school. Prior to data collection,
participants from both studies used Timez Attack during computer lab time. Study 1 participants' teacher had previously taught them the 12s times-tables while Study 2's teacher had not.
Research Methods
Quantitative Research Measures
We used the teachers' existing instrument to calculate and asses student proficiency of
multiplication facts, consisting of a 100-item multiplication facts test limited to 12s. It included
both 12 X (a number from 1-12) and (a number from 1-12) X 12 problems. It was distributed
before and after the TA playing session, or teacher-led instruction in the Study 2 control group.
The same instrument was used as both a pre- and post-test and can be found in Appendix A.
A pre/post-Multiplication Attitudes Survey (MAS) was developed to access (1) attitudes towards
learning multiplication facts and (2) attitudes towards and perceptions of Timez Attack. The pre-
MAS contained eight questions. The post-MAS contained twelve questions. Questions 1-6 on
both the pre-MAS and post-MAS were designed to measure students‘ self-efficacy, along with
physiological and emotional states involved in multiplication math fact mastery. Questions 1, 2,
4 and 5 were identical while 3 and 6 were only slightly different on the pre- and post-tests.
Questions 7 and 8 on the pre-test were designed to provide information regarding home learning
of multiplication facts and previous exposure to Timez Attack. Questions 7-12 on the post-tests
elicited information regarding perceived utility of and motivation to use Timez Attack.
Response options for the first seven questions on the pre-test and first ten questions on the post-
test involved a five-point Likert-type scale. Five possible responses were provided -- ―NO, no, ?,
yes, YES‖ -- with the two extreme options being in bolder type to make it easier for students to
interpret the rating choices. Instructions on how to interpret the scale were verbally explained to
students in advance by saying, ―Circle the big ‗YES‘ if you think what the statement says is
completely true and you very much want to say yes, circle the little ‗yes‘ if you think it is
basically true,‖ and so forth. Similar methods of administration were used by Nichols et al.
(1990). Response options for question 8 on the pre-test included four options: ―never, a few
times, many times, almost every day.‖ (See Qualitative Research Methods regarding open-ended
questions 11 and 12 on the post-MAS)
Likert-type questions measuring self-efficacy and psychological and emotional states regarding
multiplication fact mastery were adapted from the Fennema-Sherman Survey (Fennema &
Sherman, 1976). This is one of the most widely used math attitude surveys in math education.
Changes to Fennema-Sherman survey questions included changing the word ―math‖ to
―multiplication math facts.‖ Other adaptations were done in order to make the questions age-
appropriate (see Table 1). The complete pre- and post-MAS can be found in Appendix A.
Table 1 -- Likert-type Questions Pre/Post MAS
Index Statements Pre/Post
Self-Efficacy
I can learn new multiplication math facts easily.
I am good at memorizing multiplication math facts.
Remembering new multiplication facts is hard for me
Pre/Post
Pre
Post
Psychological/Emotional states I like learning multiplication math facts. Pre/Post
Learning new multiplication math facts scares me.
I usually do not worry about remembering
multiplication math facts.
Learning new multiplication math facts makes me feel
bored
Learning new multiplication math facts makes me feel
uneasy and confused.
Pre/Post
Pre
Post
Pre
Home influence I study my multiplication math facts at home.
Pre
Exposure to Timez Attack I have played Timez Attack.
Post
Perception of TA Utility and
Motivation to Use
I will play Timez Attack at home.
I think playing Timez Attack would help me know my
multiplication math facts.
Learning to play Timez Attack was easy.
Playing Timez Attack is a fun way to learn my
multiplication math facts.
Post
Post
Post
Post
Qualitative Research Methods
Qualitative data collection methods included participant responses to open ended questions 11
and 12 on the post-MAS (see Table 2), field note observations, and short informal interviews.
Field notes were taken during all stages of the studies. There was no specific focus for the
observations, and observers did not have a specific checklist of items to watch for. Rather,
observers were instructed to note anything that seemed interesting about the students‘ actions
and statements while participating in our evaluation. So, though we shared knowledge of the
evaluation questions, our observations may be particular to the individual observers.
During Timez Attack gameplay, however, observers did particularly note participants,
enjoyment, frustrations, improved self-efficacy and learning (including conceptual
understanding) of multiplication facts. Interview questions, also not specifically predetermined,
were intended to illicit participants‘ feelings and attitudes toward learning multiplication facts by
playing Timez Attack, along with their feelings toward Timez Attack game play in general.
Some questions regarding learning with Timez Attack included, ―Why do you throw the balls at
the gate?‖ ―Why do you pick up the spiders?‖ and ―Do you think Timez Attack is useful?‖
Other questions meant to determine Timez Attack game play satisfaction included, ―If you could
change anything about Timez Attack what would you change?‖ and ―What did you like most
about playing Timez Attack?‖ All student responses to open-ended questions as well as field
note observations and interviews can be found in Appendix B.
Table 2 -- Short Response Indices and Statements Post-MAS
Index Statements Pre/Post
Timez Attack Utility/Attitude What I love about Timez Attack is… Post
What I hate about Timez Attack is… Post
Data Collection Procedures
All participants were given a three-minute test on the 12s times table in their regular classroom,
followed by the MAS pre-test. All Study 1 participants then played thirty minutes of Timez
Attack on the 12s level. After playing the game, Study 1 students again took the 12s timed test
followed by the MAS post-test. During all stages of data collection, researchers took field notes
including observations of and short interviews with students. Table 3 illustrates data collection
procedures for Study 1.
Table 3 -- Study 1 data collection procedures Time Study 1 (Prior 12s learning) 15 minutes
All Participants - Times-test and Pre-MAS Observations and Interviews
30 minutes
All Participants - Timez Attack (12s level) Observations and Interviews
15 minutes All Participants - Times-test and Post-MAS Observations and Interviews
In Study 2, after administrating the pre-test to all study participants, Group 1 (the experimental
group consisting of six 2nd graders and five 3rd graders) played the multiplication by 12s level
of Timez Attack for thirty minutes. During this same time Group 2, the control group (consisting
of six 2nd graders and five 3rd graders), were taught the 12s time table through teacher-led
instruction.
Teacher instruction was carried out in the students‘ regular classroom by their classroom teacher,
and consisted of multiple representations of multiplication. The teacher helped students access
prior knowledge about arrays, applied that knowledge to the new fact (12s), discussed the
problem-solving strategy of breaking up 12s facts into the sum of 10s and 2s, and drilled using a
kinetic game involving rhythm.
At the end of thirty minutes Group 1 and Group 2 were again given a three-minute 12s times test.
The post-MAS was then administered only to Group 1. At this point in Study 2, Group 1 went
back to their classroom for teacher-led instruction and Group 2 was also allowed to play the 12s
level of Timez Attack. As with Group 1, Group 2 played Timez Attack at level 12 for thirty
minutes and was given the 12s times test followed by the post-MAS. Similar to Study 1,
researchers recorded observations of and short informal interviews with Study 2 participants
throughout the various stages of this experiment. Table 4 illustrates data collection procedures
for Study 2.
Table 4 -- Study 2 data collection procedures
Time Study 2 (No prior 12s learning) 15 minutes
Group 1 and 2 participants - Times-test and Pre-MAS Observations and Interviews
30 minutes
Group 1 (experimental group)- Timez Attack (Level 12) Observations and interviews Group 2 (control group)- Teacher instruction of 12s Observations and Interviews
15 minutes Group 1 - Times-test and Post-MAS Group 2 - Times-test
30 minutes
Group 1- Teacher instruction of 12s Observations and interviews Group 2- Timez Attack (Level 12) Observations and interviews.
15 minutes Group 2 - Times-test and Post-MAS
Data Analysis
Quantitative Data Analysis
In scoring the times tests, evaluators determined students‘ overall correctness and the correctness
of each individual 12s math fact (i.e. 12 x 1, 12 x 2 …12 x 12) by considering any written
answer to a problem as an "attempt," every correct answer as a "correct response," and every
incorrect attempt as an "error." These definitions allowed evaluators to measure changes in
attempts, changes in the number of correct responses, changes in the number of errors, and ratio
of errors to attempts, or the "error rate." Changes in correct responses, errors, and the error rate
were calculated and also analyzed using Excel in order to measure the achievement of mastery
by the students. Changes in attempts may reflect not only mastery, but also changes in student
self-efficacy.
The coding of Likert-type questions (YES, yes, ?, no, NO) from the pre/post-MAS ranged from
numbers 1 to 5 with 1 coinciding with ―NO‖ and 5 coinciding with ―YES.‖ Coding of ―never, a
few times, many times and almost every day‖ response question ranged from 1-4 with 1 being
―never‖ and 4 being ―almost every day. ‖
All Likert-type responses and changes in identical Likert-type student pre/post-MAS responses
were then calculated, summarized, compared and analyzed using Excel tables and graphs.
Information resulting from pre/post-times test analysis and Likert-type pre/post-MAS can be
found in written, table and graph form in the Presentation of Results section of this report.
Qualitative Data Analysis
At least two evaluators coded each open-ended question response, observation, and interview,
and sought agreement on the interpretation. Codes were not pre-determined but emerged during
data collection and analysis. A complete list of categories used in coding qualitative data, along
with other tables and graphs used in qualitative data analysis can be found in the Presentation of
Results section of this report.
Presentation of Results
Summary of Findings
The primary questions to be answered in this evaluation were
1. Does playing TA improve multiplication fact mastery?
2. Does playing TA contribute to positive changes in students‘ multiplication self-
efficacy?
Our evaluation showed an improved proficiency in multiplication of 12s amongst students who
played Timez Attack. The tables and discussion that follow will explain the observed
differences. On average, students answered 11-13 more questions correctly on the post-test after
they played the game than they scored on the pre-test.
Our evaluation also showed that there was a positive change in students' self-efficacy toward
multiplication. The tables and discussion that follow will explain the observed differences in the
self-efficacy measures as well.
Timez Attack and Student Mastery of Multiplication Facts As discussed in the methodology section, the evaluation took the form of two separate studies.
Study One
The first study was an experimental group consisting entirely of students in the third grade and
did not have a corresponding control group. The only treatment applied to the experimental
group between the pre and post tests was playing Timez Attack. This group had already studied
the 12s and many had "passed off" these facts in normal classroom instruction. The results of the
evaluation study on this group showed students attempted an average of 10.3 more problems on
the post-test than on the pre-test. The students answered an average of 11.1 more problems
correctly, made an average of .8 fewer errors, and reduced their error rate from 2.8% of attempts to .06% of attempts.
Table 5 -- Students in Study #1
Description Pre-Test Post-Test Change Avg Change
# of Students in Class 24 24
# Questions
Attempted 968 1,215 +247 +10.3
Attempted (% of
Possible1)
40.3% 50.6%
Correct Answers 941 1,207 +266 +11.1
Correct % (of
Attempts) 97.2% 99.3%
Errors 27 8 -19 -.8
Error Rate (of
Attempts) 2.8% .06%
1 Total Attempts possible is 100 problem per test multiplied by the number of students
Study Two
The second study conducted for this evaluation differed from the first study in three critical
aspects:
1. The students are in a class composed students in the 2nd and 3rd grades.
2. The students had not yet studied the 12s multiplication facts.
3. The study students were divided into an experimental group and a control group.
The experimental group was taught and practiced the 12s facts using Timez Attack. The control
group was taught and practiced the 12s facts through instruction from the classroom teacher.
Students received pre- and post-tests to evaluate the affect of the different modes of instruction
and practice.
Experimental Group
The results from the experimental group (Table 6) showed that students attempted an average of
11.5 more problems while answering an average of 12.9 more problems correctly. The students
also made an average of 1.5 fewer errors and reduced their error rate from 21.7% to 6.6% of
attempts following their Timez Attack experience.
Table 6 -- Students in Study #2 - Experimental Group
Description Pre-Test Post-Test Change Avg Change
# of Students in Class 11 11
# Questions
Attempted 161 287 +126 +11.5
Attempted (% of
Possible1)
14.6% 26.1%
Correct Answers 126 268 +142 +12.9
Correct % (of
Attempts) 78.3% 93.4%
Errors 35 19 -16 -1.5
Error Rate (of
Attempts) 21.7% 6.6%
1 Total Attempts possible is 100 problem per test multiplied by the number of students
Control Group
The results from the control group (Table 7) showed that students attempted an average of 10.5
more problems while answering an average of 15.1 more problems correctly. The students made
an average of 1.5 fewer errors and reduced their error rate from 35.2% to 4.4% of attempts
following the classroom instruction received from their teacher.
Table 7 -- Students in Study #2 - Control Group
Description Pre-Test Post-Test Change Avg Change
# of Students in Class 11 11
# Questions
Attempted 179 295 +116 +10.5
Attempted (% of
Possible1) 16.3% 26.8%
Correct Answers 116 282 +166 +15.1
Correct % (of
Attempts) 64.8% 95.6%
Errors 63 13 -50 -4.5
Error Rate (of
Attempts) 35.2% 4.4%
1 Total Attempts possible is 100 problem per test multiplied by the number of students
Additional Question of Interest
Study group two consisted of one class that was a combined class of 2nd and 3rd grade students.
The random assignment of students into the experimental and control groups resulted in groups
that were approximately the same of 2nd and 3rd grade students in each group. When combined, there were 11 students in each grade who were part of the evaluation study.
The evaluation team considered the possibility that grade level may influence the performance of
students. In order to address this question, further analysis segregated Study Two's groups into
2nd or 3rd grade students in order to examine the results more granularly (Table 8 and Table 9).
These data show 3rd grade students achieved significantly higher improvement than the 2nd
grade students in both the experimental and control groups.
3rd grade students in the experimental group attempted an average of 16.7 more problems,
answered an average of 17.2 more problems correctly, and reduced their error rate from 18.7% of
attempts to 6.9% of attempts following game play. This compares to the results for 2nd grade
students in the experimental group who attempted an average of 5.8 more questions, answered
and average of 7.8 more questions correctly, and reduced their error rate from 27.8% to 6.0%.
Date for the control group show similar results. The 3rd grade students attempted an average of
12.2 more problems, answered an average of 19.0 more problems correctly, and reduced their
error rate from 38.0% of attempts to 2.5% of attempts following teacher instruction. This
compares to the results for 2nd grade students who attempted an average of 9.2 more questions,
answered and average of 11.8 more questions correctly, and reduced their error rate from 30.1%
to 2.1%.
The data held consistent in grade level comparisons following the final assessment where 3rd
grade students attempted 14.6 more problems on average while answering and average of 19.5
more problems correctly and reducing the error rate from 28.0% to .03%. The 2nd grade students
attempted an average of 9.5 more problems, answering and average of 12.6 more problem
correctly and reducing their error rate from 30.1% to 2.1%.
Table 8 -- Students in Study #2 - 3rd Grade Students Only
1 Total Attempts possible is 100 problem per test multiplied by the number of students
Table 9 -- Students in Study #2 - 2nd Grade Students Only
1 Total Attempts possible is 100 problem per test multiplied by the number of students
Description
Experimental Group Control Group Combined Treatments Pre-Test
All of the data analyzed suggest that Timez Attack (both instead of and combined with teacher-
led instruction) supports significant positive results for students learning multiplication facts,
similar to results from teacher-led classroom instruction.
Timez Attack and Student Self-Efficacy In addition to the pre- and post-tests of multiplication facts, we employed survey instruments,
observation, and student interviews to assess students' experiences playing TA. Our focus in this
section is on students' mathematics self-efficacy, which is their sense of their abilities to master math facts and their general learning capabilities.
The surveys were distributed to Study 1 as well as Study 2 (both experimental and control
groups). Table 12 summarizes the analysis of the survey responses to these questions.
The student responses to question 1 suggests an increased positive feeling that "I can learn new
multiplication math facts easily" for all participants. Study 2's experimental group showed a
much greater increase in this sense of self-efficacy than the control group, and significantly
higher than students exposed to TA in Study 1.
Students who played Timez Attack--both in Study 1 and in Study 2's experimental group--agreed
more with the statement "I like learning multiplication facts" after playing Timez Attack,
whereas student agreement with question 2 declined in Study 2's control group.
Responses to question 4, "Remembering new multiplication facts is hard for me", showed nearly
identical increased disagreement in both control and experimental groups, suggesting that self-
efficacy in terms of memory simply improved with practice.
Question 5 attempted to measure anxiety about math, stating "I usually do not worry about
remembering multiplication math facts". Agreement with this statement went up in both Study 1
and Study 2's experimental group, but down slightly in Study 2's control group.
Table 12 -- Student Responses to Likert-type Self-Efficacy Survey Questions
Question 1: I can learn new multiplication facts easily.
Question 2: I like learning multiplication facts.
Question 4: Remembering new multiplication facts is hard for me.
Question 5: I usually do not worry about remembering multiplication math facts.
Findings from Qualitative Data:
Seven observers recorded their impressions watching students engage with TA, while one
observer recorded her observation of the control group. Students also responded to two open-
ended post-survey questions about their experience with TA. We coded all statements and, in the
course of coding, 14 main codes emerged. They are described below, with any relevant sub-
codes and examples taken from the data:
Table 13 -- Descriptions and Examples of Codes
Code Description Examples
Comparison
The statement represents a students‘
comparison of Timez Attack to another
game or another type of mathematics
learning activity. Such statements were
sub-coded as either Positive or Negative,
with Positive indicating a preference for
Timez Attack, and Negative indicating a
preference for another activity.
Positive: "I play a few other math
video games. It’s pretty good. Better.
… I like it more.”
Negative: "It was pretty good. It’s a
little bit less good because there’s
this [other] game I really like.”
Conceptual
Understanding
These observation notes give evidence
that the students might be developing a
conceptual understanding of the process
behind or meaning of multiplication as
repeated addition.
“It is 7X12 so since there is 7 balls in
each [and] I count by sevens to get
the answer.“
Emotional
Response
These statements include any action or
declaration by students that give insight
into what emotional response they may
be having to playing the game. Such
statements were sub-coded as either
Positive or Negative. Positive sub-codes
correspond with positive emotions (such
as enjoyment or happiness), while
Negative sub-codes correspond with
negative emotions (such as frustration or
confusion).
Positive: Student shouted, “This is
fun!”
Negative: It was obvious he knew the
math facts and was looking to use
them in the game, but the pace of the
game and confusing instructions
frustrated him.
Engagement
Statements with this code reflect how
engaged the students were while playing
the Timez Attack program. Statements
were sub-coded as Positive or Negative.
Positive sub-codes indicate active
engagement with the game, and Negative
sub-codes indicate disengagement from
the game.
Positive: He was engaged in the
game throughout the entire session.
Negative: Looked very bored pushing
the button to throw the balls. Not
looking at screen when doing this.
Game
Construct
This code focuses on the basic
appearance and design of the game. Sub-
codes are Graphics, Audio, Timing, and
Characters.
Graphics: When the colors shine out
at one point in a fantastical way, she
"ooohs."
Audio: The students interacted
differently with the game. [Some
students] used the headphones. [Two
others] did not.
Timing: “Umm... how when the
spiders come out, there are blue
things... it like turns blue near them
and then you can't get them--it takes
a long time to get them.”
Characters: Commenting about the
characters while playing it.
Gameplay
Statements given this code reflect the
dynamic and interactive nature of gaming
as students play Timez Attack.
Picked up the spider characters and
threw them over the edge of the
walkway opposite the wall where they
were supposed to be thrown for the
game. Played the game instead of the
math.
Hawthorne
Effect
These statements reflect an influence of
the evaluator/observer on the student
trying to take the test or play the game.
Very nervous about being watched.
Edgy each time someone is looking
over his shoulder. Looks up at person
watching and has a difficult time
proceeding with game.
Learning
These observation statements contain a
reference to students learning
multiplication facts using Timez Attack.
"I like how it's a fun game and you
can learn in a fun game."
Problem-
Solving
Strategy
These statements are about students
demonstrating any type of problem-
solving strategy to solve multiplication
problems.
Student was using his fingers as
manipulatives as soon as he
encountered a multiplication fact. He
actually took his fingers off of the
controls to figure the answer.
Self-Efficacy
Statements regarding students‘ self-
efficacy reflect how the student viewed
himself as a mathematician and how
competent he feels in his mathematical
abilities. Such statements might include
insights into a student‘s confidence, or
lack thereof, in doing mathematics. Sub-
codes were either Positive or Negative,
where Positive indicates a high level of
self-efficacy and Negative indicates a
lower level of self-efficacy.
Positive: “The game makes me feel
better at math.”
Negative: “This is hard… I can’t
think of it!”
Social
These statements involve an interaction
between two or more students or
observers while playing Timez Attack.
Such statements give insight into what
students are saying/doing to each other,
how they interact with those around them,
Asked her neighbor for help. He
showed her what to do.
and how their interactions are influencing
their use of Timez Attack.
Technical
Issue
Statements indicating issues with the
game not working due to problems with
the hardware/software.
Froze a bit, then started working.
Usability
These statements indicate how easily the
students are able to use the game and
achieve their goals. Sub-codes were
Positive or Negative, with Positive
indicating student facility with the game
and Negative indicating difficulties had
by the students.
Positive: Used mouse combined with
direction arrows on keyboard to
move more smoothly.
Negative: Still confused by game
controls and movements.
Usage
These observation notes deal with the
extent to which the student has used
Timez Attack in the past.
"I played it a lot last year at school.”
There were 335 total coded statements from these observation protocols, informal interviews,
and survey responses. Because multiple codes were allowed for some statements, a total of 486
codes were assigned. The following table and accompanying chart display the frequency of code
occurrences:
Table 14 -- Frequency of Codes
Code Frequency
Emotional Response Positive 61
103 Negative 42
Gameplay 84
Usability Positive 26
58 Negative 32
Learning 41
Engagement Positive 22
35 Negative 13
Social 31
Game Construct
Graphics 6
29 Audio 6
Timing 9 Characters 8
Self-Efficacy Positive 14
24 Negative 10
Usage 21
Conceptual Understanding 18
Comparison Positive 16
17 Negative 1
Technical Issue 12
Problem-Solving Strategy 8
Hawthorne Effect 5
TOTAL: 486
Three tables listing all of the student comments noted during evaluator observations or recorded
as responses to the open-ended post-survey questions can be found in Appendix B.
Discussion of Findings
Based on the data above, we found the following:
Achievement Comparison between TA and Classroom Instruction
Was there a difference in students' abilities to memorize multiplication facts?
First, we split a 3rd grade class in half and gave them all a pretest. Then, for
30 minutes, one half experienced classroom instruction while the other half
played TA. Last, all the students took a post test. The data from pre- and post-tests show no significant difference between the two groups. On the
bright side, students in both of these groups attempted more problems and
made fewer errors on the post-test than other groups that experienced only
one of these.
Grade/Age Differences
Not surprisingly, 3rd graders who played TA made larger gains than 2nd
graders who played TA for the same amount of time. It might follow to
reason that the young tender minds of 2nd graders should thus be protected
from TA. However, this same reasoning would also require that scrawny
individuals should be disallowed from gyms, sick people should be barred
from hospitals, and ignorant children should be prevented from attending school at all. In the face of all this, the ambitious educator will see
tremendous opportunity. It is likely that if 2nd graders practice more than 3rd
graders, those 2nd graders will eventually exceed 3rd grade performance. We
highly recommend further evaluation to provide evidence for this assertion.
Anxiety and Self-efficacy
In the group that experienced 30 minutes of direct classroom instruction,
there was no significant difference in reported self-efficacy from before to
after. However, when we compared pre- to post- testing of the group that
played TA for 30 minutes, there was a significant positive change in self-
efficacy. Students who only played TA also reported much less anxiety
toward learning multiplication facts after they had played for half an hour.
Peer Interaction
Students frequently engaged in social interactions with their peers during
game play. This suggests that social networking applications within the TA
game play structure could lead to greater usage and deeper effects on
student self-efficacy.
Learning vs. Gameplay
Students enjoyed playing the game. When asked what they disliked about the game, they most
often mentioned the robot, falling off of bridges, or dying and having to restart. Although these
items were intentionally built into the game, some math teachers (who we talked to on the side)
were concerned that it took too much time away from student learning.
Marketing
We found that TA helps students learn their multiplication facts about as well as traditional
classroom approaches. Therefore, it is a good supplement to traditional classroom teaching for
visually and conceptually teaching the times tables. We also found that playing TA improves
students' self-efficacy of multiplication skills more than traditional classroom approaches to
teaching multiplication facts. These two facts suggest that in order to effectively market TA to
school districts, teachers, and administrators, TA ought to be advertised as a powerful
supplement to classroom instruction that helps to build student confidence and mastery. This
marketing approach will also bring with it the idea that these benefits can be accessed outside of
the time and social constraints inherent in traditional classroom approaches.
Conclusions
The focus of this evaluation project was to determine if playing TA improves
multiplication fact mastery and/or contributes to positive changes in
students’ multiplication self-efficacy. According to the data collected, playing TA does improve mastery of multiplication facts, but this improvement was
roughly equal to that of the control group. This suggests that TA is equally
as effective as classroom instruction in helping students achieve
multiplication mastery. Additionally, students who received teacher
instruction first and then played TA exhibited the highest improvement
overall. While this improvement could be attributed to time on task, it shows
that time spent playing TA is time "on task" and its use extends beyond the
simple entertaining nature of other educational games.
According to the results of the multiplication tests and the student math
attitude surveys, it appears that playing TA results in a positive change in
students' multiplication self-efficacy. The pretest/post-test results showed
that in all of the study groups, students attempted more questions on the
post-tests. This indicated a higher level of self-efficacy (as students appear to be more confident in attempting more test items). This increase in
multiplication self-efficacy was roughly equivalent to the increase that was
experienced by those who received only classroom instruction. This indicates
that playing TA has the positive effects of improving students’ multiplication
self-efficacy beyond the classroom setting.
The results from our study indicate that TA is useful as a supplement to
classroom instruction. Teachers can only devote a limited amount of
individualized attention to each student, while playing TA allows for an
unlimited practice experience with multiplication facts outside of classroom
instruction. For those students who may feel anxiety or experience difficulty
in learning multiplication facts in classroom settings, TA may help provide an
alternative environment for students to learn by trial and error without time or social pressures. Games like TA can offer alternative modes of learning
that may reach students who are not as prepared to learn in traditional
classroom settings. As more is required from students and teachers, it will
become critically important to leverage technology to help everyone use
time more efficiently in the classroom and in the home.
Possible Confounding Variables in Findings
The summary of findings is supported by the quantitative and qualitative data collected for this
evaluation. However, there are some confounding variables that may be influencing these
reported results that are unrelated to specific research questions. Some of the variables that we
consider pertinent to the analysis include the following:
1. The students took the same test three different times within a compressed time period.
The frequency of testing may have allowed students to experience greater confidence
unrelated to the methods of instruction. This may have also allowed the students to
incorporate strategies in the taking of the test that build upon the prior test experience to
short-cut answering without considering each problem.
2. The students in Study 2 were separated into two different groups with one group led by a
teacher and the other group using Timez Attack for instruction. There may be variations
in time-on-task spent learning the math facts based on whether the student was
participating in the classroom instruction or the game. This may be one reason for the
measures of improvement in the quantitative analysis of these two groups.
3. The students were immersed in math-related activities throughout the duration of the
study. This focus may have influenced performance on the assessments. A controlled
follow-up with some interval of time between assessments and engagement in the
evaluation exercise would inform whether the students had really improved in
proficiency by retaining the facts over a longer term.
Recommendations
The results and findings of this evaluation report can be helpful in two major ways. First, the
findings can be utilized to drive the enhancement of Timez Attack game programming and
marketing strategies. Timez Attack can be really engaging for students, but teachers and other
school personnel are largely unaware of how the game will provide a benefit for the students.
Parents may also experience this confusion about the learning benefits of Timez Attack game
play. We suggest that Big Brainz gear its marketing toward helping teachers, administrators, and
parents understand that Timez Attack has been shown to provide a fun, confidence building way to complement the teacher in the classroom by producing higher times-table competencies than
would otherwise be attained. It could be helpful to provide information on the website addressed
to the different stakeholder groups such as parents, teachers, school administrators, etc. Building
rapport with these stakeholders can build market penetration and client satisfaction.
Another recommendation is making the information of students' progress more readily available
to teachers. The inclusion of more granular data reporting for the teacher could include
measurements of time spent on encounters, time spent moving around, number of correct and
failed attempts for each problem, sortable by multiplier/multiplicand, etc. Additional reporting
metrics will add value to the game when a buying decision is considered.
Future Evaluation Plan
Our initial findings can provide direction for a more comprehensive and thorough evaluation of
the Timez attack software with a more robust environment, conditions, and subject pool. In an
effort to promote continuous improvement, we submit here some suggestions for future
evaluations.
We recommend that a future evaluation work with students at the beginning of their 3rd-grade
year before they have been exposed to the multiplication tables. This will reduce the lurking
variable of students' previous exposure to the multiplication facts. Also, we recommend that the
evaluation be more longitudinal in nature with a greater number of students. This will allow for
more concrete, trustworthy results, increasing the strength of the marketing strategy.
Additionally as part of our data analysis, we discovered that for the 12s multiplication facts,
students improved on some numbers much more than they did on others. 3,4,5,11,12 seem to be
the numbers that most students got wrong, and interestingly enough, 3,4,10, and 12 were the
numbers that students attempted more. Also, these seem to be the numbers where students made
the most improvement. If we look between the lines, we see that 5,7,8,9, and 11 were the least
attempted, and 6-9 were the numbers where the least improvement was made. We did not
include these results in the evaluation because there are too many lurking variables, such as how
often the questions show up on the test, where they are placed on the test, and students'
individual test-taking strategies. However, we do recommend that a future evaluation examine
which questions are more problematic for students (controlling for the lurking variables by
having students use the same strategy and having the questions more evenly distributed). This
information could affect how the game could focus more/less on certain multiplication facts,
which would be a significant marketing advantage.
Below are listed other possible questions and methods for future evaluations.
Evaluation Questions:
1. What effect does game play have on the way students feel about their ability to learn and
use math functions?
2. How do different groups use the game?
Public school students
Public school teachers
Other school personnel (librarians, lab technicians, SPED, etc.)
School Administrators
Home schoolers
3. What influence does the teacher have on the attitudes of the students with respect to the
game?
4. Does the confidence developed by game play transfer to other math areas? In what ways?
5. Does the confidence developed by game play transfer to other curricular subjects? Which
and in what ways?
6. What other methods are used to teach the multiplication tables in traditional classrooms?
How much time do they take? How do their effects
compare with Timez Attack?
7. How do teachers feel about Timez Attack?
8. What do students remember the most after playing Timez Attack?
9. What is the effect of Timez Attack as a remediation intervention?
Evaluation Methods:
In this evaluation, two classes of second and third grade students were observed before and after
they played Timez Attack. There were some serious factors that reduced the effectiveness of this
evaluation, which should be remedied in future evaluations. It is suggested that:
1. Students have no previous experience with the Timez Attack game.
2. Students are observed who have not begun to learn their multiplication facts.
3. Feedback from teachers be intentionally solicited as part of the evaluation.
4. Only online versions of the game be used so that usage analytics can be
acquired.
5. More than two classrooms be used in the study.
6. More than one school demographic be used in the study.
7. Multiple observations be conducted over a period of time.
8. The study starts with the beginning and not the end of the school year.
9. More interviews and case studies are generated to provide a thick description of
the experience of the clients.
10. Students be chosen at random to participate in the study to enhance validity of
the study.
11. Student achievement be assessed over time to analyze long-term recall of math
facts.
These suggestions can help to drive a deeper and more meaningful exploration of the effects that
Timez Attack is achieving on the various stakeholder groups.
Appendices
Appendix A - Data Collection Instruments
Pre-Multiplication Attitudes Survey
Read each sentence.
Circle ―NO‖ if you think what is says is not true AT ALL
Circle ―no‖ if you think the sentence is mostly not true
Circle the ―?‖ if you don‘t know or are not sure
Circle ―yes‖ if you think it is mostly true
Circle ―YES‖ if you think it is VERY true
1. I can learn new multiplication math facts easily
NO
no
?
yes
YES
NO
no
?
yes
YES
2. I like learning multiplication math facts
3. I am good at memorizing multiplication math facts
NO
no
?
yes
YES
4. Learning new multiplication math facts scares me.
NO
no
?
yes
YES
5. I usually do not worry about remembering
multiplication math facts.
NO
no
?
yes
YES
6. Learning new multiplication math facts makes me feel
bored.
NO
no
?
yes
YES
7. I study my multiplication math facts at home.
NO
no
?
yes
YES
8. I have played Timez Attack…(Circle the correct
answer.)
NO
no
?
yes
YES
Post-Multiplication Attitudes Survey
Read each sentence.
Circle ―NO‖ if you think what is says is not true AT ALL
Circle ―no‖ if you think the sentence is mostly not true
Circle the ―?‖ if you don‘t know or are not sure
Circle ―yes‖ if you think it is mostly true
Circle ―YES‖ if you think it is VERY true
1. I can learn new multiplication math facts easily.
NO
no
?
yes
YES
2. I like learning multiplication math facts.
NO
no
?
yes
YES
3. Remembering new multiplication facts is hard for me.
NO
no
?
yes
YES
4. Learning new multiplication math facts scares me.
NO
no
?
yes
YES
5. I usually do not worry about remembering
multiplication math facts.
NO
no
?
yes
YES
6. Learning new multiplication math facts makes me feel
uneasy and confused.
NO
no
?
yes
YES
7. I will play Timez Attack at home.
NO
no
?
yes
YES
8. I think playing Timez Attack would help me know my
multiplication math facts.
NO
no
?
yes
YES
9. Learning to play Timez Attack was easy.
NO
no
?
yes
YES
10. Playing Timez Attack is a fun way to learn my
multiplication math facts.
NO
no
?
yes
YES
11. What I love about Timez Attack is… (Write you answer below.)
12. What I hate about Timez Attack is… (Write your answer below.)
Multiplication with 12s Pre/Post-test
Appendix B - Student Comments
Student Comments Recorded by Observers During TA Play
Shouted “This is fun!”
Said she had played the game “a few times."
Exclaimed “I’m Good!” when he defeated the robot after answering all of the facts in an
end of level checkpoint. “Yes!”
“Finally” when completing the end of level checkpoint after she had lost to the robot on
two previous attempts
“When do we get the dvd so we can take it home? I wish I could load it today at home
and play it now.”
"This is hard." "I can't think of it."
"Watch this, it's funny!" she says when she was at a challenge door.
"I played it a lot last year at school. I think it's really cool. It's, I don't know.” (shrugs)
"It's pretty fun. I don't know, it's just fun."
"I knew what it was, but I can't find the keys. It's hard."
"What the heck?" one student says when a challenge door is missing the second bottom numeral. Another girl said the same thing had happened to her.
"I really like it. I just enjoy doing my times tables. This is one of my favorite ways. It's
times tables made into a game so double the fun." He said he had played it a long time at
school, and liked the lava world because "it's the hardest one."
"Do you know if you wait, the spiders just walk to you?"
"It was fun. It was pretty good (compared with other games he has played). It's a little bit
less good because there's this game I really like. You go around catching stuff and stuff. "
"Yes I did it" a girl pumps her fist when she defeats the robot.
"I hate this part. I always fall off the edge" when she's at the bridge, trying to cross.
"Finally I'm going to the next level. Are you on the third level? I'm on the third level. I passed you. Have you been keeping track of your levels?"
"Look at me!" she says this as she falls off the cliff.
"I've played the older version at school. Don't we get to download this at home? My mom
will probably let me do that. I play a lot [of games]. I play Wii. I play a few other math
video games. It's pretty good. Better (Timez Attack is vs. those others). I like it more. I
like video games where you run around. The running around and fighting the bad guys. Do we get to play again? I want to play right now.”
"I thought it was fun because you can do challenges and stuff but math is involved. I've
never gotten past the dungeon" "It is pretty much the same fun [as other games]."
"It was awesome! It was really fun to play." She most liked going against the robot but
said it was " a little hard to learn the keys.”
"You can't backspace [when at a challenge spot]. It sucks."
Interview- Q: What did you like the most about Timez Attack? “Helps us learn times
tables. Parents can't tell us not to use it. I like it the most cause it's long.”
Q: What did you NOT like about Timez Attack? “When you really want to finish it, try to go
too fast, and make dumb mistakes.”
"Yes" when he beats final boss
"I can't see. It's too dark on the thing."
"Yes" "This is really cool"
"I don't know how to work it."
"It's angrifying" the thing in the game doesn't com back if you fall
"My computer's frozen."
"I can't think of it!"
"Ah, 12s are hard!"
Interview – Q: What do you like about Timez Attack? “It's not like having a piece of paper
or flashcards. It's like a game. It's fun. And it teaches you and makes it so you can
memorize.”
Q: What don't you like about Timez Attack? “ Umm... how when the spiders come out,
there are blue things... it like turns blue near them and then you can't get them--it takes a
long time to get them.”
Q: Do you think playing Timez Attack is useful? “It's good practice.”
Q: if you could change anything about Timez Attack, what would you change? “On the last
level, make it easier to get the dragon.”
Q: Do you play at home? “Not much. I play one or two times in three or two weeks.”
Q: Do you have any siblings? Do they play at home? “I have three younger siblings, but they don't play. My older brother used to play when he was in third grade, but he passed
all of them, so he stopped.”
"I don't know how to get out.”
"Did I just come up from there?"
Interview - Q: Why do you throw the snails at the wall? “ Well, they represent each group
in the array.”
Q: So, they help you count up? “Yeah.”
Interview – Q: Why do they have those dots? “Because they... it's sort of like an array."
Interview - Q: Did you know your 12s coming in? “Sort of. A few of them--that would have
been useful to know!"
"No!" he exclaimed quietly when he missed another question.
"Now I'm fighting the big boss, I'm guessing."
"No!" "This is so fun!"
''Holy cow you are good”
“Not another one of these guys”(robot) “This is fun”
Interview - Q: Is it easy to play the game? “Kind of easy (the multiplication can be
hard).”
Interview - Q: Have you played the game before? “Yea, at school.“
Shouting “yea” when she is correct, shouting “oh no” when wrong.
Interview - Q: Why do you throw balls at the gate? “It helps you figure out the right
answer so you can get in.”
Interview - Q: Why do you throw balls at the gate? “You have to throw the balls to get in
the gate.“
Interview - Q: Why do you throw balls at the gate? “Counts until last one then that is the
answer .”
Interview - Q: Have you play this before “Yes.”
Q: Where? “At school and home. At home it has slugs. I like slugs better.”
Q: Why are there groups with 5 balls on the gate? “Not sure.”
Interview - Q: Why do you hit the spider? “Helps you count to get answer.”
Q: What do you like about the game? “You do not have to ask anyone the answer if you forget.”
Q: Anything else? “It makes me get me answers faster.”
Q: What do you not like about the game? “When I get the wrong answer.”
Claps when right, says “ah sweet”
Interview - Q: Why is it counting by two when you throw the ball? “Because it is 2 x 12, so you have to count by 2’s.”
“I do not like battling the robot. He beats me up.” “Oh no, I killed him” “Not sure how I
get to next level. Am I in it?”
Says “phew” when he gets them right.
Interview - Q: Why are there groups of balls on the gate? “They help you figure out the
answer.”
Q: How? “Since it is 9X12 you have 9 groups. Wait, I guess I have 9 in every group and then I would have to count twelve of them.”
Interview - Q: Why are there groups of balls on the gate? “To get the answer “
Q: Why is there 7 in each group? “It is 7X12 so since there is 7 balls in each I count by
sevens to get the answer.“
“YES!” at checkpoint, with handpump.
Interview - Q: Why do you throw the balls? “Because it counts up and gives the answer if
you need it. “
Likes the game “because it's a mix between a game and times tables.”
Likes the game “better than memory cards.”
Game “makes me feel better at math. “
“I just did it!”
“Ah, 12s are hard!”
“Oh, it's . . .; I can't do it . . .” “Yes!"
“Okay, think.”
“Yes!” when he found it was right.
“I just have to beat the sixes and I'm done. Can I play the game?”
“Yes! I got it done!”
“Oh! Oh my Gosh!”
“Ah!”
“Turn the other way!”
“Door, another door, weird.” “Yes! Oh, bam bam ...”
“Oh!”
To his friend- “It shows you the ones you need to practice to beat the game.”
“Okay.” “Ah.” “Ch, ch.”
“Oh no!”
“Ah! I forgot that one!”
Gives her friend suggestions- “Just walk off the edge.”