Test

Equivalnauts Drill and Practice

Cory Robertson

Description of Project Equivalnauts (located here) is a drill and practice educational game meant to assess students on their ability to convert between fractions, decimals, and percents; and, their ability to identify equivalent numbers between the three formats. The game was created in a coding program called SCRATCH. The program uses standard Flash and Java language, but presents it in a visual, puzzle-piece format to help expedite the programming process for veteran programmers, as well as create a lower barrier of entry for novice programmers to explore coding.

SCRATCH Programming Screen

In the game, Equivalnauts, students are in control of a spaceship that they must pilot around five different systems in order to mine asteroids for fuel. In each system (level), their ship is assigned a fraction (e.g. 6/8). In that system, on each asteroid is a number that may or may not be an equivalent number of their ship’s assigned number (e.g. 75% is equivalent to 6/8, 25% is not). The students must first simplify the fraction they’ve been given, then fly around the system (level) mining the asteroids that have an equivalent fraction, decimal, and percent. In each system (level), there are three equivalent numbers for each student to find. Once they’ve found all three, the program takes them to the next system (level) where they are assigned a new number, and new sets of equivalents to find. The ship is controlled using the four arrow keys on the keyboard to move, and pressing the “X” key to mine an asteroid. When students feel they have found an equivalent number (asteroid),

they simply fly over it (using the arrow keys) and press “X” to mine (select) that asteroid (equivalent number). Students receive feedback on each selection. If they select the correct asteroid, they receive confirmation that the number they mined was an equivalent number. If they select an asteroid that was not an equivalent number, they receive feedback to let them know that they selected incorrectly. The entire game consists of a Splash screen of instructions, five systems (levels), and a final, Kill Screen with information on how to play again, or take part in a final assessment of their skills. The game is web-based, and compatible with either Flash or HTML5. It is fully embeddable in any web site or wiki. It is permanently hosted on SCRATCH’s community site, where users can vote on the game, as well as leave comments for the programmer.

Project Calendar

Learner Analysis

Item Weaker Learners Average Learners Stronger Learners

**All learners are current or recently re-designated EL’s***

Age 11-12 10-11 10-11

Education Level 5th/6th Grade 5th/6th Grade 6th Grade

Reading Level 1 yr. Below Grd Lvl On Lvl Above/On Lvl

Motivation Medium High High

Prerequisite Knowledge

Learners know that fractions, decimals, and percents have equivalents



Prerequisite Skills Ability to convert common fractions to decimals, percents, and simplify

Ability to convert common fractions to decimals, percents, and simplify

Ability to convert common fractions to decimals, percents, and simplify

Facility with a computer

Basic Approaching Advanced

Familiarity with the Internet

Basic Approaching Advanced

Typing Ability Basic Basic Basic

Access to computers

2 32 station labs, 4 in-classroom



Access to the Internet

All computers Internet-connected



Time availability 45 minutes/week in lab, 1-2 hours/week

45 minutes/week in lab, 1-2 hours/week

45 minutes/week in lab, 1-2 hours/week

in classroom in classroom in classroom

Concept Analysis Standards

● 5th Grade Number Sense 1.2: Interpret percents as a part of a hundred; find decimal and percent equivalents for common fractions and explain why they represent the same value; compute a given percent of a whole number.

● 6th Grade Number Sense 1.1: Compare and order positive and negative fractions, decimals, and mixed numbers and place them on a number line.

Learning Objectives

● Learners will identify fractions, decimals, and percents ● Learners will be able to convert fractions to decimals and percents ● Learners will be able to identify numerical equivalents for fractions, decimals, and

percents Assessments The software will present the learner with 5 stage-based opportunities to identify the fraction, and locate its equivalents. A student’s progress through the 5 stages will serve as an assessment of their ability to convert and identify equivalents. Since, as with most DP software, students can easily click or move around until they stumble upon the correct answer, a post-test with the same numbers will be presented at the end of the session. It will be a Google Form that they will be linked to at the end of the software session.

Task Analysis

Needs Analysis

Hardware: PC/Mac/Linux Details and Comments

RAM: 1GB Minimum

Monitor Resolution: 800x600 Minimum

Sound Card: On-Board

Speakers or Headphones: Either

For lab setting, headphones

Internet Connection: DSL/Cable For uploading test Flash file

CD-ROM: NA Internet-based

Processor: 1Ghz Minimum

Hard Drive Capacity: 100mb Install of SCRATCH (Needed for development only)

Software: PC/Mac/Linux Details and Comments

Operating System: Windows XP or >, Mac OSX, Linux (any current build)

Browser: Firefox 3.x, Internet Explore 6 or >, Chrome (any build), or Safari

Word Processor: NA

Spreadsheet: NA

Authoring System required: SCRATCH Needed for development only

Testing System required: Flash-installed browser Needed for development only

Image Authoring tool: GIMP or Photoshop Needed for development only

Storyboard

Beta Screenshots

Coding Screen (left), Main Gameplay Screen (right)

Originally, flying over the yellow “Home” button at the top left would take the user back to the

beginning. No feedback on wrong answers, only correct ones.

Coding Screen (left), End of Game Screen (right)

At time of beta-testing, a one sentence confirmation was all students received to let them know

they met the game’s objective. No way to restart game other than reload page.

Usability Test Description For the Usability Test, three random students from 6th grade were chosen. Students were given no background or prior knowledge on the game. No instructions were given on how to play the game, as part of the testing process was to determine whether or not the in-game instructions were sufficient enough for players. Testing was done with one student at a time so as to be able to have time to speak and reflect with the student on their experience with the game as they are using it. The students were asked to take part in three tasks throughout the testing.. No time limit was put upon the students; however, I did record how long it took each student to finish, and whether it appeared to be gameplay or math concepts that accounted for their fluency. Below is the Usability Testing form that I used, along with the results from each of the three students. Usability Form Data *Changes Made After Testing Highlighted in RED **Changes Skipped After Testing Due to Time/Technology Limitations Highlighted in BLUE Student 1 Tasks Notes Task 1 (directions) – Read the directions provided by the game, and explain what the objective/goal of the game is, and how do you control it?

Student read very quickly; was able to tell me how to control (arrows and X), but not clear on goal of game, “About fractions?” Didn’t correct student.

Task 2 (main gameplay) – Play the first level. How do you know when you are right? How do you know when you are wrong? Do you know how to get in-game help? What if you want to start over?

Was able to control just fine; pressed “X” when on asteroids, as per game’s instructions. Pressed “X” on only fractions, even ones not equivalent. Clicked on Help, which provided help with controls, not content (student needed content help). Student was frustrated that he had to sit through all six startup Help Screens.

Task 3 (game completion) – Complete the game. What did you not like about the game? What did you like? What frustrated you? What was too easy?

After about 8 minutes on World 1, student completed the rest of the game at about 2-3 minutes per World, for a total of 18 minutes on the game. Student did not like that it was about math. Student did like flying the ship around, freely, “Being able to fly anywhere on the screen.” Student was frustrated by the math concepts (may be why student didn’t lik the math). Student thought it was too easy to

simply press “X” until student selected all the right answers.

Student 2 Tasks Notes Task 1 (directions) – Read the directions provided by the game, and explain what the objective/goal of the game is, and how do you control it?

Student read directions, quickly; was able to repeat to me all directions, almost verbatim. Understood it was about converting numbers, asked me for a paper/pencil.


Student was very successful with the math. Had trouble finding where the ship’s assigned number was (top right in red). Student didn’t need help, but when I asked where she would go, student pointed to the help button.


Student completed game in about 10 minutes. Student did not like that it was only 5 levels. Student liked flying around and “X-ing” on asteroids. Student was frustrated that there was no way to repeat a level, only start from beginning. Student felt the math was too easy, “Except for the six-eighths one.”

Student 3 Tasks Notes Task 1 (directions) – Read the directions provided by the game, and explain what the objective/goal of the game is, and how do you control it?

Student took a lot of time reading directions (not sure if it was a fluency or comprehension situation). Student was able to repeat main points of directions, but did not remember that there are only 3 equivalent numbers on each level. Did not correct student.


Student quickly got 2 equivalent numbers, was looking for third, but did not know the game would automatically take him to next level after mining the third answer. Student clicked on help, only saw help with controls (need help on objective/game rules). Student accidentally “flew” Home (flew ship over home button while traveling to another asteroid), and had to start again.


Student finished game in 15 minutes (including starting over 5 minutes in). Student was very frustrated that the game started over by a mistake of flying. Student enjoyed the flying. Student felt the math was a little too difficult, and

that the gameplay was too easy, “There’s no jumping or finding hidden things.”

Changes Made After Usability Testing

Coding Screen (left), New Main Gameplay Screen (right)

Based on user feedback, the following changes were made:

1. Flying Home was changed to “Click Here to Go Home” a. Change was made due to user accidentally flying over yellow box and restarting

game 2. Students now receive negative feedback (in addition to the already included positive

feedback) to let them know that they have incorrectly chosen an equivalent number a. Change was made due to users first choosing a wrong answer, receiving no

feedback, and not sure if they were playing the game correctly. They were forced to play by the game’s arbitrary rules of only positive feedback, which required too much trial-and-error on their part.

Changes, cont.

New Help Screens With More Information


1. Added in three new instruction screens that provided the additional information, based on user issues and feedback:

a. Listed how many total levels in the game (5) b. Deliniated that equivalent numbers included fractions, decimals, and percents

(previously, only the term “equivalent” was used) c. Added the RED addition to the instructions to help visually draw attention to where

users will find their ship’s assigned number. d. Included a notice that once the user has selected all three equivalent numbers, they

would automatically be sent to the next level (previously, this was not included, and relied on the user to experience the auto-forwarding without warning).

Changes, cont.

In-Game Help Screen


1. In-Game Help screen major overhaul a. Originally, in-game Help Screen was a repeat of the 6 startup help screens b. User feedback was that they needed help, but did not like scrolling through the “story”

or concepts that they already knew c. Created truncated Help Screen with controls and content information. d. Help screen contains information on most common misconceptions of the game as

per usability testing (would be changed over time if more future testing were to take place)

Changes, cont. User-Suggested Changes Not Implemented

1. Brevity of Game – The game is only five levels as a proof of concept. I felt that for an end-of-unit assessment, five levels of three equivalents each (a total of 15 numbers to convert and compare) was enough to assess student mastery.

2. Cannot Restart Level, Only Full Game – Due to the complicated nature of the programming (broadcasting and receiving certain keywords based on user interaction), I was not able to find a timely solution (within a week) to add this feature. If given more time, I would be able to work on it.

3. “No Jumping or Hidden Things” – This is a foundational issue in that jumping and searching for hidden items is not a part of the game’s mechanics or objectives. While this was that particular user’s preferred style of gameplay, it did not fit the design of this game. Perhaps a different educational game involving a platformer style of gameplay would suit the user better.

End Game/Kill Screen

Even though users did not have feedback on the end-game issues, the following changes

were made: 1. Added end-game text to let the user know the game was over, and that they were

successful. a. Also added a line for potential additions to the game: “...and explore more worlds

later!” 2. Added a “Play Again” button for those who wish to play the game from the beginning 3. Added a text box with the URL to the Google Form that now contains a list of 5 questions on

finding equivalent numbers. This is to address the concern that users could simply click around until they find the right answer. They still can click until they find the right answer, but the Form will serve as a true assessment of their skills (the Form deals with the same numbers used in the game, but asked for reverse information; e.g. “Find an equivalent for 25%”).

Implementation Details on Implementation The final product was tested on a class of 32 6th grade GATE students. The students involved, though GATE, had a wide range of math abilities. 76% of the class was considered Proficient on the Math portion of the California Standardized Test at the end of their 5th grade year. In retrospect, of the three students I randomly chose to test the program, two of them were Basic, while the other was Advanced. The implementation took place in a computer lab setting. Based on my experience with the usability testers requesting a pencil and paper, the students were asked to bring a whiteboard and marker with them to the lab. Rather than having the student access SCRATCH’s web site to play the game – which also has thousands of other games, and could be potentially distracting – I embedded the game in my wiki, and wrote the address (clrobertson.wikispaces.com) on the board. Before the students started, I spoke to them briefly. I told them that they were going to try out a program, and that it was going to be math based, and have game-like features. I told them that I didn’t want to tell them any more about the game, as part of this was to see what they could gather from the information in the game (same as I told the original testers). I also told them that should they have questions during the game, they would need to try to find the answers within the game (i.e. click on the Help button; though, I did not tell them that). I was nervous about the potential questions they’d have, as part of me wanted to see if they struggled for a little, would they be able to find the answer? Or, would they struggle not because they needed help, but because of a flaw in the program? And, how would I handle that? I told the students that at the end of the game there were instructions on what to do next. Without telling them specifically about their options, I told them that they have two choices at the end of the game (play again, or take the test), and that it was their choice on how to proceed. The students were given 30 minutes to finish, though none needed that entire time. Evaluation of Implementation The implementation went very well. Overall, the students seemed to enjoy the experience, though a few issues occured.

Issues • One student managed to load all 5 worlds at once, each layered on the other, to create a

cluttered, unreadable screen. In talking with him about what series of keys he pressed to get there, he wasn’t able to restate what happened, nor was I able to replicate it later. Obviously, something went wrong, yet I can find nothing in the code or in playing the game to cause this. Additionally, no other user experienced this issue.

• More than half of the students clicked through the startup instructions without reading, and were lost from the very beginning. A few raised their hand, asking for help, and when I approached them, they said, “What do I do?” I said the same thing to each student, “If I wasn’t here, and you didn’t know what to do, where would you look?” All of them immediately clicked on the Help button.

• A few students flew around randomly pressing “X” until they got the right answer. Ultimately, their guessing caught-up with them on the post-test; however, prior warning of the post-test perhaps would have alleviated this guess-and-testing.

• Suprisingly, three to four students did not know what the game meant when it said, “Use the arrow keys to control the ship.” A student next to them reached over and pointed to the arrow keys.

• At the kill screen (end of game), half of the class opted to play again. Those who decided to take the post-test had to type the URL into the address bar, as the game did not support hotlinking.

Verbal Feedback I asked students to share with a partner what they did and did not like about the game: • Some said that they would rather control with the mouse, where others said that they liked

using the keyboard • A few said the math was too hard, with a few more who spoke about how easy it was • They all seemed to enjoy the game’s scenario and objectives (flying around and mining

asteroids) • The majority of them were upset that it was so short (five levels), which I suppose is more of a

compliment than a complaint of the game.

Suggestions for Improvement • I would add a portion to the startup instructions that lets the user know they will be assessed

on the concepts they explore in the game • Instead of saying “arrow keys” in the instructions, I would add a screenshot of the arrow keys

on a keyboard so that, visually, more users could understand the controls • While I wasn’t able to find any way to hotlink from the game to another site, I would post in the

SCRATCH community forums to see if someone has orchestrated a way to do this (in a very superficial search, I found that users were creating their own code commands that go beyond the basic ones included in the SCRATCH base install; mods).

• In-line with hotlinking, I would also spend more time exploring a way to allow a user to repeat a certain level, not just the entire game

• I would also spend time researching and implementing a “Lives” system, whereby three incorrect answers would result in the user starting the level over. I originally had this coded out (though never implemented in the game), but it would require a user to start the entire game over after three spent lives (much like the 80’s NES games). I surmized most students at this age are not used to having to start all over, and would be frustrated by this, so I left it

out. Would I be able to resolve the restarting a level issue, I would implement the coding I did for the “Lives” system.

Reflection on Project Overall, I thoroughly enjoyed this project. I had a lot of fun planning, creating, and sharing this project with the students. What was most beneficial to me on this project was stepping outside my comfort zone, and using a program I had wanted to explore for a while, but never seemed to find the time. Forcing myself to use a program I was not familiar with at all (except for a brief 20 minute CUE-Tip at the 2010 CUE conference), was my biggest reward for this project, as it instilled in me more confidence to push and challenge myself in all areas of my profession. Having to plan ahead for each step of this project was also new to me. I’m not typically a planner. I really like to sit down, explore, click around, and start molding and shaping until something comes to life; then, I start to fine tune the mistakes and errors. For a project like this, having to plan each step, and really work out the kinks and errors before I even had a chance to make them was just as new to me as the actual program I used. I feel it helped me grow, and see that there are times where the careful planning and preventing of errors can result in a much stronger, and more appealing product. This has been my most enjoyable project in this program, so far. Any time where I can try something new, and work with kids on something that differs from their daily classroom life, I am truly happy. While this game doesn’t even come close to competing with the games they play (even in the educational setting), it was a very positive experience to see them enjoy interacting with a program I created, and to see all that hard work and many hours I put in trying to figure out how to get the game to do what I wanted it to do. And, that was the most difficult, but rewarding part: making technology do what I knew it could, and having think rationally about how it could be achieved.

Sample of Source Code of Project Main Code

Ship Sprite Code

Asteroid Sprite Code (code differs for each asteroid, a total of 45 in game) Wrong Answer Right Answer

Perma-Link to Playable Version of Game (SCRATCH’s web site)

Link to Full Code of Project

Test

Documents

equivalent number asteroid

asteroid equivalent

equivalent fraction

equivalents learners

equivalent numbers

new number

system level

common fractions