A serious game for traffic accident investigators

A serious game for traffic accident

investigators

1. INTRODUCTION

All new police officers recruited by Dubai Police gothrough the same training process, which consistsof lectures and on-the-job training (designed fortheir specific rank category). The training is admin-istered by Dubai Police Academy which alsoaccepts recruits from other Gulf CooperationCouncil (GCC) countries, Republic of Yemen, andPalestine. The training duration varies from 6months to 4 years, based on the rank category. Aftergraduation, local recruits are assigned to police sta-tions and departments in which they receive furtheron-the-job training.

The objective of the research presented in thispaper is to assess the suitability of a virtual environ-ment for training traffic investigators. In particular,a game-based environment has been used toenhance the performances of the police officers toaddress some of the problems identified with thecurrent training process in Dubai Police (BinSubaihet al, 2005a). The general problem with training forpractical skills is well documented (Aldrich, 2005)(Carless, 2005). Using only lectures lacks interactionand engagement. Additionally, in the training cours-es, we observed that the time allocated for a coursewas not sufficient to cover all the various accidenttypes. The participants in this study complained of

Interactive Technology & Smart Education (2006) 4: 329–346© 2006 Troubador Publishing Ltd.

Ahmed Binsubaih, Steve Maddock and Daniela RomanoDepartment of Computer Science, University of Sheffield, Regent Court,

211 Portabello Street, Sheffield, UKEmail: {a.binsubaih, s.maddock, d.romano}@dcs.shef.ac.uk

In Dubai, traffic accidents kill one person every 37 hours and injure one person every 3 hours. Novice traffic accidentinvestigators in the Dubai police force are expected to ‘learn by doing’ in this intense environment. Currently, they useno alternative to the real world in order to practice. This paper argues for the use of an alternative learning environment,where the novice investigator can feel safe in exploring different investigative routes without fear for the consequences.The paper describes a game-based learning environment that has been built using a game engine. The effectiveness ofthis environment in improving the performance of traffic accident investigators is also presented. Fifty-six policementook part in an experiment involving a virtual traffic accident scenario. They were divided into two groups: novices (0 to2 years experience) and experienced personnel (with more than 2 years experience). The experiment revealed signifi-cant performance improvements in both groups, with the improvement reported in novices significantly higher than theone reported in experienced personnel. Both groups showed significant differences in navigational patterns (e.g. dis-tances travelled and time utilization) between the two training sessions.

Keywords: Game-based learning, traffic investigation, comparative study, performance, virtual environments.

VOL 3 NO 4 NOVEMBER 2006 329

problems similar to those found in the literature. The on-the-job training suffers from impractical-

ity, varying levels of exposure, and lack of uniformassessment. The impracticality issue arises from thenature of the real environment, which hindersrepeatability and exploration, two elements whichare very important in any training environment. Inparticular, in a real traffic accident, exploration isvery difficult to achieve. Issues such as the possibil-ity of a traffic jam meaning that a road has to becleared as soon as possible, the bewildering heatduring the day in Dubai, the intolerance of the peo-ple around and of those involved in the traffic acci-dent that want to get away, etc. make it very diffi-cult for an investigator to do his job. For a novice,the pressure of such problems, including the fear ofembarrassment in front of the public and his col-leagues, induces him to avoid exploration. In addi-tion, in the real world it is impossible to reproducea situation in an identical manner so that the sametasks can be practised again and again.

A further undermining problem is the varyinglevel of accident exposure that the various officersare subject to. Accident types and frequency differfrom one area to another. Due to the fact that a newinvestigator is assigned within a jurisdiction to aparticular police station and a particular patrol unitduring on-the-job training, he might only beexposed to a limited range and number of accidents.The third issue is the lack of uniform assessment.The experienced investigator uses his own subjec-tive judgement to decide whether a new recruit hascompleted the training and the lack of objectivemetrics can undermine this judgment.

A virtual environment is not a magic wand to solveall the problems highlighted above. However, it canhelp to improve exploration, is highly repeatable,allows the trainee to practice without any fear andoffers a uniform assessment process. Furthermore,game-based learning is engaging and gives thetrainees an active role in the learning process, foster-ing enquiry and reflection. The study presented herealso suggests that game-based learning has strongpotential in addressing these problems across bothpopulation samples, novices that need to practiceand experienced officers that need to refresh theirskills on cases that occur less frequently.

This research work contributes to knowledge inthis area in two ways. First it provides an example ofa valid learning environment where participants canpractice their traffic investigation skills and exploredifferent paths. Second it demonstrates statisticallythe training effectiveness of such an environmentand adds to the body of literature researching theeffectiveness of 3D environments as training tools.

Section 2 describes the related work and givesexamples of the use of this technology in trainingfor different domains. Section 3 describes the virtu-al training environment. Section 4 details the exper-iment design method, development, and measuresof performance. Section 5 reports the results fol-lowed by a discussion in section 6. Finally, section 7presents our conclusions.

2. RELATED WORK

Game-based learning environments (also describedas serious games) are gaining wide acceptance inmany domains due to a number of contributing fac-tors. First, there has been a shift in the approachtaken to develop games. This has changed fromgames being developed from scratch to reusing com-ponents. The advent of game-independent gameengines (an idea reported to have first surfaced withQuake (Lewis & Jacobson, 2002)), combined withreduction in hardware and game engines costs,means that more low-budget projects are possible.For instance, Unreal engine is shipped with theUnreal Tournament game (for less than $50) and canbe modified using an inbuilt editor and a scriptinglanguage. In this research we have used the Torquegame engine, which costs $100 for a single license.Other freely-available engines that have been used inresearch are OGRE (Romano et al., 2005), Freeciv(Ulam et al., 2004), and Stratagus (Marthi et al.,2005).

Second, there has been interest from particularapplication areas, e.g. the military. The U.S.Department of Defense (DOD) has shown muchinterest in the applicability of the advances in thegames industry for training. In 1996, this led theDOD to ask the National Research Council’sComputer Science and Telecommunications Boardto examine the possibilities of collaborationbetween the games industry and the DOD in thetechnical advances in simulation and modelling.This led to a report published in 1997 entitled“Modeling and Simulation – Linking Entertainmentand Defense” (Zyda & Sheehan, 1997). The reportmade VR researchers aware of the capabilities ofgames engines and the overlap that exists betweengames and VR (Zyda, 2005).

Third, the Serious Games Initiative1 of 2000(Dobson, 2006) aimed to link video games to seri-ous applications and publicize the field of seriousgames. It also founded the Games for Health2 annu-al conference. Many domains have benefited fromthis trend of converting a game that is primarilybuilt for entertainment to a serious game with

BinSubaih et al: A serious game for traffic accident investigators

INTERACTIVE TECHNOLOGY & SMART EDUCATION330


education or training taking precedence over enter-tainment. In the following sections we presentapplications from three domains: military, health-care, and first responders.

2.1 Military

There have been a number of military applications.For example, America’s Army was built on top ofthe Unreal engine with the primary aim of recruit-ment. However, many consider it to be the mostsuccessful serious game to date (Harz, 2006).Moreover it is considered to be the game that begana revolution in thinking about the potential role ofgames for non-entertainment domains (Zyda, 2005).Although the training ability of the game is yet tobe proven, the anecdotal evidence described in(Zyda, 2005) showed that it succeeded in helpingnew army recruits to pass rifle ranges and obstaclecourses.

Another example is Ambush!, which enablessquads to experience and respond to ambush situa-tions using 3D simulations (Diller & Roberts, 2004).It was developed as a modification of an existingcommercial game engine (Operation Flashpoint) injust six months. A preliminary evaluation was con-ducted to measure its effectiveness. 18 subjects wereevaluated including two officers and 16 enlisted per-sonnel of the 1st Brigade, 25th Infantry DivisionStryker Brigade Combat Team (SBCT). The resultsshowed a satisfaction level of 5.88 out of 7. Subjectsalso felt positive (6.72 out of 7) about its effective-ness for tactics, techniques, and procedure training.The likelihood of using the system for training wasat 6.72 out of 7 and the likelihood of recommendingit to others was at 6.77 out of 7. Subjects reportedliking the visual realism and disliking the difficultyof the controls. One Airman wrote: “I was in anambush last year. This [Ambush!] is as close to realas you can get without being in danger” (Chathman,2005).

A third example is the Tactical LanguageTraining System (TLTS) (Johnson et al., 2004),where the objective is to help learners acquire com-munication skills in foreign languages and cultures.The learner interacts in a 3D environment wherethey can speak and select gestures for their charac-ters. The 3D environment was built using theUnreal Tournament 2003 game engine. An evalua-tion with seven college-age subjects reported thatthe game was fun and interesting and they were gen-erally confident that with practice they would beable to master the game. It was reported(Chathman, 2005) that it was used for personnel dis-

patched to Iraq, although there are no reports yetfrom returnees. However one beta tester wrote: “Ilearned more in 1 day with this [TLTS] than I did ina whole tour in Iraq.”

2.2 Healthcare

This application area has grown quite rapidly to thepoint that a games for health conference is heldannually. The usage varies from therapy to trainingprocedural skills. For example, using components ofFull Spectrum Warrior3 , virtual environments werecreated to treat patients suffering from PostTraumatic Stress Disorder (PTSD) (Pair et al.,2006). The initial trials created environmentsresembling scenes from the Iraq war. These trialsinvolved two patients and provided anecdotal evi-dence to show that the environment helped to cog-nitively reframe their experience in a positive wayand also to reduce their nightmares. Re-Mission(Re-Mission, 2006) is another example of the use ofgames in healthcare. It was developed by a non-profit organization called HopeLab with the aim toproduce “an innovative solution to improve thehealth and quality of life of young people withchronic illness”. A trial test on 375 cancer patientsshowed that patients who played the game exhibit-ed an increase in the quality of life, knowledgeabout cancer, and ability to manage the side effects.

2.3 First Responders

First responders are those who are expected to be thefirst at any incident. These include police officers,fire-fighters, hazardous material technicians, emer-gency medical providers, etc. First RespondersSimulation and Training Environment (FiRSTE) is avirtual reality training system developed to replicate areal-life environment for first responders. It wasdeveloped using the Half-Life game engine. An initialexperiment developed was a virtual terrorist attackon a computer science building (Hall et al., 2004)which aimed to measure the effects of an intenseenvironment (explosions and fires) on learning. Theresults showed that the arousal levels were dramati-cally higher following explosions in the environment.

Another first responders application isUnrealTraige (McGrath & Hill, 2004), built usingthe Unreal engine, which simulates emergencyresponse to a mass casualty plane crash. The objec-tive is to provide fire-fighters and emergency med-ical technicians with an environment to rehearse firesuppression and primary triage. Hazmat: Hotzone


was also developed on top of Unreal engine to trainfire-fighters how to handle hazardous material. Forthe police force, the commercial game SWAT 44

could be used for training. In this game, a player canbe a team leader of non-player characters (NPCs) orjoin teams of human players. The game as it standshas a number of scenarios that can be used as theyare and the ability to modify the game potentiallymakes it a very good platform for SWAT training.

3. THE VIRTUAL TRAINING ENVIRONMENT

In traffic investigation training courses, there arethree domains of learning: knowledge, skills, andattitudes. In this training environment we mainlyfocus on knowledge and skills. Figure 1 shows anexample of the accidents we travelled to as part of afield study we conducted to get first-hand under-standing of the learning environment for novices.The instructional objective of this training is toenable participants at the end of the experiment tobe able to carry out the following procedures:

• Search for and identify clues, and secure them bymarking their position

• Park the patrol vehicle at an appropriate spot tohelp secure the accident scene and warn oncom-ing traffic of the accident.

• Use traffic cones to secure the accident scene.• Photograph the accident scene (e.g. vehicles,

clues, whole scene).• Take required measurements to enable recon-

struction of the accident scene.• Draw the accident scene – this is required to

complete the accident file.

3.1 Development

The training environment was built on top of a soft-ware architecture developed at the University ofSheffield (BinSubaih et al., 2005b) which has a com-ponent called Game Space that is linked to twogame engines (Torque and a bespoke engine). Wechose to use the Torque game engine because itcatered for the features required for this project. Tobuild a game using this architecture there are four



Figure 1 Images from the field study


steps. First, you create the game world (e.g. charac-ters, roads, vehicles, buildings, etc) using modellingtools such as 3D Studio Max and export them to theTorque format. Second, you create the interface ofthe game – Torque provides an interface builder(GUI Editor). Third, you create a game model usinga tool called OntRAT (BinSubaih et al., 2005c),which comes with the architecture. Finally, you addthe game behaviour through an API or using thePython scripting language.

3.1.1 Game WorldFigure 2 shows a view of the 3D environment. Theroads were created using 3D Studio Max and thebuildings and vehicles are from freely-availableonline sources (3D Cafe5 and TurboSquid6), whichwe modified to suit our requirements, e.g. the policecar was customised and damage was added to othervehicles. The characters are also from onlinesources and are modified to fit in with Arabic cul-ture. The faces for the models were generated usingFaceGen7, and then exported to the Torque format.The game world was built using the Mission Editortool provided as part of the Torque engine. Eachactive object (i.e. one that needs to be manipulatedas opposed to a decorative object) has to be given aunique ID which will then be mapped to its associ-ated object in the Game State (see Section 3.2.4).

Our architecture supports a text-to-speech synthe-sizer, however the Arabic version lacks quality andinstead we used recorded actors’ voices for dia-logues.

3.1.2 InterfaceThe interface shown in Figure 2 was created usingthe Torque GUI Editor. The menu on the right hasthe following items starting from the top: timer,compass8, camera, measuring wheel, traffic cones,markers, investigator folder, and radio. In addition,four green arrows appear at the bottom of the menuwhen an object is selected, in order to allow theobject to be moved. The user can navigate using thekeyboard arrows, and the mouse wheel is used tolook up and down. Figure 3 shows another interface,developed in Java, to display Arabic text. At the timeof development of the system this was necessary, asTorque 1.3 did not support Unicode (which is need-ed to display Arabic text). A more recent version ofTorque (version 1.4) has added support for Unicode.

3.1.3 Game ModelThe architecture stores a game model using ontolo-gies (Chandrasekaran et al., 1999). This is comprisedof objects and properties and allows inheritance.The difference between it and object-orientedclasses is that it can be created on-the-fly and does


Figure 2 Example of the view of the 3D environment

not need a hard-coded representation. This makesit easier to build and understand. Our ultimate aimis to allow domain experts to build these to unifythe terminologies used and share accident experi-ences by reconstructing scenarios. Figure 4 shows asample of the game model.

3.1.4 Game StateThe game state resides inside the Game Space com-ponent and holds objects that have virtual represen-tation in the game world (i.e. active objects). Wecall these objects the mapped objects and they needto have the same unique ID used in the game world.The game state can be added using the Pythonscripting language and it gets stored in a persistentdatabase (MySQL).

3.1.5 Game BehaviourTo understand the behaviour required we give the

following brief walkthrough of a session. A gamesession starts with the investigator standing besidehis patrol vehicle waiting for an incident call. Uponreceiving and accepting the deployment, the inves-tigator is put into a car and gets driven to the acci-dent scene – we are training the officer in chargerather than the driver. During the travel his role isto communicate with the operation room to findout more details about the incident (such as whoreported it, seriousness, number of vehiclesinvolved, etc.). After arriving at the accident scene,the investigator is placed outside the vehicle and hecan start attending to the accident. His first role isto secure the accident scene by clicking on thepatrol vehicle and moving it to an appropriate spot.Then he can search for injured people and requestadditional resources (i.e. an ambulance) from theoperation room. After that he can carry out othertasks such as asking questions, examining the scene,placing markers, taking photographs, taking meas-urements, etc.

The behaviour is controlled from the GameSpace which receives updates and sends actions tothe game engine. To demonstrate this we considerthe action of requesting an ambulance. When theinvestigator clicks on the radio icon, the click issent to the Game Space which updates its GameState. This state is monitored by a behaviourcontroller which, in this instance, sends an action tothe game engine to display the operator interfaceshown in Figure 3 and synthesize a messageacknowledging the action and requesting the inves-tigator to specify what resources are required. Thenthe user can select an ambulance from the resources



Figure 4 Sample of the game modelFigure 3 An interface developed in Java to display Arabic text


and specify the number required. Similarly to theclick action this is updated to the Game State. Ifthere are ambulances free at the time of requestthen the behaviour controller creates a finite statemachine (FSM) to handle the resource. Thisinvolves dispatching the ambulance to the scene,dropping the paramedic, walking the paramedictowards the investigator to debrief him, and waitingfor further instructions. If nothing further isrequired he heads back to the ambulance and drivesaway.

3.1.6 Logged Data For After Action ReviewThe user’s actions that are logged by the system,along with the time, are: navigation, object selec-tion, photographing, moving objects, addingobjects, measuring, resource interactions, and ques-tions. These are used for after-action debriefing andalso stored in the investigator profile. After a num-ber of these actions a dialogue box appears and asksthe participant to record the reason the action wastaken, e.g. measuring, placing cones, etc. The rea-son is recorded to help the trainer gain understand-ing into the participant’s thinking and for the par-ticipant to reflect on his actions.

3.1.7 Software and HardwareThe architecture used was developed with Java,Python, and MySQL database. The training envi-ronment runs on a windows-based PC. For thisexperiment we used two relatively high-specificationsystems: a desktop PC (17-inch, 512MB RAM and32MB graphics card) and a laptop (15.4-inch, 2GBRAM and 256MB graphics card). We tested both tomake sure that the game play remained smooth bykeeping the average frame rate at approximately 20frames per second (FPS). We believe the differenceof 1.6 inches in screen size is too small to have hadany significant effects on presence or performance.The research we found that points to the effect ofscreen size on presence (Laarni et al., 2005) and taskperformance (Tyndiuk et al., 2004; Patrick et al.,2000) was for screens where one screen was morethan double the size of another.

4. EVALUATION OF THE VIRTUAL TRAINING ENVIRONMENT

In this section, we report on the training effective-ness of the game-based environment we havedesigned and developed for the Dubai traffic inves-tigator. Measurements for two groups of investiga-tors are presented: novices and experienced person-nel.

4.1 Method

Fifty-six participants were selected randomly fromtraffic investigators in the Dubai Police force. Wewanted two main groups: novices and experienced.The one restriction we placed on any candidate wasthe ability to use a computer. The sample averageexperience is 6.69 years (SD=8.87 and median=1). Allthe participants were males. Seven participantswere dropped for various reasons (2 for study leave,1 for special assignment, 1 for sick leave, 1 felt pres-surized by the experiment and requested to stopafter the first training session, 1 due to simulatorsickness, and 1 due to unrecorded data in the sec-ond training session).

The experiment design is shown in Figure 5. Itconsists of two primary sessions. The first sessionhas three parts: agreeing and signing the experimentconfidentiality agreement, pre-test, and first ques-tionnaires. All participants went through the firstthree parts. After that the pre-test results were cal-culated and they were used to divide participantsinto two groups (A and B) with similar performanceaverages. Group A is the control group and group Bis the one that was trained. These groups (A and B)are further divided into two groups based on theirexperience (novices and experienced): novices A,novices B, experienced A, and experienced B.

The control groups have two main roles. Thefirst role is to control the experiment stages toensure that the pre and post tests are of similardifficulty levels. The second role is to use theirresults to measure the effect training has by com-paring them against the trained groups. In session2, as shown in Figure 5, groups A and B followeddifferent routes. Groups A only took part in thepost-test whereas Groups B went through fourparts: familiarization, training, post-test, and asecond questionnaire.

4.2 Measures of Performances

4.2.1 Tasks MeasuredThe investigator performance was measured basedon the successful completion of the followingtasks: securing the accident scene (parking policecar (10.5%) and placing cones (7%)), photographingthe accident scene (17.5%), taking appropriatemeasurements (21%), placing markers at importantclues (14%), and drawing the accident scene (30%).Two trainers approved the marking scheme.

4.2.2 Training SessionThe training session accident scenario involves a


collision between two vehicles as shown in Figure 2.One of the drivers is slightly bruised. The investiga-tor has 30 minutes to complete the investigation.Before the session starts the investigator is told thathe will receive two reminders: one after 15 minutesand one 5 minutes from the end, which will alsoremind him that the drawing should be also accom-plished within the 30 minutes. After the sessionends the trainee goes through self-assessing wizardswhich show a list of tasks that had to be completedfor the different tasks, and the user is asked to tickthe ones accomplished. The system then generatesthe results. The user examines the results for 10minutes before the next session starts. All the ses-sions were video taped for further analysis.Participants who do not achieve a score of 70% orabove are asked to take the training session oncemore. This session proceeds in a manner similar tothe first.

4.2.3 Pre- and Post-TestThe pre- and post-tests consisted of two parts: a

written test and a drawing test. The written testcomprised of a short explanation of how the acci-dent happened, a 2D drawing of the accident scene,and a set of questions. In the drawing test theinvestigator is given a description of a differentaccident to the one used in the written test andthen allowed to draw the accident scene by exam-ining a 3D environment. The accident scenariosused are shown in Figure 6. To stop any learningfrom taking place, no self-assessment is conductedafter each test and the marking is done by the facil-itator afterwards.

5. RESULTS

5.1 Performance

Figure 7 plots the performance distribution of allthe participants for the pre- and post-tests. Table 1shows the total average performance scored andthe improvement that occurred. There are two



Figure 5 Experiment design


important checks to perform before analysing theresults. The first check needs to confirm that thepre- and post-tests are of similar difficulty levels andthe second must examine that the grouping processhas managed to divide the groups equally by per-formance. To verify the first check we examine the

differences in the average performances in pre- andpost-test levels for the control groups (novices Aand experienced A). The average performance dif-ferences in the novices A and experienced Aare 2.79% and 2.53% respectively. The t-testresults shown in Table 2 confirmed that there is no


Figure 6 Pre and post written tests (written and drawing)

Table 1 Average performance score and improvement

significant difference between the pre- and post-tests for both novices and experienced which sug-gests that the difficulty level is similar.

The second check verifies the grouping processwhich had to divide the novices and experiencedinto groups of equal level of performances. By com-paring the pre-test results for novices A and B wefind that they scored 37.25% and 40.04% respec-tively for a difference of 2.79%. The t-test con-firmed that there is no statistically significant dif-ference between the pre-tests for novices A and Bas it was t(24)=-0.85 (p>0.05, t critical two-tail=2.06). Similarly there was no significant differ-ence between the experienced groups A and B whoscored 49.33% and 51.86% respectively for a differ-ence of 2.53%. The t-test result was t(21)=-0.89(p>0.05, t critical two-tail=2.08). This shows thatthe grouping process achieved its aim.

Analysing the performance in Table 2 shows thatboth trained groups (novices B and experienced B)have managed to improve their performances by36.17% (t(15)=17.01, p<0.05, t critical two-tail=2.13)and 23.54% (t(13)=7.88, p<0.05, t critical two-tail =2.16) respectively. These results confirm that thetraining condition managed to significantly improvethe performance. The results stayed significanteven when progressively reducing p (alpha) to 0.005and 0.001.

The difference of improvements between novicesB and experienced B shows that the formerimproved by 12.63% more. A t-test is used to exam-ine the significance of this difference. At the pre-

tests the t-test confirmed that there was a signifi-cant difference between the two groups (t(28)= -4.13,p<0.05, t critical two-tail=2.05). If at the post-testthis gap still exists (i.e. more than the t critical) thenwe can conclude that there is no significant differ-ence, and vice-versa. The t-test for the post-testswas t(28)=0.23 (p>0.05, t critical two-tail=2.05) whichconfirms that the significant difference found at thepre-test was nullified at the post-tests which impliesthat the improvement is statistically significant.

Figure 8 shows the learning trends of all thegroups. The lines for novices A and experienced Aseem to exhibit a very small improvement, men-tioned above, which verifies that the two tests areof equal difficulty levels. However the two B groupsshow an interesting line of one drop and two rises.The drop occurs between the pre-test and the firsttraining session where novices B and experienced B



Table 2 t-Test: paired two sample of means

Figure 7 Performance distribution for both groups


dropped by 9.07% (t(15)=3.74, p<0.05, t critical two-tail=2.13) and 15.85% (t(13)=3.95, p<0.05, t criticaltwo-tail=2.16) respectively for an average drop of12.46%. This is then followed by sharp risesbetween training 1 and training 2. Novices B rose by45.14% (t(15)=-14.29, p<0.05, t critical two-tail=2.13)and experienced B rose by 31.37% (t(13)=-7.77,p<0.05, t critical two-tail=2.16). The average rise is38.26%. The second small rises recorded occurredbetween training 2 and post-test 2 where novices Brose by 0.1% (t(15)=-0.03, p>0.05, t critical two-tail=2.13) and experienced B rose by 8.02% (t(13)=-2.159, p>0.05, t critical two-tail=2.16) for an averagerise of 4.06%. The t-tests show that the first dropsand the first rises for both groups are statisticallysignificant whereas the second rise is not. Section 6discusses probable causes of the exhibited trends.Table 3 shows the breakdown of performances bytasks. The objective is to examine if there are anyindicators that can suggest the suitability of thistraining for some tasks more than others. The bestaverage improvement for both groups (novices andexperienced) occurred for the photographing task(45.08%) and the worst improvement recorded wasfor parking the police car task. If we examine eachgroup individually we see that photographing is thehighest in the experienced but it comes fourth innovices (after placing cones, taking measurements,and marking tasks, respectively). T-tests confirmedthat the significant improvements for novices

occurred in the following tasks: measuring, mark-ing, photographing, placing cones, and drawing.Experienced personnel, however, showed signifi-cant improvements in: measuring, marking, photo-graphing, and drawing.

It was not surprising to see the photographingtask top the best improved task for the experiencedinvestigator, because of what was found during anearlier study (BinSubaih et al., 2005a). In this earlierstudy, despite the fact that the investigator isexpected to accompany and instruct the photogra-pher to the important clues that need to be pho-tographed, it became a habit with a number ofinvestigators to allow the photographer to wanderalone and take the photographs that he judgedappropriate. The problem with that is that the pho-tographer is not aware of the sequence of actionsthat led to the accident and thus cannot determinethe clues that need to be photographed. Some pos-sible explanations for the investigator’s behaviourhere are: time urgency and the culture of collabora-tion. There might also be an element of trustbetween the investigator and the photographer asthey have most likely previously worked togetheron a number of occasions. As one experiencedinvestigator revealed in the debriefing session, theyinitially accompanied and instructed the photogra-pher, but with time this became a lower priority. Inthe virtual training environment the trainees haveto take pictures themselves. This might provide an


Figure 8 Learning trends

explanation for why the photographing task was themost improved task for experienced investigators.

5.2 Accident Scene Navigation

Two navigational patterns were examined: distancestravelled and time spent in motion. Novices trav-elled on average 332.91 meters (SD 190.73) in training1 compared to 560.01(SD 164.74) meters in training2. The t-test result was t(15)=-5.495 (p<0.05, and tcritical two-tail). It confirmed that there is a signifi-cant difference between the two training sessions.Experienced investigators travelled on average380.34 (SD 163.24) meters in training 1 compared to585.72 (SD 163.94) meters in training 2. The t-test(t(13)=-3.396, p<0.05, t critical two-tail=2.160) forexperienced investigator also confirmed that there isa significant difference between the distances trav-elled in the two training sessions. Contrasting train-ing 1 for novices and experienced investigators didnot reveal any significant difference (t(28)=0.726,p>0.05, t critical two-tail=2.048). Similarly no signif-icant difference was found when contrasting training2 for novices and experienced investigators(t(28)=0.427, p<0.05, t critical two-tail=2.048).

The second pattern is the time spent in motionwhich includes movement and rotation. Novices onaverage spent 5:26 minutes in training 1 and 8:08 min-utes in training 2. A t-test confirmed that the differ-ence between the two training sessions was signifi-cant (t(15)=-5.78, p<0.05, t critical two-tail=2.13).Experienced investigators spent on average 6:30 min-utes in training 1 and 9:14 minutes in training 2. A t-test confirmed that the sessions are significantly dif-ferent (t(13)=-3.59, p<0.05, t critical two-tail=2.16).Comparing training 1 for novices and experiencedinvestigators showed that there is no significant dif-ference (t(28)=1.35, p>0.05, t critical two-tail=2.048).Similarly no significant difference was found betweennovices and experienced investigators for training 2(t(28)=1.58, p>0.05, t critical two-tail=2.048).

Both navigational patterns examined showed sig-nificant difference between training 1 and training2. Both novices and experienced investigators spentmore time and travelled more in the second trainingsession compared to the first. However they failedto show any statistically significant difference whencomparing novices to experienced investigators.

5.3 Sense of Presence

A presence questionnaire similar to the one in (Slater,1999) was used to measure the subjective experiencefelt by the participants of ‘being there’ in the accidentscene. It contains 23 questions with scores between 1to 7 and one open-ended question. Table 4 showsthat both groups recorded very similar presence aver-ages. The t-test also confirmed that there is no sig-nificant difference between the two groups (t(28)=-0.68, p>0.5, t critical two-tail=2.05). To examine ifthere exists any correlation between presence felt andthe performance achieved we performed a Pearsonproduct-moment correlation test which revealed thatfor the novices there is a small negative correlation(r=-0.16) and for experienced investigators there is asmall positive correlation (r=0.15). Combining bothgroups the correlation becomes insignificant (r=0.07).This interpretation is based on (Cohen, 1988) whosuggested that, for correlations in psychologicalresearch, a correlation between 0.10 and 0.29 (or–0.29 and –0.10) is considered small. Figure 9 showsthe correlation diagram.

The correlation found between presence and per-formance was small and not what we believed wouldhappen. We believed that novices (younger genera-tion) would be more at ease with this type of envi-ronment and would feel more presence compare tothe older participants. However this was not thecase as the novices showed negative correlationcompared to positive correlation shown by theexperienced investigators. The exhibited positivecorrelation is much smaller than the one reported in



Table 3 The breakdown of performances by tasks

Table 4 Presence


VERTS (Youngblut & Huie, 2003) which was at0.42. One explanation for this could be the use ofdialogue boxes to ask the student to reflect aftereach action to justify what he did (e.g. after a meas-urement is taken the system asks why he performedthat measurement). The boxes could have brokenthe sense of presence.

One explanation of the fact that the experiencedinvestigator’s sense of presence increases with per-formance, instead of decreasing as for the novices,could be that the nature of their expertise is basedon the ‘expert recording set of schema’, whichguides their problem-solving, and which novices donot have (Chi et al., 1998). We speculate that expertsuse the stimuli from the simulation as a trigger forprevious memories and have a more “internal” expe-rience thus paying less attention to the presence-breaking stimuli (e.g. dialogue boxes). Novices mayhave a more “external” experience due to their lackof experience and pay more attention to the pres-ence-breaking stimuli.

5.4 Comments

At the end of the experiment, participants wereasked open-ended questions to describe their expe-rience, likes and dislikes, and any suggestions theymay have. Figure 10 presents the comments madeby 24 out of the 30 participants who were trained, inthree categories: positive comments, negative com-ments, and suggestions.

6. DISCUSSION

The results revealed that the training condition hada statistically significant effect for improving the

performances of novices and experienced investiga-tors. This remained significant even when reducingthe alpha level to 0.005 and 0.001. The results alsoshowed that the improvement recorded for noviceswas higher and statistically more significant com-pared to the one found in the experienced investi-gators. However this did not hold when reducingthe alpha level to 0.005 and 0.001. Therefore witha confidence of at least 95% we can conclude thatnovice and experienced participants trained on thisenvironment for one hour and forty minutes shouldexpect their average performance to improve by36.17% and 23.54% respectively.

6.1 Performance Trends

Interesting learning trend lines were exhibited inFigure 8. Both trained groups’ trends showed onedrop followed by two rises. The control groups didnot exhibit any significant differences between thepre-test and post-test.

6.1.1 Sharp DropsThe relatively sharp drop occurred between the


Figure 9 The presence and performance correlation

Figure 10 Comments made by 24 participants

pre-test and the first training session (training 1).We do not have definitive answers of why thisoccurred. However we can speculate on a number offactors that might have contributed to it. The firstfactor is the unfamiliarity with the environmentdespite the efforts made in the familiarization stageto avoid this problem. We expected the steps wetook of supplying participants with a written tutori-al and video demonstration days before the secondsession, and giving each a maximum of 30 minutestraining, would have addressed this issue.Unfortunately some of the participants did not readthe tutorial or view the demonstration and for thosewe instructed them to take some time before thesecond session to do so. The familiarization timesrecorded show that only 4 novices compared to 8experienced investigators had to redo this sessionbecause they could not complete the tasks the firsttime round. On the second run all 4 novices man-aged to finish on time compared to only 4 of the 8experienced investigators. When asked about thereasons why they could not finish on time the issuesraised were difficulty with navigation and unfamil-iarity with using 3D technology. These issues wereaddressed by giving them more time to practiceuntil they were happy to move to the next stage.Some exhibited the inability to press-and-hold a keywhen navigating. Instead they opted for repetitivefast clicking which wasted some time and we had tospecifically instruct them to try to press-and-hold tospeed up their navigation. Others exhibited issueswith the 3D technology by their body movement(e.g. tilting their head to the right or left to lookbehind things) and by the type of questions theyasked (e.g. how can I make the picture of the carlook larger). Of the 4 experienced investigators thatcould not finish the second round of the familiar-ization session on time, only 2 needed the full-allowed time to complete training session 1. Thissuggests that the effect of unfamiliarity on the sharpdrop could only have affected 2 experienced partic-ipants and it did not contribute to the novices drop.

The second factor is the insufficiency of timeallocated for the training session compared to thatallocated to the pre-test. The recorded averagetimes to complete training session 1 for novices andexperienced investigators were 25:32 minutes and27:08 minutes respectively. Eleven novices (i.e.68.8% of all novices) compared to eight experiencedinvestigators (i.e. 57.1% of all experienced investiga-tors) managed to finish the training session beforetime was up. These findings suggest that the proba-bility of this factor being the cause of the drop ishigher in the experienced investigators than it is inthe novices.

The third factor is the varying difficulty levels ofthe accident scenarios, i.e. the training scenario isharder than the pre-test scenario. Since the per-formances achieved by the control groups con-firmed that there is no statistically significant dif-ference between the pre- and post-tests, then prov-ing that there is no significant difference betweenperformances achieved by the trained groups fortraining 2 and the post-test should suggest a reduc-tion of the effect of this factor. This was confirmedby the t-tests which showed that the difference isnot significant. The experienced participants aremore likely to be affected by this factor when com-paring their average to the novices.

Table 5 summaries the effects of the three factorson the novices and experienced investigators. Forthe three factors, the effect on the experiencedinvestigators is higher than on the novices. Fromthe three factors, we believe that time was the mostinfluential, followed by unfamiliarity and difficultylevel. Time might have added pressure as it wasmentioned as one of the causes for one experiencedparticipant to drop out, as he felt pressurized. Aphysiological sensing device might have provided uswith a stronger sense of whether or not pressure wasa factor. However, we would still have to determineif pressure was linked to time or other experimentsettings.

6.1.2 First Sharp RisesThis section examines the causes for the sharp risesthat occurred between training 1 and 2 (45.14% fornovices and 31.37% for experienced investigators).The t-tests described in the results section con-firmed that both rises are statistically significant.The unfamiliarity factor that affected the dropmight have also contributed inversely to this risesince each participant had another 30 minutesworking with the system which accounts for a com-plete familiarization session with two runs. Butsince the unfamiliarity only affected 2 experiencedparticipants during the sharp drop, this suggeststhat its effect is very small if not negligible.

The other two factors (time and difficulty levels)that affected the drop are unchanged – the trainingsession time remained 30 minutes and the accidentscenario used is the same. The novices’ average timefor training 2 compared to training 1 increased from



Table 5 Factors effect probability on both groups


25:32 minutes to 30:04 and the experienced investi-gators’ time increased from 27:08 to 31:06. Since wehave used the same scenario for both training ses-sions, students might have memorized what needsto be done for this specific accident scenario andjust applied that. In games terminology they mighthave learned to beat the game. However, if this wasthe case, it would not explain why their average per-formance stayed high at the post-test which uses adifferent accident scenario. This suggests that thereis another factor that is influencing this sharp riseand we believe it is the training condition/factor weintroduced. The results are statistically significantand the comments were very positive which pointstowards this factor.

6.1.3 Second Small RisesThe second small rises recorded occurred betweentraining 2 and post-test 2 for novices and experi-enced investigators and were 0.1% and 4.06%respectively. None of these is statistically signifi-cant as confirmed by the t-tests.

6.2 Addressing the Current Training Issues

Besides improving the performance there are otherfindings that suggest the potential suitability of thisenvironment to address the problems with the cur-rent training process at Dubai police discussed inthe introduction to this paper. The first issue raisedwith the on-the-job training is the impracticality ofthe environment due to the lack of repeatability andexploration. Our VR environment allows studentsto repeat practice as many times as they feel neces-sary to improve their skills. Since they can practiceon their own they can explore different optionswithout fear of failure or embarrassment.

The second issue was the varying levels of expo-sure. The architecture used allows multiple scenar-ios to be run on it. This feature means that we canhave different scenarios to suit the different kindsof accident types the trainers feel necessary for thestudent to experience. We have developed threetraffic accident scenarios on this platform. Thethird issue was the lack of uniform assessment. Inthis environment we showed how the implementedperformance metrics were more systematic and fair.

The study also suggests that there is a potential forcreating a social interaction around the training envi-ronment between students themselves and with thetrainers. This was facilitated by the ability to recordstudents’ missions in their profiles which means thatthey can be shared and may be used for training pur-

poses. Another potential use for the environment isas a platform for sharing the experiences of an ageingworkforce. The environment records users’ missionsfor after-action review. This data can be used to shareexperiences. A sample of the data generated for eachmission is shown in Figure 11. The figure shows threeoutputs: the path taken by the investigator to scanthe accident scene, the recorded interactions orderedto show how he prioritized the tasks, and finally adetailed report of his score.

Although the game-based environment managedto address the issues with the current real-life train-ing, we do not see it as replacement but rather as asupplement. The trainer is still at the heart of thegame-based training process, with three major roles.The first role is the creation of the accident scenar-ios and their instructional objectives. The second roleis in breaking the training scenarios into manageablechunks that could be administered during the class-room or as homework assignments. The third role isrunning and overseeing the after-action review ses-sions. The best way to augment the game-based envi-ronment requires more research to find the mostapplicable to the traffic investigation domain. A panelsession in the recent game developers conference pro-posed options such as: using games that work in smallbites, building support tools to help in-class use, andchanging the nature of the class structure.

7. CONCLUSION AND FUTURE WORK

Serious games are finding wide acceptance and usagein many domains. The main motivation of our workwas to examine the applicability of this technologyin the police domain. We have demonstrated how aserious game to train police traffic accident investi-gators can be built on top of a game engine. We havealso run an empirical study to compare the effective-ness of such an environment as a performanceenhancement tool. The results showed that therewas a significant improvement in both the novicesand experienced personnel that were tested. Theresults also showed that the improvement recordedin novices was significantly higher than the onereported in experienced personnel. However theenvironment did not find any significant differencein the navigational patterns between novices andexperienced personnel. However, we are not sure ifthis improvement would be transferred to the realenvironment and in the future work we would like totest this hypothesis. The positive comments madeby participants in the experiment give us confidencethat there is a strong possibility of this happening.


ACKNOWLEDGMENTS

The project was sponsored by a grant from Dubaipolice. The authors wish to thank the trainers whohelped in developing the performance metrics and giv-ing us invaluable feedback on the environment. Wealso wish to thank the police officers who took part inthe experiment and those who assisted us in testingthe environment. Finally, we would like to thank theactors whose voices were used in the experiment.

NOTES

1. http://www.seriousgames.org2. http://www.gamesforhealth.org3. http://www.fullspectrumwarrior.com4. http://www.swat4.com5. http://www.3dcafe.com6. http://www.turbosquid.com

7. http://www.facegen.com8. Adopted from Garage Games Modding Community9. One novice participant failed to return the presence

questionnaire10. Two participants time were not recorded because the

application was closed prematurely11. Can Serious Games Work in 45-Minutes?

https://www.cmpevents.com/gd06/a.asp?option=G&V=3&id=425144 (accessed August 2006)

REFERENCES

Aldrich, C. (2005) Learning by Doing: The Complete Guideto Computer-Based Simulations. Pfeiffer Wiley, ISBN:0787977357.

BinSubaih A., Maddock S. and Romano D.M. (2005a)Comparing the use of a tabletop and a collaborativedesktop virtual environment for training police offi-cers to deal with traffic accidents: Case study.International Conference on Engineering Education, July25–29th, Gliwice, Poland, Volume 2, pp. 94–100,



Figure 11 Data recorded for after action review


ISSN 1562-3580.BinSubaih A., Maddock S. and Romano D.M. (2005b)

Game Logic Portability. ACM SIGCHI InternationalConference on Advances in Computer EntertainmentTechnology ACE 2005, Computer Games Technologysession, June 15-17th, Valencia, Spain, pp. 458–461,ISBN 1-59593-110-4.

BinSubaih A., Maddock S. and Romano D.M. (2005c)OntRAT: Ontology-Based Rules Acquisition Tool.m-ICTE2005, 3rd International Conference on multime-dia and Information & Communication Technologies inEducation, June 8-10th 2005, Caceres, Spain, Volume3, pp. 978–983.

Carless, S. (2005) Postcard From SGS 2005: Hazmat:Hotzone – First-Person First Responder Gaminghttp://www.gamasutra.com/features/20051102/car-less_01b.shtml, November 2, (2005) (accessed June2006).

Chandrasekaran, B., Josephson, J. and Benjamins, V.(1999) What are ontologies and why do we needthem? IEEE Intelligent Systems, Jan/Feb (1999), 14(1),pp. 20–26.

Chetley, A. and Vincent, R. (2003) Learning to share learn-ing: An exploration of methods to improve and share learn-ing. A report prepared for the UK Commission forHealth Improvement, March 2003. http://www.healthcomms.org/resource/learning-chi.html(accessed June 2006)

Chathman, R. (2005) A Tale of Training Superiority, Games,and People Stuff, August 2005, DARPATech 2005, pp51–55.

Chi, M., Glaser, R. and Farr, M. (1988) The Nature ofExpertise. Hillsdale, NJ: Erlbaum.

Cohen, J. (1988) Statistical power analysis for the behavioralsciences (2nd ed.) Hillsdale, NJ: Lawrence ErlbaumAssociates.

Diller, D. and Roberts, B. (2005) DARWARS Ambush!A Case Study in the Adoption and Evolution of a Game-based Convoy Trainer by the U.S. Army. SimulationInteroperability Standards Organization, 18-23September 2005.

Dobson J. (2006) Serious Games Initiative/Games ForHealth At E3 2006: A Look Back. 26-5-2006,http: / /seriousgamessource.com/features/fea-ture_052606_games_for_health.php. (accessed June2006)

Hall, R.H., Wilfred, L.M., Hilgers, M.G., Leu, M.C.,Walker, C.P. and Hortenstine, J.M. (2004) VirtualTerrorist Attack on the Computer Science Building:A Research Methodology. Presence-Connect, 4(4),http://www.presence-connect.com.

Harz, C. (2006) Games for Learning: SeriousEntertainment. April 26, 2006, http://mag.awn.com/index.php?ltype=pageone&article_no=2863.(accessed June 2006)

Johnson, W. L., Marsella, S. and Vilhjalmsson, H. (2004)The DARWARS Tactical Language Training System.The Interservice/Industry Training, Simulation andEducation Conference, 2004, Orlando, FL.

Laarni, J., Ravaja, N., Saari, T. (2005) Presence experi-

ence in mobile gaming. Proceedings of DIGRA 2005.Lewis, M. and Jacobson, J. (2002) Game Engines in

Scientific Research. Communications of the Associationfor Computing Machinery (CACM), NY: ACM 45(1),2002.

Marthi, B., Russell, S. and Latham, D. (2005) WritingStratagus-playing agents in Concurrent Alisp, IJCAI2005 workshop on Reasoning, Representation, and Learningin Computer Games. Edinburgh, Scotland, July–August2005.

McGrath, D. and Hill, D. (2004) UnrealTriage: A Game-Based Simulation for Emergency Response. TheHuntsville Simulation Conference, October 2004.Sponsored by The Society for Modeling andSimulation International.

Pair, J., Allen, B., Dautricourt, M., Treskunov, A.,Liewer, M., Graap, K., Reger, G. and Rizzo, A. (2006)A Virtual Reality Exposure Therapy Application forIraq War Post. Proceedings of the IEEE VR2006Conference (Alexandria, Virginia (March 2006)), pp.64–71.

Patrick, E., Cosgrove, D., Slavkovic, A., Rode, J.,Verratti, T. and Chiselko, G. (2000) Using a largeprojection screen as an alternative to head-mounteddisplays for virtual environments. In Proceedings ofCHI 2000, vol. 2.

Re-Mission™ (2006) Outcomes Study: A Research Trialof a Video Game Shows Improvement in Health-Related Outcomes for Young People with Cancer.http://www.hopelab.org/remission.html. Visited14–June-2006. (accessed June 2006)

Romano, D.M., Sheppard, G., Hall, J., Miller, A. and Ma,Z. (2005) BASIC: A Believable, Adaptable SociallyIntelligent Character for Social Presence. PRES-ENCE 2005, 8th Annual International Workshop onPresence, 21–22 September 2005, University CollegeLondon, London, UK.

Slater, M. (1999) Measuring Presence: A Response to theWitmer and Singer Questionnaire, Presence:Teleoperators and Virtual Environments, 8(5), pp.560–566.

Tyndiuk, F., Lespinet-Najib, V., Thomas, G. and Schlick,C. (2004) Impact of large displays on virtual realitytask performance. Afrigraph 2004: 61–65

Ulam, P., Goel, A. and Jones, J. (2004) Reflection inaction: Model-based self-adaptation in game playingagents. In D. Fu and J. Orkin, editors, Challenges inGame Artificial Intelligence: Papers from the AAAIWorkshop. San Jose, CA: AAAI Press.

Youngblut, C. and Huie, O. (2003) The RelationshipBetween Presence and Performance in VirtualEnvironments: Results of a VERTS Study, IEEEVirtual Reality Conference 2003 (VR'03).

Zyda, M. and Sheehan, J. eds. (1997) Modeling andSimulation: Linking Entertainment and Defense, NationalAcademy Press, Washington, DC. Sept. 1997, ISBN:0309058422

Zyda, M. (2005) From Visual Simulation to VirtualReality to Games, IEEE Computer, September 2005,pp. 25–32.


Ahmed BinSubaih is a PhD candidate at the University of Sheffield. He received a MEng in software engineering from the University

of Sheffield in 2000. His research interests include game architecture and serious games. He is a member of the eLearning Guild.

Steve Maddock is a lecturer in computer science at the University of Sheffield. His research interests include computer facial model-

ling and animation, human figure animation, procedural modelling, and surface deformation techniques. He received a PhD in com-

puter science from the University of Sheffield in 1999. He is a member of Eurographics and ACM SIGGRAPH.

Daniela M. Romano is a lecturer in computer science at the University of Sheffield. Her research interests include the creation of vir-

tual environments for education and entertainment, as well as all aspects of virtual reality from VR technology to the sense of pres-

ence. After her masters degree in Computer Science at the University of Bari, Italy, she worked in industry for six years. Then, after

obtaining an ESRC CASE Studentship, she moved into academic research and completed her PhD in Computer Based Learning at the

University of Leeds in 2001.



A serious game for traffic accident investigators

Documents