Serious gaming for behavior change: The state of play

Björn Schuller, Ian Dunwell, Felix Weninger, Lucas Paletta

Pervasive Serious Gaming for Behavior Change –

The State of Play

ABSTRACT

Digital Games can change the way we behave – be it explicitly or implicitly, and whether

we are aware of it or not. This bears great potential for Serious Gaming, where behavior

change can be guided so as to be targeted, meaningful, and helpful for the player.

Particularly effective is the analysis of the human in every-day situations to provide an

immediate feedback loop. This, however, requires mobile behavior and affect analysis

“on the road” – a challenge in its own right. We summarize the current state-of-play in

this field and provide two selected show-cases: The ASC-Inclusion project aims to

improve affective and social behavior of children with Autism Spectrum Condition by

multimodal analysis and feedback. The MASELTOV project is designed as playful

pervasive aid for migrants to ease their daily-life interaction with locals.

Keywords: Introductory and Survey, Affective computing applications, Affect sensing

and analysis

s Richard Lindgard put it, “If you would read a man’s Disposition, see him

Game; you will then learn more of him in one hour, than in seven Years

Conversation.” In this sense, serious games hold great promises for affect and

behavior analysis of human players. Moreover, they allow changing behavior in a

positive way playful and pleasant to the user. A major lack, however, is often

generalization to the real-world, where the changed behavior needs to be applied.

Pervasive computing holds the promise to overcome this gap, as behavior can be partly

learnt in the environment where it needs to be applied. Furthermore, in current serious

games for behavior change the area of human-human interaction is rarely addressed,

although it bears great application potential in the inclusion of individuals on the autistic

A

spectrum or other target groups that experience difficulties in human-human

communication. In order to implement serious games for behavior change in the context

of human-human interaction, a holistic approach is needed that takes into account

behavioral cues from multi-sensory input, possibly including speech, video (facial

expressions and gestures), and physiological sensors. In this light, let us first give a

definition of serious games and then reflect the state of play in affect and behavior

analysis in these games, before taking a short glimpse at two exemplary case studies from

the context of teaching appropriate behavior in human-human interaction. Then, to fill in

the last piece of the puzzle, we will discuss ‘going mobile’ in automatic multi-modal

analysis of human behavior.

SERIOUS GAMES

The concept of “serious gaming” is often used to describe the use of digital gaming

technology to address a specific set of learning objectives, or behavioral goals. As such,

they often seek to build upon the increasingly pervasive role games play as an

entertainment medium to provide an engaging and entertaining way to communicate

educational content, and in turn, an efficient way of behavior analysis.

In this article, we focus particularly on the case of using games to induce a change in the

behavior of players, an objective which games have sought to achieve through a range of

means. Common to all these methods is the central role the game plays as either a

medium for conveying educational messages, or encouraging certain activities through

game-based elements such as competition or rewards. In the following section, these

underlying methods are examined in more detail, with the roles and potentials knowledge

transfer, gamification, and social learning may hold as tools for inducing behavioral

change explored.

BEHAVIOR ANALYSIS AND FEEDBACK IN SERIOUS GAMES

Two unique traits of games make them particularly interesting as tools for analyzing and

changing affect and behavior of players. The first is their universal appeal, coupled with

their ability to reach certain demographics traditionally resistant to other forms of direct

messaging or intervention, such as adolescents. The second is the ability games hold to

capture and retain a user’s attention for a significant period of time; consider, for

example, the average 90-minute usage of the online serious game Code of Everand

amongst its 100,000 players when compared to the 3-minute visit duration shown for

many static websites [1]. Yet, how to fully utilize this contact time to achieve a

behavioral outcome without compromising the ‘fun’ element of the game remains a

demanding task, particularly as behavioral outcomes can prove difficult to measure: self-

reported planned behavior can often deviate from that observed [2], and observation of

large samples over an extended period of time is seldom practical. However, it may be

possible to glean some insight into the behavioral impact of a serious game through the

analysis of player’s interactions within the game itself. The notion of video games as

research instruments is well established [3]; yet, how to understand the unique data that

can be captured through play remains a central topic of research.

As players might adopt an “intuitive” approach to play, whereby they willfully explore

wrong choices and worst cases as well as correct actions, understanding their level of

knowledge or attitude is likely not as simple as equating this to the “correctness” of their

in-game actions [4]. Indeed, a behaviorist paradigm in which a game attempts to replicate

intended behaviors in a virtual or gaming context has been argued as ineffective in many

cases, as players seek to defeat the game by circumventing rather than attaining its

intended behavioral outcomes [5]. Hence, it is important to explore how large-scale

capture of data from players might be ethically achieved, and used to more effectively

identify behavioral and attitudinal trends amongst them. Subsequently, games might be

adapted either to individual users [6], or in response to large-scale understanding of

efficacy.

Several models have been employed in serious games seeking to invoke a change in the

behaviors of players. The first is based upon knowledge transfer, conveying educational

content to learners so as to better inform their decision-making based on knowledge of

the consequences of certain behavior. Particularly amongst younger audiences, this can

prove effective: A positive behavioral outcome in young cancer sufferers who played the

game Re-Mission was observed in a randomized control trial when compared to an

entertainment game serving as a placebo [7]. In this case, the game sought to educate

players in the nature of their cancers and treatment, to support them in adherence to

treatment programs with short-term negative side-effects but long-term benefits. Hence,

the behavioral model underpinning this game was one of information transfer, exploiting

the engaging and entertaining medium of the game to appeal to a young audience who

may be more resistant to less immersive materials. A similar route has been taken with

games tackling childhood obesity [8], with the emergence of gaming hardware enabling a

more active user experience also being explored as a means of encouraging exercise

through gaming [9]. Whilst the benefits in terms of energy expenditure might not be

greater than other activities away from the confines of a digital gaming environment,

mobile devices and alternate-reality based gaming are increasingly allowing novel

approaches to combining gaming with active lifestyles [10, 11].

However, in some areas, knowledge transfer alone is unlikely to result in significant

changes in behavior. One example here is road safety, in which studies have shown that

in developed countries such as the UK, the problem does not stem from a lack of

knowledge, but failure to routinely apply it in practice [12]. There are multiple causal

factors behind this, such as social pressure, perception of risk, and negative reinforcement

cycles, where unsafe behavior goes unpunished until a serious accident occurs [13]. More

generally, these can be applied to a wide range of public health issues, such as smoking

and obesity. Environmental concerns also can be related to scenarios where individuals

know the correct behavior, but fail to apply it, leading to effects such as the “tragedy of

the commons”, whereby an individual’s knowledge of the long-term consequences of

their collective actions is outweighed by their short-term individual gain [14]. A range of

projects have sought to encourage individuals to lower their consumption through either

transferring information through play [15], use of location-based services [16], or

pervasive approaches which link consumption monitoring to game mechanics [17]. As

with healthcare, younger audiences have seen particular attention as a target audience for

serious games, with a range of games tackling power consumption explicitly developed

with such an audience in mind. Further, public engagement has also seen attention as a

key area in which games might be used [18].

More generally, gamification might serve as a means to create incentivization without

actual cost, rewarding people with virtual trophies, achievements, or other rewards given

intrinsic value through peer-recognition. Many online communities reward positive

behaviors with such awards, and the interface between real and virtual community may

prove a fertile ground for exploring how virtual rewards might be used to influence real-

world behaviors. A study of gamification in a mobile context for university students

demonstrated both the potential of the approach to engage students, but also several

drawbacks [19]. Game-based approaches are not universally welcomed, and in this case

could be perceived as making a resource less valued as a learning resource. The

“strictness” of game rules and level of difficulty are also noted as challenging to effect

without leading to usability issues. Given the recognized importance of usefulness and

ease-of-use in technology acceptance [20], these findings suggest gamification must be

carefully and selectively applied to avoid a negative outcome. This could be achieved by

adaptivity on an individual level, for example giving users the choice between the initial

resource and its gamified form, though this assumes users would be able to

introspectively select the ideal resource for their learning needs, a theory partly

contradicted by a number of studies [21]. A more comprehensive solution, therefore,

should seek to understand the learner more fully and provide them with the optimum

resource based on this understanding, a task which is the subject of continued research

[22].

Across sectors, an important next step is in understanding how to use the rich data

collected during play to adapt, personalize, and enhance the impact of serious games with

respect to specified behavioral objectives. Effective feedback is noted as central to the

efficacy of learning and behavioral outcomes in a range of studies, for example a minor

adjustment to the implementation of feedback during the development of the serious

game Triage Trainer showed a significant impact on overall efficacy [23]. From these

findings, a model was created which expresses the multiple levels of game-based

feedback based on an established generic framework, noting that whilst games can

readily support immediate, evaluatory feedback to learners, higher levels of interpretive,

probing, and supportive feedback either require sophisticated machine-driven analysis

techniques, or the involvement of a training professional. Supporting such professionals,

and adopting a tutor-centric as well as a learner-centric view of serious games, can prove

a key component of an effective development methodology [24]. An experiential

viewpoint based on Kolb’s established, cyclic model of learning in which action and

experience are met with reflection and conceptualization [25] can be complicated by the

level of abstraction introduced by a game; to address this feedback one must recognize

and adapt to the learner’s capacity to learn independently, as well as ensure a continued

match between learner ability and task difficulty, hence inducing a “flow” experience in

which the learner exclusively focuses on task [26].

A wide range of studies have sought to examine and identify behaviors unique to social

networks constructed in online games. Relationships formed in gaming communities may

prove shorter term and less stable than those in non-gaming environments, though they

may also prove more task-centric and engaging [27]. Peer or e-leader driven gaming

communities may offer another avenue for engendering behavioral change through social

learning principles and paradigms [28]. As social networks are increasingly used, how

these networks might best be understood and utilized to create models for social change

is a topic of ongoing research for the deployment of games.

TWO EXAMPLES

Let us now exemplify by two currently developed serious games how the next generation

of serious games can be used for improving the quality of life of individuals experiencing

difficulties in human-human interaction – by that, the focus will be on automatic analysis

of (affective) behavior: teaching individuals to ‘hit the right note’ in their verbal and non-

verbal expressions including emotional expressivity, speaking style, and body language.

Figure 1: Screenshots from the two projects. Top-row ASC-Inclusion: the virtual game

world (a research camp in which the children play a scientist researching on emotion,

left-most), one of the contained mini-games (the child players have to match facial

expression and speech by emotion, left of middle), karaoke-style emotion training (right

of middle), and a reward item (collectible card with one of several planets and its

description). Bottom-row MASELTOV: the virtual world where players first train in a

playful way (left two images) and the pervasive embedding of real-world interactions and

tasks that are evaluated for assignments of coins in the virtual world (right two images).

Autism Spectrum Condition Inclusion

The first example aims to help individuals with Autism Spectrum Conditions (ASC) that

often have social communication difficulties and restricted and repetitive behavior

patterns. Their affinity for computerized environment has led to several attempts to teach

emotion recognition and expression, and social problem solving using computer-based

training. As intervention is more effective early in life, a playful serious game approach

for the support of younger individuals with ASC could significantly promote their social

inclusion. The European ASC-Inclusion project1 creates and evaluates the effectiveness

of such an internet-based gaming platform. It combines a virtual game world with affect

and behavior analysis by users’ gestures, facial and vocal expressions using a standard

microphone and webcam for affective and social behavior training through mini-games

(cf. Figure 1, top-row). The exercises are partly “karaoke”-style imitation tasks of audio

and video clips that display target emotions from private and social categories. Feedback

1 http://www.asc-inclusion.eu

is given and in free exercises, the children’s affect is measured and information is given

to them in a dimensional emotion space.

Obviously, a particular potential is given by a mobile distributed approach: The children

can receive feedback in every-day life situations and the game can give tasks for real-life

social situations. In addition, parents and therapists can be offered on-line update and

feedback by distribution of the affect and behavior analysis. A first step into this direction

is currently the distribution of the vocal affect analysis – this allows playing “on the road”

on smart phone devices, as the game itself runs via internet browser.

Mobile Assistance for Social Inclusion and Empowerment of Immigrants with

Persuasive Learning Technologies and Social Network Services

As a second example, the European MASELTOV project2 recognizes the major risks for

social exclusion of immigrants from the local information society and identifies the huge

potential of mobile services for promoting integration and cultural diversity:

Everywhere/everytime – pervasive assistance is crucial for more efficient and sustainable

support of immigrants. Language understanding, local community building, and

consciousness and knowledge for the bridging of cultural differences shall be fostered via

the development of innovative social computing services that motivate and support

informal learning for the appropriation of highly relevant daily skills. A mobile assistant

embeds these novel information and learning services such as ubiquitous language

translation, navigation, administrative and emergency health services that address

activities towards the social inclusion of immigrants in a pervasive and playful manner:

Besides a virtual world, MASELTOV develops a mixed reality game (cf. Figure 1,

bottom-row) in which the user is applying her language skills in various, critical

situations, such as, during shopping, and for navigation in the urban environment. The

mobile service supports her in the situation as well as receives feedback from the user in

order to measure and estimate performance success. The success of an applied dialogue

in terms of the emotion and frustration of the user is sensed with the smartphone in situ,

using recent computational audio-based affective computing. Advanced human factors

2http://www.maseltov.eu

studies with wearable interfaces are further applied to extract the decisive parameters of

affective and attention oriented content in audio. Next, wearable eye-tracking glasses data

are interpreted with semantic 3D mapping of attention [29], bio-signal sensing, and

classification to automatically extract from a huge data analysis the decisive parameters

for dialogue evaluation. In practicing dialogue communication, the user can gain credits

in the serious game.

MASELTOV embeds an easily scalable context recognition framework [30] that receives

contributions from various context feature generating services; it evaluates the user

behavior and from this maps to appropriately motivating actions in the form of

recommendations. From long-term dialogue assessments with multimodal mobile context

awareness on the basis of affect and attention sensitive services it classifies the language

learning behavior of the recent migrant. The recommender system then instantiates –

according to the individual human factors profile and the measured performance –

personalized motivating games, in order to change the behavior of the user. For example,

to reinforce the training on interaction with local citizen, the rewarding of dialogue

supporting activities will be increased, such as, by doubling virtual credits in return for

dialogue specific language learning and measured communication in shopping scenarios.

GETTING GAME BEHAVIOR ANALYSIS MOBILE

Let us now sketch the challenges and opportunities of serious gaming enabling affect and

behavior analysis in mobile and pervasive environments. As is evident from the case

studies above, analyzing players’ behavior, basic emotions, and more subtle “states” such

as interest, confusion, frustration, or stress, can be of vital importance in a serious game.

Besides the voice, video from a smart phone camera [31] or physiological measurement

from mobile sensors [32] can be exploited.

A particularly engaging form of player’s behavior analysis is to blend the game with real-

world events. This does, however, require the games to be able to perform players’

(affective) behavior analysis “in the wild” [33]. In particular, it concerns real-time and

incremental analysis to provide low-latency system responses to changes in the user’s

state. A prototype of such real-time, incremental human behavior analysis has been

successfully implemented in the SEMAINE system3.

Mobile behavior analysis may also foster increased usage: For example automatic speech

recognition has been massively spread by deployment in mobile services. In a virtuous

circle, its usage in daily life has increased the availability of realistic data for research and

development of improved recognition technology, and even self-improvement of systems

[34]. Thus, implementing mobile behavior analysis applications opens an avenue to

remedy the scarcity of labeled, realistic data from the target domain and target users.

Ready-to-use mobile affect recognition services are currently emerging. Figure 2 shows a

simplified view of a human affect or behavior recognition system enhanced by a

distribution for shared mobile and server processing as used in the ASC-Inclusion and

MASELTOV projects. Components present in a standalone recognizer are depicted in

blue color while additional components required for a distributed client-server

architecture oriented on the ETSI standard for distributed speech recognition are shown

in green color. Let us first discuss the parts of a state-of-the-art affect and behavior

recognizer: The input signals such as voice (as is increasingly featured in games since

N64's Voice Recognition Unit as used in “Hey You, Pikachu!”, and more recently in

“Truth or Lie” or “Rainbow Six: Vegas”, Nintendo DS’s “Mario Party 6” and several

singing and Microsoft Kinect games), text, video or physiological data are captured,

typically from according sensors, and (optionally) preprocessed, including enhancement

of the signal of interest in noisy and disturbed conditions. Low-level descriptors (e.g.,

spectral bands or symbols) are extracted on a time frame-by-frame basis [35]. Chunking

(segmentation) then refers to the process of grouping frames into meaningful units, such

as words or connected movements, etc. This process is optional, if dynamic or recurrent

modeling is used in the recognition step. Otherwise, after grouping frames into chunks,

functionals such as statistical moments, percentiles, or peaks, can be applied. Semantic

features such as lexical or action units and other behavioral events can be converted to a

vector space representation, generally resulting in frequencies of occurrence features such

as “Bag-of-Words” vectors, or are related to open-domain on-line knowledge sources,

e.g., to determine their semantic distance from affective or behavioral concepts. In real-

3 http://www.semaine-project.eu

time systems, the chunking has to be applied based on human activity detection, which

can be already a challenging problem in adverse environments.

Figure 2: Schema of a mobile affect or behavior analysis system. s(k) represents the signal at

discrete time step k, x the feature vector, and y the gold standard label.

A database of feature-vector-label pairs is used to train the affect or behavior model and

potentially a temporal context model used in classification or regression. First results with

affect recognition on autistic children’s speech from the ASC-Inclusion project suggest

that binary classification of arousal and valence can be achieved with over 80% accuracy

while there is still room for improvement if a more fine-grained emotion categorization is

desired (43% accuracy on nine classes). In the context of serious gaming for therapy

purposes, these results highlight the necessity of appropriate confidence measures given

by the system in order to prevent inappropriate interventions taken by the system.

Ultimately, the information is forwarded to the game: In the ASC-Inclusion and

MASELTOV examples, gaming is centered around emotion analysis. In other games, the

entry point may be at the control of the dynamic difficulty setting [36] or the reaction of

non-player characters. The other way round, the game state can also be forwarded to the

recognition engine as contextual knowledge provision, e.g., if a stress-full or particularly

emotional sequence is started.

Moving to a distributed architecture, one aims to reduce the required transmission

bandwidth, and decrease storage cost. Information reduction also ensures privacy, as not

all (feature) information, e.g., from a microphone or camera is transmitted. This is

important considering the rather private nature of affect and behavior. In [35],

compression rates of 20 to 40 were found feasible without expecting significant decreases

in accuracy applying a sub vector quantization algorithm. In Figure 2, we can observe an

important feedback loop from result encoding to feature encoding, as the distributed

architecture allows future mobile services to rely on existing mobile services, generating

behavioral features and sending them to a server performing affect and behavior

recognition or vice versa.

Ideally, affective end user systems should be free in their choice of a server-side

recognition engine. While there are already standards for distributed speech recognition

and generic communication protocols such as web services widely used, there is a need

for standardization of feature extraction for affect and behavior recognition in general.

Standardization of recognition results to be sent to the client side for interpretation – here

the serious game management unit – is currently achieved by markup languages for

description of behaviour or affective states such as the W3C’s Emotion Markup

Language (EmotionML).

Pursuing affect and behavior recognition ‘on the go’ further immediately implies the

requirement of environmental robustness, particularly against (generally) non-stationary

noise sources and reverberation in the case of audio analysis or rotation, low lighting

conditions, and occlusions in the case of video analysis, etc. Besides compensation of

such disturbances, distributed recognition will need to cope with various transmission

channels and potential package loss.

There is another important feedback loop found in this system architecture, which allows

continuous improvement of the system by semi-supervised and active learning to collect

better suited data from the target group for labeling by the system or, e.g., crowd-

sourcing.

While a system like that in Figure 2 is not implemented so far, several parts are already:

semi-supervised learning, evaluation of transmission noise, and noise-robust processing

[37] leading to first mobile engines such as in [38].

LESSONS LEARNED AND FUTURE AVENUES

Summarizing, this review showed the great promises that serious gaming for behavior

change holds for the society – be it for inclusion and empowerment of minorities such as

autistic children or migrants, or for training on the job or cases of emergency [39]. It

further made it evident that behavior and affect analysis of the users’ can lead to

provision of corrective feedback along-side adapting the progress of the game’s difficulty

in a motivational way. Fully automatic affective and behavioral analysis and fully

automatic in-game-feedback are possible with today’s technology as long as used in

‘less’ serious games or with care-takers and professionals kept in the loop at regular

intervals. In particular pervasive solutions can train the user in a playful way in every-day

life standard situations and allow close-to-real-life simulations. This requires to bring

affect and behavior analysis “on the road” – best by distribution as demonstrated and

ensuring of sufficient robustness to face “out-of-the-lab” conditions.

This will lead to new research questions in this field such as low energy consumption or

situational context knowledge exploitation, e.g., based on location sensitivity, besides

optimal game integration. The rapid growth of social networks then increasingly offers a

platform for deploying games to large numbers of users. Ethical methods for data capture

from these users, coupled with analysis techniques which seek to interpret the resultant

“Big Data” on their affect and behavior and subsequently adapt the game in response will

play an increasing role in delivering more efficient and targeted solutions. In parallel,

going from mobile to pervasive computing will in particular address the questions of

localized scalability. In the long run, “invisibility” can then be reached in the sense that

the gaming blends into the real-world in a positive sense – the game stops unnoticed and

the learnt behavior persists.

ACKNOWLEDGEMENT:

The research leading to these results has received funding from the European

Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreements

No. 289021 (ASC-Inclusion) and No. 288587 (MASELTOV).

REFERENCES

[1] C. Holsing and R. Olbrich, "Effect of user-generated content on website stickiness: the

case of social shopping communities," in Proc. 14th Annual International Conference on

Electronic Commerce, Singapore, Singapore, 2012.

[2] D. J. Terry, M. A. Hogg, and K. M. White, "The theory of planned behaviour: Self-

identity, social identity and group norms," British Journal of Social Psychology, vol. 38,

pp. 225-244, 1999.

[3] E. H. Calvillo Gamez, P. Cairns, J. Gow, J. Back, and E. Capstick, "Video games as

research instruments," in Proc. 28th International Conference on Human Factors in

Computing Systems, New York, NY, USA, 2010, pp. 4493–4496.

[4] I. Dunwell, P. Petridis, M. Hendrix, S. Arnab, M. Al-Smadi, and C. Guetl, "Guiding

Intuitive Learning in Serious Games: An Achievement-Based Approach to Externalized

Feedback and Assessment," in Proc. 6th International Conference on Complex,

Intelligent and Software Intensive Systems (CISIS), Palermo, Italy, 2012, pp. 911-916.

[5] S. Egenfeldt-Nielsen, "Beyond Edutainment: Exploring the Educational Potential of

Computer Games," IT-University Copenhagen, 2005.

[6] J. Westra, F. Dignum, and V. Dignum, "Scalable adaptive serious games using agent

organizations," in Proc. 10th International Conference on Autonomous Agents and

Multiagent Systems, Volume 3, Taipei, Taiwan, 2011.

[7] P. M. Kato, S. W. Cole, A. S. Bradlyn, and B. H. Pollock, "A Video Game Improves

Behavioral Outcomes in Adolescents and Young Adults With Cancer: A Randomized

Trial," Pediatrics, vol. 122, pp. 305-317, 2008.

[8] K. W. Cullen, K. Watson, T. Baranowski, J. H. Baranowski, and I. Zakeri, "Squire's

Quest: intervention changes occurred at lunch and snack meals," Appetite, vol. 45, pp.

148-51, Oct 2005.

[9] E. J. Lyons, D. F. Tate, S. E. Komoski, P. M. Carr, and D. S. Ward, "Novel approaches to

obesity prevention: effects of game enjoyment and game type on energy expenditure in

active video games," Journal of diabetes science and technology, vol. 6, pp. 839-48,

2012.

[10] J. D. Johnston, A. P. Massey, and R. L. Marker-Hoffman, "Using an alternate reality

game to increase physical activity and decrease obesity risk of college students," Journal

of diabetes science and technology, vol. 6, pp. 828-38, 2012.

[11] S. Gobel, S. Hardy, V. Wendel, F. Mehm, and R. Steinmetz, "Serious games for health:

personalized exergames," in Proc. ACM Multimedia, Firenze, Italy, 2010.

[12] M. S. Zeedyk, L. Wallace, W. Carcary, K. Jones, and K. Larter, "Children and road

safety: increasing knowledge does not improve behaviour," British Journal of

Educational Psychology, vol. 71, pp. 573-594, 2001.

[13] M. A. Elliott, C. J. Armitage, and C. J. Baughan, "Using the theory of planned behaviour

to predict observed driving behaviour," British Journal of Social Psychology vol. 46, pp.

69-90, 2007.

[14] G. Hardin, "The Tragedy of the Commons," Science, vol. 162, pp. 1243–1248, 1968.

[15] A. Gustafsson, C. Katzeff, and M. Bang, "Evaluation of a pervasive game for domestic

energy engagement among teenagers," Computers in Entertainment, vol. 7, pp. 1-19,

2010.

[16] P. Centieiro, T. Romao, and A. E. Dias, "A location-based multiplayer mobile game to

encourage pro-environmental behaviours," in Proc. 8th International Conference on

Advances in Computer Entertainment Technology, Lisbon, Portugal, 2011.

[17] R. N. Madeira, A. Silva, C. Santos, B. Teixeira, T. Romao, E. Dias, and N. Correia,

"LEY!: persuasive pervasive gaming on domestic energy consumption-awareness," in

Proc. 8th International Conference on Advances in Computer Entertainment Technology,

Lisbon, Portugal, 2011.

[18] A. N. Antle, J. Tanenbaum, A. Bevans, K. Seaborn, and S. Wang, "Balancing act:

enabling public engagement with sustainability issues through a multi-touch tabletop

collaborative game," in Proc. 13th IFIP TC 13 International Conference on Human-

Computer Interaction, Part II, Lisbon, Portugal, 2011.

[19] Z. Fitz-Walter, D. Tjondronegoro, and P. Wyeth, "Orientation Passport: using

gamification to engage university students," in Proc. 23rd Australian Computer-Human

Interaction Conference, Canberra, Australia, 2011.

[20] F. D. Davis, "Perceived usefulness, perceived ease of use, and user acceptance of

information technology," MIS Quarterly, vol. 13, pp. 319–340, 1989.

[21] D. Kostons, T. v. Gog, and F. Paas, "Self-assessment and task selection in learner-

controlled instruction: Differences between effective and ineffective learners," Computer

Education, vol. 54, pp. 932-940, 2010.

[25] A. Komedani, T. Kojiri, and T. Watanabe, "Modeling understanding level of learner in

collaborative learning using bayesian network," in Proc. 9th International Conference on

Knowledge-Based Intelligent Information and Engineering Systems, Part II, Melbourne,

Australia, 2005.

[23] S. Jarvis and S. de Freitas, "Evaluation of a Serious Game to support Triage Training: In-

game Feedback and its effect on Learning Transfer," in Proc. IEEE Conference in Games

and Virtual Worlds for Serious Applications, Genoa, Italy, 2009.

[24] M. Q. Tran and R. Biddle, "Collaboration in serious game development: a case study," in

Proc. Conference on Future Play: Research, Play, Share, Toronto, Ontario, Canada, pp.

49-56, 2008.

[25] D. A. Kolb, Experiential learning: experience as the source of learning and development.

Englewood Cliffs, N.J: Prentice-Hall, 1984.

[26] M. Cziksentmihalyi, Finding Flow: The Psychology of Engagement with Everyday Life

London: Routledge, 1997.

[27] M. Balint, V. Posea, A. Dimitriu, and A. Iosup, "An analysis of social gaming networks

in online and face to face bridge communities," in Proc. 3rd International Workshop on

Large-scale System and Application Performance, San Jose, California, USA, 2011.

[28] A. Bandura, Social Learning Theory: General Learning Press., 1977.

[29] J. Schrammel, E. Mattheiss, S. Döbelt, L. Paletta, A. Almer, and M. Tscheligi,

“Attentional Behavior of Users on the Move Towards Pervasive Advertising Elements,”

in Pervasive Advertising, Springer HCI Series, 2011.

[30] N. Dimakis, J. Soldatos, L. Polymenakos, A. Bürkle, U. Pfirrmann, and G.

Sutschet,“Agent-based architectural framework enhancing configurability, autonomy and

scalability of context-aware pervasive services,” in Autonomous Agents and Multi-Agent

Systems, Springer Berlin / Heidelberg, 2010, vol. 21(1), pp. 36-68.

[31] A. Campbell and T. Choudhury, “From Smart to Cognitive Phones,” IEEE Pervasive

Computing Magazine, IEEE, Vol. 11(3), pp. 7-11, July – Septmber 2012.

[32] M.Z. Poh, N.C. Swenson, and R.W. Picard, "A Wearable Sensor for Unobtrusive, Long-

term Assessment of Electrodermal Activity," IEEE Transactions on Biomedical

Engineering, vol.57, no.5, pp.1243-1252, May 2010.

[33] L. Atallah and G. Yang, "The use of pervasive sensing for behaviour profiling - a

survey", in Proc. Pervasive and Mobile Computing, 2009, pp.447-464.

[34] M. Gilbert, I. Arizmendi, E. Bocchieri, D. Caseiro, V. Goffin, A. Ljolje, M. Phillips, C.

Wang, and J. Wilpon, “Your Mobile Virtual Assistant Just Got Smarter!,” Proc.

INTERSPEECH 2011, Florence Italy, pp. 1101-1104: ISCA.

[35] W. Han, Z. Zhang, J. Deng, M. Wöllmer, F. Weninger, and B. Schuller, “Towards

Distributed Recognition of Emotion in Speech,” in Proc. 5th International Symposium on

Communications, Control, and Signal Processing, Rome, Italy: IEEE, 2012.

[36] G. Chanel, C. Rebetez, M. B trancourt, T. Pun, “Emotion Assessment From

Physiological Signals for Adaptation of Game Difficulty,” IEEE Transactions on

Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 41, no. 6, pp. 1052-

1063, 2011.

[37] K. K. Rachuri, M. Musolesi, C. Mascolo, P. J. Rentfrow, C. Longworth, and A. Aucinas,

“EmotionSense: A Mobile Phones based Adaptive Platform for Experimental Social

Psychology Research,” in Proc. of ACM UbiComp, Copenhagen, Denmark, 2010.

[38] S. S. Hussain, C. Peter, and G. Bieber, “Emotion Recognition on the Go: Providing

personalized services based on emotional states,” in Proc. Mobile HCI 2009, Bonn, 2009.

[39] M. Kobes, N. Oberijé, and K. Groenewegen - Ter Morsche, “Serious Gaming for

Behavioural Assessment and Research in Case of Emergency. An Evaluation of

Experiments in Virtual Reality”, in Proc. SimTecT 2009 Simulation Conference and

Exhibition, Adelaide, Australia, 2009.

AUTHORS:

Björn W. Schuller (M’05) s a visiting key researcher at JOANNEUM RESEARCH in

Graz/Austria and a tenured senior lecturer at Technische Universität München (TUM) in

Munich/Germany. His research focuses on Affective Computing and Computer Audition.

He received the habilitation, doctoral, and diploma degrees in Electrical Engineering and

Information Technology from TUM, and was with the CNRS-LIMSI in Orsay/France

(2009-2010) and visitor in the Imperial College London’s Department of Computing

(2010). He is president-elect of the HUMAINE Association and coordinator of the

European ASC-Inclusion project and the European Cluster for Digital Games for

Empowerment and Inclusion. (JOANNEUM RESEARCH Forschungsgesellschaft mbH,

DIGITAL - Institute for Information and Communication Technologies, Steyrergasse 17,

8010 Graz, Austria, [email protected])

Ian Dunwell Ian Dunwell is a Senior Researcher at Coventry University’s Serious Games

Institute. His research principally focuses on the development and application of effective

design and evaluation methods for serious games. Having obtained his BSc in Physics

from Imperial College London, he holds an MSc and Doctorate awarded by The

University of Hull and is an Associate of the Royal College of Science. (Serious Games

Institute, Coventry Innovation Village, Cheetah Road, Coventry, West Midlands, CV1

2TL, UK, [email protected])

Felix Weninger (M'11) is a researcher in the Intelligent Audio Analysis Group at TUM's

Institute for Human-Machine Communication. His research focuses on environmental

robustness and software engineering of real-world speech analysis applications. He

received his diploma in computer science from Technische Universität München (TUM),

Germany. (Technische Universität München, Institute for Human-Machine

Communication, Arcisstrasse 21, 80333 München, Germany, [email protected])

Lucas Paletta is a key researcher at JOANNEUM RESEARCH in Graz/Austria, leading

a research studio on human factors technologies and services. His research focuses on

mobile/wearable context awareness and computational attention. He received the doctoral

and diploma degrees in Computer Science from Graz University of Technology, and was

visiting researcher at The Johns Hopkins University (1995) as well as Fraunhofer IAIS

(1998-2000). He is the coordinator of the European MASELTOV project. (JOANNEUM

RESEARCH Forschungsgesellschaft mbH, DIGITAL - Institute for Information and

Communication Technologies, Steyrergasse 17, 8010 Graz, Austria,

[email protected])