Designing and Evaluating Touchless Playful Interaction for ASD Children

DRAFT

Designing and Evaluating Touchless Playful Interaction for ASD Children

Laura Bartoli (1), Franca Garzotto (2), Mirko Gelsomini (2), Luigi Oliveto (2), Matteo Valoriani (2) (1)

Associazione Astrolabio (Firenze), (2)

Politecnico di Milano (1)

[email protected], (2)

[email protected]

ABSTRACT Limited studies exist that explore motion-based touchless applications for children with ASD (Autism Spectrum Disorder) and investigate their design issues and the benefits they can bring to this target group. The paper reports a structured set of design guidelines that distill our experience gained from empirical studies and collaborations with therapeutic centers. These heuristics informed the design of three touchless games that were evaluated in a controlled study involving medium functioning ASD children at a therapeutic center. Our findings confirm the potential of motion-based touchless applications games in technology-enhanced interventions for this target group.

Categories and Subject Descriptors K.3.0 [Computers and Education]: General; H.5.2 [Information Interfaces and Presentation]: Multimedia Systems, User Interfaces

General Terms: Design, Human Factors

Keywords Autistic children, motion-based touchless interaction, learning, therapy

1. BACKGROUND The Autistic Spectrum Disorder (ASD) is a general term for a group of complex disorders of brain development, characterized, in varying degrees, by difficulties in social interaction, verbal and nonverbal communication and repetitive behaviors often accompanied by sensorimotor impairments. Autism, estimated to affect 1 of every 88 children, is marked by the presence of impairments along a triad of dimensions: social interaction, communication, and imagination. Children with autism show a great variance of symptoms, ranging from a delay or a total lack of spoken language to a severe impairment in the use of nonverbal behaviors that regulate social interaction, to a failure to develop peer relationships appropriate to age. ASD children also show imagination inability, manifested in the difficulty to generalize between environments, in a limited range of imaginative activities and in a difficulty in figuring out future events and abstract ideas. This reflects to a lack of spontaneous make-believe play or social imitative play and tendency to repetitive and stereotyped patterns of activity. Other behavioral symptoms include hyperactivity, short attention span, impulsivity, aggressiveness, self-injurious behavior, and temper tantrums. Studies conducted to consider the effectiveness of digital technologies for ASD children reveal that these tools are in general well received [16]. A digital environment provides stimuli that are more focused, predictable, and replicable than conventional tools. It also reduces the confusing, multi-sensory

distractions of the real world that may induce anxiety and create barriers to social communication. In addition, digital tools can exploit the benefits of visually based interventions adopted in existing therapeutic practices such as video modeling [6]. Existing products and prototypes for autistic children exploit a variety of technologies and interaction modes, from desktop to multitouch mobile devices, tangibles and digitally augmented objects, robots [11], and more recently, touchless motion based environments, enabling users to interact using body movements without any physical contact with digital tools. The goal of our research is to design, develop and evaluate touchless motion based games that can be used for educational and therapeutic purposes in different contexts - school, therapeutic center, home - to improve autistic children’s skills in the motor, cognitive, and social sphere. In previous papers [2][3], we reported a survey and a discussion of the state of the art on touchless motion-based interactive technology for autistic children [29][15][23][14]. This review pinpoints that the number of empirical studies in this field is small, the understandings of the neurological mechanisms underlying autism are limited, and the theoretical underpinnings are controversial [17][8][13][5]. Little is known about how touchless motion-based interaction works for autistic children and how it can be designed to promote specific skills. This papers provides an overview of our activities in this field (section 2) and presents our recent contributions to the above research issues, which include: i) a set of design guidelines (section 3) for motion-based touchless games for autistic children; ii) three new motion-based touchless games for low-medium functioning autistic children that were designed according to these guidelines (section 4); iii) empirical evidence of the learning

potential of these games (sections 5 and 6).

2. THE WOKFLOW Our research on motion-based touchless technology for ASD children involved the research and development team at our lab (3 HCI engineers, 1 interaction designer, 1 psychologist) and … from 3 different therapeutic centers: Centro Benedetta D’Intino in Milan (CBD), Associazione Astrolabio in Florence (AA) and San Camillo Hospital in Turin (SC). Overall, 18 specialists (6 neuropsychiatric doctors, 4 language therapists, 4 motor therapists, 4 special educators) and 19 autistic children have participated in our research so far. Children can be deemed as “medium-low functioning”, for their cognitive level and for the severity of their deficits in at least one of the three ASD dimensions (social interaction, communication, imagination). Our overall work unfolded along various steps: field observations of children’s normal daily activities (i.e., without any technology) at CBD; a first empirical investigation at AA of children interacting with commercial motion based touchless games (discussed in [2]); a partial replication of this study at CBD; definition of design guidelines; development of three new games designed according to the above guidelines; iterative evaluation of progressive prototypes; summative evaluation of the final version of these games at AA.

IDC’14, June 17 – 20, 2014, Aarhus, Denmark.

DRAFT

Observing children’s activities in their normal context enabled us to understand what happens in their real life context and where the opportunity lies for technology. We attended as observers 5 afternoon sessions at CBD in which 10 autistic children participated in their normal activities at the center. All sessions were videotaped and videos were discussed with CBD experts.

Four commercial Kinect games were then installed initially at AA (as reported in [2]) and later at CBD, in both cases for a period of two and a half months. Overall, these products were used by 15 children (5 at AA and 10 at CBD) who were videotaped during play, enabling us to collect over 30 hours of videos. The games, belonging to the “packages” MS Kinect Sports and MS Rabbids

Alive & Kicking, were selected by therapists, after the analysis of over 150 entertainment products available on the market. At AA, observations and video recordings of children’s play were complemented with standardized therapeutic tests on attention, as discussed in [2] and [3]. For each study, the gathered materials (videos, therapists’, observers’ notes, clinical measures) were initially analyzed independently by our team and by each local team of specialists; then focus groups were organized at each therapeutic center to discuss and compare design observations and results. This work on the field enabled us to assess the learning benefits of motion based touchless gaming in relationship to attentional skills and emotional or behavioral aspects. From these studies we also constructed a more accurate description of how children behave during gameplay, what works, what doesn’t, which are the points of strength and weakness of the products used. This knowledge was distilled into a preliminary set of design guidelines. They informed the design of three new games

that have similar game logic as the commercial games used but revised UX characteristics. The progressive prototypes of these games were evaluated at 3 therapeutic centers, involving overall 15 medium-low functioning children, leading to design revisions and refinements of our guidelines. The final version of the design guidelines are reported in the following section. The final version of the games was evaluated in a controlled study performed at AA, as discussed in sections 5 and 6.

3. DESIGN GUIDELINES Our guidelines take into account the specificity of ASD, particularly in relationship to medium-low functioning children, the characteristics motion-based touchless interaction, and the learning potential of this paradigm. We have defined general and goal specific guidelines. General guidelines consider high-level design principles and concern general interface/ interaction features. Goal specific guidelines focus on design features related to specific learning goals. As such, they are classified in 3 categories, respectively associated to learning skills in the motor, cognitive, and social dimension.

3.1 General Guidelines

One game per child There is no such thing as an "average" child with autism. Each child manifests unique strengths and skill deficits. Things that are reinforcing or rewarding to one individual may be unpleasant for another person [26]. Any play activity must therefore be oriented to addressing the unique capability and needs of each individual child, which implies that a game must support a high degree of customizability. It must enable caregivers to adapt a gaming experience to the individual skills and preferences of each child, customizing multimedia contents, rewards, play time, body movements.

Evolving tasks To support the evolving needs of a child over time, game should support increasing levels of motor and cognitive complexity of game tasks [20]. It should enable the progression along a continuum of game sessions involving activities that are similar but, when the child has acquired and consolidated the proper skills, are progressively more demanding in terms of motor, cognitive and social skills required. Unique goal Within a play session, children should have a unique, explicit, well focused game goal to reach (e.g. “hit as many moving objects as possible”, “avoid as many falling asteroids as possible”). The goal should be associated to one single task and a clear set of movements the child can afford (e.g. ‘moving arms to hit objects’) to promote the cognitive process related of organizing movements to achieve a given objective. Instructions Understanding the goal and tasks should be facilitated before playing and should be reinforced during the whole game session. Children, especially those with a delay in or a total lack of verbal language, can benefit from visual means for communication, like PCSs (Picture Communication Symbols) and iconic images representing the movements to be performed and the body parts to be used. Rewarding After a good performance, offering a rewarding stimulus which is "valued" or "liked" by the child increases motivation, enhances player’s engagement and implicitly improves her skills [24]. In our experience, medium-low functioning autistic children might not value much quantitative performance results (e.g., points or extra time won) as rewards. What seems to act as a stronger motivator and a positive reinforcement is a video or audio effect that creates fun, e.g., the play of cartoon videos, funny animations, cheerful music, and applauses. In case of scarce game performance, these elements should not be necessarily removed at all, but can be reduced and completed with something that fosters children to do better, e.g., visual instructions, in order to help managing frustration issues. Repeatability and Predictability Autistic subjects appear to have a well-established affinity with interactive technologies. Repeatability plays an important role to achieve mastery and provide control of the rate of learning. Repeating the same routines (tasks) improves not only individuals’ mastery but also gives them the predictability they need as well as clear expectations of the next future. Transitions The same game is typically played many times in the same configuration. Eventually, the child will need to move from a level in which she has proved to be successful to the next one, which unlocks new challenges and opportunities. It must be very easy for the caregiver and the child to repeat a game, as well as to move to the “next level”. The time of restarting a session or switching from one level to the next one must be minimized, to reduce the risk of a child’s loss of concentration during the transition. Minimalistic graphics Visual items should be cheerful and aesthetically nice, but always strictly functional to the goal. Children may be distracted from visual elements that are not strictly relevant for the current task and may lose attention. In addition, too many visual stimuli may induce anxiety as children may not be able to discriminate and

DRAFT

interpret single elements within a group. Graphic elements should have clearly distinguishable shapes and should not overlap. For some children, the use of colors may need to be reduced. Some subjects, for example, can work with black and white images only, getting very nervous when dealing with other colors. Clear audio Sound or music can be used not only to provide feedbacks on actions. They should be played during moments when nothing happens on the screen or there is a transition from one game configuration to another, or at the end of the game to complement visual rewards effects. At the same time, it is important to remember that autistic children easily reach a point in which too many audio stimuli are perceived just as a mass of noise that they cannot interpret and creates extra stress. Hence similar principles mentioned for visual elements (see “Minimalistic Graphics”) apply to sound: Sound elements should be cheerful, clear, simple, and functional to the game task and to the need of keeping children’s attention. Dynamic stimuli Dynamic stimuli such as animations and music should be provided along the entire game session. When visual elements remain static and nothing else happens, the child may lose concentration and move her attention to something outside the game. A prolonged static situation may trigger abnormal behaviors, such as stereotyped movements or motor rigidity (e.g. motionless gazing at the static image on the screen), which typically must be “unlocked” by a caregiver’s intervention. Serendipity In any game activity and for any, enjoyment increases when visual or audio effects create wonder or surprise. This is also true, at some degree, for autistic children. For these subjects sensory stimuli should balance phenomena that are more predictable and consistent with the ongoing task (e.g., audio feedbacks on child’s movements) with serendipitous effects (e.g., a new different object appears and disappear on the screen), always having in mind the risk of attention loss. Avateering An avatar is a virtual representation of the child inside the game that offers an immediate visual feedback to a player’s actions. It enforces the perception of “self” (“It looks like me!” – a child during our experiments). The game space becomes like a “virtual mirror” where the player sees herself, establishes a connection between her movements and the system’s reaction, and develops imitative abilities (see also Guideline “Developing imitative abilities” in sect 3.3). In addition, avatars can be a means to direct children’s focus of attention on the effects of their movements rather than the body movements per se, to promote static and dynamic motor skills (see General Guideline “Motor control through “external” attention”). Hence avatars should be incorporated in a game even when they are not strictly functional to the game narrative structure, e.g., they do not represent characters of a “story”. We identified three different kinds of Avatars: Articulated Avatar: all parts of the body are represented using

simple shapes (points, lines, circles…) and follow players’ movements

Pointing Avatar: the user is represented by a single image that follows the movements of a single part of the user’s body.

Real Avatar: the user is shown as a silhouette of her body, which can be filled with the image of the child captured by the color camera.

3.2 Goal specific guidelines: Motor skills

Motor control through “external” attention To promote static and dynamic motor skills (i.e., movement control, balance, or postural control) it is important to provide audio-visual stimuli associated to each movement or still position needed to perform a game task, in order to direct a child focus of attention on the effects of her movements rather than the body movement per se. This guideline is also grounded on a number of studies on non-disabled persons pinpoint that fostering “external” attention focus (on the effects of one’s movements) rather than on the action itself (“internal” attention focus) can boost motor learning. [21] found an increase in motor performance when directing an individual’s attention “externally” compared to directing attention “internally”. The advantage of focusing attention on the movement effect might be that it allows unconscious or automatic processes to control the movements required to achieve this effect. When persons focus on their body they might be more likely to consciously intervene in these control processes and may inadvertently disrupt the coordination of a number of relatively automatic (reflexive and self-organizing) processes that normally control the movement. Increasing gross motor skills Gross motor skills enable such functions as walking, jumping, kicking, sitting upright, lifting, throwing, as well as head control, trunk stability, maintaining balance, balancing position from one foot to the other [19]. To promote the development of these skills, interaction can comprise the actions above and in general, movements that involve the large muscle groups and the whole body. As motor skills generally develop from the center to the body outward and head to tail, the progression of tasks (see General Guideline “Evolving Task”) should include a progression of movements initially involving the whole body and then increasingly more peripheral parts. Increasing postural stability To promote postural balance and control, the game can involve tasks that require to maintain still positions (e.g., keeping head or arms steady), to keep the line of gravity of the body with minimal sway, or to balance position from one foot to the other. Task complexity can be increased by requiring alternating dynamic movements and static gestures, or progressively augmenting the time during which motion-less positions must be maintained (offering appropriate rewards as time proceeds). Increasing coordination Various types of tasks can promote motor coordination [9][25]: Tasks that involve coordinated movement of different parts of

the body, or eye-rest of the body coordination. For example, to hit a virtual object, still or moving, in different positions or with varying speed, the child can use legs and arms alternation, shift of left and right arm/leg, or the combination of both arms/legs.

Tasks that require distinguishing left/right or forward/backward, determining the distance between objects, combining movements into a controlled sequence, remembering the next movement in a sequence

Tasks that require to apply visuo-spatial memory, which concerns to perception of spatial relationships among objects (e.g., in jigsaw puzzles) and is thought to be associated to coordination deficits.

3.3 Goal specific guidelines: Cognitive Skills

Promoting perceptual learning and attention skills Perceptual learning forms important foundations of complex cognitive processes (i.e., language) and is defined as “the process

DRAFT

of learning improved ability to respond to the environment” [27]. These improvements range from simple sensory discriminations - focusing on, and discriminating between, certain stimuli - to identification of items as belonging to the same or different category, to complex categorizations of spatial and temporal patterns relevant to real-world expertise. Perceptual learning is widely seen as tightly coupled with various forms of attention: sustained attention - the ability to direct and focus cognitive activity on specific stimuli; selective attention - the process by which a person can selectively pick out one message from a mixture of messages occurring simultaneously; weighted attention, which entails making a distinction between relevant and irrelevant stimuli. Different types of tasks and visual contents can promote perceptual learning: Tasks that require the child to stabilize her gaze and track with

her body moving objects on display (sustained attention) Tasks that require moving the entire body or some body parts to

hit or avoid specific moving objects (sustained attention). Tasks that involve movements to express simple sensory

discriminations, e.g., distinguishing colors, shapes, sizes, position of objects (selective attention)

Tasks that require to recognize a visual shape among multiple shapes or to reproduce a similar shape with the body (a body position (selective attention)

Tasks and visual contents that require the child to focus on similarities or on differences such as pointing to and gripping visual elements that represent different food types or the same food type (selective attention)

Tasks that involve more complex categorizations of spatial and temporal properties of the objects on display in situations relevant to a child’s own life, e.g., “What do you eat at breakfast? Grasp only the food you eat at breakfast”) (weighted attention)

Tasks that require the child to select information relevant for a task, and ignore irrelevant information, e.g., hitting only “target” elements among various falling objects (weighted attention)

Increasing space awareness Space awareness can be increased by tasks that involve moving in and out a constrained space (e.g., defined by a circle drawn on the floor) or rotating the body to change perspective on a virtual 3D space on the screen. Increasing body awareness To increase awareness of body and body structure, multimedia contents and movements should emphasize the identification of body parts, how they work and how they fit together. For example: Playing songs that focus on body parts, such as head, shoulders,

knees and toes, and asking the child (while singing) to touch the parts of the body in the song, or to move and shake them (clapping, stopping and head nodding)

Involving more than one child and requiring children doing interactions while certain body parts are touching the whole time, such as hands, elbows or heads, proving feedbacks when a player disconnects (see also Guideline “Engaging in human-human interaction”)

Developing imitative capability Autistic children show limited imitative capability, which is manifested by the lack of spontaneous make-believe play or social imitative play [10]. The (realistic or schematized) shape of the player’s body and her movements in the game virtual space are fundamental to promote the development of imitative skills (see

also General Guideline “Avateering”). Other means are the inclusion of tasks like: imitating the movements of characters on display; forming static shapes using the body that correspond to shapes or images on display; performing gross and fine motor movements that animate characters on the screen (objects, animals, vegetables, humans), e.g., in a storytelling contexts; imitating the movements of co-players (adults or peers).

3.4 Goal specific guidelines: Social Skills

Not only multiplayer Most touchless games can be developed for being played by both a single individual and two or more persons without changing game logic. However, moving together in front of a display and performing independent or complementary tasks do not necessarily trigger social interaction. This is especially true for autistic children [28] who do not engage with others spontaneously. Social interaction must be supported “by design”, with multiplayer tasks that are explicitly conceived to promote and exert social skills (see next guidelines). Motivating human-human interaction The game should include tasks that require movements of more than one player to be completed, i.e., jumping together to overcome an obstacle, creating a body shape that simulates a character on the screen with 3 legs and 4 arms. Visual elements and rewards should emphasize the concept of “together” and acknowledge the benefits of doing things cooperatively. Also avatars can act as a social cue that may influences children’s perceptions, leading them to perceive the experience as more ‘‘social’’. Motivating communication Children with ASD are often self-absorbed and seem to exist in a private world where they are unable to successfully communicate with others. It is important to give them a motivation to make the effort of sending and receiving messages. To this end, a task that requires players to give verbal and non-verbal mutual instructions to reach a goal (e.g., “while I move here you have move there”) offers a motivating opportunity for communication. Increasing joint attention Joint attention is the intentionally shared focus and interest of two individuals on the same object or event. It is achieved for example when one individual alerts another to an object by means of eye-gazing, pointing, or other verbal or non-verbal indications, and the other person looks back to her after looking at the object. Joint attention abilities are important for many aspects of language development, socio-emotional development and the ability to take part in normal relationships, and are negatively affected by autism. To support joint attention it is important to include tasks where one or more of the situations below take place: one child has to call the attention of another child, or an

adult, toward a target objects, e.g., through a pointing gesture two children have to coordinate each other in order to find a

target object and catch it simultaneously two children have to alternate in the same or different tasks

(“turn taking” ) and regulate one’s behavior to the one of the co-player.

4. OUR GAMES Using the guidelines described in the previous sections, three different motion-based games have been designed and developed: Bubble Game, Space Game and Shape Game. All games are strongly customizable according to the characteristics and learning needs of the specific child that plays with the system. This feature leads to a large number of playing

DRAFT

opportunities even within the same game. The customization can be done by the therapist during the treatment sessions or by the children’s parents at home, due to the simplicity and ease of use of the configuration functions and interface. The most relevant parameters that can be modified are game speed, object density and enabled body parts. Games can also be personalized using different graphic themes for background and graphic elements or adding specific multimedia reward. The games automatically collect the scores obtained by players and the information relative to the played configuration, and generate a session report that can be used to evaluate or tune the treatment.

4.1 Bubble Game The clinical goal of this game is to improve children speed and accuracy of movements and motor-visual coordination between visual elements. The body parts which can be used are head, one or two hands, one or two feet. The children have to catch as many appearing objects as possible. They are free to play until time expires; at the end of the game the resulting score is shown on screen, representing the total number of collected objects. A more positive final reward (e.g. an applause and a golden cup) is given when the score is greater than a threshold calculated considering the various parameters (e.g. the total number of elements and the children allowed body parts). The child avatar is an Articulated Avatar; it helps the child to recognize all body parts in order to better link her body with the stick figure on the screen and to improve the body awareness.

Figure 1: Bubble Game

Figure 1 shows a child playing a particular level of Bubble Game where she can use all body parts to catch and burst the appearing bubbles. On top of the screen the information about the total game time, the current score and the enabled body parts are displayed. The game levels can be personalized using various parameters, the most relevant are: total game time; total number of elements; fade time of objects; objects size; objects position; body part to be used to hit (head, left hand, right hand, left foot, right foot or a combination of them) and graphic background.

4.2 Space Game The main goal of this game (Figure 2) is to increase speed and accuracy of movements coordination and selective and sustained attention. This activity also elicits movement coordination and pace. The child has to move her entire body to avoid falling objects. She is represented by a Pointing Avatar that can move only horizontally from a side of the screen to the other, following her position. This kind of avatar has been chosen because children use only one point of their body (e.g. the hand) or the body as a whole to interact with the game space and do not need an articulated stick figure. They can concentrate only on falling objects, reducing the noise generated by useless body part movements and it also allows children with special motor disabilities to play the game.

The player has an initial number of “lives” and if she loses all of them before the game time is elapsed, she loses the game. If the child still has at least one life when the time runs out, she wins the game and the relative reward is shown on screen. Beside the falling objects to be avoided, some special award elements can be caught in order to gain additional lives (e.g. stars). The number of these objects can be defined by therapists according to the overall level of difficulty. Game configurations are determined by multiple parameters such as game total time, total number of elements, objects falling time, number of initial lives, number of additional lives, body part to be used to hit (body, left hand, right hand), objects dimensions (small/medium/large), and graphic background. In Figure 2 the top-left of the screen presents the remaining lives and time, while on the right side the current score is shown.

Figure 2: Space Game

4.3 Shape Game The main goal of this game is to promote the following skills: identification of the correlation between the body and the shown image; body awareness and limits; body consciousness in the space; abstraction ability; imitative skills and motor control auto-regulation.

In Shape game children have to replicate a particular shape shown on screen using their body. While the motion sensor captures the user’s silhouette, the child must overlay the shape until a certain threshold level (predefined for each game level) and within a certain time limit. Moreover, the child has to keep the correct position for a given time frame. The game can be played by a single child or by two children who have to collaborate together to mimic the shape on the screen. If time elapses before children correctly overlay the shape, the game ends without any particularly cheerful reward; when children overlay the shape at the expended degree, the game ends showing a multimedia reward.

Figure 3: Shape Game

A Real Avatar helps children to recognize themselves on the screen and improves body awareness and body schema. As shown in Figure 3, the left part of the screen contains the graphic element representing the remaining time and indicates the coverage percentage level.

DRAFT

Therapists can create customized shapes in order to address different children’s needs or to stimulate them in performing some specific body movements. For customization purposes, shapes can be extracted from any pre-existing image and edited with classical digital paint tools (brush and eraser) or can be generated on the fly by children or therapists using the shape capturing tool integrated in the game, as illustrated in Figure 4.

Figure 4: Shape Game Capturing Tool

5. EMPIRICAL STUDY In order to validate the effectiveness of our games, we led an empirical controlled study. Our first goal was to evaluate the benefits of our games monitoring the improvement of the abilities of children. Our second goal was to compare the effectiveness of our games respect to the commercial ones, used in the previous study [2].

5.1 Study Variables The variables considered in the evaluation phase can be divided into two different categories: Performance and Clinical variables. The Performance Variable measures how better the child complete a specific task; this measure is related to the specific game that is played by the child. The Clinical Variables include attention aspects and the capability of integrating visual and motor skills. For attention, we analyzed two variables [7] [27]: Selective Attention, the capability to focus on an important

stimulus ignoring competing distractions Sustained Attention, the capability to hold the attention for the

time needed to conclude an activity In order to evaluate the capability of children to integrate visual and motor skills, we investigate: Visual Perception, the ability the mind and eye to interpret the

surrounding environment by objectively processing visual information

Motor Coordination, the harmonious functioning of body parts that involve movement, including gross motor movement, fine motor movement, and motor planning.

Visuo-Motor Integration, the ability to control body movement guided by vision.

5.2 Participants We recruited 10 medium-low functional children who can be considered as homogenous from a clinical perspective. In addition, information was collected using questionnaires compiled by parents, in order to confirm that all children have comparable behavioral characteristics (e.g., a very limited set of favorite activities and similar stereotyped behaviors, both verbal and motor). The children (9 males, 1 female, aged 6-8 years) were randomly assigned either to the treatment group (G1) or the control group (G2). Both groups had regular therapeutic treatments during the study period. The treatment group G1 attended extra sessions during which they used our games. During the entire testing

period, parents of all children were warned not to let them play with motion-based game consoles such as Nintendo Wii nor Xbox with Kinect at home.

5.3 Procedures The study was performed at the therapeutic center AA (Association Astrolabio) in order to investigate behavioral reactions and social, motor and cognitive skills before and after the use of our games. Before the game treatment, a preliminary phase (T0) took place, when both groups G1 and G2 were subjected to standardized clinical tests to define the initial functional profile and to establish the “starting point” of each child. While G2 continued its regular activities, G1 attended regular meetings and additional therapeutic sessions using our games and, at the end of the testing period (T1), both groups were evaluated. G1 had been analyzed along a 3-months period (from October to December 2013) for a total amount of 10 individual weekly meetings of 45 minutes. Initial and final assessment needed 2 individual meetings. For each session, children played each game for 10 minutes, gradually increasing the level difficulty. The meetings were video-recorded, using two cameras placed in front and on the back of the children, simultaneously capturing children movements and the game visual interface respectively. A total of 15 hours of video recording were collected. During the game sessions, the therapist was sitting or standing aside the child and outside the Kinect sensing area, taking notes and intervening when needed. All sessions took place in the same room without modification of the environment settings. 5.3.1 Assessment In order to evaluate the Performance variable, we propose a Global Weighted Score (GWS) represents a weighted score depending on the specific game parameters.

Global Weighted Score (GWS) Each game has different tasks to achieve, so three different GWS variables have been defined.

Bubble Game GWS = #Collected_Bubble * Difficulty_Level Where the #Collected_Bubble represents the number of hit items, while the Difficulty_Level is the category assigned by the therapist to the played configuration.

Space Game GWS = #Item_Avoided * Difficulty_Level Where the #Item_Avoided represents the number of avoided objects during the game.

Shape Game GWS = (Difficulty_Level * Steady_Time *

Coverage_Threshold * (1 – (Final_Time / Max_Time)) Where the Steady_Time is the time in which children have to maintain their position to cover the shape; Coverage_Threshold is the percentage of the shape that the children have to cover in order to win; Final_Time is the time spent to correctly cover the shape and Max_Time is the total time available for the specific level. For the clinical variables we adopted 3 evaluation tests: Modified Bells Test (MBT), Subtest Wisc_IV and Visual-Motor Integration (VMI).

Modified Bell Test (MBT) The Modified Bells Test, adopted in many therapeutic centers in our country, is a child-oriented adaptation of the method proposed in [12][4] and consists in a sequence of cancellation tasks. In each task, a child marks as many similar items as possible from a paper of cluttered items. Target stimuli (images or shapes, e.g., of bells, with equal size and orientation) are randomly intermixed with other different ones. The evaluation of the attention process is based on the measurement of two indicators: accuracy (the total number of target items, identified in the maximum time –

DRAFT

normally 2’) and speed (the number of target items identified in the first 30”).

Supplementary Subset Wisc IV The Wechsler Intelligence Scale for Children (WISC) [22] is an individually administered intelligence test for children that can be completed without reading or writing. The supplementary subset, one of the 15 Wisc IV subsets, evaluates in particular the selective attention. The test consists of a random deletion and a structured deletion task. For each trial of 45 seconds the child has to find and mark the requested figures of a given topic (e.g. animals).

Developmental test of Visual-Motor Integration (VMI) Visual-Motor Integration Test [18] is commonly used to identify significant difficulty in visual-motor integration and to determine the most appropriate intervention plan. This test was designed to measure both the integration between visual perception and motor coordination, and each of these two components separately. VMI is a paper-pencil based task, consisting in 27 items in which the subject copies a developmental sequence of geometric shapes in a predefined time frame. Since it does not need particular knowledge (numbers, letters) it is particularly suitable for ASD children.

5.3.2 Games Configurations All of the games were set with an initial homogeneous setup for all children. In each session, the child started from the level which he played the previous session, with the goal to redeem confidence and reach a satisfactory concentration level. All games configurations are divided in 5 levels with growing difficulty. Table 1 summarizes Bubble Game configurations.

Table 1: Bubble Game configurations

Levels Enabled body parts

Time [sec]

Time fade [sec] #Elements Appear

after burst

1 Head, hands, feet 120 infinite 90 on

2 Hands, feet 120 8 120 off 3 Hands, feet 120 5 180 off 4 Hands 120 4 180 off 5 Head, feet 120 4 180 off

Table 2 Space Game configurations. In the first 3 levels the graphical theme is “space” thus children have to avoid black satellites, planets and meteorites. The fourth level theme is “air” showing butterflies, airplanes and flowers, while the fifth level, displaying bubbles, shells and starfishes, has the “water” theme. The different graphic setting, especially in the last two levels, requires different degrees of difficulty in the identification of objects of different semantic values (star space vs. starfish), where the first increases life points and the second one decreases them.

Table 2: Space Game configurations

Level

# Lives

Max Time [sec]

Time fall

down [sec]

# Objects

# Bonus

Dimensions

Graphic theme

1 5 90 6 25 15 medium space 2 4 90 5 50 10 medium space 3 3 90 4 60 7 medium space 4 2 90 3 90 6 small air 5 1 90 3 110 5 small water

We also decided to augment the objects’ density, increasing the number of flows of falling items, from 4 flows in level 1 to 10 in level 5.

Table 3 summarizes the experiment configurations of Shape. Each level is progressively more demanding in terms of cognitive and motor skills. In particular, L4 focuses on improving static balance skills while L5 involves motor planning abilities, since most of the figures in this level do not have a human shape.

Table 3: Shape Game configurations

Level Coverage Pose time [sec]

Game time [sec]

Additional difficulties

1 60% 2 30 No 2 60% 2 30 No 3 70% 3 30 No 4 80% 3 30 Static balance 5 90% 3 40 Abstraction

6. RESULTS

6.1 Results of Clinical Test Comparing the evaluation between the initial (T0) and final (T1) time, the use of our games shows important improvements of the treatment Group (G1) in terms of attention and visuo-motor abilities. In addition, surveys, compiled by parents during clinical meetings, according to the standardized tests and clinical observations, highlight an improvement of children’s daily activities and skills.

6.1.1 Attention parameters Figure 5 and 6 show each child score at the beginning (T0) and at the end (T1) of the test phase. Children from c01 to c05 belong to the treatment group (G1) while from c06 to c10 apply to the control group (G2).

Figure 5: Bell Test - Selective Attention

The treatment group G1, between T0 and T1, has an average percentage increase of selective attention +72.38% [+14.43% average increase on the reference scale] and a sustained attention’s one of +33.48 [+19.29% average increase on the reference scale]. On the contrary, G2 was not subjected to considerable variations and it remained steady on the investigated variables (the average percentage increase of selective attention is +3.07% [0.43% average increase on the reference scale] and of sustained attention of +11.23% [+6.86% average increase on the reference scale]).

c01 c02 c03 c04 c05 c06 c07 c08 c09 c10

G1 G2

T0 31 24 39 21 35 28 27 16 40 32

T1 56 48 51 40 56 30 25 18 40 33

10

20

30

40

50

60

DRAFT

Figure 6: Bell Test - Sustained attention The clinically significant results on G1 were confirmed by the deletion subtest of WISC IV, obtaining an average percentage increase of 32.38% [+14.00% average increase on the reference scale] (Figure 7).

Figure 7: Wisc IV – Selective Attention

6.1.2 Visuo-motor Integration Overall general trends of G1 show a rise in the evaluated parameters during the entire testing period. Contrarily to G1, G2 do not obtain a significant variation (Table 4). Table 4: Comparison between Treatment and Control groups

TEST G1 G2

VMI Avg. increase 138.92% 8.14% Avg. increase

(ref. scale) 13.61% -0.14%

MOTOR COORDINATION

Avg. increase 1136.42% 18.80% Avg. increase

(ref. scale) 91.67% 1.21%

VISUAL PERCEPTION

Avg. increase 820.71% 62.92% Avg. increase

(ref. scale) 11.4% 3.73%

Concerning the visuo-motor integration skill, G1 shows uneven trends due to each child’s individual peculiarity.

Figure 8: VMI Test – Integrated result

Figure 9: VMI Test - Motor coordination skill

Figure 10: VMI Test - Visual-Perception skill

6.2 Play Performance Children have been progressively able to deal with complex play levels and improve their scores during their treatment period.

6.2.1 Bubble Global Weighted Score (average per session) still increases for every child except c05 who experienced complications in L4 and L5 (Figure 11). Overall, the average number of hit bubbles from level L1 through L5 increases until L3 and decreases after L4. This means that L3 can be considered as the game “sweet spot”.

Figure 11: Bubble - GWS average per session

Table 5: average # collected bubbles per difficulty level Session L1 L2 L3 L4 L5

S1 30 91 52 S2 24 76 91 S3 27 59 75 58 S4 37 64 79 49 38 S5 94 57 26 S6 64 33

6.2.2 Space For all children, 28% of plays ended before completing the level because the number of collisions with objects that had to be avoided surpassed the number of life points and bonus items to collect.

c01 c02 c03 c04 c05 c06 c07 c08 c09 c10

G1 G2

T0 92 77 99 62 91 78 65 84 93 91

T1 117 107 122 92 118 83 71 90 122 93

5060708090

100110120130

c01 c02 c03 c04 c05

t0 36 16 30 44 31

t1 50 22 44 54 36

10

20

30

40

50

60

c01 c02 c03 c04 c05 c06 c07 c08 c09 c10

G1 G2

T0 18 2 2 21 30 2 1 3 8 9

T1 37 10 4 18 61 4 1 3 9 5

0

10

20

30

40

50

60

70

[%]

c01 c02 c03 c04 c05 c06 c07 c08 c09 c10

G1 G2

T0 19 0,4 3 5 2 0,02 0,5 3 2 4

T1 27 2 6 32 94 0,02 1 1 9 7

0

20

40

60

80

100

[%]

c01 c02 c03 c04 c05 c06 c07 c08 c09 c10

G1 G2

T0 1 0,2 2 25 61 0,05 0,03 1 2 32

T1 1 5 34 39 90 0,1 0,07 1 2 58

0

20

40

60

80

100

[%]

0

50

100

150

200

250

300

s1 s2 s3 s4 s5 s6

c01 c02 c03 c04 c05

DRAFT

Figure 12: Space - GWS average per session

However, along the entire period, the Global Weighted Score (average per session) linearly increased and collisions (objects not avoided) decreased or remained constant.

6.2.3 Shape Our findings show that the time for completing the tasks at a specific level of difficulty decreases with the treatment. After a first initial phase, children began understanding new and creative strategies combining them to complete the tasks. Overall Global Weighted Score (average per session) increases over the time for all the children with predictable fluctuations (Figure 13).

Figure 13: Shape – GWS average per session

6.2.4 Further Observations The systematic analysis of the qualitative data gathered during the study (13 hours of video recording, therapist’s and observers’ notes, questionnaire-based feedbacks from parents) is still ongoing. Still, some facts have already emerged that are worth to be mentioned. During the first play sessions of Space Game, some children did not have a correct understanding of the correlation between their body and their virtual representation on the screen, in that they identified themselves too much with avatars: every time an avatar was hit by a falling object (which instead should be avoid) they said “Ahia!!” as if they were really hit and felt pain. This behavior is not surprising; the mechanisms for correct coupling perception/action are weak in cognitive disorders like autism. What is interesting is that the perceptual identification “own-body/avatar” gradually mitigated up to disappearing with the progression of game sessions. Another observed behavior is the eye triangulation that eventually occurs during play between the child, the therapist and game screen. This phenomenon was judged very positively by therapists, because the marked impairment in eye-to-eye gaze is a typical symptom of the social interaction deficits that characterizes autism. Finally, parents filled questionnaires after the study and reported evident improvements in their children’ motor skills, execution of daily tasks at home, and attitude towards social play with peers. The positive comments about the so called “Kinect treatment” spread by word of mouth among many families attending the therapeutic center, and many parents asked explicitly to organize an additional study and to have their children involved.

6.3 Discussion Our findings provide empirical evidence that motion-based touchless games have a learning potential for autistic children, to promote attention, (integration of) motor and visual skills. This is witnessed by clinical tests and by the increasing level of play performance manifested with the progression of the treatment. Some observations arise from the comparison of the outcomes of the study reported here, hereinafter referred to as “Study 2”, with our previous study reported in [2][3], hereinafter referred to as “Study 1”, where we used commercial games with similar game logic. Both studies were performed at the same therapeutic center, with the same setting, number and duration of sessions. In Study 1 the children exposed to the game treatment had a slightly higher clinical profile (medium functioning) than participants of Study 2. Clinical variables considered in Study 1 were selective and sustained attention and were measured using Bell Test only. The most crucial difference of Study 1 is the use of commercial Kinect games. Study 1 showed positive results in the attention sphere that Study 2 confirms using a more robust research design, i.e., involving a control group and suing more than one standard method to measure the clinical variables.

Table 4: Comparison between Study 1 and Study 2 TEST Study 1 Study 2

Selective Attention

Avg increase 43.40% 72.38% Avg. increase

(reference scale) 14.71% 14.43%

Sustained Attention

Avg increase 15.29% 33.48% Avg. increase

(reference scale) 10.43% 19.29%

The attention effects measured in Study 2 are more evident than in Study 1, as shown in Table 4. The average percentage increase of selective attention is higher in Study 2 (from 43.4% to 72.38%) while the average percentage relative to the reference scale is substantially the same in the two studies. In Study 2 both measures of sustained attention are higher. The percentage increase relative to the reference scales in Study 2 is even more relevant if we consider that participants have lower capabilities than those in Study 1. In conclusions, the comparison between the two studies seems to suggest that the games developed according to our guidelines are more effective than the commercial games used in the first study, which have a similar game logic but are not explicitly designed for autistic children. In this respect, Study 2 can support the effectiveness of our design heuristics. The study reported in this paper certainly has a number of limitations. Tests were administered only once before and after the whole period of intervention. Autistic children’s behavior may considerably vary depending on various circumstances of the day. We could only control for the local contextual factors during game sessions (e.g., the physical setting). Repeated test could have offered more reliable results. The tester sample included five children only. This small size is comparable to of most existing research addressing autistic children in relationship to technology, and is not much lower than the participants’ size in most behavioral studies involving disabled children. Still, results measured on such a small number of subjects can only indicate a trend, and require further validation. In addition, the profile of our participants is considered “homogenous from a clinical perspective” (medium-low functioning subjects with low-moderate levels of cognitive deficit, sensory-motor dysfunction, and motor autonomy) but each autistic child has unique characteristics, as it is evident from the different measures of the clinical parameters at the beginning of the study. Hence many subjective variables that are difficult to control may have affected our results, and the heterogeneity of the target group

0

20

40

60

80

100

120

s1 s2 s3 s4 s5 s6

c01 c02 c03 c04 c05

0

200

400

600

800

1000

s1 s2 s3 s4 s5 s6

c01 c02 c03 c04 c05

DRAFT

makes meaningful comparative measures near impossible. Still, the therapists involved in our study were impressed by each subject’s increment relative to her starting point and relative to the reference scale of each measured parameter that according to their experience, tells us more about the learning potential of an intervention than absolute increments and their average values.

7. FINAL REMARKS This paper provides various contributions to the current state of the art in motion touchless technology for autistic children. We define a set of design guidelines for touchless games devoted to this target group that, to our knowledge, are novel. They are grounded on a vast experience on the field, distilling the know-how of many specialists - neurological doctors, therapists, educators, and our team. These guidelines have informed the design of 3 highly customizable touchless games. These tools have been used in a controlled empirical study devoted to assess the learning benefits of our tools and also to indirectly validate the effectiveness of our design guidelines. The results from this study confirm and extend the findings of prior research [2][3], providing additional empirical evidence that touchless gaming does have a strong potential to improve attention and motor/ visual skills in medium-low functioning autistic children. Still, all our results have to be considered as tentative and deserve further research. Considering the complexity of autism and the little we know about this disorder, our set of design guidelines should certainly be improved, refined, and validated. Additional empirical studies are needed to confirm the effectiveness of touchless interaction for autistic children’s learning. In particular, more complex longitudinal studies should be performed to assess the persistency of benefits and their generalization to environments outside the treatment one, and to address some key research questions: to which degree are the skill improvements we measured a long-term achievement? to which degree can they be translated to other contexts and moments of participants’ life? Our games are freely available on http://www.m4allproject.eu/ or by contacting the authors. We hope the IDC community will help us disseminate them to families with ASD children and therapeutic centers. We also hope that other researchers will replicate our study, to address the above research questions, or to perform new studies that consider different profiles of autistic children or explore different learning variables other than the ones considered in our study.

8. ACKNOWLEDGMENTS This work is partially supported by by the European Commission under grant “M4ALL-Motion Based Interaction for All” (# 2012-3969-531219 - Life Long Learning Program 2012). The authors are grateful to the children and families from Associazione Astrolabio, Centro Benedetta D’Intino, and San Camillo Hospital who participated in our studies.

9. REFERENCES [1] Alcorn, A., Pain, H. et al. 2011. Social Communication between

Virtual Characters and Children with Autism. Volume 6738, 2011, pp 7-14, Springer.

[2] Bartoli L., Corradi C., Garzotto F., Valoriani M. 2013. Exploring Motion-based Touchless Games for Autistic Children’s Learning. IDC 2013, 102-111

[3] Bartoli L.,Garzotto F., M. Valoriani. Virtual, Augmented Reality and Serious Games for Healthcare V. 1, 2014, chap. 23, Springer

[4] Biancardi A., Stoppa E. 1997. Modified Bell Test. Erikson. [5] Broaders S. C. et al. 2007. Making children gesture brings out

implicit knowledge and leads to learning. J. Experimental

Psychology 136 (4), 2007, 539-550. Springer.

[6] Blum-Dimaya, A., et al. Teaching children with autism to play a video game using activity schedules and game-embedded simultaneous video modeling. Education and Treatment of

Children, 33(3), 2010, 351-370 [7] Brickenkamp, R., Zillmer, E. 1998. The d2 test of attention.

Hogrefe & Huber Pub. [8] Evans, M. 2012. Gestural Interfaces in Learning. Proc. Of Int.

Conf. on Society for Information Technology & Teacher

Education 2012, 3337-3340. AACE. [9] Fabio R. 2003. L’Attenzione – fisiologia, patologie e interventi

riabilitativi. Franco Angeli. [10] Farneti P., Savelli L. 2013. La mente imitativa. Franco Angeli. [11] Ferrari E., Robins E., Dautenhahn K. 2009. Therapeutic and

educational objectives in robot assisted play for children with autism. Proc. RO-MAN 2009, 108-114, IEEE

[12] Gauthier L, Dehaut F, Joanette Y 1989. The bells test: A quantitative and qualitative test for visual neglect. Int. J. of

Clinical Neuropsychology, 11, 49-54. APA. [13] Grandhi, S. A., Joue, G., Mittelberg, I. 2011. Understanding

naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces. Proc. CHI 2011, 821-824. ACM.

[14] Herrera G., Casas X., Sevilla J., Rosa L., Pardo C., Plaza J., Jordan R., Le Groux S. 2012, Pictogram Room: Natural Interaction Technologies to Aid in the Development of Children with Autism. Annuary of Clinical and Health Psychology (8), 2012, 39-44. ACPS.

[15] Hsueh H.L., Ly A. Socio-emotional Learning Through Play and Reflection - http://memu.leehsueh.com/

[16] Kaliouby R., Robinson, P. 2005. The Emotional Hearing Aid: an Assistive Tool for Children with Asperger Syndrome. Universal

Access in the Information Society, 4 (2), 121-134. Springer. [17] Keay-Bright W., Howarth I., 2012, Is simplicity the key to

engagement for children on the autism spectrum? Personal and

Ubiquitous Computing, 16 (2), 2012, 129-141. Springer. [18] Keith E. Beery. 2008. VMI Developmental Test of Visual-Motor

Integration. Giunti O.S. [19] Kurtz L. 2007. Understanding Motor Skills in Children with

Dyspraxia, ADHD, Autism, and Other Learning Disabilities. Jessica Kingsley Publishers.

[20] Lee W. J., Huang C. W., Wu C. J., Huang S. T., Chen G. D. 2012. The Effects of Using Embodied Interactions to Improve Learning Performance. Proc. ICALT 2012, 557-559. IEEE

[21] McNevin, N. H., Shea, C. H., & Wulf, G. 2003. Increasing the distance of an external focus of attention enhances learning. Psychological Research, Feb. 2012, 67, 22–29. Springer.

[22] Orsini A., Pezzuti L., Picone L. 2012. WISC-IV Wechsler Intelligence Scale for Children-IV. Giunti O.S.

[23] Pares N., Masri P. van Wolferen, G., Creed C. 2005. Achieving dialogue with children with severe autism in an adaptive multisensory interaction: the "MEDIATE" project. IEEE Trans.

Visualization and Computer Graphics, 11, 734-743. IEEE. [24] Sylva K., Jolly A., Bruner, J. S. 1976. Play: its role in

development and evolution. Penguin. [25] Sugden D., Henderson S., 2007, Movement Assessment Battery

for Children, 2nd Edition. Pearson [26] Toner N., 2008, Conducting Preference Assessments on

Individuals with Autism and other Developmental Disabilities. [27] Van Zomeren, A. H., Brouwer, W. H. 1994. Clinical

neuropsychology of attention. Oxford University Press. [28] Xaiz C., Micheli E. 2008. Gioco e interazione sociale

nell’autismo. Erickson. [29] Zalapa, R. and M. Tentori, 2013. “Movement-based and tangible

interactions to offer body awareness to children with autism”. Proc. of UCAmi ’13, 1-6. Springer.