A Comparative Study between Tangible and Gesture based ...sandeepjagtap.com/researchpaper.pdf · (education), offices, libraries, gaming, etc. Most commonly performed gestures in

A Comparative Study between Tangible and Gesture based interactions for 3D Object Manipulation

Aziz Khilawala IUPUI

Indianapolis, US [email protected]

Pratik Nalawade IUPUI

Indianapolis, US [email protected]

Pushkar Joshi IUPUI

Indianapolis, US [email protected]

Pruthviraj Narayanaswamy

IUPUI Indianapolis, US

[email protected]

Sandeep Jagtap IUPUI

Indianapolis, US [email protected]

INTRODUCTION We describe a study comparing the usefulness and learnability of tangible interaction (TI) versus gesture interaction (GI) while manipulating a three-dimensional object. [1, 2, 3] We developed a prototype using Intel RealSense technology which allows people to interact with a three-dimensional model using a tangible object and mid-air hand gestures.[22] In one version users used the application using tangible object and gestures both, and in other participants used gestures only.

Figure 1. User interface of the prototype used during the study

With the advent of touchscreen and motion-sensing technologies, systems can not only recognize gestures and whole body interactions [5, 6] (e.g. Leap Motion sensor and Microsoft Kinect) but also tangible

objects (e.g. Microsoft’s Surface Dial, interactive tabletop displays). [2,7,8] Although keyboard and mouse based input are still preferred, at times it may not be a suitable solution for all types of users [9]. Gesture based interaction now provide realistic and affordable opportunities for people. It is a natural form of interaction. Instead of the typical use of mouse and keyboard [10]. For our study, we made use of an automobile car scenario in which users can manipulate the automotive object in three-dimensional space. User can rotate car model with a tangible object in one hand and change its color properties using gestures with the other hand. A body of literature suggests that the usage of tangible objects coupled with gestural input for education aids in a higher learning gain. Interaction with the tangible object improves the user experience while working with 3D models.[3] Also, the use of tangible objects coupled with hand gestures can help users make sense of abstract and unfamiliar digital representations. [11,12,8] Furthermore, it can make interaction appear more playful, promoting use and engagement. The scenarios that can employ three-dimensional object manipulation includes classroom settings or any collaborative learning environment which involves interaction with three-dimensional objects. For example, automotive industry and CAD (Computer-aided design) based applications.

The use of Intel RealSense Technology while designing 3D object manipulation applications utilizing only the in-air hand gestures limits the interaction opportunities with which user could interact with the system. This motivated us to design and develop a prototype application supported with tangible interaction using a physical object.Though Intel RealSense can accurately sense hand position, sensing hand orientation while manipulating an object is not robust.[13] The use of tangible object can counter this difficulty. To measure both the system's performance (tangible and gesture based), we compared each system on its ease of use (how easy it is for users to interact), learnability (how quickly can users learn the interactions), satisfaction (are users able to accomplish all the possible tasks using this application) and usefulness (System's ability, to be used advantageously for several purposes).

RELATED WORK By studying the related work we wanted to explore the usage of gesture-based interactions in various domains of ubiquitous computing and where gesture controlled user interface (GCUI) can be used more efficiently. As intuitiveness is a very vague term so the aim of our study was what other process and techniques have authors used so that we could use it for testing the usability of our gesture vocabulary. Relevant prior work includes studies of human gesture based on memorability, intuitiveness, and classifying hand gestures into categories. Gesture based application Yikai Fang and others [14] have proposed an automatic hand tracking method based on the depth information that is available and used in applications like maps. Niels Henze and others [15] have proposed a refined process for deriving gestures from constant user feedback and tested the process with different users. These works elaborate on how we can use gesture-based interaction in various mobile applications. Cuccurullo and others [16] created an application which utilized user’s body as the gesture based input and captured it with Kinect technology to control the flow of the presentation. The paper also highlighted the challenges of vision based recognition technology.

While above applications are in social environments, we are focusing our prototype in the automotive industry, where industry profession work on 3D car model and perform actions like rotation, scaling, and changing other properties. Piumsomboon and others [17] conducted an elicitation study and published a recommended set of gestures for augmented reality based on the results of the study. Buchanan and others [18] conducted a study to understand Multi-touch interaction with 3D objects on a 2D multi-touch display. Gestures were classified either in physical or metaphorical nature and were compared with each other. Virtual reality technology is extending in all the places like schools (education), offices, libraries, gaming, etc. Most commonly performed gestures in VR is 3D object rotation and transformation [19]. We are using Intel RealSense to recognize the gestures, which has limitations like recognizing the 360° hand movement [13]. To overcome that limitation we are introducing the usage of a tangible object which will help in performing gestures in the real world. Heuristics for evaluating gestures Michael Nielson and others [20] gave a procedure on how to design and develop the gestures which are ergonomically intuitive. What we found that MYO is technologically based vocabulary, which has a limited number of gestures and if used, has a high probability that we will have to force the available functions upon these gestures. For user studies we are assigning gestures to specific functions. For testing the usability of our gestures we will use the following heuristics which are given by their work: 1. Use of simple and natural dialogue which are more familiar to the people. 2. Speak the user’s language 3. Minimize user memory load 4. Be consistent 5. Provide feedback 6. Provide clearly marked exits 7. Provide shortcuts 8. Provide good error messages 9. Prevent errors In their paper, Moniruzzaman Bhuiyan and others [10] tested a gesture based prototype (open gesture) app and collected qualitative data. We will use the similar line of questions to collect and further analyze the data to derive conclusions. In this paper, we will be evaluating the gestures on

effectiveness and efficiency using common usability metrics like task completion time and task success rates along with Fisher's Exact test to analyze likert scale ratings. Learnability and Memorability Nacenta and others [21] in their work on Memorability of gestures concluded that User-defined were more memorable (by 44%) than the pre-defined gestures in three experiments that involved testing the gestures memorability a day after the users were first exposed to those gestures. Jude and others [22] conducted a user study to understand user preferences in gestures and concluded that grasp gesture was preferred over pinch or grab gesture. We are using a combination of pre-defined gestures and a tangible object and determine if this combination is learnable for the users.

RESEARCH QUESTION(S) With the comparative study described in this paper we try to validate our hypothesis that the use of tangible interaction using a physical object along with the hand gestures can significantly improve usability in terms of ease of use, learnability, and satisfaction. More specifically,

● Does the introduction of Tangible object make the interface more useful ?

● Does the Tangible object make the interface more learnable ?

● Does the use of Tangible object make the interface easier to use ?

● How to create a good tangible object based on qualitative data?

We examined the above questions by testing the two versions of our application, one that used tangible object supplemented by hand gestures, and one that used only gestures.

This study can contribute in Human Computer Interaction (HCI) research to identify the strengths and weaknesses of tangible interaction versus gesture interaction in the use case of three dimensional object manipulation. Our findings can help researchers and designers in the design of tangible based interactive systems for engagement and collaborative use.

PROTOTYPE AND TANGIBLE OBJECT The prototype under evaluation is based on Intel RealSense technology [4]. Intel RealSense uses camera sensor to measure depth and hand tracking. It provides us a set of seven predefined gestures which are used to grab, hold, scale and select objects in three-dimensional space [24]. Grab/Release gesture was used to select objects on the interface, and Engage gesture was used to track user’s hand movement to mimic cursor.

Figure 3. The seven predefined gestures available

with Intel RealSense. Tangible object We used a cube as tangible object for interaction due its affordance to distinguish different perspective in 3 dimensional space. Also we limited size of tangible object so that user could comfortably hold using one hand.

Figure 4. Tangible object used for evaluations. Interface The interface for both interaction was purposefully kept similar except the additional hot spots [fig 5] in case gesture based interaction.

Figure 5. Graphical User Interface of tangible and gesture based interaction.

Figure 6. Graphical User Interface of gesture based interaction.

RESEARCH METHODOLOGY

Figure 7. Participant interacting with prototype using tangible object.

We used both qualitative and quantitative research methods to understand the strength and weakness of Tangible Interaction (TI) with Gesture Interaction (GI). We used usability evaluation method to test the prototype to benchmark system's performance on ease of use, learnability, usefulness, and satisfaction using both interaction methods. Additionally, a semi-structured interview was conducted to understand the cause-and-effect relationship of user's behavior and system's response. During the testing, we switched between the order of interaction method and tasks to counter balance carry over effect i.e. to eliminate participant’s learnings from one session to the next. The pilot study of prototype with tangible object revealed the limitation of Intel RealSense technology to accurately detect object’s movement with image recognition. At the end, we decided to use the wizard of Oz technique for TI in order to give fair advantages against GI. Participant For the study, we recruited 16 participants within the age range of 22-28 years comprising an even distribution of both genders. The participants were recruited by posting advertise on unviersity classified. Most of the participants were university students at IUPUI. The selected participant had prior experience with computers and limited or no experience with the gesture or tangible object based interactions.

Participant Male Female

n = 16 10 (62.5%) 6 (37.5%)

Table 1: Participant’s gender distribution.

Age Participants

22 2

23 3

24 5

25 2

26 2

27 2

(22-27) Total=16

Table 2: Participant's age distribution.

Usability Evaluation The usability sessions were conducted in a controlled environment at IUPUI's Audio Video Lab. The users were subjected to perform the following task using both TI and GI 1.Rotate car clockwise or anti-clockwise around vertical axis (Y) 2.Change the car’s color (RED, GREEN, BLUE) In order to direct participant to rotate car in certain orientation, we made use of physical replica of car to demonstrate the same. The participant perform similar task using both tangible and gesture interaction. After interacting with the system using i.e. either TI or GI participant evaluated system’s interaction using the following questionnaire: Learnability: It was easy to learn to use this system. Ease of Use: It was easy to use this system. I was able to efficiently complete the tasks and

scenarios using this system. I can master using this system with minimal effort. Usefulness: I believe I could become productive quickly using this system I would definitely use such system in future. I will prefer this interaction method over traditional input method in this scenario. Satisfaction: I felt comfortable using this system. Overall, I am satisfied with this system. The questions were designed to assess system's ease of use, learnability, usefulness and satisfaction. The responses were recorded based on the Likert scale. Participants were also asked to think aloud while performing the task in order understand participants behavior. Additionally, the sessions were audio and video recorded along with task metrics like task times and task success rates. Each session lasted for 30-45 minutes. At the end of the session, users were asked to describe their experience using TI and GI. Users were also asked to pick their favorite interaction and to comment on it. Occasionally users were asked probing questions to understand the reasoning behind their distinct behavior. At the end of the interview, the users were asked to provide suggestions or recommendation. RESULTS Does the introduction of Tangible object make the interface more useful ?

Figure 7 : Usefulness of Gesture Interaction and

Tangible Interaction.

To answer this question we analysed the quantitative likert scale data that was collected after user completed all the given tasks with Tangible Interaction were completed. After analyzing the data three themes emerged which were ultimately contributing to the usefulness of the system.

1. Productivity: Four out of sixteen users strongly agreed that Tangible Interaction will increase their productivity while six users agreed that it will improve their productivity. While three users were neutral and 2 disagreed and only 1 participant strongly disagreed that it mentioned that it might even be counter productive.

2. Use in future: Ten out of sixteen participants believed that they would use the Tangible Interaction system in the future and for their work.

3. Preference over gesture based system: 50% of the participants prefered the gesture based system while 50% prefered the gesture based interactions.

From three sections we can say that users felt Tangible Interaction is useful in different scenarios.While majority of the participants believed that TI system will improve their productivity and they’d use it future,only half of the participants agreed that they would prefer the object over gesture. Does the Tangible object make the interface more learnable ?

Figure 8 : Learnability of Gesture Interaction

and Tangible Interaction . Among the sixteen participants, 80% of the participants agreed that TI system is easy to learn

while 100% of the participants agreed that GI system is easy to learn. Qualitative Analysis also revealed that most users found the system to be learnable which ties in with the quantitative data. Fisher exact test found no significant impact (p=0.2652) on the learnability of the system by the introduction of a tangible object. Does the use of Tangible object make the interface easier to use ?

Figure 9 : Ease of Use of Gesture Interaction and

Tangible Interaction.

We gathered ease of use data by capturing how the users perceived the system when interacting with it and their performance on the tasks in Tangible Interaction (TI) and Gesture Interaction (GI). The analysis of Likert scale data showed that 61 % of users agreed that the Gesture interaction was easy to use while 75% of users agreed Tangible interaction was easy to use. A Fisher exact test found no statistically significant (p-value = 0.3122) impact on the interactions by the introduction . The results in the table show that when Gesture interaction was quicker than Tangible interaction. It was also observed that the Tangible interaction was significantly quicker when the user performed Gesture interaction first. Qualitative analysis revealed that using gestures and tangible object at the same time was a taxing task and added additional cognitive load to the user.

Task 1 Task 2 Task 3

Gesture Interaction

14.16 s 11.66 s 15 s

Tangible Interaction

15.5 s 9.3 s 13.6 s

Table 3 : Gesture Interaction first - Average

times.

Task 1 Task 2 Task 3

Gesture Interaction

12.25 s 8.87 s 10.37 s

Tangible Interaction

20.37 s 14.62 s 13.5 s

Table 4 : Tangible Interaction first - Average

times. Does the introduction of Tangible object improve the satisfaction of the interface?

Figure 10 : Satisfaction of Gesture Interaction and Tangible Interaction .

Satisfaction of the user was determined by understanding the user's comfort on Likert scale and by analyzing post task comments. Fisher exact test revealed no statistical ( p = 0.5677) improvement in the user satisfaction by the introduction of Tangible object.

Based on Qualitative Analysis we found the some additional insights

1. Most of the users complained about accuracy and sensitivity of both the interactions. Many users struggled when the object hand crossed their gesture hand and the system failed to adapt to the change. Users also complained about performing repetitive actions as the RealSense camera sometimes failed to recognize their actions

2. When interacting with the tangible object 3 of our users suggested some form of haptic feedback from the tangible object when any action is completed successfully. 5 users expected to have some form of input via clicking or tapping on the tangible object instead of always having it in front of the camera

3. Users also raised a concern on selecting an object when interacting with multiple objects on screen. They expected some gesture for selection in Gestural interaction and also that the tangible object should allow them to switch between objects

DISCUSSION From the evaluation results, it became clear that users were able to learn and use the gestures-only system better when compared with the tangible-and-gestures based system. Also, introducing gestures with tangible interaction created some pain points for the participants as they were not able to perform two finger gesture to rotate the model using right hand and rotate the model using cube shaped tangible object. The qualitative data - “My right hand bumps into my left side and I can’t really use the cube to rotate the car so easy” indicated that users’ hands collided with each other and obstructed the interaction. The hypothesis could not be confirmed as the the data did not indicate any statistically significant change between Gesture interaction and Tangible interaction in our experimental condition. While the data gave us the comparison between the two system TI and GI, we also received some complaints from users about both of the systems, in which they compared both these systems with the traditional mouse and keyboard system. Accuracy and sensitivity of prototype was not upto the level of traditional system, and we suspect this to be the reason behind the low agree scores on the Likert

Scale when asked if they would prefer these systems over traditional systems Although both systems scored high on their learnability for both novice and expert computer users, the system would not be used due to the low scores on the perceived usefulness of the system for manipulating 3D objects Users expected different forms of interaction with the tangible object other than the camera recognizing the rotation of the object. A click or a tap on the side of the object can be utilized that can change the orientation of the object or help the user to select one object from multiple objects. An interactive tangible object could also eliminate the requirement of gestures creating a Tangible object only interaction. This form of system would require another round of comparative study to understand its usefulness, ease of use and learnability. There were multiple comments on the form factor of the tangible object with suggestions of making the object same as the object on screen or using a circular object. Although a square object gives the user an idea of edges and orientation, the current task of horizontal rotation could be achieved with a cylindrical object that works as a radio dial and offers precise control over object orientation. In the present study, we explored rotate and changing the color property of the car model. It would be interesting to explore if more diverse tasks such as zoom in/out, simulation control, integrating multiple 3D parts can influence user preferences with respect to Tangible Interaction and Gesture Interaction. CONCLUSION The results of this study suggest that there is an advantage of using tangible interaction to increase the productivity. Product designers looking to develop 3D object manipulation in learning or industry can leverage tangible interaction to increase productivity. The tangible object can also be enhanced to support additional functionalities thereby increasing productivity further. However, several of our findings from the study suggest that introduction of tangible object does produce significant impact on ease of use, learnability and satisfaction.

FUTURE WORK The findings from the study merit further investigation on the following topics. Tangible object form factor and feedback The shape and form factor of the tangible object needs further analysis. The shape and form factor of object and its perceived functionality seem to be related to each other. Additionally, further study is required to understand the feedback mechanism for the tangible object itself, especially in haptic and visual nature. Comparative study between tangible-only system and gesture based system The study in this paper compares gesture interaction with tangible and gesture interactions. In future, similar comparison between gestures-only interaction with tangible-only interaction can provide additional insights.

REFERENCES 1. Gesture Interaction, The Interaction Design

Foundation, https://www.interaction-design.org/literature/topics/gesture-interaction

2. Tangible Interaction: The Glossary of Human Computer Interaction,The Interaction Design Foundation, https://www.interaction-design.org/literature/book/the-glossary-of-human-computer-interaction/tangible-interaction

3. Sanmathi Dangeti, Yingjie Victor Chen, and Chunhui Zheng. 2016. Comparing bare-hand-in-air Gesture and Object-in-hand Tangible User Interaction for Navigation of 3D Objects in Modeling. In Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '16). ACM, New York, NY, USA, 417-421. DOI: https://doi.org/10.1145/2839462.2856555

4. Intel® RealSense™ Technology, Intel, http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html

5. Robert Walter. 2015. Whole Body Interaction with Public Displays. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, New York, NY, USA, 235-238.

DOI: https://doi.org/10.1145/2702613.2702614

6. S. Mitra and T. Acharya, “Gesture recognition: A survey,” Syst. Man, Cybern. Part C Appl. Rev. IEEE Trans., vol. 37, no. 3, pp. 311–324, 2007.

7. A closer look at Microsoft’s crazy Surface Dial puck, The Verge, http://www.theverge.com/2016/11/10/13586658/microsoft-surface-dial-hands-on

8. Joyce Ma, Lisa Sindorf, Isaac Liao, and Jennifer Frazier. 2015. Using a Tangible Versus a Multi-touch Graphical User Interface to Support Data Exploration at a Museum Exhibit. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 33-40. DOI: https://doi.org/10.1145/2677199.2680555

9. S. Trewin and H. Pain, “Keyboard and Mouse Errors due to Motor Disabilities,” International Journal of Human-Computer Studies, Vol. 50, No. 2, 1999, pp. 109-144.

10. Bhuiyan, M., & Picking, R. (2011). A Gesture Controlled User Interface for Inclusive Design and Evaluative Study of Its Usability. Journal of Software Engineering and Applications, 04(09), 513-521

11. Horn, M.S. The role of cultural forms in tangible interaction design. In Proc. TEI 2013, ACM Press (2013), 117-124.

12. Hornecker, E. and Buur, J. Getting a grip on tangible interaction: a framework on physical space and social interaction. In Proc. CHI 2006, ACM (2006), 437–446.

13. Song, P., Goh, W. B., Hutama, W., Fu, C. W., & Liu, X. (2012, May). A handle bar metaphor for virtual object manipulation with mid-air interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1297-1306). ACM

14. Fang, Y., Chai, X., Xu, L., & Wang, K. (2009). Hand tracking and application in map navigation. Proceedings of the First International Conference on Internet Multimedia Computing and Service - ICIMCS '09

15. Henze, N., Löcken, A., Boll, S., Hesselmann, T., & Pielot, M. (2010). Free-hand gestures for music playback. Proceedings of the 9th International Conference on Mobile and Ubiquitous

Multimedia - MUM '10 16. Cuccurullo, S., Francese, R., Murad, S.,

Passero, I., & Tucci, M. (2012, May). A gestural approach to presentation exploiting motion capture metaphors. In Proceedings of the International Working Conference on Advanced Visual Interfaces (pp. 148-155). ACM.

17. Piumsomboon, T., Clark, A., Billinghurst, M., & Cockburn, A. (2013, April). User-defined gestures for augmented reality. In CHI'13 Extended Abstracts on Human Factors in Computing Systems (pp. 955-960). ACM.

18. Buchanan, Sarah, et al. "Towards user-defined multi-touch gestures for 3D objects." Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces. ACM, 2013.

19. Soh, J., Choi, Y., Park, Y., & Yang, H. S. (2013, November). User-friendly 3D object manipulation gesture using Kinect. In Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (pp. 231-234). ACM.

20. Nielsen, M., Störring, M., Moeslund, T. B., & Granum, E. (2004). A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. Gesture-Based Communication in Human-Computer Interaction Lecture Notes in Computer Science, 409-420.

21. Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of pre-designed and user-defined gesture sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1099-1108.

22. Alvin Jude, G. Michael Poor, and Darren Guinness. 2016. Grasp, Grab or Pinch? Identifying User Preference for In-Air Gestural Manipulation. In Proceedings of the 2016 Symposium on Spatial User Interaction (SUI '16). ACM, New York, NY, USA, 219-219

23. Varga, E., Verlinden, J., Klaas, O., Langenhoff, L., Steen, D. V. D., & Verhagen, J. (2008, July). A study on intuitive gestures to control multimedia applications. In Proceedings of the IADIS International Conference on Interfaces and Human Computer Interaction

24. Intel® RealSense™ Technology, Intel, http://www.intel.com/content/www/us/en/support/emerging-technologies/000006242.html

25. Schneider, B., Jermann, P., Zufferey, G., & Dillenbourg, P. (2011). Benefits of a tangible interface for collaborative learning and interaction. IEEE Transactions on Learning Technologies, 4(3), 222-232.

26. Song, P., Goh, W. B., Hutama, W., Fu, C. W., & Liu, X. (2012, May). A handle bar metaphor for virtual object manipulation with mid-air interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1297-1306). ACM

27. Neff, M., Kipp, M., Albrecht, I., & Seidel, H. P. (2008). Gesture modeling and animation based on a probabilistic re-creation of speaker style. ACM Transactions on Graphics (TOG), 27(1), 5.

28. Sardana, A., & Bairagi, A. K. (2010, April). Gest: exploring gestural interactions. In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 4777-4782). ACM.

29. Speelpenning, T., Antle, A.N., Doering, T., and van den Hoven, E. Exploring how tangible tools enable collaboration in a multi-touch tabletop game. In Proc. INTERACT 2011, Springer-Verlag (2011), 605–621.

30. Zaman, B., Vanden Abeele, V., Markopoulos, P., and Marshall, P. Editorial: the evolving field of tangible interaction for children: the challenge of empirical validation. Personal and Ubiquitous Computing 16, 4 (2011), 367–378.