the molecular control toolkit - Controlling 3D molecular graphics via gesture and voice

The Molecular Control Toolkit: Controlling 3D Molecular Graphics via Gesture and Voice

Quan Le HCI Lab

17th Mar, 2015

Sabir, K. ; Garvan Inst. of Med. Res., Australia ; Stolte, C. ; Tabor, B. ; O'Donoghue, S.I. IEEE Symposium on Biological Data Visualization 2013, Atlanta, Georgia, USA

Contents

Introduction

Outstanding

Toolkit Overview

Toolkit Architecture

Toolkit Design

Results

Future Work

Conclusion

2

Fig 1. Molecular Control Toolkit using Leap Motion and Aquaria

Introduction

3

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

- Three-dimensional (3D) molecular graphics systems are widely used in the life sciences.

- Biological systems requires use of molecular graphics to examine atomic-resolution, protein, DNA, …

- Precise control 3D molecular structures can be challenging for occasional users, event experts.

Introduction

4

Fig 3. Aquaria application is accessed by http://aquaria.ws

Fig 2. Molecular Control Toolkit runs as a standalone Java application

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

- Molecular Control Toolkit supports multiple gestures and voice recognition.

- Apply Toolkit to Aquaria molecular graphics system.

Outstanding

- The common interactions: translation, rotation, zooming, selection of specific items, change display mode, …

- Current applications: • In 1996, Pavlovic: using hand gesture and speech interface.

• Multi-touch display: applies six degrees of freedom (6DoF*), only uses finger gestures.

• Sony’s EyeToy, Playstation Eye, uses audio gesture control.

• Nintendo Wii and Sony Move: requires a physical device to manipulate objects in 3D space.

5

* 6DoF: Six degrees of freedom, comprising of three axes for rotation and three for translation.

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Outstanding

- Molecular Control Toolkit provides a full six degree of freedom.

- Ability to redefine the center of rotation, integrated voice commands.

- Lower cost, more intuitive to learn, interact and communicate.

6

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Six degrees of freedom (6DoF)

7

Fig 4. The ship movements on the wave

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

The ship movements on the wave:

Translation:

1 – Moving up and down

2 – Moving left and right

3 – Moving forward and backward

Rotation:

4 – Turning left and right

5 – Tilting forward and backward

6 – Tilting side to side

Molecular Control Toolkit

- Supports two gesture devices: Leap Motion and Microsoft Kinect to control and interact with a Java Application.

- Provide voice commands.

8

Fig 5. Molecular Control Toolkit interacts with Aquaria via Leap Motion and MS Kinect

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Toolkit Architecture

- Gesture Listener

- Device Connectors

- Gesture Dispatcher

- Voice Dispatcher

9

Fig 6. Molecular Control Toolkit architecture

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Toolkit Listeners

Connectors & Dispatchers

Toolkit Design

• Connector for Kinect gesture recognition:

10

Fig 7. Support gesture controls for Kinect

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Toolkit Design

• Connector for Leap Motion gesture recognition:

11

Fig 8. Support gesture controls for Leap Motion

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Toolkit Design

• Connector for Kinect voice recognition:

- Microsoft Kinect Speech API.

- Recognize the words from a custom grammar dictionary.

12

Fig 9. Supported voice commands for Molecular Control Toolkit

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Results

13

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Fig 10. Participants background experience

- 18 staff members from Medical Research in Sydney, Australia.

- 20 ~ 30 minutes for training.

- Three tasks: Rotating, Selecting and Zooming.

Results

14

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Fig 11. Pilot study comparing mouse/keyboard, Leap Motion and Kinect controls for molecular graphics.

Future Work

15

- Voice command:

• The vocabulary should be expanded.

- Support other new gesture devices: Create Interactive Gesture Camera, PrimeSense, …

- Provides gesture tracking algorithms for both Leap Motion and Kinect at the same time.

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

Conclusion

16

Demo: https://www.youtube.com/watch?v=XU_6TWM9ke8

Introduction Outstanding

Toolkit Overview Toolkit Architecture

Toolkit Design Result

Future work Conclusion

- Molecular Control Toolkit is the first system designed to enable life scientists to control molecular graphics using gestures and voice commands.

- Supports Leap Motion and MS Kinect.

Thank you!

17