Best Practices in Natural Interaction Design (Gestures, Posture, Speech, Gaze)

1

Humanizing Interaction with Devices

www.nuia.com

Interaction with devices is about to change

2www.nuia.com

NUI vs. GUI

3

•  NUI hardware is still in a quite early stage

•  Precision and recognition rate are much lower

–  A gesture may not be recognized immediately/definitely and a gaze click might not exactly be at the place the user looks at

•  Interfaces can be more intuitive, but on the other hand also not as clear as traditional user interfaces

–  Which voice commands or gestures are available?

•  Direct manipulation of objects displayed to the user is possible

www.nuia.com

Best practices - General

•  Do not try to recreate traditional interfaces

–  Design for as little learning as possible and use familiar interaction principles

•  Think of usage scenarios, e.g. in open space office a speech interface may not be optimal due to distraction of other users. Similarly, do not require the users to perform the same hand gesture a hundred times since they will tire of it quickly.

•  Actively plan for false positive and false negatives: Just like human interaction misunderstandings are common. Today users expect devices to respond error free to any input, so you need strategies

–  Direct feedback ideally with option to continue or undo

–  “Double check” by combining sensors, based on history and competing detection algorithms

–  Minimize requirements regarding precision

•  Test, evaluate with several people and environments and tune the user experience

4www.nuia.com

Best practices – Gesture control

5

•  Use gestures for simple commands and direct manipulation

•  Use natural and intuitive gestures, ideally inspired by RL or at least iconic

•  Stay consistent: The meaning of gestures should not change wildly with context

•  Performing the gestures should be possible with both hands

•  Keep in mind cross-cultural audiences and user variation, since a gesture can mean very different things in different places and users can have very different habits

•  Do not occlude the relevant content with the users hands

•  Give feedback about the camera FOV. The hands should not be "just lost" because the user accidentally left the camera's FOV

•  Make sure recognition is robust across different platforms with different camera angle and user distance (Notebook, AIO, Add-on)

www.nuia.com

Best practices – Voice control

6

•  Use voice for more complex commands and when not within range of other sensors

•  Voice commands should be based around natural language usage

•  Use synonyms for the commands to be more flexible and tolerant in interpreting the input

•  Avoid using a single word as command (ideally two or three words)

•  Either avoid dangerous commands or implement confirmation/undo due to possible false negatives

•  Be aware of pauses and interjections during speech input

•  Consider environmental noise and social appropriateness

•  Always show the status of the speech engine (listening, not listening, processing sound)

•  If you implement voice feedback make it short and intelligent

www.nuia.com

So why not just use these new sensors?

7

•  Sensors cannot just be connected to existing devices and applications due to missing hooks

•  Uncertainty which vendor will “win the race”

•  Multiple sensors will conflict with each other without “rules”

•  Very few interaction standards have been established

www.nuia.com

NUIA provides a standardized integration layer

8

•  NUIA connects all next generation sensors with any environment

–  Eye Trackers, 3D Cameras, Microphones, IoT Devices, Wearables, Mobiles and AR/VR Devices

•  NUIA incorporates the user context in real-time, enabling the devices to interact intelligently and automatically with us

•  Hundreds of use cases with most used applications and web services already integrated

•  NUIA SDK enables rapid prototyping of new use cases to save time from idea to implementation

•  Install and use - No source code changes in controlled applications necessary

www.nuia.com

100+ Application Extensions

NUIA Software & SDK

Your Environment(Workstation, HUD, IoT Device,

Control Panel, 3D Cave, …)

9

NUIA platform as base for all products and projects

Your Application

?

Next Generation Sensors(Eye Tracking, 3D Cameras, Voice Recognition, Mood Detection,

User Authentication, VR/AR Headsets, Neuro Headsets, …)

www.nuia.com

10

Humanizing Interaction with Devices

[email protected]

„… [NUIA’s] vision for reshaping how the masses interact with computers could become a reality sooner rather than later.“ - TechCrunch

www.nuia.com