Intel real sense handson

Tips and Trikes from

Real Case Studies

Matteo Valoriani

[email protected]

@MatteoValoriani

#RealSense

PhD at Politecnico of Milano

CEO of Fifth Element

Speaker and Consultant

Intel Software Innovator: Perceptual Computing

Microsoft MVP on Kinect

Who I am…

2

[email protected]

@MatteoValoriani

Intel® RealSense™ Hands-On Lab - Milan

3Intel® RealSense™ Hands-On Lab - Milan

Follow me on

Twitter or the

Kitten gets it:

@MatteoValoriani


You have to be a magician…

or at least a good illusionist

https://www.youtube.com/watch?v=u_diRgwPCS8

https://www.youtube.com/watch?v=u_diRgwPCS8

UI evolution

Intel® RealSense™ Hands-On Lab - Milan 5

Understands 4 basic types of input


Categories of

Input

Capabilities Features

Hands • Hand and Finger Tracking

• Gesture Recognition

• 22-point Hand and Finger Tracking

• 9 static and dynamic mid-air gestures

Face • Face Detection and

Tracking

• Multiple Face Detection and tracking

• 78-point Landmark Detection (facial features)

• Emotion Recognition (7 emotions, coming post-Beta)

• Pulse Estimation

• Face Recognition (Coming post-beta)

Speech • Speech Recognition • Command and Control

• Dictation

• Text to Speech

Environment • Segmentation

• 3D Scanning

• Augmented Reality

• Background Removal

• 3D Object / Face / Room Scanning (Coming post-beta)

• 2D/3D Object Tracking

• Scene Perception (coming post-beta)

Understands Hardware Limits


Competitive

technologies focus

on a living-room

experience or a

sub-set of Intel

RealSense

technology

features

Designed for close-range interactions


120 cm

Intel®RealSense™

3D camera

56°(v) x 72° (v)

20 cm

Leap, RealSense, Kinect


2,5 cm 60 cm 2 m 4 m

Vertical Rages


60cm

58 c

m

120cm

56°

20cm

17 c

m

70cm 35cm73 c

m

Effective Range

Gestures Range

Effective Range

3D Facial Traking Range


115cm

Vertical Misalignment


56°

Horizontal Rages


60cm

87cm

120cm

72°

20cm

24cm

Effective Range

Gestures Range

Effective Range



170cm

70cm 35cm108cm

50cm

Capture Volumes


The user is performing a hand gesture outside of the capture volume.

The camera will not see this gesture

Evaluate different settings and environment


RealSense Camera use IR light and Sunlight can blind the

camera!!!

• Check exposition during all day

• Verify that there isn’t direct light on the camera

Indoor/Outdoor


RealSense isn’t a Rugged device:

• Check temperatures (+3/33°)

• Check humidity

Indoor/Outdoor (2)


Comfortable positions

Your users are not GORILLAS!!!


User posture may affect design of a gesture


Input variability


Feedback, feedback, feedback,…


View of user:

• User Viewport

• User Overlay

Feedback, feedback, feedback,…


… where actions performed for some other purpose or unconscious signs are interpreted in order to influence/improve/facilitate the actors' future interaction or day-to-day life (from Alan Dix)

• The interaction is not purposeful from the person side, but it is designed “to happen”

• It “happens” in relation to signs which are not done for that (body temperature, unconscious reactions such as blink rate, or unconscious aspects of activities such as typing rate, vocabulary shifts (e.g. modal verbs), actions done for other purposes, …

• It is designed for people acting

Manage Incidental Interaction



Takeaways

Gestures should have a clear cognitive association with the semantic

functions they perform and the effects they achieve. Intuitiveness can be

enforced by appropriate interface and feedbacks.

The semantics of gestural patterns that belong to everyday life or

common task should be as consistent as possible to their “conventional”

meaning, but also take into account that intuitiveness is strongly

associated with users’ cultural background, general knowledge, and

experience.

Semantic intuitiveness

Gestural communication involves more muscles than keyboard interaction

or speech. Gestural commands must therefore be concise and quick, and

minimize user’s effort and physical stress.

Two types of muscular stress are known: static, the effort required

maintaining a posture for a fixed amount of time; dynamic, related to the

effort required to move a portion of the body through a trajectory.

Minimalize Fatigue

It must be easy for the user to learn how to perform and remember

gestures, minimizing the mental load of recalling movement trajectories

and associated actions.

The learning rate depends on tasks, user experience, skills, as well as the

size of the gesture language (more gestures decrease the learnability

rate).

Favor ease of learning (Learnability) 1/2

The gestures that are most natural, easy to learn and are immediately assimilated by the user are those that belong to everyday life, or involve the least physical effort. These gestures should be associated to the most frequent interactions.

Complex gestures can be more expressive and give more control, but have a higher learnability burden.

Hence there is clearly a tension between design requirements, among which a compromise must be made: naturalness of gestures, minimum size of the gesture language, expressiveness and completeness of the gesture language.

Favor ease of learning (Learnability) 2/2

Users can perform unintended gestures, i.e., movements that are not

meant to communicate with the system they are interacting with.

The “immersion syndrome” occurs if every movement is interpreted by

the system, whether or not it was intended, and may determine

interaction effects against the user’s will.

Intentionality (Immersion Syndrome) 1/2

The designer must identify well-defined means to detect the intention of

the gestures, as distinguishing useful movements from unintentional ones

is not easy.

Body tension and non-relaxed posture of users can be used to make

explicit the user intention to start interaction, issue a command, or

confirm a choice.

The tense period should be short to not generate fatigue.

Intentionality (Immersion Syndrome) 2/2

Appropriate feedback indicating the effects and correctness of

the gesture performed is necessary for successful interaction,

and to improve the user's confidence in the system.

Not-self-revealing

Intel RealSense

Inspiration Projects


https://software.intel.com/sites/campaigns/perceptualshowcase/

winners.htm

http://www.intel.com/content/www/us/en/architecture-and-

technology/realsense-jim-parsons-flight-attendant-to-mars.html

Intel® Perceptual Computing Showcase


https://software.intel.com/sites/campaigns/perceptualshowcase/winners.htm

http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-jim-parsons-flight-attendant-to-mars.html

34

• The robot is built using Legos* Mindstorm

• A standard Ultrabook is place on top

• It acts as the brains

• As well as the interface

• Intel 3D Cameras are used to capture the

environment & steer the robot around

• It also allows for some facial & gesture recognition

Rover the

Self-Driving Car

Intel® RealSense™ Hands-On Lab - Milan

Space Between

• Used both Gesture & Voice commands in this

Puzzle Oriented Game about teaching a creature to

survive in a virtual world

• Used Unity* 3D as its game engine


3D Head Scanner

• Rapidly (about a minute) does a full scan of

the user’s head for avatar creation or other

uses such as fashion & beauty applications

• In the original app, users could change hair

styles, skin complexion, etc.

• Built as a Unity* project, it could be used by

other applications & games for a wild array of

different usage models


Stargate SG1

Gunship V2

• An evolution of the award

winning Perceptual Computing

game

• Controller-free, immersive UX

• Two-handed, multi-axis camera

and fire controls

• Menu and in-game voice

commands


Redwall: The

Warrior Reborn

• Facial landmark mapping to

character avatars

• Fully controller free UI

• Expression sensitive NPC

interactions.


Twitter Question!


WHAT IS THE EFFECTIVE

RANGE FOR 3D FACIAL

TRACKING?


#RealSense

@IntelRealSense

Intel real sense handson

Technology

d camera

user posture

d object face room

multiple face detection

hand gesture

check temperatures

check exposition

check humidity indooroutdoor