Forge - DevCon 2016: Introduction to building for HoloLens

You Had Me at HoloWorld

Dona SarkarPrincipal Product Person

1994-2004 tech

2005-2015 tech

2016-???

the future is screenless

we are analog creatures

Human Environment

Machine

Tech

HoloLens: Windows 10 computer

design. explore. communicate.

HoloLens blends interactive holographic content and your physical environment.

“What are holograms good for?”

experiences and information embedded in your environment

Transportive Storytelling

transportivestorytelling

exploration and collaboration

Making and seeing 3D in 3D

Complement existing devices and workflows

“What are holograms not good for?”

Challengesin holographic design INDOOR FOCUS BATTERY LIFE NEAR CLIP PLANE ARM FATIGUE

Universal Windows

Platform

Tools

Ecosystem

Windows

Store

Windows

Holographic

Unity

DirectX

Visual Studio

Emulator

Build with Windows

Holographic Voice Commands

Spatial Mapping

Gesture Input

Spatial Sound World Coordinates

Gaze Input

• used for targeting and intent

• uses position and orientation of your user's

head, not their eyes, to determine their gaze

vector

• laser pointer from between the eyes

• as the user looks around app intersects this

ray, both with its own holograms and with

the spatial mapping mesh to determine

what virtual or real-world object your user

may be looking at.

• implement a cursor in HoloLens that

represents the user's gazeGaze Input

• allows interaction with hologram with hands or clicker• gestures recognized by tracking position of either or

both hands that are visible to the device• hands: ready state + pressed state• clicker• within the gesture frame

• 2 main gestures:• Air-tap – activate a hologram• Bloom – return to Shell

• app can recognize more by combining presses and releases with movement of your hand or using a clicker

Gesture Input

• users can gaze at a hologram and speak your command

• voice input is powered by the same engine that supports

speech in all other Universal Windows Apps.

• even without specifically adding voice support to your

app, your users can activate your holograms simply by

saying "select“

• “Hey Cortana” will enable you to do a multitude of things:

check weather, check battery, take a picture, etc.

• Manipulation commands:

• Face me

• Bigger | Enhance

• SmallerVoice Commands

• holograms appear visually where the user's gaze

is pointing but sound can come from all directions;

above, below, behind, to the side, etc.

• use this feature to draw attention to an object that

might not currently be in the user's view

• user can perceive sounds to be emanating from a

source in the mixed-reality world.

• Walk toward it and away

• Behind the user

• sounds should come from realistic places. Eg: bird

chirp from the sky not the groundSpatial Sound

• detailed representation of real-world surfaces in

the environment

• apps align with user expectations by providing

familiar real-world behaviors and interactions.

• Placement

• Occlusion

• Physics

Spatial Mapping

- xyz coordinate system- relative to the real world: 1 meter in the

real world is 1 meter in your app- used to stabilize your hologram in a

realistic way

World Coordinates

Did you know…- You can port any Windows 10 PC or Mobile app to HoloLens as a 2D app?

- You can see & record what the HoloLens is seeing by using the Windows Device Portal on a PC? Great

way to do demos for a large group.

- You can take a screenshot or a video

- “Cortana, take a picture”

- “Cortana, start recording”

…and immediately share on social

- The sharing experience (multiple people seeing the same hologram) can be used out of the box from the

HoloToolkit

- People feel like holograms are real

- If you place a holographic elephant in the room, people will walk around it, not through it- If you are in a shared experience and others are shooting holographic projectiles at you, you’ll duck

Gaze

Input

Gesture

Input

Voice

Input

Spatial

Anchors

Spatial

Mapping

Spatial

Sound

Cursor

prefabs

Sharing

prefab

Spatial

prefabsUtilities Shaders Tests

Holograms Materials Models Scripts Textures Shaders

App Experience – you build

HoloToolkit – you get open source for this

Windows Holographic – we made this

• Documentation

• Forums

• Academy

• Open Source

dev.windows.com/holographic

Dona Sarkar twitter.com/donasarkarinsider.windows.com