23047 Designing the Collaborative Virtual Workspace ...on-demand.gputechconf.com/gtc-eu/2017/presentation/... · Consistency in Presentation Each environment and device has its own

1www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.

www.esi-group.com

MUNICH | 10 – 12 OCT 2017

23047 DESIGNING THE COLLABORATIVE VIRTUAL WORKSPACE INFRASTRUCTURE

Jan Wurster, ESI GroupRoom 14a | 12 OCT 2017 | 23047 2:30pm to 3:00pm

2www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.

ENGINEERING WORKSPACEScourtesy of Getty Images

3www.esi-group.com


ENGINEERING WORKSPACEScourtesy of BMW Group

4www.esi-group.com


ENGINEERING WORKSPACES

5www.esi-group.com


ENGINEERING WORKSPACEScourtesy of Valmet Automotive

6www.esi-group.com

Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.

“Augmented And Virtual Reality Fuel The Future Workplace“

Forbes Mag, Nov 2016

7www.esi-group.com


Evolution of Engineering Workspaces

Once upon a time: Physical Prototypes ascentral element of a shared workspace

▪ Interacting with peers

▪ Continuous validation andpractical engineering

Established VR: Virtual Prototypes in Virtual Environments

▪ All but removes the need for physical prototypes, saving time, effort and resources

▪ Great for Collaboration!

Cost, complexity, maintenance and availability in digital prototypes sometimes lead to less validation andcollaboration.

8www.esi-group.com



Workplace VR!▪ Finally: Affordable immersive hardware!

However .. ▪ Collaboration?

▪ Efficient use of expensive hardware resources?

▪ Complexity of setup and maintenance?

▪ Software affordability?

9www.esi-group.com



10www.esi-group.com



Workplace VR!▪ Finally: Affordable immersive hardware!

However .. ▪ Collaboration?

▪ Efficient use of expensive (and noisy) resources?

▪ Complexity of setup and maintenance?

▪ Software affordabiblity?

▪ Augmentable and integration into decisionworkflows?

11www.esi-group.com



• Offer the benefits of VR in engineering to non-experts

• Adopt to and support changes in infrastructure

• Integrate and support processes in the 4.0 enterprise

• Lower the technical obstacleswithout compromising quality ofresults

• Use the opportunity of mobile VR ubiquity - match today‘s flexible workstyles

SOFTWARE AFFORDABILITY

HARDWARE ACCESSIBILITY

INTUITIVE USABILITY

Virtualization& Cloud

Technology

Immersive

Virtual Reality Experience

12www.esi-group.com


Collaborative Virtual Workspace Challenges

SOCIALCOLLABORATION

HETEROGENOUSDEVICE LANDSCAPE

INFRASTRUCTURE INTEGRATION

COLLABORATIVEINTERACTION

13www.esi-group.com


Collaborative Virtual Workspace Challenges

SOCIALCOLLABORATION

HETEROGENOUSDEVICE LANDSCAPE

INFRASTRUCTURE INTEGRATION

COLLABORATIVEINTERACTIONRealtime Stereoscopic

Panoramic Visualization

Virtualization

On Premise Cloud Computing

Supporting Presence

14www.esi-group.com


Collaboration and SharingCOLLABORATIONPRESENCE

15www.esi-group.com


Collaboration and SharingCOLLABORATIONPRESENCE DEVICES

1:NExplain and

share

PRESENT

N:1Understand and

discuss

LEARN & DISCUSS

N:NCollaboratively Investigate,

Evaluate and Decide

INTERACT & COLLABORATE

16www.esi-group.com


SpatialStream

Collaboration and Sharing

Communication and Presence

▪ Voice is a keyrequirement in training, collaboration orpresentation scenarios.

▪ Avatars and presence area necessary ‚storytelling‘ device.

▪ Multimodality throughSpatial Audio feedbackhelps locate thepresenter and points ofinterest

COLLABORATIONPRESENCE DEVICES

Client

Client Client

Presenter

Visualization Session Instance

ODS Stream Stream Casting

RTCMultiConnection

Interactions

P2P Voice / Audio

Interactions

17www.esi-group.com


Collaboration and Sharing

Consistency in Presentation▪ Each environment and device has its own unique capabilities

▪ Decisions taken in collaborative discussion should not be influenced by presentation

▪ Definitely not a ‚lowest common denominator‘ approach - but visual consistency is keyfor presenting decision-relevant content

Diversity in Expertise▪ Individual proficiency should not keep users from developing the necessary insight to

decide and discuss.

Heterogeneous Device Landscape▪ Developing market with new devices to be added as they become available

COLLABORATIONPRESENCE DEVICES

18www.esi-group.com


Decoupled Visualization to a Heterogeneous Device Landscape

Personal VR HMDs

Projection-based VRCAVE, Powerwall

Ce

ntr

aliz

ed V

R

Imm

ers

ive

VR

Mobile VRCardboard, GearVR, Daydream

mo

bile

VR

19www.esi-group.com



Personal VR HMDs


Ce

ntr

aliz

ed V

R

Imm

ers

ive

VR


mo

bile

VR

20www.esi-group.com


Displaying Streamed Immersive Content

Decoupled Remote Visualization

STR

EAMNative Thin Client for projection-

based / immersive VR▪ Custom protocol for device data▪ ODS Streaming for

▪ presentation-only scenarios▪ low bandwidth / high latency networks

▪ 1:1 Output Streaming pipeline for▪ Interactive scenarios▪ High bandwidth / low latency networks

▪ GPU decoding for lowest possiblelatency

“Delivering Immersive Experiences through GPU Virtualization & Streaming”

Jan Wurster, Solution & Technology Expert, ESI Group

— Source: GTC 2017, s7203

21www.esi-group.com



Personal VR HMDs


Ce

ntr

aliz

ed V

R

Imm

ers

ive

VR


mo

bile

VR

“Immersive Cloud Experiences with NVIDIA GRID & ESI Virtual Reality Technology”

Andreas Mank, Team Leader Visualization, ESI Group

Jan Wurster, Solution & Technology Expert, ESI Group

— Source: GTCE 2016, s6147

22www.esi-group.com


Stereoscopic Panoramics


LEFT

RIGHT

Top / Bottom Format at >= 2048x2048 resolution

Omnidirectional Stereo▪ Low visualizuation latency

▪ Feasible bandwidth for streaming

▪ Interactive / dynamic content

▪ Straightforward through Raytracing

▪ Reasonable Quality Approximation through Realtime Rasterization

23www.esi-group.com


Omni-Directional Stereo Image Synthesis

Stereoscopic Panoramics

STEREO CAMERA SETUP ODS APPROXIMATED ODS APPROXIMATED

ReferencesP.D. Bourke. “Synthetic Stereoscopic Panoramic Images”Google Developers – „Rendering Omni‐directional Stereo Content“

24www.esi-group.com


Rasterizing Realtime Stereoscopic Panoramics


LEFT

RIGHT


CUDA

OPENGLGRID vGPUTesla P40

Quadro>= Maxwell

v2

Helios

Optix

VIS

UA

LIZA

TIO

N S

ERV

ICE

25www.esi-group.com


LEFT

RIGHT


Realtime Stereoscopic Panoramics


CUDA


Quadro>= Maxwell

v2

Helios

Optix

VIS

UA

LIZA

TIO

N S

ERV

ICE

References(*) NVIDIA Gameworks Examples – Single Pass Cubemapshttp://nvidiagameworks.github.io/GraphicsSamples/CubemapRenderingSample.htm

Helios ODS Pipeline – OpenGL

Single Pass Render to Cubemaps

NVIDIA Simultaneous Multiprojection (*)

Scene Data OmniDirectional

Stereo ReprojectionSample cubemap to Left / Right Equirect

Stack into output buffer for streaming

http://nvidiagameworks.github.io/GraphicsSamples/CubemapRenderingSample.htm

26www.esi-group.com


LEFT

RIGHT

Realtime Stereoscopic Panoramics


CUDA


Quadro>= Maxwell

v2

Helios

Optix

VIS

UA

LIZA

TIO

N S

ERV

ICE

Helios ODS Pipeline – OptiX

Scene Data OmniDirectional

Stereo RT Program

Stack into output buffer for streaming

OptiXBackend

27www.esi-group.com


Low Latency Streaming


WebRTC▪ Reasonable level of support

▪ Stable implementation

▪ Designed to handle the challenges of streaminginteractive content▪ Copes with varying input FPS

▪ Does not require the application to always maintain a fix timebase

▪ Does not require added media containers such as mp4

▪ Standard HTM5 <video> element supported, gpu decodingavailable on most devices

▪ Supports point to point a well as casting to multiple end points for collaborative sharing!

28www.esi-group.com




LEFT

RIGHT

CUDA

OPENGLNVIDIA Capture

STR

EAM

ING

SER

VIC

E

webRTC NVENC

VideoCapturer

Streaming

“3D Cloud Streaming for Mobile and Web Applications”

Tyler Gibson, Senior Software Engineer, Microsoft

— Source: GTC 2017, s7859

https://github.com/CatalystCode/3dtoolkit

https://github.com/CatalystCode/3dtoolkit

29www.esi-group.com




LEFT

RIGHT

OPENGLNVIDIA Capture

STR

EAM

ING

SER

VIC

E

Streaming

CUDAwebRTC NVENC

VideoCapturer

Streaming Pipeline

NVENCGPU

EncodingHTML5

Web Client

Custom webRTC

VideoCapturer

NvIFROpenGLBuffer

Transfer

OPENGLFBO

30www.esi-group.com


Displaying Streamed Immersive Content


Tracking & Projection

Warping & Lens Correction

Decoding

Immersive UI Overlay

WEB

CLI

ENT

STR

EAMWebVR Client for mobile devices

▪ Device APIs for efficient GPU-enabledDecoding

▪ Equirectangular images texturemappedto left / right eye spheres

▪ Client-side User Interface Overlay▪ Device APIs for Warping and Lens

Correction

Compatible to GearVR, Cardboard, Rift, Vive, Daydream

Available at least in Firefox, ChromePreview available for Microsoft Edge

31www.esi-group.com


Immersive Datacenter InfrastructureINFRASTRUCTUREDEVICES

32www.esi-group.com


Immersive Datacenter Infrastructure

Scaling▪ Easily add resources to support growing

environments

▪ Integrate remote sites, homeoffice locations, mobile service personnel

Heterogeneous Use Case Scenarios▪ Complete and amend the workspace continuously

through new ‚job to be done‘ scenarios

Heterogeneous Device Landscape▪ Developing market with new devices to be added

as they become available

INFRASTRUCTUREDEVICES

SOFTWARE AFFORDABILITY

HARDWARE ACCESSIBILITY

INTUITIVE USABILITY

Virtualization& Cloud

Technology

Immersive

Virtual Reality Experience

33www.esi-group.com


On Premise / Enterprise Cloud Stack

NVIDIA GRID | HypervisorBare Metal

Immersive Datacenter Infrastructure

vGPU vGPUvGPUvGPU

Tesla GPUGPUEnterprise Storage

Ceph, Swift, ..

IC.IDOInstance(Master)

IC.IDOInstance(Slave)

IC.IDOInstance(Slave)

Visualization Cluster

vrifyCloudInstance

IC.IDOInstance(Master)

Enterprise Network

Thin / Mobile Client ImmersiveSetups

Application & Visualization

VirtualizationDatacenter

INFRASTRUCTURE

Immersive Projection Setups

IDO.Cooperate Cluster

34www.esi-group.com


Immersive Cloud Experiences & mobile VR

IndustrialDesigners

Design Engineers

Manufacturing& Integration

Engineers

Maintenance & repair

operators

Makers & Enthusiasts

Engineering Schools

Educators

35www.esi-group.com


Acknowledgements and Team

‣ NVIDIA GRID• http://www.nvidia.com/object/nvidia-

grid.html

‣ NVIDIA Capture• https://developer.nvidia.com/capture-sdk

‣ Don Breda• NVIDIA – ESI Alliance Manager

‣ NVIDIA DevTech Team

‣ Alexander Langs• Single Pass Cubemap Rendering

• Stereoscopic Panoramics

‣ Thomas Gierlinger• Stereoscopic Panoramics

• Streaming

‣ Markus Hocke• Streaming - webRTC implementation

36www.esi-group.com


THANK YOU

[email protected]

May 8-11, 2017 | Silicon Valley

http://icido.com http://vrify.esi-group.com

mailto:[email protected]

23047 Designing the Collaborative Virtual Workspace ...on-demand.gputechconf.com/gtc-eu/2017/presentation/... · Consistency in Presentation Each environment and device has its own

Documents