Page 1
1www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.
www.esi-group.com
MUNICH | 10 – 12 OCT 2017
23047 DESIGNING THE COLLABORATIVE VIRTUAL WORKSPACE INFRASTRUCTURE
Jan Wurster, ESI GroupRoom 14a | 12 OCT 2017 | 23047 2:30pm to 3:00pm
Page 2
2www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of Getty Images
Page 3
3www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of BMW Group
Page 4
4www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACES
Page 5
5www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
ENGINEERING WORKSPACEScourtesy of Valmet Automotive
Page 6
6www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.Copyright © ESI Group, 2017. All rights reserved.
“Augmented And Virtual Reality Fuel The Future Workplace“
Forbes Mag, Nov 2016
Page 7
7www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Once upon a time: Physical Prototypes ascentral element of a shared workspace
▪ Interacting with peers
▪ Continuous validation andpractical engineering
Established VR: Virtual Prototypes in Virtual Environments
▪ All but removes the need for physical prototypes, saving time, effort and resources
▪ Great for Collaboration!
Cost, complexity, maintenance and availability in digital prototypes sometimes lead to less validation andcollaboration.
Page 8
8www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Workplace VR!▪ Finally: Affordable immersive hardware!
However .. ▪ Collaboration?
▪ Efficient use of expensive hardware resources?
▪ Complexity of setup and maintenance?
▪ Software affordability?
Page 9
9www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Page 10
10www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
Workplace VR!▪ Finally: Affordable immersive hardware!
However .. ▪ Collaboration?
▪ Efficient use of expensive (and noisy) resources?
▪ Complexity of setup and maintenance?
▪ Software affordabiblity?
▪ Augmentable and integration into decisionworkflows?
Page 11
11www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Evolution of Engineering Workspaces
• Offer the benefits of VR in engineering to non-experts
• Adopt to and support changes in infrastructure
• Integrate and support processes in the 4.0 enterprise
• Lower the technical obstacleswithout compromising quality ofresults
• Use the opportunity of mobile VR ubiquity - match today‘s flexible workstyles
SOFTWARE AFFORDABILITY
HARDWARE ACCESSIBILITY
INTUITIVE USABILITY
Virtualization& Cloud
Technology
Immersive
Virtual Reality Experience
Page 12
12www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Collaborative Virtual Workspace Challenges
SOCIALCOLLABORATION
HETEROGENOUSDEVICE LANDSCAPE
INFRASTRUCTURE INTEGRATION
COLLABORATIVEINTERACTION
Page 13
13www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Collaborative Virtual Workspace Challenges
SOCIALCOLLABORATION
HETEROGENOUSDEVICE LANDSCAPE
INFRASTRUCTURE INTEGRATION
COLLABORATIVEINTERACTIONRealtime Stereoscopic
Panoramic Visualization
Virtualization
On Premise Cloud Computing
Supporting Presence
Page 14
14www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Collaboration and SharingCOLLABORATIONPRESENCE
Page 15
15www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Collaboration and SharingCOLLABORATIONPRESENCE DEVICES
1:NExplain and
share
PRESENT
N:1Understand and
discuss
LEARN & DISCUSS
N:NCollaboratively Investigate,
Evaluate and Decide
INTERACT & COLLABORATE
Page 16
16www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
SpatialStream
Collaboration and Sharing
Communication and Presence
▪ Voice is a keyrequirement in training, collaboration orpresentation scenarios.
▪ Avatars and presence area necessary ‚storytelling‘ device.
▪ Multimodality throughSpatial Audio feedbackhelps locate thepresenter and points ofinterest
COLLABORATIONPRESENCE DEVICES
Client
Client Client
Presenter
Visualization Session Instance
ODS Stream Stream Casting
RTCMultiConnection
Interactions
P2P Voice / Audio
Interactions
Page 17
17www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Collaboration and Sharing
Consistency in Presentation▪ Each environment and device has its own unique capabilities
▪ Decisions taken in collaborative discussion should not be influenced by presentation
▪ Definitely not a ‚lowest common denominator‘ approach - but visual consistency is keyfor presenting decision-relevant content
Diversity in Expertise▪ Individual proficiency should not keep users from developing the necessary insight to
decide and discuss.
Heterogeneous Device Landscape▪ Developing market with new devices to be added as they become available
COLLABORATIONPRESENCE DEVICES
Page 18
18www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Decoupled Visualization to a Heterogeneous Device Landscape
Personal VR HMDs
Projection-based VRCAVE, Powerwall
Ce
ntr
aliz
ed V
R
Imm
ers
ive
VR
Mobile VRCardboard, GearVR, Daydream
mo
bile
VR
Page 19
19www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Decoupled Visualization to a Heterogeneous Device Landscape
Personal VR HMDs
Projection-based VRCAVE, Powerwall
Ce
ntr
aliz
ed V
R
Imm
ers
ive
VR
Mobile VRCardboard, GearVR, Daydream
mo
bile
VR
Page 20
20www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Displaying Streamed Immersive Content
Decoupled Remote Visualization
STR
EAMNative Thin Client for projection-
based / immersive VR▪ Custom protocol for device data▪ ODS Streaming for
▪ presentation-only scenarios▪ low bandwidth / high latency networks
▪ 1:1 Output Streaming pipeline for▪ Interactive scenarios▪ High bandwidth / low latency networks
▪ GPU decoding for lowest possiblelatency
“Delivering Immersive Experiences through GPU Virtualization & Streaming”
Jan Wurster, Solution & Technology Expert, ESI Group
— Source: GTC 2017, s7203
Page 21
21www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Decoupled Visualization to a Heterogeneous Device Landscape
Personal VR HMDs
Projection-based VRCAVE, Powerwall
Ce
ntr
aliz
ed V
R
Imm
ers
ive
VR
Mobile VRCardboard, GearVR, Daydream
mo
bile
VR
“Immersive Cloud Experiences with NVIDIA GRID & ESI Virtual Reality Technology”
Andreas Mank, Team Leader Visualization, ESI Group
Jan Wurster, Solution & Technology Expert, ESI Group
— Source: GTCE 2016, s6147
Page 22
22www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Stereoscopic Panoramics
Decoupled Remote Visualization
LEFT
RIGHT
Top / Bottom Format at >= 2048x2048 resolution
Omnidirectional Stereo▪ Low visualizuation latency
▪ Feasible bandwidth for streaming
▪ Interactive / dynamic content
▪ Straightforward through Raytracing
▪ Reasonable Quality Approximation through Realtime Rasterization
Page 23
23www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Omni-Directional Stereo Image Synthesis
Stereoscopic Panoramics
STEREO CAMERA SETUP ODS APPROXIMATED ODS APPROXIMATED
ReferencesP.D. Bourke. “Synthetic Stereoscopic Panoramic Images”Google Developers – „Rendering Omni‐directional Stereo Content“
Page 24
24www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Rasterizing Realtime Stereoscopic Panoramics
Decoupled Remote Visualization
LEFT
RIGHT
Top / Bottom Format at >= 2048x2048 resolution
CUDA
OPENGLGRID vGPUTesla P40
Quadro>= Maxwell
v2
Helios
Optix
VIS
UA
LIZA
TIO
N S
ERV
ICE
Page 25
25www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
LEFT
RIGHT
Top / Bottom Format at >= 2048x2048 resolution
Realtime Stereoscopic Panoramics
Decoupled Remote Visualization
CUDA
OPENGLGRID vGPUTesla P40
Quadro>= Maxwell
v2
Helios
Optix
VIS
UA
LIZA
TIO
N S
ERV
ICE
References(*) NVIDIA Gameworks Examples – Single Pass Cubemapshttp://nvidiagameworks.github.io/GraphicsSamples/CubemapRenderingSample.htm
Helios ODS Pipeline – OpenGL
Single Pass Render to Cubemaps
NVIDIA Simultaneous Multiprojection (*)
Scene Data OmniDirectional
Stereo ReprojectionSample cubemap to Left / Right Equirect
Stack into output buffer for streaming
Page 26
26www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
LEFT
RIGHT
Realtime Stereoscopic Panoramics
Decoupled Remote Visualization
CUDA
OPENGLGRID vGPUTesla P40
Quadro>= Maxwell
v2
Helios
Optix
VIS
UA
LIZA
TIO
N S
ERV
ICE
Helios ODS Pipeline – OptiX
Scene Data OmniDirectional
Stereo RT Program
Stack into output buffer for streaming
OptiXBackend
Page 27
27www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Low Latency Streaming
Decoupled Remote Visualization
WebRTC▪ Reasonable level of support
▪ Stable implementation
▪ Designed to handle the challenges of streaminginteractive content▪ Copes with varying input FPS
▪ Does not require the application to always maintain a fix timebase
▪ Does not require added media containers such as mp4
▪ Standard HTM5 <video> element supported, gpu decodingavailable on most devices
▪ Supports point to point a well as casting to multiple end points for collaborative sharing!
Page 28
28www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Low Latency Streaming
Decoupled Remote Visualization
LEFT
RIGHT
CUDA
OPENGLNVIDIA Capture
STR
EAM
ING
SER
VIC
E
webRTC NVENC
VideoCapturer
Streaming
“3D Cloud Streaming for Mobile and Web Applications”
Tyler Gibson, Senior Software Engineer, Microsoft
— Source: GTC 2017, s7859
https://github.com/CatalystCode/3dtoolkit
Page 29
29www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Low Latency Streaming
Decoupled Remote Visualization
LEFT
RIGHT
OPENGLNVIDIA Capture
STR
EAM
ING
SER
VIC
E
Streaming
CUDAwebRTC NVENC
VideoCapturer
Streaming Pipeline
NVENCGPU
EncodingHTML5
Web Client
Custom webRTC
VideoCapturer
NvIFROpenGLBuffer
Transfer
OPENGLFBO
Page 30
30www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Displaying Streamed Immersive Content
Decoupled Remote Visualization
Tracking & Projection
Warping & Lens Correction
Decoding
Immersive UI Overlay
WEB
CLI
ENT
STR
EAMWebVR Client for mobile devices
▪ Device APIs for efficient GPU-enabledDecoding
▪ Equirectangular images texturemappedto left / right eye spheres
▪ Client-side User Interface Overlay▪ Device APIs for Warping and Lens
Correction
Compatible to GearVR, Cardboard, Rift, Vive, Daydream
Available at least in Firefox, ChromePreview available for Microsoft Edge
Page 31
31www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Immersive Datacenter InfrastructureINFRASTRUCTUREDEVICES
Page 32
32www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Immersive Datacenter Infrastructure
Scaling▪ Easily add resources to support growing
environments
▪ Integrate remote sites, homeoffice locations, mobile service personnel
Heterogeneous Use Case Scenarios▪ Complete and amend the workspace continuously
through new ‚job to be done‘ scenarios
Heterogeneous Device Landscape▪ Developing market with new devices to be added
as they become available
INFRASTRUCTUREDEVICES
SOFTWARE AFFORDABILITY
HARDWARE ACCESSIBILITY
INTUITIVE USABILITY
Virtualization& Cloud
Technology
Immersive
Virtual Reality Experience
Page 33
33www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
On Premise / Enterprise Cloud Stack
NVIDIA GRID | HypervisorBare Metal
Immersive Datacenter Infrastructure
vGPU vGPUvGPUvGPU
Tesla GPUGPUEnterprise Storage
Ceph, Swift, ..
IC.IDOInstance(Master)
IC.IDOInstance(Slave)
IC.IDOInstance(Slave)
Visualization Cluster
vrifyCloudInstance
IC.IDOInstance(Master)
Enterprise Network
Thin / Mobile Client ImmersiveSetups
Application & Visualization
VirtualizationDatacenter
INFRASTRUCTURE
Immersive Projection Setups
IDO.Cooperate Cluster
Page 34
34www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Immersive Cloud Experiences & mobile VR
IndustrialDesigners
Design Engineers
Manufacturing& Integration
Engineers
Maintenance & repair
operators
Makers & Enthusiasts
Engineering Schools
Educators
Page 35
35www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Immersive Cloud Experiences & mobile VR
https://youtu.be/-FdNpHjtWDc https://youtu.be/iotacBK_8d0
Have Daydream or Cardboard? Experience the 360 Stereo Recording of a live session!
See the demo as recorded live on-site during GTC 2017!
Page 36
36www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
Acknowledgements and Team
‣ NVIDIA GRID• http://www.nvidia.com/object/nvidia-
grid.html
‣ NVIDIA Capture• https://developer.nvidia.com/capture-sdk
‣ Don Breda• NVIDIA – ESI Alliance Manager
‣ NVIDIA DevTech Team
‣ Alexander Langs• Single Pass Cubemap Rendering
• Stereoscopic Panoramics
‣ Thomas Gierlinger• Stereoscopic Panoramics
• Streaming
‣ Markus Hocke• Streaming - webRTC implementation
Page 37
37www.esi-group.com
Copyright © ESI Group, 2017. All rights reserved.
THANK YOU
[email protected]
May 8-11, 2017 | Silicon Valley
http://icido.com http://vrify.esi-group.com