Rising trends in 3D User Interfaces trends in 3D User Interfaces Introduction to Qt 3D & Qt Canvas3D Nils Christian Roscher-Nielsen Product Manager, The Qt Company Content • Trends

Rising trends in 3D User InterfacesIntroduction to Qt 3D & Qt Canvas3D

Nils Christian Roscher-NielsenProduct Manager, The Qt Company

Content

• Trends in 3D

• Qt 3D Value Proposition

• Highlights of Qt 3D features

• The road ahead

© 20152

Trends in 3D

3

© 20154

Industries embracing 3D user interfaces

Industrial Automation

Geological Modeling

Car IVI Systems Medical / Learning

3D Modeling / Animations Product Visualization

Virtual Reality

GOOGLE CARDBOARD

STEAM VR / HTC VIVE SONY MORPHEUSOCULUS RIFT

SAMSUNG GEAR VR

© 20155

Augmented / Mixed Reality

© Microsoft © Magic Leap

© 20156

360º Imaging & Video

Google CARDBOARD & JUMPwith GoPro Odyssey

RICOHTHETA

NOKIA OZO

© 20157

3D scanning

© 20158

9

• Games• Virtual Tours - Google Street View • 360 videos - YouTube and Facebook• 3D Design and engineering – SpaceX, Autodesk• Real Estate - Matterport Inc• Teaching - Case Western Reserve University • Automotive – Mini Goggles

Virtual & Augmented Reality Examples of Use Cases

© Mini

© Matterport Inc

© Microsoft

© 2015

10

• Head movement• Hand gestures• Torso tracking• Controllers

• Mouse• Game controllers (with

movement tracking)

• Speech

Interaction with VR & AR

© Oculus Rift

© 2015

3D Printing

© Microsoft © Formlabs

© 201511

© 201513

Pioneer in Modern User Experiences

• For the past 20 years, Qt has delivered all the tools and libraries for creating the best possible UX for your end users• Classic desktop look-and-feel• Fluent, modern touch-based UX

• Today, the future of UIs is going towards 3D• Qt is pioneering there with you!

• It has been possible to integrate 3D content with Qt UIs easily• Qt is now providing means to create 3D content easily

• Qt Canvas3D• Qt 3D

New 3D features in Qt

Qt Canvas3D module• About WebGL• About Canvas3D• three.js

Qt3D Module• Key Features• Building Blocks• Future of Qt 3D

© 201514

Building Blocks for Qt and 3D

Summary

2015

Building Blocks for Qt and 3D

Qt Scenegraph / OpenGL

Tools&

Integration to 3rd Party

Tools

Data Visualization

Data VisualizationCanvas3DCanvas3DQt 3DQt 3D

Integration to 3rd Party 3D

Engines

Integration to 3rd Party 3D

Engines

ServicesServices

Focus of this presentation

© 201516

New 3D features in Qt 5.5

• Focus on making integration of 3D content and creation of 3D user interfaces easier than ever

• Qt Canvas3D 1.0• WebGL based 3D canvas for Qt Quick• 3D content creation with JavaScript• Integration to popular three.js 3D library

• Qt 3D 2.0 Technology Preview• High level 3D API built on top Qt & OpenGL• C++ and Qt Quick APIs• Use any rendering technique available

Canvas3DNew Qt Quick Element in Qt 5.5 for 3D Drawing

© 201518

About WebGL – What Is It?

WebGLA low level, state based, 3D vector graphics rendering API for HTML JavaScriptOften described as “OpenGL ES 2 for the web”

Khronos Group• Non-profit technology consortium that manages WebGL API (plus OpenGL/ES etc.)• WebGL 1.0 is based on OpenGL ES 2.0• Initial WebGL Standard Release in 2011• Stable Release 1.0.2 in 2013• https://www.khronos.org/registry/webgl/specs/latest/1.0/

About WebGL – Browser Support

DesktopGoogle Chrome 9Mozilla Firefox 4.0Safari 6.0Opera 11Internet Explorer 11

MobileSafari from iOS 8 Android BrowserGoogle Chrome 25Internet Explorer on WP 8.1Firefox for MobileFirefox OSTizenUbuntu Touch…

© 201519

Widely supported in modern web browsers:

Motivation for Qt Canvas 3D

WebGL is a nice and productive environment for implementing 3D contentQtQuick is a nice and productive environment for doing 2/2.5D UI→ Combining these two makes for a very productive environment

There is a lot of innovation around WebGL, allowing porting of that content to QtQuick on top of Canvas3D makes a lot of sense.

Qt Canvas3D allows developers to make 3D QtQuick applications that re-use existing 3D assets.

There are a lot of resources on the web to get started with WebGL, these resources translate to Qt Canvas3D as well.

All developers who know WebGL can easily use Qt Canvas3D efficiently

© 201520

21

• Qt Canvas 3D• WebGL based 3D canvas for Qt Quick• 3D content creation with JavaScript• Requires OpenGL (ES) 2.0 or higher

• Integration to three.js 3D library• Increases the abstraction level• Utilizes the innovation around WebGL

• Qt 5.6 new features• Qt Quick items as textures• Rendering directly to Qt Quick scene background or foreground

Qt Canvas

What is Canvas3D?

• Qt 5.5 module• 3D canvas component• Used in a QtQuick scene. • Requires OpenGL or OpenGL ES.

Canvas3D provides• WebGL like (non-conformant)

context API• All the functions from WebGL 1.0.3

© 201522

© 201523

About the Implementation

Implementation is based on analysis of the behavior of typical WebGL apps. There are two stages to all the WebGL apps:

1. Initialization (initializeGL):• Start loading resources (textures, models etc.)• Note: Loading is asynchronous, so rendering WILL start before all resources are loaded.

Must be handled gracefully in the rendering code.2. Drawing (paintGL):

• Handle inputs, state changes, transitions and animations.• Render each frame of 3D content animation.

You get two signals to QtQuick that you handle in your JavaScript code initializeGL and paintGL

Incidentally, this is a familiar pattern from QOpenGLWidget :initializeGL(), paintGL(), resizeGL()

© 201524

About the Implementation

In HTML WebGL canvas element has separated “logical layout size” and “physical pixel size”.This is also reflected in the design and behavior of the Canvas3D.

onResizeGL signal notifies whenever the canvas width, height or devicePixelRatio changes, this maps to onWindowResize handlers in most HTML/WebGL content.

WebGL/HTML Canvas3D/QtQuick

Layout Size canvas.style.widthcanvas.style.height

canvas.widthcanvas.height

Drawing Size canvas.widthcanvas.height

canvas.pixelSize.widthcanvas.pixelSize.height

three.js

© 201526

Motivation

Writing low level OpenGL code is not very productive.

A 3D scene graph library helps with productivity by offering higher level abstraction:• Load a 3D model from a file and place it in the scene.• Place the lights in the scene.• Place the camera in the scene.• DONE! Start rendering the content..

What is three.js?

three.js Upstream:• Three.js is one of the most active open source WebGL

based scene graph libraries.• Implemented 100% with JavaScript on top of WebGL• A lot of WebGL content on the web uses three.js.• See: http://threejs.org

three.js Canvas3D Port:• Enables easy porting of three.js content

from the Web to QtQuick• Port 3D content, Implement 2D UI with QtQuick,

Enhance with QtQuick transitions and animations.

© 201527

© 201528

What’s Next?

Canvas3D 1.0 in Qt 5.5• Uses the new TypedArrays from QtQuick JavaScript Engine (V4VM)

Will compile and run only on Qt 5.5Uses less memory and performs even better than the technology preview versions

• Porting some of the Khronos WebGL conformance tests and fixing bugs found in those tests.• Implementing three.js examples for release with Qt 5.5.• Bugfixing…

Qt 5.6• Aiming to bundle three.js as part of Qt as atested release.• Looking at 3D tooling to make the developer experience smoother• Prototyping move of the rendering to the QtQuick scene graph rendering thread, this would allow us to

integrate Qt Multimedia and rendering of QtQuick elements as textures. It is still early days, but looking quite promising..

Qt 5.7 and beyond• We are working with the Automotive industry—and all of our customers—to create a great roadmap.

Qt 3D 2.0Technology Preview in Qt 5.5

© 201530

Motivation for Qt 3D

• Modern GPU’s are very powerful and offer fantastic capabilities • Beyond what could be done with Qt3D 1.x.

• Modern CPU’s on all platforms are now multicore designs• Qt3D 1.x engine could not leverage this.

• Leverage modern rendering algorithms like deferred rendering• Not supported in Qt3D 1.x.

Qt3D 2.0 is a completely re-engineered and re-written Qt3D module. The goal is to make Qt 3D more relevant in the modern world.

© 201531

Qt 3D Key Features

• Extensible and flexible architecture• Fully configurable renderer that allows to quickly implement any

rendering pipeline that you need• 3D content adaptation can be done during runtime or based on

environment• Scalability from OpenGL 2.0 to OpenGL 4.3• C++ and QML language API’s• Rendering tasks can be distributed across several GPU cores by a

scheduler for improved performance

© 201532

Qt 3D 2.0 Building Blocks

• Based on a framework that maintains the object graph using queue of jobs that can be spread to multiple execution threads.

• The object graph can be parsed to build the rendering graph (for visual elements) or e.g. to build a audio graph (for audible elements).

• This flexible framework allows plugging in new “aspects” to the engine like audio or physics.

http://www.kdab.com/overview-qt3d-2-0-part-1

© 201533

Qt 3D 2.0 Building Blocks

• Qt 3D 2.0 is fully data driven• You can define how the rendering of the scene is done using render graphs.

• Any new 3D rendering algorithm that appears in the latest scientific papers can be implemented with Qt 3D 2.0.

• Qt 3D enables easy use of advanced OpenGL programming • Instancing.

What’s Next?

• Qt 3D 2.0 Technology Preview has been released• API can change based on user feedback• Current platform support is limited (mainly tested on desktop only)• Parts of documentation are still missing• Second technology preview available with Qt 5.6

• Target for Qt 3D 2.0 final is Qt 5.7• Qt 5.6 will already bring many improvements. • API will be finalized• Input methods (touch, mouse and keyboard) improved• Integration with Qt Quick 2 content improved• Picking—select specific 3D objects from a scene• Documentation and examples improved

Summary

36

The near future will bring big changes to the industries• How we interact with technology changes• User experiences are becoming more immersive• 3D content creation and consumption will change radically

Qt is a technology that actively takes part in the development and enable our users to create the technology of tomorrow

Summary

© 201537

Summary

• Many leading 3D design tools and 3D solutions have been made with Qt

• Qt has the building blocks that make integrating 3D / OpenGL content easy

• Qt 5.5 introduced 2 new 3D technologies.• Qt Canvas3D & Qt 3D

• High level API for defining e.g. 3D objects, lights and cameras• Extensible architecture• Wide support for 3D formats (through assimp and three.js libraries)• Integration of 3D content and creation of 3D user interfaces is super

© 201538

Comparison between Canvas3D and Qt3D

• Canvas3D• QML/JavaScript, low level API• Provides an easy and widely used solution when combined with three.js 3D

library• Possibility to utilize the application code also in Hybrid and Web applications

• Qt3D• Both QML and C++ APIs• Extensible architecture multithreaded architecture• Qt3D 2.0 technology preview is available in 5.5 and 5.6.

[email protected]

Than you for listening.Happy 3D coding!