WT-4151, Efficient Delivery of 3D Web Contents with Khronos and MPEG Technologies, by Khaled Mammou

EFFICIENT 3D WEB CONTENT DELIVERY WITH KHRONOS AND MPEG TECHNOLOGIES

| EFFICIENT 3D WEB CONTENT DELIVERY | NOVEMBER 19, 2013 2

AGENDA

Introduction

Graphics Library Transmission Format

Conclusions

REST 3D

Open 3D Graphics Compression MPEG

Introduction


INTRODUCTION – CONTENT IS KING

rest3d

[email protected]


CONTENT CREATION

3Dapp 3D model

Images

Animations

Physics

Audio

Script

…


WEB CONTENT PUBLICATION

3Dapp 3D model

Images

Animations

Physics

Audio

Script

…

Web app

html

js

images

css

3D assets

WebGL

json bin (dds, typedarray)


Web Browser

http

CDN

Cache

Authentication

WEB CONTENT PUBLICATION

3Dapp 3D model

Images

Animations

Physics

Audio

Script

…

Web app

html

js

images

css

3D assets


Rest3d … to the rescue

http://en.wikipedia.org/wiki/Representational_state_transfer

The REST architectural style was developed by W3C Technical Architecture Group (TAG) in parallel with HTTP 1.1, based on the existing design of HTTP 1.0. The World Wide Web represents the largest implementation of a system conforming to the REST architectural style

www.rest3d.com

https://groups.google.com/forum/#!forum/3d-rest

discuss the principles of a shared RESTful http API dedicated to 3D media

https://github.com/amd/rest3d/

AMD open-source experimental implementation

Client:

- Workstation style UI in the web browser

- COLLADA and glTF javascript parser

- WebGL renderer

Server:

- node.js http server, server cache, database proxy

- XML native database, Xquery processing

http://www.rest3d.com




https://groups.google.com/forum/



https://github.com/amd/rest3d/


CLOUD (rest3d) PIPELINE M

od

eler

Max

Maya

Blender

Dat

aban

k COLLADA

fbx

Edit

or Desktop

Or web app

Ass

ets Database

Queries

Upload/download

Pro

cess

ing Conversion (glTF)

Compression

Image conversion

Web

Ap

p

Turbulenz

Three.js

Cesium

…WebGL…

Desktop

Browser

Cloud

rest3d

rest3d rest3d

Demo

[email protected]


graphics library Transmission Format

COLLADA WG initiative (Khronos)

‒ work-in-progress, not yet a ratified specification

‒ specification and implementation (converter) work at the same time

‒ runtime asset format for GL based APIs

Principles

‒ Streamline rendering: size, speed and ease of implementation

‒ Offload processing from runtime

‒ Built on top of GL Based APIs and Typed Arrays


glTF & Open3DGC Integration

‒ First cut, Open3DGC glTF extension is a work-in-progress

‒ geometry: POSITION, NORMAL, multiple sets of TEXCOORD, COLOR and generic attributes

‒ animations: TRS (Translation, Rotation, Scale)

‒ skinning: weights and bone indices

Demo example: Virtual City (www.3drt.com)

geometry animation total

COLLADA raw: 5469 kb

gzip: 1612 kb

glTF 680 kb

1297 kb raw: 1977 kb

gzip: 878 kb

glTF Open3DGC-

binary

198 kb 297 kb raw: 496 kb

gzip: 429 kb

glTF Open3DGC- ascii 355 kb 533 kb raw: 888 kb

gzip: 390 kb

graphics library Transmission Format

http://www.3drt.com/

Demo

Open3DGC

[email protected]

MPEG


OPEN-3DGC: WHAT IS IT?

Efficient Implementation of MPEG 3D Graphics Codecs

‒ Encoders in C++ for fast server side compression

‒ Decoders in javascript and C++ to target mobile and web applications

‒ Available under MIT license https://github.com/amd/rest3d/tree/master/server/o3dgc

Codecs

‒ 3D meshes with multiple attributes

‒ Geometry (e.g., positions, normals, texture coordinates…)

‒ Skinning (e.g., animation weights and joint IDs)

‒ Animations

‒ Bone-based animations

‒ Morphing (coming soon)

MPEG

https://github.com/amd/rest3d/tree/master/server/o3dgc


OPEN-3DGC: HOW EFFICIENT IS IT?

Compression Efficiency

‒ Gzip (default level=6)

‒ OpenCTM (default settings)

‒ Open3DGC and Webgl-loader

‒ Positions on 14 bits

‒ Normals and texCoords on 10 bits

Open3DGC is 5x-9x more efficient than Gzip, 1.3x-2.4x more efficient than OpenCTM and 1.2x-1.5x more efficient than webgl-loader

0

100

200

300

400

CAD (3748 models)

3D Scanned (78 models)

MPEG dataset (1211 models)

Size

(M

Byt

es)

Gzip

OpenCTM

Webgl-loader + Gzip

Open3DGC-ASCII + Gzip

Open3DGC-Binary

http://www.gzip.org/

http://openctm.sourceforge.net/



https://github.com/amd/rest3d/tree/master/server/o3dgc

https://code.google.com/p/webgl-loader/




OPEN 3DGC: HOW FAST IS IT?

Javascript Decoding Speed

‒ Desktop machine

‒ Windows® 64-bit

‒ AMD Phenom™ II X4 B95 CPU @ 3.0GHz

‒ 8GB RAM

‒ Chrome

‒ Smart phone

‒ Samsung Galaxy S4

‒ Android 4.2.2

‒ Chrome

Number of triangles

Desktop decoding time (ms)

Smart phone decoding time (ms)

“Hand” 100K 130 1045

“Dilo” 54K 85 768

“Octopus” 34K 65 457

Decoding speed will become even more critical with dense 3D meshes generated by 3D digitization technologies (e.g., 3D scanners)


1, 3, 7 1, 7, 6 1, 6, 2 1, 2, 4 1, 4, 5

Connectivity 3D Mesh Compression

‒ Indexed Face Set

‒ Geometry: positions

‒ Connectivity: list of triangles

‒ Redundancy

‒ Indexes repeated multiple times

‒ No need to preserve triangles and vertices order

‒ No need for 32-bit precision for positions/attributes

‒ Neighbour vertices exhibit high geometry correlations

Requires 192 bits per vertex!

1 2 3 4 5 6 7

X1, Y1, Z1 X2, Y2, Z2 X3, Y3, Z3 X4, Y4, Z4 X5, Y5, Z5 X6, Y6, Z6 X7, Y7, Z7

Geometry

X3, Y3, Z3 X7, Y7, Z7 X6, Y6, Z6 X2, Y2, Z2 X4, Y4, Z4 X5, Y5, Z5 X1, Y1, Z1

7, 5, 6 7, 4, 5 7, 3, 4 7, 1, 2 7, 2, 3

Reordered geometry

Reordered Connectivity

Lossy geometry compression and lossless connectivity encoding reduce the stream size to 10-20 bpv

V2

V6

V7

V3

V1

V4

V5

OPEN-3DGC: WHAT’S UNDER THE HOOD?


Connectivity

Encoder

Arithmetic or ASCII

Encoding

*Mammou’09+ K. Mammou, T. Zaharia, F. Prêteux, “TFAN: A low complexity 3D mesh compression algorithm” Computer Animation and Virtual Worlds, Vol. 20(2-3), pp. 343–354, 2009

TFAN-based Connectivity Encoding

Prediction Uniform

Quantization Geometry

Compressed

stream

3D Mesh Compression

‒ Algorithm based on TFAN codec *Mammou’09+

‒ Triangular meshes with attributes

‒ Arbitrary topologies (e.g., manifold or not, open/closed, oriented or not, arbitrary genus, holes…)

‒ MPEG-SC3DMC (Scalable Complexity 3D Mesh Coding) published in 2010



Connectivity

Encoder

Arithmetic or ASCII

Encoding



Prediction Uniform


Compressed

stream

3D Mesh Compression



‒ Arbitrary topologies (e.g., manifold or not, open, closed, holes…)





3D Mesh Compression ‒ Uniform quantization

‒ Map real numbers to integers

‒ Maximum quantization error

Uniform quantization reduces the number of bits per vertex for positions

from 96 bpv to 24 bpv

q=8 q=6 q=4 q=10 Original


Connectivity

Encoder

Arithmetic or ASCII

Encoding



Prediction Uniform


Compressed

stream

3D Mesh Compression







3D Mesh Compression Algorithm

‒ Geometry prediction

‒ Exploit connectivity information

‒ Differential and “parallelogram” prediction

Processed region

B

P Unprocessed region

P’

B

A

B’

C




‒ Geometry prediction

‒ Exploit connectivity information

‒ Differential and “parallelogram” prediction

‒ Adaptively choose the best predictor

Processed region

B

P Unprocessed region

B

Geometry prediction reduces the number of bits per vertex for positions

from 24 bpv to 10.3 bpv

P’

A

B’

C



Connectivity

Encoder

Arithmetic or ASCII

Encoding



Prediction Uniform


Compressed

stream

3D Mesh Compression








‒ Triangle FAN

‒ Each two successive triangles share a common edge

‒ All triangles share a common vertex (i.e., center of the TFAN)

‒ All the triangles have the same orientation

‒ Described by enumerating the vertices in their traversal order

7 indices instead of 15 for IFS

(1,3,7,6,2,4,5)

Implicitly encodes adjacency information




‒ TFAN-based connectivity compression

‒ Decompose the mesh into a set of TFANs ‒ Traverse the vertices from neighbour to neighbour

‒ Rename vertices according to the traversal order

‒ For each vertex, group its incident non-visited triangles into TFANs

V’7

V’8

V’9

V’10

V’1

V’2 V’3

V’5

V’6

V’4

V1

V4 V6

V5 V7

V3

V8

V10 V2

V9








‒ Encode TFANs ‒ Distinguish 10 topological configurations

‒ Use local indices instead of absolute ones








‒ Encode TFANs ‒ Distinguish 10 topological configurations

‒ Use local indices instead of absolute ones

‒ Entropy encoding ‒ Exploit statistical redundancy

TFAN-based connectivity compression reduces the number of bits per vertex for

connectivity from 96 bpv to 3.1 bpv

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10 Fr

eq

ue

ncy

TFAN configurations


Conclusions


CONCLUSIONS

Content delivery is becoming crucial for Web and mobile 3D applications

Rest3D paves the way toward a unified standard to negotiate and exchange 3D data

glTF provides a efficient out-of-the-box 3D content delivery solution

Open3DGC leverages MPEG codecs to offer a complete set of 3D graphics compression technologies


DISCLAIMER & ATTRIBUTION

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors.

The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes.

AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Windows is a trademark of Microsoft Corp.

ATTRIBUTION

© 2013 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names are for informational purposes only and may be trademarks of their respective owners.

Backup slides



Encoding Example




‒ Encoding example Vertex re-ordering

Neighbors list

Bitstream

10



10 1,(7,1)

Vertex re-ordering

Neighbors list

V1 → V1 {}

Bitstream

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10


‒ Encoding example



Vertex re-ordering

Neighbors list

V1 → V1 {V4}

V6 → V2

V4 → V3

Bitstream

10 1,(7,1) 2,(4,1)

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V4, V5}

V6 → V2

V4 → V3

V5 → V4

Bitstream

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10

10 1,(7,1) 2,(4,1),(7,2)





Vertex re-ordering

Neighbors list

V1 → V1 {V5, V2, V9, V7}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V9}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1)

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V7,V10}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

V10 → V8

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2)

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V7,V10,V8}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

V10 → V8

V8 → V9

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1)

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V3}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0

V10 → V8

V8 → V9

V3 → V10

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Vertex re-ordering

Neighbors list

V1 → V1 {V8}

V6 → V2

V4 → V3

V5 → V4

V2 → V5

V9 → V6

V7 → V7

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0

V10 → V8

V8 → V9

V3 → V10

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10






Decoding Example



Neighbors list

{}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10


‒ Decoding example



Neighbors list

{V3}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V3, V4}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V4}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V6}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V7, V8}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V7, V8, V9}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V9}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{V10}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10





Neighbors list

{}

Bitstream

10 1,(7,1) 2,(4,1),(7,2) 0 0 1,(4, 1) 2,(6,2),(4,1) 0 0 0 0

C1

C2

C3

C6

C7

C8

C4

C5

C9

C10




WT-4151, Efficient Delivery of 3D Web Contents with Khronos and MPEG Technologies, by Khaled Mammou

Technology