Top Banner
Hao Su (UCSD) Leonidas Guibas (Stanford) Michael Bronstein (Università della Svizzera Italiana) Evangelos Kalogerakis (UMass) Jimei Yang (Adobe Research) Charles Qi (Stanford) Qixing Huang (UT Austin) 3D Deep Learning Tutorial@CVPR2017 July 26, 2017
40

3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Jun 29, 2018

Download

Documents

hoangphuc
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Hao Su (UCSD) Leonidas Guibas (Stanford) Michael Bronstein (Università della Svizzera Italiana)Evangelos Kalogerakis (UMass)Jimei Yang (Adobe Research) Charles Qi (Stanford) Qixing Huang (UT Austin)

3D Deep LearningTutorial@CVPR2017

July 26, 2017

Page 2: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Schedule

• Opening remark 1:30PM-1:40PM • Deep learning on regular data (MVCNN&3DCNN) 1:40PM-2:45PM • Break 2:45PM-3:00PM • Deep learning on point cloud and primitives 3:00PM-4:15PM • Break 4:15PM-4:30PM • Deep learning on meshes (Intrinsic CNN) 4:30PM-5:45PM

2

Page 3: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Outline

Overview of 3D deep learning

3D deep learning algorithms

3

Page 4: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Outline

Overview of 3D deep learning

3D deep learning algorithms

4

Background3D deep learning tasks

Page 5: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The world around us is comprised of 3D geometry

5

Page 6: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Broad applications of 3D data

6

Robotics

Page 7: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Broad applications of 3D data

7

Robotics Augmented Reality

Page 8: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Autonomous driving

Broad applications of 3D data

8

Robotics Augmented Reality

Page 9: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Autonomous driving

Broad applications of 3D data

` 9

Robotics Augmented Reality

Medical Image Processing

Page 10: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Autonomous driving

Broad applications of 3D data

` 10

Robotics Augmented Reality

Medical Image Processing

Historically, most 3D visual computing techniques focus on single models, lacking robustness

Page 11: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Lacking 3D data has been the major bottleneck

11

Stanford bunny

Utah teapot

Princeton shape benchmark

[Shilane et al. 04]

1800 models in 90 categories

Status as of 2010:

Page 12: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Recent rise of Internet 3D models

12Nowadays millions of 3D models in online repositories

Page 13: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

……

Recent rise of Internet 3D models

13

Growing market of crowd-sourcing for 3D modeling

Nowadays millions of 3D models in online repositories

Page 14: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

……

Recent rise of Internet 3D models

14

Growing market of crowd-sourcing for 3D modeling

An opportunity of Data-driven 3D Visual Computing

Nowadays millions of 3D models in online repositories

Page 15: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Title Text 15

> 30,000,000 units

Page 16: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The surge of 3D deep learning

• Arguably started from 2015 along with of big 3D datasets (ShapeNet & ModelNet)

• Very active due to huge industry interests!

16

CVCG

ML• Robotics• Autonomous driving• Virtual/augmented reality• Smart manufacturing• …

Page 17: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning tasks

17

3D geometry analysis

3D synthesis

3D-assisted image analysis

Page 18: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning tasks

18

3D geometry analysis

Classification Parsing(object/scene)

Correspondence

Page 19: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning tasks

19

3D synthesis

Monocular 3D reconstruction Shape completion Shape modeling

Page 20: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning tasks

20

3D-assisted image analysis

Query

Results

Cross-view image retrieval Intrinsic decomposition

Page 21: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

All about Data and Network

21

3D geometry analysis

3D synthesis

3D-assisted image analysis

Page 22: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

All about Data and Network

22

3D geometry analysis

3D synthesis

Page 23: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Outline

Overview of 3D deep learning

3D deep learning algorithms

23

3D Representation issueDeep learning on different 3D representations

Page 24: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

24

Images: Unique representation with regular data structure

Page 25: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

25

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 26: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

26

Novel view image synthesis

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 27: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

27

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 28: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

28

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 29: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

29

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 30: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

30

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 31: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

The representation issue of 3D deep learning

31

Rasterized form (regular grids)

Geometric form(irregular)

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Page 32: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning algorithms (by representations)• Projection-based

32

[Su et al. 2015][Kalogerakis et al. 2016]…

VolumetricMulti-view

[Maturana et al. 2015][Wu et al. 2015] (GAN)[Qi et al. 2016][Liu et al. 2016][Wang et al. 2017] (O-Net)[Tatarchenko et al. 2017] (OGN)…

Page 33: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

3D deep learning algorithms (by representations)• Projection-based

33

[Defferard et al. 2016][Henaff et al. 2015][Yi et al. 2017] (SyncSpecCNN)…

VolumetricMulti-view

[Qi et al. 2017] (PointNet)[Fan et al. 2017] (PointSetGen)

Point cloud Mesh (Graph CNN) Part assembly

[Tulsiani et al. 2017][Li et al. 2017] (GRASS)

[Su et al. 2015][Kalogerakis et al. 2016]…

[Maturana et al. 2015][Wu et al. 2015] (GAN)[Qi et al. 2016][Liu et al. 2016][Wang et al. 2017] (O-Net)[Tatarchenko et al. 2017] (OGN)…

Page 34: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Cartesian product space of “task” and “representation”

34

3D geometry analysis

3D synthesis

Page 35: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Fundamental challenges of 3D deep learning

35

Can we directly apply CNN on 3D data?

Page 36: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Fundamental challenges of 3D deep learning

36

Can we directly apply CNN on 3D data?

Page 37: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Fundamental challenges of 3D deep learning

37

3D has many representations:

multi-view RGB(D) imagesvolumetric

• Can directly apply CNN• But has other challenges

Rasterized form (regular grids)

Page 38: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Fundamental challenges of 3D deep learning

38

3D has many representations:

multi-view RGB(D) imagesvolumetricpolygonal meshpoint cloudprimitive-based CAD models

Geometric form(irregular)

Cannot directly apply CNN

Rasterized form (regular grids)

Page 39: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Outline

Overview of 3D deep learning

3D deep learning algorithms• Deep learning on regular structures• Deep learning on meshes• Deep learning on point cloud and parametric models

Title Text 39

Page 40: 3D Deep Learning - Stanford University3ddl.stanford.edu/CVPR17_Tutorial_Overview.pdf · Correspondence. 3D deep learning tasks 19 3D synthesis Monocular ... • Deep learning on point

Schedule

• Opening remark 1:30PM-1:40PM • Deep learning on regular data (MVCNN&3DCNN) 1:40PM-2:45PM • Break 2:45PM-3:00PM • Deep learning on point cloud and primitives 3:00PM-4:15PM • Break 4:15PM-4:30PM • Deep learning on meshes (Intrinsic CNN) 4:30PM-4:15PM

40


Related Documents
3D Deep Learning for Biological Function Prediction …3D Deep Learning for Biological Function Prediction from Physical Fields Vladimir Golkov 1, Marcin J. Skwark2, Atanas Mirchev

3D Deep Learning for Biological Function Prediction …3D...

Category: Documents
Convolutional-Recursive Deep Learning for 3D Object ...web.stanford.edu/class/cs331b/2016/presentations/paper3.pdf · Convolutional-Recursive Deep Learning for 3D Object Classification

Convolutional-Recursive Deep Learning for 3D Object...

Category: Documents
Geometric Deep Learning on 3D Meshes an overview

Geometric Deep Learning on 3D Meshes an overview

Category: Documents
3D ShapeNets: A Deep Representation for Volumetric Shapes · 3D ShapeNets: A Deep Representation for Volumetric Shapes ... To train our 3D deep learning model, ... surface correspondence

3D ShapeNets: A Deep Representation for Volumetric...

Category: Documents
Deep Learning 3D Dose Prediction for Conventional Lung ...

Deep Learning 3D Dose Prediction for Conventional Lung ...

Category: Documents
IntrA: 3D Intracranial Aneurysm Dataset for Deep Learningopenaccess.thecvf.com/content_CVPR_2020/papers/Yang...IntrA: 3D Intracranial Aneurysm Dataset for Deep Learning Xi Yang1 Ding

IntrA: 3D Intracranial Aneurysm Dataset for Deep...

Category: Documents
Deep Learning Methods for 3D Human Pose Estimation under ...

Deep Learning Methods for 3D Human Pose Estimation under ...

Category: Documents
Multiple Imaging Modalities 3D Deep Learning for ...

Multiple Imaging Modalities 3D Deep Learning for ...

Category: Documents
Computer Vision System Design: Deep Learning, 3D Vision ...€¦ · Computer Vision System Design: Deep Learning, 3D Vision, ... Tools to enable art historians to navigate vast databases

Computer Vision System Design: Deep Learning, 3D Vision...

Category: Documents
PointNet: Deep Learning on Point Sets for 3D Classification ... · PDF filePointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Charles R. Qi* Hao Su* Kaichun

PointNet: Deep Learning on Point Sets for 3D Classification...

Category: Documents
EuRAD: Automotive Radar Dataset for Deep Learning Based 3D ...

EuRAD: Automotive Radar Dataset for Deep Learning Based 3D.....

Category: Documents
Review 3D Deep Learning on Medical Images: A Review

Review 3D Deep Learning on Medical Images: A Review

Category: Documents