Page 1
OBJECT-AWARE
GUIDANCE FOR
AUTONOMOUS SCENE
RECONSTRUCTIONLigang Liu, Xi Xia, Han Sun, Qi Shen,
Juzhan Xu, Bin Chen, Hui Huang, Kai Xu
© 2018 SIGGRAPH. All Rights Reserved
University of Science and Technology of China
Shenzhen University
National University of Defense Technology
Page 2
© 2018 SIGGRAPH. All Rights Reserved 1
Photography &
Recording Encouraged
Page 3
Background
• Commodity RGB-D sensors
© 2018 SIGGRAPH. All Rights Reserved 2
Microsoft Kinect PrimeSense Intel RealSense
Page 4
Background
• RGB-D sensor allows real-time reconstruction
© 2018 SIGGRAPH. All Rights Reserved 3
KinectFusion[Izadi et al. 2011]
Page 5
Background
• Other real-time reconstruction methods
© 2018 SIGGRAPH. All Rights Reserved 4
Voxel Hashing[Nießner et al. 2013]
ElasticFusion[Whelan et al. 2015]
Page 6
Background
• Indoor scene reconstruction -> 3D object models
© 2018 SIGGRAPH. All Rights Reserved 5
Page 7
Background
• Human scanning is a laborious task [Kim et al. 2013]
© 2018 SIGGRAPH. All Rights Reserved 6
Time
consuming
Inaccurate
scanning
Page 8
Background
• Modern robots are more and more reliable and controllable.
© 2018 SIGGRAPH. All Rights Reserved 7
Unimation, 1958 Fetch, 2015
Page 9
Motivation: Autoscanning with Robots
© 2018 SIGGRAPH. All Rights Reserved 8
AutomaticNever feel
tired
Stable and
precise
Page 10
Existing Works: Single Objects
• High quality scanning and reconstruction of single object [Wu et al. 2014]
© 2018 SIGGRAPH. All Rights Reserved 9
Page 11
Existing Works: Unknown Scenes
• Two pass scene reconstruction and understanding.
• Can only use low-level information in first exploration pass.
© 2018 SIGGRAPH. All Rights Reserved 10
exploration & reconstruction[Xu et al. 2017]
segmentation & recognition[Nan et al. 2012]
First Pass Second Pass
frontier-based exploration[Yamauchi et al. 1997]
Page 12
Existing Works: Unknown Scenes
• Two pass scene reconstruction and understanding.
• Can only use low-level information in first exploration pass.
© 2018 SIGGRAPH. All Rights Reserved 11
reconstruction & segmentation[Xu et al. 2015]
object recognition[Xu et al. 2016]
First Pass Second Pass
Page 13
The Main Challenge
• How to automatically achieve scene reconstruction and understanding in one
pass?
© 2018 SIGGRAPH. All Rights Reserved 12
Page 14
Motivation
• Human explore unknown scenes object by object!
© 2018 SIGGRAPH. All Rights Reserved 13
Page 15
Our Solution
© 2018 SIGGRAPH. All Rights Reserved 14
• Key idea: using recognized objects as a guidance map
Page 16
We need to
© 2018 SIGGRAPH. All Rights Reserved 15
Page 17
Phase 1: The Next Best Object Problem
© 2018 SIGGRAPH. All Rights Reserved 16
Which object should I
scan next?
Object of Interest (OOI)
Page 18
Overview
© 2018 SIGGRAPH. All Rights Reserved 17
Objectness
Objectness Based Segmentation
Objectness Based
Global Exploration Planning
Objectness Based
Local View Planning
Page 19
Model-Driven Objectness
• Objectness should measure both similarity and completeness
© 2018 SIGGRAPH. All Rights Reserved 18
Page 20
Partial Matching
© 2018 SIGGRAPH. All Rights Reserved 19
Database
Query Database Model
Page 21
Partial Matching
© 2018 SIGGRAPH. All Rights Reserved 20
Query Database Model
3DMatch [Zeng et al. 2016]
Page 22
Partial Matching
© 2018 SIGGRAPH. All Rights Reserved 21
Query Dataset Model
Page 23
Model-Driven Objectness
© 2018 SIGGRAPH. All Rights Reserved 22
Similarity CompletenessObjectness
Page 24
Next Best Object
© 2018 SIGGRAPH. All Rights Reserved 23
Distance Orientation Size
Objectness
Page 25
Technical Challenge
• How to segment and recognize objects during reconstruction?
© 2018 SIGGRAPH. All Rights Reserved 24
Missing data
Recognition and segmentation constitute a chicken-egg problem
Page 26
Pre-segmentation
© 2018 SIGGRAPH. All Rights Reserved 25
Indoor object Scanned Model
[Whelan et al. 2015] [Tateno et al. 2015]
Pre-segmented Components
Page 27
Post-segmentation
• Couples segmentation and recognition in the same optimization
© 2018 SIGGRAPH. All Rights Reserved 26
Page 28
Post-segmentation
© 2018 SIGGRAPH. All Rights Reserved 27
Page 29
Post-segmentation Results
© 2018 SIGGRAPH. All Rights Reserved 28
Pre-segmentation Post-segmentation
Page 30
Database Construction
© 2018 SIGGRAPH. All Rights Reserved 29
Page 31
Database Construction
Two advantages:
• Decrease the difference between CAD model and scanned model
• Segmented components & component pairs can make retrieval easier
© 2018 SIGGRAPH. All Rights Reserved 30
Page 32
Phase 2: The Next Best View Problem
© 2018 SIGGRAPH. All Rights Reserved 31
Which view of the OOI
should I scan next?
?
??
?
Page 33
Next Best View
© 2018 SIGGRAPH. All Rights Reserved 32
Page 34
© 2018 SIGGRAPH. All Rights Reserved 33
Page 35
Evaluation
• Virtual scene dataset
© 2018 SIGGRAPH. All Rights Reserved 34
SUNCG (66 scenes) ScanNet (38 scenes)
Page 36
Comparison
• Comparing object recognition with PointNet++ [Qi et al. 2017]
© 2018 SIGGRAPH. All Rights Reserved 35
Page 37
Comparison
• Comparing Rand Index of segmentation
© 2018 SIGGRAPH. All Rights Reserved 36
Page 38
Comparison
• Comparing object coverage rate and quality against tensor field guided
autoscanning [Xu et al. 2017]
© 2018 SIGGRAPH. All Rights Reserved 37
Page 39
More Results
© 2018 SIGGRAPH. All Rights Reserved 38
Page 40
Conclusion
© 2018 SIGGRAPH. All Rights Reserved 39
• Objectness based segmentation
• Pre-segmentation
• Post-segmentation
Key techniques:
• Objectness based reconstruction
• The next best object (NBO)
• The next best view (NBV)
Page 41
Limitations
© 2018 SIGGRAPH. All Rights Reserved 40
No similar models Cluttered scenes
Page 42
Future Works
© 2018 SIGGRAPH. All Rights Reserved 41
Combine image-based method Driverless car with LiDAR
Page 43
Thank you for your attention!Data and code are available:
http://kevinkaixu.net/projects/nbo.html
© 2018 SIGGRAPH. All Rights Reserved
Page 44
Comparison
• Comparing object coverage rate and quality against tensor field guided
autoscanning [Xu et al. 2017]
© 2018 SIGGRAPH. All Rights Reserved 43
Depth noise
Page 45
Time Table
© 2018 SIGGRAPH. All Rights Reserved 44
Page 46
Robot
© 2018 SIGGRAPH. All Rights Reserved 45