LSD-SLAM: Large-Scale Direct Monocular SLAM

1

LSD-SLAM: Large-Scale Direct Monocular SLAM

Jakob Engel, Thomas Schps, Daniel CremersTechnical University Munich

Monocular Video Camera Motion and Scene Geometry

Computer Vision GroupTechnical University of Munich

Jakob Engel

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

2

Live Operation

real-time operation on laptop (no GPU)LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

3

(Some) Related Work

DTAM: Dense Tracking and Mapping in Real-Time. Newcombe, Lovegrove, Davison; ICCV 11

MonoSLAM: Real-time single camera SLAM. Davison, Reid, Molton, Stasse; PAMI 07

Structure from Motion Causally Integrated Over Time. Chiuso, Favaro, Jin, Soatto; PAMI 02

SVO: Fast Semi-Direct Monocular Visual Odometry. Forster, Pizzoli, Scaramuzza; ICRA 14

Parallel Tracking and Mapping for Small AR Workspaces. Klein, Murray; ISMAR 07

Scalable monocular SLAM. Eade, Drummond; CVPR 06

Visual Odometry.Nistr, Naroditsky, Bergen; CVPR 04

Scale Drift-Aware Large Scale Monocular SLAM. Strasdat, Montiel, Davison; RSS 10

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Extract & MatchFeatures

(SIFT / SURF / BRIEF /...)

Input Images

4

Track:min. reprojection error

(point distances)

Map:est. feature-parameters

(3D points / normals)

abstract images to feature observations

Input Images

Track:min. photometric error

(intensity difference)

Map: est. per-pixel depth

(semi-dense depth map)

keep full image

LSD-SLAM: whats new?Keypoint-Based Direct (LSD-SLAM)

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

5

...and why do that?

can only use & reconstruct corners can use & reconstruct whole imageLSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Depth Estimation

Input Video

6

Current KF

Tracking

Overview

Add to Map

Map Optimization

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

7

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KFAdd to Map

Map Optimization

8

Tracking

KF image KF depth

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

8

Tracking

KF image KF depth back-warped new frame

Camera Pose in

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

8

Tracking

KF image KF depth back-warped new frame

Camera Pose in

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

( forward-compositional Lucas-Kanade)

8

Tracking

minimize using Gauss-Newton Algorithm

KF image KF depth back-warped new frame

Camera Pose in

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

9

Tracking multi-resolution (track large motions)

Huber norm instead of L2 (outliers & occlusions)

statistical normalization (respect depth- and pixel-noise)

single core timings:320x240: 5-10ms640x480: 20-30ms

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

10

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KFAdd to Map

Map Optimization

11

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Add to Map

Map Optimization

Depth Estimation

Current KF

Take KF?

Createnew KF

Refine KF

12

Depth Estimation

pixelwise filtering (exploit video)small-baseline large baseline

information selectiononly do stereo if sufficientinformation gain

edge-preserving smoothing distance-based KF selection

image inverse depth inverse depth variance

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

[Engel, Sturm, Cremers; ICCV 13]

13

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KF

Take KF?

Createnew KF

Refine KF

Add to Map

Map Optimization

14

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KF

Take KF?

Createnew KF

Refine KF

Add to Map

Map Optimization

15

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KF

Take KF?

Createnew KF

Refine KF

Add to MapSim(3) alignmentto all nearby KFs

Optional: FabMap for large loops

Map OptimizationSim(3) pose-graph

16

Global Mapping

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

with (warped point)

16

Global Mapping

Direct Tracking with scale (on Sim(3)):

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

with (warped point)

16

Global Mapping

Direct Tracking with scale (on Sim(3)):

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

with (warped point)

16

Global Mapping

Direct Tracking with scale (on Sim(3)):

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

+ GN optimization + multi-resolution + Huber norm + statistical norm.

with (warped point)

16

Global Mapping

Direct Tracking with scale (on Sim(3)):

Optimize pose-graph on Sim(3)

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

+ GN optimization + multi-resolution + Huber norm + statistical norm.

17

Global Mapping

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

18

Overview

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Input Video640x480 @ 30Hz

TrackingSE(3) alignment

to current KF

Depth Estimation

Current KF

Take KF?

Createnew KF

Refine KF

Add to MapSim(3) alignmentto all nearby KFs

Optional: FabMap for large loops

Map OptimizationSim(3) pose-graph

19

Results

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

6 minutes, 640x480@50fps: 16.000 Tracked Frames, 800 Keyframes; 11.000 Constraints; 51 Million Points

20

Results

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

12 minutes, 640x480@50fps:36.000 Tracked Frames, 1.000 Keyframes; 18.000 Constraints; 100 Million Points

21

Results

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Semi-Dense Visual Odometry for AR on a Smartphone; T. Schps, J. Engel, D. Cremers; ISMAR 14.

22

Key Ingredients

Direct Tracking

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

22

Key Ingredients

Direct Tracking

Semi-Dense Stereo filter over many small-baseline frames strict information selection

Pose-Graph on Sim(3)

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

23

LSD-SLAM

LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

Large-scale direct mono-SLAM Fully direct (no keypoints / features) Real-time even on CPU Open-source code & data-sets

Slide Number 1Slide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Slide Number 29Slide Number 30Slide Number 31