Robot Vision: Introduction - Graz University of Technology · Vision: Algorithms and Applications. Springer. 2010 Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer

Robot Vision:

Introduction

Ass.Prof. Friedrich Fraundorfer

SS 2020

About me

Ass. Prof. Dr. Friedrich Fraundorfer

Email: [email protected]

Institut für Maschinelles Sehen und Darstellen

Inffeldgasse 16/II

+43 (316) 873 - 5020

Consultation hours after email-appointment

2

Course schedule

14 lectures

▫ Tuesdays, 14:30-16:00, HS i11 (Dates in TUG-Online)

Course grade

▫ Written exams multiple times per term

Accompanied by practical

Lecture webpage

▫ https://www.tugraz.at/institutes/icg/education/coursepages/710088_RobotVi

sion/

3

Practical

Practical consists of 3 programming assignments

Groups of 2 students -> group enrollment in TC (will be opened after the

lecture)

Programming in C/C++ and OpenCV

Assignments:

▫ Camera calibration and stereo

▫ Feature matching and epipolar geometry

▫ Deep learning for depth estimation

Deliverables (submitted via TC):

▫ Source code

▫ Report (PDF)

Practical details this week in practical slot (4.3.2020)

4

Lecture material

Slides will be made available on the web-page

Relevant publications and book sections will need to be consulted (links

will be available)

Lecture will be recorded and recordings are visible for you in the Teach

Center

5

Richard Szeliski. Computer

Vision: Algorithms and

Applications. Springer. 2010

Richard Hartley and Andrew

Zisserman. Multiple View

Geometry in Computer Vision.

2004

Classroom activity

What is robot vision?

What do you think you will learn about?

6

Cameras for safe navigation

[Image credit: NASA (public domain)]

Cameras for safe navigation

8

Cameras for safe navigation

Self driving cars

10

Self driving cars

[Image credit: Mapilliary]11

Robotic grasping & household robotics

[Image credit: Andy Zeng MIT]

Flying robots

13

Flying robots

14

Lecture topics

Projective geometry

Image formation and camera calibration

Geometric algorithms (Fundamental matrix, Essential Matrix,

Triangulation)

Robust estimation (Ransac)

Features and matching

SfM

Bundle adjustment

Stereo matching

Deep learning for depth estimation

Depth cameras

15

Projective geometry

[Image credit: Charles Gunn]

Projective geometry: Measuring in images

[Source: Flickr]

17

Projective geometry: Measuring in images

[Source: KITTI]

18

Projective geometry: Measuring in images

19

Image formation and camera calibration

Z

X

YX1

x1

C

y

x

Cx

Cy

Cz

20

Geometric algorithms

21

𝑥′𝑇𝐹𝑥 = 0 … Epipolar constraint

pp’x’

x

X world point

C C’T

R

Robust estimation

Ransac – Random sample consensus

22

Feature detection and matching

23

Structure-from-Motion (SfM) concept

24

Bundle adjustment

25

C

y

x

xi

Xi

z PXi

min𝑃𝑗,𝑋𝑖

𝑖

𝑗

𝑥𝑖,𝑗 − 𝑃𝑗𝑋𝑖

Dense matching process

Left View Right View

lI rI D

Disparity image*

26

Deep learning for depth estimation

input image

depth image (output)

depth CNN

27

Depth Cameras

28