Top Banner
Unsupervised Learning of Hierarchical Spatial Structures Devi Parikh, Larry Zitnick and Tsuhan Chen
56

Unsupervised Learning of Hierarchical Spatial Structures

Feb 22, 2016

Download

Documents

Ed Sippel

Unsupervised Learning of Hierarchical Spatial Structures. Devi Parikh , Larry Zitnick and Tsuhan Chen. Our visual world…. Intro Approach Results Conclusion. What is an object?. What is context?. … hierarchical spatial patterns. Goal. Intro Approach Results Conclusion. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Unsupervised Learning of  Hierarchical Spatial Structures

Unsupervised Learning of Hierarchical Spatial Structures

Devi Parikh, Larry Zitnick and Tsuhan Chen

Page 2: Unsupervised Learning of  Hierarchical Spatial Structures

2… hierarchical spatial patterns

Our visual world…

What is an object?What is context?

Intro

Approach

Results

Conclusion

Page 3: Unsupervised Learning of  Hierarchical Spatial Structures

3

Goal

Unsupervised!

Intro

Approach

Results

Conclusion

Page 4: Unsupervised Learning of  Hierarchical Spatial Structures

4

Related work

[Todorovic 2008]

[Fidler 2007] [Zhu 2008]

[Sivic 2008]

Fully unsupervised

Structure and parameters learnt

From features to multiple objects

Intro

Approach

Results

Conclusion

Page 5: Unsupervised Learning of  Hierarchical Spatial Structures

5

ModelRule based

c2

c4

c1

c2

c3

r1 0.9

0.1

0.60.7

0.6

Intro

Approach

Results

Conclusion

Page 6: Unsupervised Learning of  Hierarchical Spatial Structures

6

c2

r2

c1

c2

c3

r1 0.9

0.1

0.60.7

0.6

ModelRule based

Intro

Approach

Results

Conclusion

Page 7: Unsupervised Learning of  Hierarchical Spatial Structures

7

c2

r2

c1

c2

c3

r1 0.9

0.1

0.60.7

0.6

ModelHierarchical rule-based

Intro

Approach

Results

Conclusion

Page 8: Unsupervised Learning of  Hierarchical Spatial Structures

8

Rules R

Image-parts V

Model

Codewords C

Features F

Intro

Approach

Results

Conclusion

Page 9: Unsupervised Learning of  Hierarchical Spatial Structures

9

Model NotationV = {v} instantiated image-parts

rv rule corresponding to instantiated part v

Ch(rv) = {x} children of rule rv

includes instantiated children Ch(v) and un-instantiated children

Intro

Approach

Results

Conclusion

Page 10: Unsupervised Learning of  Hierarchical Spatial Structures

10

Model

Intro

Approach

Results

Conclusion

Page 11: Unsupervised Learning of  Hierarchical Spatial Structures

11

Inference

Intro

Approach

Results

Conclusion

Page 12: Unsupervised Learning of  Hierarchical Spatial Structures

12

Inference

Intro

Approach

Results

Conclusion

Page 13: Unsupervised Learning of  Hierarchical Spatial Structures

13

Inference

Intro

Approach

Results

Conclusion

Page 14: Unsupervised Learning of  Hierarchical Spatial Structures

14

Inference

Intro

Approach

Results

Conclusion

Page 15: Unsupervised Learning of  Hierarchical Spatial Structures

15

Inference

Intro

Approach

Results

Conclusion

Page 16: Unsupervised Learning of  Hierarchical Spatial Structures

16

Inference

Intro

Approach

Results

Conclusion

Page 17: Unsupervised Learning of  Hierarchical Spatial Structures

17

Inference

Intro

Approach

Results

Conclusion

Page 18: Unsupervised Learning of  Hierarchical Spatial Structures

18

Inference

Intro

Approach

Results

Conclusion

Page 19: Unsupervised Learning of  Hierarchical Spatial Structures

19

Inference

Intro

Approach

Results

Conclusion

Page 20: Unsupervised Learning of  Hierarchical Spatial Structures

20

Inference

Intro

Approach

Results

Conclusion

Page 21: Unsupervised Learning of  Hierarchical Spatial Structures

21Minimum Cost

Steiner TreeCharikar 1998

Inference

Intro

Approach

Results

Conclusion

Page 22: Unsupervised Learning of  Hierarchical Spatial Structures

22

Inference

Intro

Approach

Results

Conclusion

Page 23: Unsupervised Learning of  Hierarchical Spatial Structures

23

Generalized distance transformFelzenszwalb et al. 2001

Inference

Intro

Approach

Results

Conclusion

Page 24: Unsupervised Learning of  Hierarchical Spatial Structures

24

EM style

Initialize rules

Infer rules Update parameters Modify rules

Learning

Intro

Approach

Results

Conclusion

Page 25: Unsupervised Learning of  Hierarchical Spatial Structures

25

Initialize rules

Learning

Intro

Approach

Results

Conclusion

Page 26: Unsupervised Learning of  Hierarchical Spatial Structures

26

Inference

Learning

Intro

Approach

Results

Conclusion

Page 27: Unsupervised Learning of  Hierarchical Spatial Structures

27

Inference

Learning

Intro

Approach

Results

Conclusion

Page 28: Unsupervised Learning of  Hierarchical Spatial Structures

28

Add children

Learning

Intro

Approach

Results

Conclusion

Page 29: Unsupervised Learning of  Hierarchical Spatial Structures

29

Add children

Update parameters

Pruning children

Removing rules

Learning

Intro

Approach

Results

Conclusion

Page 30: Unsupervised Learning of  Hierarchical Spatial Structures

30

Adding rules

Randomly add rules

Learning

Intro

Approach

Results

Conclusion

Page 31: Unsupervised Learning of  Hierarchical Spatial Structures

31

Behavior Competition among rules Competition with root (noise)

Intro

Approach

Results

Conclusion

Page 32: Unsupervised Learning of  Hierarchical Spatial Structures

32

Behavior Competition among rules Competition with root (noise) Dropping children and rules Number of children Structure of DAG and tree # rules, parameters, structure learnt automatically Multiple instantiations of rules Multiple children with same appearance

Intro

Approach

Results

Conclusion

Page 33: Unsupervised Learning of  Hierarchical Spatial Structures

Experiment 1: Faces & MotorbikesIntro

Approach

Results

Conclusion

Page 34: Unsupervised Learning of  Hierarchical Spatial Structures

34

Faces and Motorbikes SIFT (200 words)

Learnt 15 L1 rules, 2 L2 rules Each L1 rule average ~7 children Each L2 rule average ~4 children

Faces & Motorbikes

Intro

Approach

Results

Conclusion

Page 35: Unsupervised Learning of  Hierarchical Spatial Structures

35

Example rules

Intro

Approach

Results

Conclusion

Page 36: Unsupervised Learning of  Hierarchical Spatial Structures

36

Patches

Intro

Approach

Results

Conclusion

Page 37: Unsupervised Learning of  Hierarchical Spatial Structures

37

Localization behavior

Intro

Approach

Results

Conclusion

Page 38: Unsupervised Learning of  Hierarchical Spatial Structures

38

Categorization behavior

Faces Motorbikes Faces Motorbikes Faces Motorbikes

occu

rren

ce

code-words first level rules second level rules

Intro

Approach

Results

Conclusion

Page 39: Unsupervised Learning of  Hierarchical Spatial Structures

39

Categorization behavior

Words Rules Tree

Words: 94 %

Tree: 100%

KmeansPLSASVM

Intro

Approach

Results

Conclusion

Page 40: Unsupervised Learning of  Hierarchical Spatial Structures

40

Edge features

Words: 55 %

Tree: 82%

Intro

Approach

Results

Conclusion

Page 41: Unsupervised Learning of  Hierarchical Spatial Structures

Experiment 2: Six categoriesIntro

Approach

Results

Conclusion

Page 42: Unsupervised Learning of  Hierarchical Spatial Structures

42

Six categories

61 L1 rules (~9 children)12 L2 rules (~3 children)

Kim 2008: 95 %

Words: 87 %

Tree: 95 %

Intro

Approach

Results

Conclusion

Page 43: Unsupervised Learning of  Hierarchical Spatial Structures

Experiment 3: Scene categoriesIntro

Approach

Results

Conclusion

Page 44: Unsupervised Learning of  Hierarchical Spatial Structures

44

Scene categories

Image Segmentation

Mean color Codeword

Intro

Approach

Results

Conclusion

Page 45: Unsupervised Learning of  Hierarchical Spatial Structures

45

Outdoor scenes

rule

s

images

Intro

Approach

Results

Conclusion

Page 46: Unsupervised Learning of  Hierarchical Spatial Structures

Experiment 4: Structured street scenesIntro

Approach

Results

Conclusion

Page 47: Unsupervised Learning of  Hierarchical Spatial Structures

47

Windows

Intro

Approach

Results

Conclusion

Page 48: Unsupervised Learning of  Hierarchical Spatial Structures

48

Object categories

Intro

Approach

Results

Conclusion

Page 49: Unsupervised Learning of  Hierarchical Spatial Structures

49

Object categories

Intro

Approach

Results

Conclusion

Page 50: Unsupervised Learning of  Hierarchical Spatial Structures

50

Object categories

Intro

Approach

Results

Conclusion

Page 51: Unsupervised Learning of  Hierarchical Spatial Structures

51

Parts of objects

Intro

Approach

Results

Conclusion

Page 52: Unsupervised Learning of  Hierarchical Spatial Structures

52

Multiple objects

Intro

Approach

Results

Conclusion

Page 53: Unsupervised Learning of  Hierarchical Spatial Structures

53

Street Scenes (PLSA)

Intro

Approach

Results

Conclusion

Page 54: Unsupervised Learning of  Hierarchical Spatial Structures

54

Dataset specific rules

irrelevant

relevantIntro

Approach

Results

Conclusion

Page 55: Unsupervised Learning of  Hierarchical Spatial Structures

55

Conclusion

Unsupervised learning of hierarchical spatial patterns Low level features, object parts, objects, regions in scene

Rule-based approach Learning: EM style Inference: Minimum cost Steiner tree

Features SIFT, edges, color segments

Intro

Approach

Results

Conclusion

Page 56: Unsupervised Learning of  Hierarchical Spatial Structures

56

Summary

I

Root

Scene

Objects

Object Parts

Features

Intro

Approach

Results

Conclusion