1 Towards Weakly- and Semi- Supervised Object Localization and Semantic Segmentation Lecturer: Yunchao Wei Image Formation and Processing (IFP) Group https://weiyc.githu b.io University of Illinois at Urbana- Champaign
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Towards Weakly- and Semi- Supervised Object Localization and Semantic
Segmentation
Lecturer: Yunchao WeiImage Formation and Processing (IFP)
Grouphttps://weiyc.github.io
University of Illinois at Urbana-Champaign
2
Fully Supervised Semantic Segmentation
Masks
…
Hard to Collect
3
4
100/C 5min/I
5
Weakly Supervised Annotations
horse personhorse
person
scribbles points image-level labelsbounding boxes
The simplest and the most efficient one
6
person
horsetable
images
annotations
…
…
Loc
Seg
Object Localization Maps
Our Targets
7
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
8
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
9
Proposal-based Localization
Learning to Segment with Image-level Annotations. PR 2016
10
Proposal-based Localization
Hypotheses-CNN-Pooling Localization Map Generation
o Exhaustedly examine each proposal to generate localization
o Introducing false negative pixels(background)
Weakness
Pascal VOC
43.2%
Learning to Segment with Image-level Annotations. PR 2016
11
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
12
Simple to Complex
Simple Images Complex Images
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017
13
Simple images with the corresponding saliency maps
Simple to Complex
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017
14
o Initial-DCNN
o Enhanced-DCNN
o Powerful-DCNN
Simple to Complex
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017
15
Flickr-Clean (40K)
Simple to Complex
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017
16
Networks Training Set mIoU
I-DCNN Flickr-Clean 44.1
E-DCNN Flickr-Clean 46.8
P-DCNN Flickr-Clean+VOC 49.8
Ablation Analysis on Pascal VOC12 val
o Collecting a large number of simple images
o Time consuming for training
Weakness
Simple to ComplexPascal VOC
43.2%
51.2%
STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017
17
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
18
Adversarial Erasing
Small and sparse object localization maps Dense and integral object localization maps
dog bird cow
Previous works Our target
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)
19
Adversarial Erasing
headconf: 1.0
body
conf: 0.8
foot
conf: 0.5
Motivation
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)
20
Adversarial Erasing
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)
Our Solution Visualizations
21
Adversarial Erasing
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)
The pipeline of weakly semantic segmentation based on AE
22
Weakness of Adversarial Erasing
o Time consuming to learn several classification networks.
o Hard to determine how many AE steps should be conducted.
0 150
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Epoch
Loss
AE-step4
AE-step3
AE-step2
AE-step1
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)
Pascal VOC
43.2%
51.2%
55.7%
23
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
24
Adversarial Complementary Learning
Imag
esC
AM
Ou
rs
…
Classification
…
GAP FC
Conv 1x1 GAP
CA
MO
urs
Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018
Revisiting CAM
25
Adversarial Complementary Learning
Detailed Framework
Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018
Erasing
Thresholding
26
Adversarial Complementary Learning
Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018
27
Adversarial Complementary Learning
Localization Comparison
Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018
28
Methods Top-1 err. Top-5 err.
Backprop on GoogleLeNet 61.31 50.55
GoogLeNet-GAP (CVPR 2016) 56.40 43.00
GoogLeNet-HaS-32 (ICCV 2017) 54.53 -
GoogLeNet-ACoL(Ours) 53.28 42.58
GoogLeNet-ACoL*(Ours) 53.28 35.22
Backprop on VGGnet 61.12 51.46
VGG-GAP (CVPR 2016) 57.20 45.14
VGGnet-ACoL(Ours) 54.17 40.57
VGGnet-ACoL*(Ours) 54.17 36.66
Adversarial Complementary Learning
Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018
Localization error on ILSVRC validation set
Pascal VOC
43.2%
51.2%
55.7%
58.8%
29
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
30
Multi-dilated Convolution
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)
3x3
In
Out
Motivation
kernel
31
o Multi-dilated Convolutional Network for Object Localization
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)
Multi-dilated Convolution
32
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)
Multi-dilated Convolution
33
o Weakly- and Semi- Supervised Semantic Segmentation
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)
Multi-dilated Convolution
34
Multi-dilated Convolution
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)
Pascal VOC
43.2%
51.2%
55.7%
58.8%
60.8%68.5%
35
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Achievements
36
Transferable Semi-supervised Network
In-category Semi-supervised Semantic Segmentation (I3S)
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
37
100/C 5min/I
38
Transferable Semi-supervised Network
In-category Semi-supervised Semantic Segmentation (I3S)
Cross-category Semi-supervised Semantic Segmentation (C3S)
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
39
Label Transfer Network(L-Net)
Prediction Transfer Network(P-Net)
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
40
denotes the standard element-wise binary cross-entropy loss
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
41
Classification Activation Map
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
42
Random Walk based self-diffusion algorithm
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
43
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
44
Transferable Semi-supervised Network
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
45
Transferable Semi-supervised NetworkPascal VOC
43.2%
51.2%
55.7%
58.8%
60.8%68.5%
64.6%
46
Transferable Semi-supervised Semantic Segmentation. AAAI 2018
47
CVPR 2017PR 2016 PAMI 2017
CVPR 2018 CVPR 2018 AAAI 2018
Proposal-based Localization Simple to Complex Adversarial Erasing
Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network
Summary
person
horsetable
images
annotations
…
…
Loc
Seg
Object Localization Maps
48
Thanks
Related Documents
