This work was supported by the National Basic Program of China, 973 Program (Project no. 2015CB351706), the Shenzhen Science and Technology Program (JCYJ20170413162256793 & JCYJ20170413162617606), the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14201918), and the CUHK Research Committee Funding (Direct Grants) under project code - 4055103. Mask - ShadowGAN : Learning to Remove Shadows from Unpaired Data Xiaowei Hu 1 , Yitong Jiang 2 , Chi-Wing Fu 1,2 , and Pheng-Ann Heng 1,2 1 The Chinese University of Hong Kong 2 Shenzhen Institutes of Advanced Technology Motivation #1: Mask-ShadowGAN Experimental Results ➢ It is very tedious to prepare the training data. ➢ The approach limits the kinds of scenes that data can be prepared. ➢ Training pairs may have inconsistent colors or shift in camera views. ⚫ Comparison using USR testing set (user study) Motivation #2: ➢ On the same background, we may have different shadows. ➢ However, the generator Gs can only produce a unique shadow image from a given shadow-free image (background). ➢ The generated shadow image cannot match different input shadow images (leftmost) and the cycle-consistency constraint cannot hold. Limitations of paired training data: Code & data: https://github.com/xw-hu/Mask-ShadowGAN Learn to remove shadows from unpaired training data: cycle-consistency loss ሚ ሚ input Discriminator Generator real shadow-free image? match match (b) Mask-guided cycle-consistency constraint (ours) ➢ On the same background, Mask-ShadowGAN can generate different shadow images. ➢ Our key idea is to first learn to produce a shadow mask from the input shadow image during the training and generate the shadow images with the help of shadow masks. shadow cycle-consistency loss ሚ ሚ (a) Learning from shadow images input shadow-free adversarial loss ሚ guide shadow identity loss ሚ input guide ሚ shadow-free cycle-consistency loss ሚ (b) Learning from shadow-free images input guide ሚ shadow adversarial loss ሚ shadow-free identity loss input Shadow Domain Shadow-free Domain (a) Cycle-consistency constraint (conventional) not match not match Trained on ISTD (paired) Trained on SRD (paired) Trained on USR (unpaired) ⚫ Comparison using SRD & ISTD testing sets (RMSE) Our Unpaired Shadow Removal Dataset - USR ➢ 2,445 shadow images (training : testing = 1,956 : 289) ➢ 1,770 shadow-free images (training) ➢ Shadows are cast by various kinds of objects, e.g., trees, buildings, traffic signs, persons, umbrellas, railings, etc. ➢ Existing datasets cover only hundreds of different backgrounds, while ours cover over a thousand different backgrounds. ⚫ Comparison with CycleGAN inputs CycleGAN Mask-ShadowGAN inputs CycleGAN Mask-ShadowGAN vs .
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
This work was supported by the National Basic Program of China, 973 Program (Project no. 2015CB351706), the Shenzhen Science and Technology Program (JCYJ20170413162256793 & JCYJ20170413162617606), the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14201918), and the CUHK Research Committee Funding (Direct Grants) under project code - 4055103.
Mask-ShadowGAN: Learning to Remove Shadows from Unpaired DataXiaowei Hu1, Yitong Jiang2, Chi-Wing Fu1,2, and Pheng-Ann Heng1,2
1 The Chinese University of Hong Kong 2Shenzhen Institutes of Advanced Technology
Motivation #1:
Mask-ShadowGAN
Experimental Results
➢ It is very tedious to prepare the training data.
➢ The approach limits the kinds of scenes that data can be prepared.
➢ Training pairs may have inconsistent colors or shift in camera views.
⚫ Comparison using USR testing set (user study)
Motivation #2:
➢ On the same background, we may have different shadows.
➢ However, the generator Gs can only produce a unique shadowimage from a given shadow-free image (background).
➢ The generated shadow image cannot match different input shadowimages (leftmost) and the cycle-consistency constraint cannot hold.
Limitations of paired training data:
Code & data: https://github.com/xw-hu/Mask-ShadowGAN
Learn to remove shadows from unpaired training data:
cycle-consistency loss
𝐺𝑓
ሚ𝐼𝑓
𝐺𝑠
ሚ𝐼𝑠𝐼𝑠
input
Discriminator
𝐷𝑓
Generator
𝐺𝑓real
shadow-freeimage?
𝐺𝑓 𝐺𝑠
match
match
(b) Mask-guided cycle-consistency constraint (ours)
➢ On the same background, Mask-ShadowGAN can generatedifferent shadow images.
➢ Our key idea is to first learn to produce a shadow mask from theinput shadow image during the training and generate theshadow images with the help of shadow masks.
𝐺𝑓
shadow cycle-consistency loss
𝑀𝑙
ሚ𝐼𝑓
𝐺𝑠
ሚ𝐼𝑠𝐼𝑠
(a) Learning from shadow images
input
𝐷𝑓
shadow-free adversarial loss
ሚ𝐼𝑓
guide
𝐺𝑠
shadow identity loss
𝑀𝑛 ሚ𝐼𝑠𝑖
𝐼𝑠
input
guide
ሚ𝐼𝑠𝐼𝑓
shadow-free cycle-consistency loss
𝑀𝑟
𝐺𝑓𝐺𝑠
ሚ𝐼𝑓
(b) Learning from shadow-free images
input
guide
ሚ𝐼𝑠
𝐷𝑠
shadow adversarial loss ሚ𝐼𝑓
𝑖
shadow-free identity loss
𝐺𝑓
input 𝐼𝑓
Shadow Domain Shadow-free Domain
(a) Cycle-consistency constraint (conventional)
𝐺𝑓 𝐺𝑠
not match
not match
Trained on ISTD (paired)
Trained on SRD (paired)
Trained on USR (unpaired)
⚫ Comparison using SRD & ISTD testing sets (RMSE)
Our Unpaired Shadow Removal Dataset - USR ➢ 2,445 shadow images (training : testing = 1,956 : 289)
➢ 1,770 shadow-free images (training)
➢ Shadows are cast by various kinds of objects, e.g., trees, buildings,
traffic signs, persons, umbrellas, railings, etc.
➢ Existing datasets cover only hundreds of different backgrounds,
while ours cover over a thousand different backgrounds.
⚫ Comparison with CycleGAN
inputs CycleGAN Mask-ShadowGAN
inputs CycleGAN Mask-ShadowGAN
vs.
Related Documents
