This work was supported by the National Basic Program of China, 973 Program (Project no. 2015CB351706), the Shenzhen Science and Technology Program (JCYJ20170413162256793 & JCYJ20170413162617606), the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14201918), and the CUHK Research Committee Funding (Direct Grants) under project code - 4055103.
Mask-ShadowGAN: Learning to Remove Shadows from Unpaired DataXiaowei Hu1, Yitong Jiang2, Chi-Wing Fu1,2, and Pheng-Ann Heng1,2
1 The Chinese University of Hong Kong 2Shenzhen Institutes of Advanced Technology
Motivation #1:
Mask-ShadowGAN
Experimental Results
➢ It is very tedious to prepare the training data.
➢ The approach limits the kinds of scenes that data can be prepared.
➢ Training pairs may have inconsistent colors or shift in camera views.
⚫ Comparison using USR testing set (user study)
Motivation #2:
➢ On the same background, we may have different shadows.
➢ However, the generator Gs can only produce a unique shadowimage from a given shadow-free image (background).
➢ The generated shadow image cannot match different input shadowimages (leftmost) and the cycle-consistency constraint cannot hold.
Limitations of paired training data:
Code & data: https://github.com/xw-hu/Mask-ShadowGAN
Learn to remove shadows from unpaired training data:
cycle-consistency loss
𝐺𝑓
ሚ𝐼𝑓
𝐺𝑠
ሚ𝐼𝑠𝐼𝑠
input
Discriminator
𝐷𝑓
Generator
𝐺𝑓real
shadow-freeimage?
𝐺𝑓 𝐺𝑠
match
match
(b) Mask-guided cycle-consistency constraint (ours)
➢ On the same background, Mask-ShadowGAN can generatedifferent shadow images.
➢ Our key idea is to first learn to produce a shadow mask from theinput shadow image during the training and generate theshadow images with the help of shadow masks.
𝐺𝑓
shadow cycle-consistency loss
𝑀𝑙
ሚ𝐼𝑓
𝐺𝑠
ሚ𝐼𝑠𝐼𝑠
(a) Learning from shadow images
input
𝐷𝑓
shadow-free adversarial loss
ሚ𝐼𝑓
guide
𝐺𝑠
shadow identity loss
𝑀𝑛 ሚ𝐼𝑠𝑖
𝐼𝑠
input
guide
ሚ𝐼𝑠𝐼𝑓
shadow-free cycle-consistency loss
𝑀𝑟
𝐺𝑓𝐺𝑠
ሚ𝐼𝑓
(b) Learning from shadow-free images
input
guide
ሚ𝐼𝑠
𝐷𝑠
shadow adversarial loss ሚ𝐼𝑓
𝑖
shadow-free identity loss
𝐺𝑓
input 𝐼𝑓
Shadow Domain Shadow-free Domain
(a) Cycle-consistency constraint (conventional)
𝐺𝑓 𝐺𝑠
not match
not match
Trained on ISTD (paired)
Trained on SRD (paired)
Trained on USR (unpaired)
⚫ Comparison using SRD & ISTD testing sets (RMSE)
Our Unpaired Shadow Removal Dataset - USR ➢ 2,445 shadow images (training : testing = 1,956 : 289)
➢ 1,770 shadow-free images (training)
➢ Shadows are cast by various kinds of objects, e.g., trees, buildings,
traffic signs, persons, umbrellas, railings, etc.
➢ Existing datasets cover only hundreds of different backgrounds,
while ours cover over a thousand different backgrounds.
⚫ Comparison with CycleGAN
inputs CycleGAN Mask-ShadowGAN
inputs CycleGAN Mask-ShadowGAN
vs.
