Page 1
Estimating natural illumination from a single outdoor image12/8/09
Estimating natural illumination from a single outdoor scene
Sagnik Dhar (107219686)Debaleena Chattopadhay (107224101)
Course Project for CSE – 591Fall ‘09
Based on, “Estimating Natural Illumination from a Single Outdoor Image” by Jean-
Francois Lalonde, Alexei A. Efros, and Srinivasa G. NarasimhanCVPR ‘09
Page 2
Estimating natural illumination from a single outdoor image12/8/09
Estimation of illumination
Page 3
Estimating natural illumination from a single outdoor image12/8/09
Why do people care about illumination ?
• Insertion of objects
• Retrieval of objects from a database
Page 4
Estimating natural illumination from a single outdoor image12/8/09
Approach
Natural Outdoor Daytime Scene
Extract Plausi-ble Cues
Output a Probable Distribution of Sun’s
Zenith angle
Sky Cue
Shadow Cue
Perez Modeling of
Skies
Simulate Normal
Distribution of sun positions
Determining Best Linear
Shadow
Vertical Surface Cue
Intensity Gradi-ents Vote For Pos-
sible Sun positions
Intensity Of The Vertical Surfaces
Weighed By Their Size Vote For Possible
Sun Directions
Compute Size Ratio Of Surfaces
To The Scene
Page 5
Estimating natural illumination from a single outdoor image12/8/09
Sky Cue
Determine Horizon Line Extract Sky Segment
Classify into Clear, Patchy or Overcast
category and Process(KNN)
Simulate Perez Sky Model to create 360 skies (each
for a sun zenith angle)(Video)
Taking our sky segment as the mean, find a probability
distribution of sun zenith angle
Page 6
Estimating natural illumination from a single outdoor image12/8/09
KNN Classification of skies
Sky
Clear Patchy Overcast
Binary segmentation to remove clouds Discard Cue
Continue
Page 7
Estimating natural illumination from a single outdoor image12/8/09
Shadow Cue
Extract ground Segment
Detect Longest Linear Shadow
Shadow pixel intensities vote for
probable sun directions
Convert to CIE L*a*b Color
Space
Detect Shadow Pixels in b channel
Find Intensity orientation of the shadow
pixels
Cluster Using K-means
Top Cluster votes for Sun
Directions
Page 8
Estimating natural illumination from a single outdoor image12/8/09
Vertical Surface Cue
Extract Vertical Surfaces
Compute surface size proportionality with respect to the scene.
Find intensity of each surfaces
Surface size weighted by intensity vote for their preferred Sun direction
Page 9
Estimating natural illumination from a single outdoor image12/8/09
Sun Zenith angle
Page 10
Estimating natural illumination from a single outdoor image12/8/09
Cue Combination
1ST
2ND3RD
4TH
271°Sky
14°Shadow
BehindVertical Surfaces
Page 11
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
Page 12
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
Page 13
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
1ST
2ND3RD
4TH
100°Sky
8°Shadow
RightVertical Surfaces
Page 14
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
Page 15
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
Page 16
Estimating natural illumination from a single outdoor image12/8/09
Results- Best Case
1ST
2ND3RD
4TH
181°Sky189°
Shadow
LeftVertical Surfaces
Page 17
Estimating natural illumination from a single outdoor image12/8/09
Results- Sky + Vertical Cue
Page 18
Estimating natural illumination from a single outdoor image12/8/09
Results- Sky + Vertical Cue
Page 19
Estimating natural illumination from a single outdoor image12/8/09
Results- Sky + Vertical Cue
1ST
2ND3RD
4TH
271°Sky
BehindVertical Surfaces
Page 20
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
Page 21
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
Page 22
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
1ST
2ND3RD
4TH
14°Shadow
BehindVertical Surfaces
Page 23
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
Page 24
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
Page 25
Estimating natural illumination from a single outdoor image12/8/09
Results- Shadow + Vertical Cue
1ST
2ND3RD
4TH
9°Shadow
BehindVertical Surfaces
Page 26
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
Page 27
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
Page 28
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
1ST
2ND3RD
4TH
5°Shadow
BehindVertical Surfaces
292°Sky
Page 29
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
Page 30
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
Page 31
Estimating natural illumination from a single outdoor image12/8/09
Results- Failure Cases
1ST
2ND3RD
4TH
16°Shadow
BehindVertical Surfaces
0°Sky
Page 32
Estimating natural illumination from a single outdoor image12/8/09
Human Validation
No. of Humans: 3
No. of Images: 58
Human Consensus
Human Contradiction
Accuracy
27 31 46%
Match with Human
Mismatch Accuracy
22 36 37%
Page 33
Estimating natural illumination from a single outdoor image12/8/09
Cue Weightage Investigation
Page 34
Estimating natural illumination from a single outdoor image12/8/09
Conclusion
• Accurate estimation of sun position – Tough problem for human visual system too.
• The photographic eye doesn’t bother to capture “informative” cues for visual systems to analyze.
• The weak cues are strongly coupled and highly susceptible to incorrect conclusion.
• Shadow detection from a single image is still in the need of better algorithms.
Page 35
Estimating natural illumination from a single outdoor image12/8/09
Thank You
To end, a quote I came across recently,
“All I see is the sun; And even when it's hiding behind a smoke of cloud, the shadow tells me all that there is...”