1 Ramesh Raskar, Computational Illumination Computational Illumination Computational Illumination Ramesh Raskar Mitsubishi Electric Research Labs Course WebPage : http://www.merl.com/people/raskar/photo/ Ramesh Raskar, Computational Illumination Computational Illumination Traditional Traditional ‘ film film‐ like like’ Photography Photography Lens Detector Pixels Image Computational Photography Computational Photography : Optics, Sensors and Computations Optics, Sensors and Computations Generalized Sensor Generalized Optics Computations Picture 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction Computational Photography Computational Photography Novel Cameras Generalized Sensor Generalized Optics Processing Computational Photography Computational Photography Novel Illumination Novel Cameras Generalized Sensor Generalized Optics Processing Light Sources
24
Embed
Computational Illumination - MIT CSAILgroups.csail.mit.edu/graphics/classes/CompPhoto06/... · 7 Ramesh Raskar, Computational Illumination Image-Based Actual Re-lighting Film the
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Programmable 4D Illumination field + time + wavelength
Programmable 4D Illumination field + time + wavelength
‘Smarter’ Lighting Equipment
What Parameters Can We Change ?What Parameters Can We Change ?
Edgerton 1930Edgerton 1930’’ss
3
Edgerton 1930Edgerton 1930’’ss
Multi‐flash sequential photography
Stroboscope(Electronic Flash)
CameraExposure
Flash Time
Ramesh Raskar, Computational Illumination
Computational Illumination:Computational Illumination:Programmable 4D Illumination Field + Time + WavelengthProgrammable 4D Illumination Field + Time + Wavelength
•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness–– Flash/NoFlash/No--flashflash
•• Light positionLight position–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)
Computational IlluminationComputational Illumination•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness
–– Flash/NoFlash/No--flashflash•• Light positionLight position
–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)
•• Temporal ModulationTemporal Modulation–– TV remote, Motion Tracking, Sony IDTV remote, Motion Tracking, Sony ID--cam, RFIGcam, RFIG
•• General lighting conditionGeneral lighting condition–– Day/NightDay/Night
Denoising Challenging Images
Available light:+ nice lighting
- noise/blurriness- color
No-flash
Flash:+ details+ color
- flat/artificial
Flash
Denoise no-flash image using flash image
Flash
No-flash
Result
Elmar Eisemann and Frédo Durand , Flash Photography Enhancement via Intrinsic Relighting
Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen, Kentaro Toyama. Digital Photography with Flash and No-Flash Image Pairs
4
+ original lighting+ details/sharpness+ color
Result
No-flash
Transfer detail from flash image to no-flash imageCross-Bilateral Filter based Approach
Varying Exposure time Varying Flash brightness Varying both
Capturing HDR Image
Flash Result Reflection LayerAmbient
Flash and Ambient Images[ Agrawal, Raskar, Nayar, Li Siggraph05 ]
Intensity Gradient Vector Projection
Intensity Gradient Vectors in Flash and Ambient Images
Same gradient vector direction Flash Gradient Vector
Ambient Gradient Vector
Ambient Flash
No reflections
6
Reflection Ambient Gradient Vector
Different gradient vector direction
With reflections
Ambient Flash
Flash Gradient Vector Residual Gradient Vector
Intensity Gradient Vector Projection
Result Gradient Vector
Ambient Flash Result Residual
Reflection Ambient Gradient Vector
Flash Gradient Vector
FlashProjection = Result
Residual =Reflection Layer
Co-located Artifacts
Ambient
Ramesh Raskar, Computational Illumination
Computational IlluminationComputational Illumination•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness
–– Flash/NoFlash/No--flashflash•• Light positionLight position
–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)–– MultiMulti--flash for depth edgesflash for depth edges
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Depth Edge Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
10
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
11
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Depth Discontinuities
Internal and externalShape boundaries, Occluding contour, Silhouettes
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Depth Edges
12
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Sigma = 9 Sigma = 5
Sigma = 1
Canny Intensity Edge Detection
Our method captures shape edges
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Our MethodCanny
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Our Method
PhotoMitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Canny Intensity Edge Detection
Our Method
Photo Result
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
13
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Shadows
Clutter
Many Colors
Highlight Edges
Mark moving parts
Basic colors
A New ProblemA New Problem
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Imaging Geometry
Shadow lies along epipolar ray
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Shadow lies along epipolar ray,
Epipole and Shadow are on opposite sides of the edge
Imaging Geometry
m
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Shadow lies along epipolar ray,
Shadow and epipole are on opposite sides of the edge
Imaging Geometry
m
14
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Depth Edge Camera
Light epipolar rays are horizontal or vertical
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Normalized
Left / Max
Right / Max
Left Flash
Right Flash
Input U{depth edges}
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Normalized
Left / Max
Right / Max
Left Flash
Right Flash
Input U{depth edges}Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Normalized
Left / Max
Right / Max
Left Flash
Right Flash
Input U{depth edges}
Negative transition along epipolar ray is depth edge
Plot
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Normalized
Left / Max
Right / Max
Left Flash
Right Flash
Input
Negative transition along epipolar ray is depth edge
Plot U{depth edges}Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
% Max compositemaximg = max( left, right, top, bottom);
% Normalize by computing ratio imagesr1 = left./ maximg; r2 = top ./ maximg;r3 = right ./ maximg; r4 = bottom ./ maximg;
% Compute confidence mapv = fspecial( 'sobel' ); h = v';d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobeld2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel
• Issues– Baseline between camera and flash– Specularities– Flash non-uniformity, area light source
• Comprehensibility– Sharp edges not captured
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Change DetectionChange Detection
Before After
Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Change DetectionChange DetectionMitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera
Change DetectionChange Detection
Reconstructed from gradient field of new depth edges
16
Ramesh Raskar, Computational Illumination
Computational IlluminationComputational Illumination•• Presence or AbsencePresence or Absence
–– Flash/NoFlash/No--flashflash•• Light positionLight position
–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)
Synthetic aperture photographyusing an array of mirrors
• 11-megapixel camera (4064 x 2047 pixels)• 18 x 12 inch effective aperture, 9 feet to scene • 22 mirrors, tilted inwards → 22 views, each 750 x 500 pixels
Computational IlluminationComputational Illumination•• Presence or AbsencePresence or Absence
–– Flash/NoFlash/No--flashflash•• Light positionLight position
–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)
• Decode signals from blinking LEDs + image – Sony ID Cam– Phoci
• Motion Capture Cameras
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher
Mitsubishi Electric Research Labs (MERL), Cambridge, MA
R F I R F I GG LampsLamps : : Interacting with a SelfInteracting with a Self--describing World via describing World via Photosensing Wireless Tags and ProjectorsPhotosensing Wireless Tags and Projectors
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Radio Frequency Identification Tags (RFID)Radio Frequency Identification Tags (RFID)
microchip
Antenna
No batteries,
Small size,
Cost few cents
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
WarehousingWarehousing RoutingRouting
Library Library
Baggage handling Baggage handling
CurrencyCurrency
Livestock trackingLivestock tracking
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Micro Controller
Memory Computer
READER
Micro Controller
Memory
Conventional Passive RFID
19
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Tagged Books in a LibraryTagged Books in a Library
IdEasy to get list of books in RF range
No Precise Location DataDifficult to find if the books in sorted
order ?Which book is upside down ?
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Where are boxes with Where are boxes with Products close to Expiry Date ?Products close to Expiry Date ?
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
READER
Micro Controller
RF Data
Memory
Conventional RFID
Computer
READER
Micro Controller
RF Data
Light
Memory
Photosensor
Computer
Conventional RF tag
Photo-sensing RF tag
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
READER
Projector Micro
ControllerRF Data
Light
Memory
Photosensor
Computer
Photosensor ?
Compatible with RFID size and power needs
Projector ?
Directional transfer,AR with Image overlay
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
c. Tags respond via RF, with date and precise (x,y) pixel location. Projector beams ‘O’ or ‘X’ at that location for visual feedback
c. Tags respond via RF, with date and precise (x,y) pixel location. Projector beams ‘O’ or ‘X’ at that location for visual feedback
a. Photosensing RFID tagsare queried via RFa. Photosensing RFID tagsare queried via RF
d. Multiple users can simultaneously work from a distance without RF collision
d. Multiple users can simultaneously work from a distance without RF collision
b. Projector beams a time-varying pattern unique for each (x,y) pixel which is decoded by tags
b. Projector beams a time-varying pattern unique for each (x,y) pixel which is decoded by tags
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
20
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
RFID(Radio Frequency Identification)
RFIG
(Radio Frequency Id and Geometry)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Prototype TagPrototype Tag
RF tag + photosensor
RF tag + photosensor
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
21
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
PatternMSB
PatternMSB
PatternMSB-1PatternMSB-1
PatternLSB
PatternLSB
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
00 11 11 00 00 X=12X=12
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang,
Dietz, Lee, Leigh, WillwacherR F I G Lamps
Visual feedback of 2D positionVisual feedback of 2D position
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
Ramesh Raskar, Computational Illumination
Computational IlluminationComputational Illumination•• Presence or AbsencePresence or Absence
–– Flash/NoFlash/No--flashflash•• Light positionLight position
–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
A Night Time Scene: Objects are Difficult to Understand due to Lack of Context
Dark Bldgs
Reflections on bldgs
Unknown shapes
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Enhanced Context :All features from night scene are preserved, but background in clear
‘Well-lit’ Bldgs
Reflections in bldgs windows
Tree, Street shapes
22
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Background is captured from day-time scene using the same fixed camera
Night Image
Day Image
Result: Enhanced Image
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Mask is automatically computed from scene contrast
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
But, Simple Pixel Blending Creates Ugly Artifacts
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Pixel Blending
Our Method:Integration of
blended Gradients
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Nighttime imageNighttime image
Daytime imageDaytime image
Gradient fieldGradient field
Importance Importance image Wimage W
Fina
l res
ult
Fina
l res
ult
Gradient fieldGradient field
Mixed gradient fieldMixed gradient field
GG11 GG11
GG22 GG22
xx YY
xx YY
II11
I2
GG GGxx YY
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Reconstruction from Gradient Field
• Problem: minimize error |∇ I’ – G|• Estimate I’ so that
G = ∇ I’
• Poisson equation
∇ 2 I’ = div G
• Full multigridsolver
II’’
GGXX
GGYY
23
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Video Enhancement using Fusion
– Video from fixed cameras• Improve low quality InfraRed video using high-quality visible video• Fill in dark areas, enhance change in intensity• Output style: better context
– Current Demo• Fusion of Night video and Daytime image
Original Video FrameEasy-to-understand
Non-photorealistic (NPR)Image or Video
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Details– Combine day and night time images
• Night videos have low contrast, areas with no detail• Same camera during day can capture static information• Dark areas of night video are replaced to provide context• Moving object (from night) + Static scene (from day)
Day time Photograph
Combine pixels depending on context, image and temporal gradient
Night time Video
(or Photo)
Enhanced Night Video (or Photo)
with context
Modified Surveillance CameraModified Surveillance Camera
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Video Enhancement
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Enhanced videoNote: exit ramp, lane dividers,
buildings not visible in original night video, but clearly seen here.
Overview of Process
Day time image: By averaging 5 seconds of day video
Original night time traffic camera 320x240 video
Mask frame (for frame above): Encodes pixel with intensity change
Input
Output
Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005
Programmable 4D Illumination field + Time + Wavelength
Programmable 4D Illumination field + Time + Wavelength
Ramesh Raskar, Computational Illumination
Computational Illumination:Computational Illumination:Programmable 4D Illumination Field + Time + WavelengthProgrammable 4D Illumination Field + Time + Wavelength
•• Presence or Absence, Duration, BrightnessPresence or Absence, Duration, Brightness–– Flash/NoFlash/No--flashflash
•• Light positionLight position–– MultiMulti--flash for depth edgesflash for depth edges–– Programmable dome (image reProgrammable dome (image re--lighting and matting)lighting and matting)