Eyes for Relighting Extracting environment maps for use in integrating and relighting scenes (Noshino and Nayar)
Dec 16, 2015
Eyes for Relighting
Extracting environment maps for use in integrating and
relighting scenes(Noshino and Nayar)
Previous Work Global illumination
– image based lighting
Mirror Ball environment maps
Eye Shape studies
Idea Find Eye in image by hand Find best fit ellipse Construct ellipsoid in 3 space from
ellipse, focal length, distance to camera and previous knowledge of eyes
Perform inverse ray tracing from camera to ellipsoid
Use law of reflection to project ray onto unit sphere and mark color.
Idea
Camera and Picture
Cornea
Environment Map
Implementation Mark points around
eye, find best fit ellipse.
Paper discusses using gradients to refine initial guess for ellipse parameters, though in most cases the results are satisfactory without.
Implementation Assume limited
perspective, so that the ratio between major and minor axis represents the angle of the plane it lies in with respect to the camera.
Assumes basically level camera.
Implementation Use a (t,theta)
parameterization for the ellipsoid, where t is the height, and theta the radial angle.
Send rays to focal point on camera and sample image (focal point in pixels)
Find reflection along normal
Implementation Issues (t,theta) is not uniformly
distributed, need far more rays for small t (where the ellipsoid is shallow) than large t
No guarantee that whole of sphere is covered, there may be holes.
Implementation Issues To solve these ideally: compute from
sphere to camera and sample image. However this is not trivial since rays are
dependent upon camera position and one would need to try several camera to eye rays to reach a certain point on the sphere.
The solution used just keeps adding rays until there are no gaps in the sphere image
Implementation Once sphere map is calculated, use in
global illumination as light source Assumes the map approximates higher
light level HDR since the reflection reduced small light to negligible levels. Still need to be careful to pick low levels for illumination
Additionally no attempt is made to remove iris texture. Need to pick dark eyes.
Implementation Rendered as a sphere map in
Cinema 4d 7.0 with new objects added.
Blade Runner Suppose we want to add a car to
the background of this image:
Blade Runner First we find the
best fit ellipse
Blade Runner Next we extract
the correct environment map.
(here rendered on a sphere)
Blade Runner Finally we render
the object using the computed environment map
(note that it is untextured)
Blade Runner Finally we composite (fiddle by
hand!)
Blade Runner
Sections not implemented The map can also be
used to unlight a model. Wave a light using the
light distribution found on the eyes, and assumption that skin is lambertian (dot product of normal and light direction) find a normal map.
Sections not implemented Since the lighting is
known and normals can be found, computing the base color is also possible.
All computations take place over set of images with different lighting so best fit matches of both normals and base color can be found.
Sections not implemented
Integration of the normals can be done as in previous work to find a 3d model which fits the lighting description.
This can also be texture mapped with the base texture, and lit as a lambertian surface with shadows.
Sections not implemented The surface can be inserted into a
new scene and lit according to the global illumination of that scene.
This illumination can of course come from eyes!
Sections not implemented