MIT Media Lab | Camera Culture Shoot Now, Relight Later Illumination Multiplexing with Lock-In Time of Flight Sensors Achuta Kadambi, Ayush Bhandari, Refael Whyte, Adrian Dorrington, Ramesh Raskar [email protected] How Can We Relight a Photograph? In this paper we propose a fusion of two popular contexts, time of flight range cameras and illumination multiplexing. Time of flight cameras are a low cost, consumer-oriented technology capable of acquiring range maps at 30 frames per second. Such cameras have a natural connection to conventional illumination multiplexing strategies as both paradigms rely on the capture of multiple shots and synchronized illumination. While previous work on illumination multiplexing has exploited coding at millisecond intervals, we repurpose sensors that are ordinarily used in time of flight imagstrategiesing to demultiplex via nanosecond coding. ICCP 2014 Illumination Multiplexing Illumination multiplexing is not a new problem, but requires fast captures. Figure from Schechner 2007 Time of Flight 3D Sensors Time of Flight 3D cameras are an emerging camera technology. Such technology forms the basis for the new Kinect. Time of Flight Operating Principle Operation of ToF range cameras. The time difference of arrival between the emitted and received code is encoded in the phase offset. Thus, phase encodes depth. Nanosecond vs Millisecond Coding Case 1: Conventional High Speed Multiplexing Case 2: Using a lock-in ToF sensor allows for nanosecond coding. Case 3: Optimizing the Nanosecond Codes is the goal. Illumination Multiplexing Illumination multiplexing is not a new problem, but requires fast captures. Figure from Schechner 2007 Demonstration: Relighting Post-Capture Demonstration: Scene Relighting. Demonstration: Multiplexing Wavelength Demonstration: Color ToF camera by multiplexing RGB lights. Hardware Prototype We validate our technique with a hardware prototype. It consists of an FPGA, Lock-in Sensor, and Solid State Illumination sources. www.media.mit.edu/~achoo/demux