Automated HDR Image Capturing System Angela Buttrick, Nate Naber, John Olender.

Automated HDR Image Automated HDR Image Capturing SystemCapturing System

Angela Buttrick, Nate Naber, John Angela Buttrick, Nate Naber, John OlenderOlender

AgendaAgenda

Project OverviewProject Overview

System DescriptionSystem Description

User InterfaceUser Interface

Operating BoundariesOperating Boundaries

Feasibility/Potential ProblemsFeasibility/Potential Problems

Test StrategyTest Strategy

Project OverviewProject Overview

HDR image – capture a much wider dynamic HDR image – capture a much wider dynamic range of intensities than a regular LDR imagerange of intensities than a regular LDR image

Project OverviewProject Overview

1.1. Remote Control Unit Remote Control Unit (RCU) takes pictures(RCU) takes pictures

2.2. Pictures are then Pictures are then uploaded to the PCuploaded to the PC

3.3. Software package then Software package then handles extracting handles extracting images and converting images and converting each series of pictures each series of pictures into HDR imagesinto HDR images

4.4. These HDR images run These HDR images run through an algorithm to through an algorithm to produce a final pictureproduce a final picture

System Description - CameraSystem Description - Camera

Still Image: JPEG (Exif 2.2 compliant)Image Compression

Auto/ ISO 50/ 100/ 200/ 400ISO Sensitivity

15 - 1/1,500 sec.Shutter Speed

/2.8/ 7.1 (W), f/3.9/ 10 (T)Maximum Aperture

3.2x Digital Zoom (Max 6.4x digital zoom when combined with optical zoom)

Digital Zoom

5.4 - 10.8mm, f/2.8 - 3.9 zoom lens (35mm film equivalent: 35 - 70mm)Focal Length

High Resolution 3.2 Megapixel CCDMegapixels

System Description - RCUSystem Description - RCU

Remote Control Unit (RCU):Remote Control Unit (RCU):– HCS12 microcontrollerHCS12 microcontroller– USB host controllerUSB host controller– Buttons/LEDs for control and Buttons/LEDs for control and

system statussystem status

– RCU will include a standard-RCU will include a standard-size Type-A female USB size Type-A female USB port, provided by the USB port, provided by the USB host controllerhost controller

– Battery four AA rechargeable Battery four AA rechargeable batteries used to power all batteries used to power all contained components. contained components.

text

Rem

ote

Con

tro

l with

H

CS

12u

C

Running

Ready

ErrorON/OFF

START

STOPBattery Pack

4 AA DuraCell~4.8 Volts, 1800mAh

+-

System Description – USB Host System Description – USB Host Controller Controller

MCU-to-USB host MCU-to-USB host controller module, controller module, specifically designed for specifically designed for adding USB host adding USB host functionality to functionality to microcontrollers microcontrollers

Allows the camera to Allows the camera to recognize the RCU as a recognize the RCU as a USB hostUSB host

Makes the USB protocol Makes the USB protocol transparent to the HCS12 transparent to the HCS12

FTDI VNC1L MCU to USB Host Controller (VDIP1 Module)

VDIP1 SchematicVDIP1 Schematic

System Description – PTPSystem Description – PTP

Picture Transfer Protocol (PTP) – used for Picture Transfer Protocol (PTP) – used for communication between HCS12<->Canon communication between HCS12<->Canon CameraCamera

Canon cameras use a series of command Canon cameras use a series of command and response blocks to send packets back and response blocks to send packets back and forthand forth

PTP Send / Receive PacketsPTP Send / Receive Packets0000000: 0000000: xx xx xx xxxx xx xx xx yy yy yy yyyy yy yy yy - 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 000000010: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 000000010: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 000000020: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 000000020: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 000000030: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 0000030: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 0000040: 02 00 00 00 0000040: 02 00 00 00 UUUU 00 00 cc VVcc VV - - xx xx xx xx SS SS SS SSxx xx xx xx SS SS SS SS0000050: ... (payload/arguments)0000050: ... (payload/arguments)

0000000: 00 00 00 00 yy yy 00 00 00 00 00 00 00 00 00 00 0000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000040: 02 00 00 00 UU 00 cc VV - xx xx xx xx SS SS SS SS0000050: kk kk kk kk

System Description – HDR System Description – HDR Software PackagesSoftware Packages

PfsPfscalibrationcalibration: : allows for the calibration of the camera allows for the calibration of the camera used to take the pictures and the recovery of HDR used to take the pictures and the recovery of HDR images from the multiple exposure images collected images from the multiple exposure images collected using the RCUusing the RCU

PfsPfstools:tools: provides a set of tools for reading and writing provides a set of tools for reading and writing HDR images. This will allow for the storage and HDR images. This will allow for the storage and previewing of raw HDR imagespreviewing of raw HDR images

PfsPfstmo: tmo: allows for tone-mapping operators to be applied allows for tone-mapping operators to be applied to HDR images, producing images which may be to HDR images, producing images which may be displayed on a LDR display devicedisplayed on a LDR display device

contains the gradient domain compression schemecontains the gradient domain compression schemetone mapping operatorstone mapping operators

Pfstmo ExamplesPfstmo ExamplesMantuik '06 Tone Mapping Operator Fattal '02 Tone Mapping Operator

User InterfaceUser Interface• Clean, easy-to-use UI

– Allows user to calibrate his/her camera for optimal HDR images

– Allows user to select a range of images to be processed

– Converts LDR->HDR image by invoking pfstools, pfscalibration, and pfstmo

– Displays resulting image to the user

Operational BoundariesOperational Boundaries

Limiting FactorsLimiting Factors– Portability – Camera can only extend up to 3 meters from the Portability – Camera can only extend up to 3 meters from the

RCURCU– Image Quality – Necessary to have stable Camera and still Image Quality – Necessary to have stable Camera and still

imageimage– Power – Limited by battery power, batteries increases portabilityPower – Limited by battery power, batteries increases portability

Four AA batteries can supply ~4.8V. 1800mAh to each component Four AA batteries can supply ~4.8V. 1800mAh to each component in the RCU requiring powerin the RCU requiring powerTypical operating conditions: Typical operating conditions:

– LEDs draw 12mA (one LED lit)LEDs draw 12mA (one LED lit)– Buttons draw 12mAButtons draw 12mA– HCS12 microcontroller draw about 100mAHCS12 microcontroller draw about 100mA– VNC1L USB host controller draw about 25mAVNC1L USB host controller draw about 25mA– Total current draw of 149mATotal current draw of 149mA– ~12 hours of use with constant use~12 hours of use with constant use

Battery condition and temperature Battery condition and temperature

Analytical ComponentAnalytical ComponentTime for HDR Automated Flow:Time for HDR Automated Flow:– Experimentation to derive time to save image to flash, Experimentation to derive time to save image to flash,

shutter speed’s effect on capture timeshutter speed’s effect on capture time

Number of photos taken per HDRNumber of photos taken per HDR– Experimentation of the optimal number to produce Experimentation of the optimal number to produce

best imagebest image– Affects total memory availableAffects total memory available– Max # of photos is Dictated by mega pixel Max # of photos is Dictated by mega pixel

size/memory card sizesize/memory card size

Analytical ComponentAnalytical ComponentWhy do we need so many exposures?Why do we need so many exposures?

Range of exposures too narrow (missing very light / very dark information from the scene)

Not enough exposures to cover the range (light/dark details in the scene emphasized too highly)

Feasibility/Potential ProblemsFeasibility/Potential Problems

• Controlling shutter speeds via the RCU• PTP/IP and Canon PTP• User Interface:

• Compiling set of images into a single HDR image• Completely transparent to the user• Invoking processes through a GUI and piping I/O quietly• Open-source software tools with the software UI

• Drop-in USB Host Controller• Battery Power

• Low-power circuitry as well as low-power modes• WAIT and STOP power-saving modes of HCS12

Testing Strategies – UI/SoftwareTesting Strategies – UI/Software

UI needs to start up properly + allow user to select a UI needs to start up properly + allow user to select a number of images onlynumber of images only

Camera calibration function must be tested with the Camera calibration function must be tested with the ability to load/save camera calibration infoability to load/save camera calibration info

Image processing must be tested with a list of images Image processing must be tested with a list of images loaded and camera successfully calibratedloaded and camera successfully calibrated

Fail-over testing will be performed in depth in order to Fail-over testing will be performed in depth in order to prevent a number of user errors from occurringprevent a number of user errors from occurring

Testing Strategies - RCUTesting Strategies - RCU SetupSetup

Layout Physical ConnectionsLayout Physical Connections Establish/Verify Canon CaptureEstablish/Verify Canon Capture

Phase 1Phase 1 Verify Camera is recognized by PCVerify Camera is recognized by PC Establish/Verify Canon Capture camera connection to PCEstablish/Verify Canon Capture camera connection to PC Send simple instruction to Camera, i.e. Zoom 2XSend simple instruction to Camera, i.e. Zoom 2X Check for completionCheck for completion Send shutter control from PCSend shutter control from PC Take pictures at various shutter speedsTake pictures at various shutter speeds Remove/Download/Verify Images used different shutter speedsRemove/Download/Verify Images used different shutter speeds

Phase 2 – Same as Phase 1 but with the HCS12 instead of the PCPhase 2 – Same as Phase 1 but with the HCS12 instead of the PC

Phase 3Phase 3 Verify Buttons properly send instructions to the CameraVerify Buttons properly send instructions to the Camera Verify LEDs show correct status during HDR capturingVerify LEDs show correct status during HDR capturing

Questions?Questions?