Kevin A. Shaw, Ph.D. Chief Technology Officer March 30 th , 2014 Santa Clara, California, USA 2014 Embedded Vision Member Meeting [email protected] R2
Aug 12, 2015
Kevin A. Shaw, Ph.D.
Chief Technology Officer
March 30th, 2014 Santa Clara, California, USA
2014 Embedded Vision Member Meeting [email protected]
R2
• Consumer games – More stable attitude
• Augmented Reality (AR) – Needs improvement, better accuracy
• Indoor Navigation – Need better accuracy; lower power
• Hyper photography – Super resolution; intraframe deblur
• Robotics – Visual odometry to detect egomotion
– Always need better accuracy
6/2/2014 2
• Construction equipment – Perimeter safety
• Context awareness – Understanding users better
• Change from mobile to wearables – Shift from mostly-pocket to always-visual
– Digital eyewear makes a big difference
• Natural interfaces – Using the same wealth of information as
humans do to understand the world
6/2/2014 3
• How to solve some of limitations of vision systems using some of these sensors
• Some limitations: – Lack of metric scale
• The Dollhouse problem
– Pose stability
• Feature point robustness
– Power consumption
• Suitable for mobile products?
6/2/2014 5
6/2/2014 6
Accelerometer
Gyroscope
Magnetometer
Barometer
Proximity
Amb. Light sensor
GPS
WiFi
Bluetooth
GSM/CDMA Cell
NFC
Camera (front)
Touch screen Camera (back)
20 sensors!
Humidity Colorimeter
CO2/VOC gas Microphones x 3
Fingerprint Thermal ambient
• What are they? – MEMS are tiny silicon structures
– MicroElectroMechanical Systems
– Leveraging semiconductor toolsets Bosch
6/2/2014 8
• Measures dynamic acceleration – Very versatile.
– Result: Vibration, tilt, & position
– Low power 1-10uA
6/2/2014 9
Hydrogen atom = 1Å
MEMS displacement Resolution ~ 0.1Å
• Used to measure rotation
• Absolute orientation reference for gyroscope
• Power is moderate: 300-1500uA
6/2/2014 10
• Gyros don’t measure angle! – They measure the rate of change
– Body rates: rotation about each axis
• Rates are relative to starting point – Depend on Accel/Mag for start
• Integrate to get angle
• Power is high: 1-5mA or more
Gyro SEM 𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0
6/2/2014 11
• Measures air pressure
• Air pressure indicates altitude
• Not good for absolute
• Resolution of 1-2 feet
• Low power: 1-5uA
Melexis.com
6/2/2014 12
• We want to know Position and Attitude (pose). – Inertial and Vision systems can each help find this
6/2/2014 14
𝑃𝑜𝑠𝑒 = 𝑝 , 𝑞 = 𝑥, 𝑦, 𝑧, 𝑞0, 𝑞1, 𝑞2, 𝑞3,
• Position seems easy: double integrate
• Angle is only a single integration.
• No problem!
6/2/2014 15
𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = 𝑣 𝑡 𝑑𝑡 + 𝑝0
𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 = 𝑎 𝑡 𝑑𝑡 + 𝑣0
𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0
• Noise Random walk – Integrating noise causes a linear walk.
6/2/2014 Sensor Platforms Proprietary and Confidential Information 16
Measured Acc error [m/s/s] for an accelerometer when sitting still.
• Noise Random walk – Integrating noise causes a linear walk.
6/2/2014 Sensor Platforms Proprietary and Confidential Information 17
dttatv )()(
Measured)( ta
6/2/2014 Sensor Platforms Proprietary and Confidential Information 18
dttvtp )()(
dttatv )()(
Measured)( ta
• Dead Reckoning – Over the past few years
significant progress has been made
– Stable solutions with consumer grade sensors
– Graph (right) uses stock sensors on Galaxy S3
– Pedestrian walking constraints aid solution
6/2/2014 20
Waypoints Measured Path
• Visual odometry – Visually tracking position (camera pose) through a space
• Tracking feature points is a powerful way to understand the world
6/2/2014 21
• Limitations – Can't tell size of objects: i.e. scale
• Doll house problem
– Hard to map points between frames over time
– Need cohesion over long time scales
– Need robustness in dark spaces & low-texture surfaces
– Need maintain vision lock (can get lost due to motion-blur)
– Enormous computational load (ready for mobile?!?)
• Can we aid the solution with more sensors?
6/2/2014 22
• Attitude estimates allow anticipated search space – Reduce computation for FP correspondence
• Power reduction with reduced/opportunistic frame rates – Can trust INS when not moving
– Or when spatial diversity is low
• Vision System can be turned on only when high resolution navigation/alignment is needed – PDR to Statue, VS for precise AR overlay
6/2/2014 23
• Need to find metric scale – Without it the world makes no sense
• Monocular / Binocular issues
6/2/2014 24
If you were a vision system, which one is real?
• Need some way to get extra scale information
– Binocular cameras (like humans do; monocular is cheaper)
– Reference object (hard to keep in sight; i.e. Ikea catalog)
– Location estimates (GPS is not available indoors)
– Mapped landmark (best but hard; humans do this)
– Inertial estimates (tend to drift, but commonly available)
– Depth cameras
6/2/2014 25
• State estimation from visual & inertial sources – Combined measurements & physical models
6/2/2014 26
],,,,ˆ,[ gak bbvqpx
• State estimation from visual & inertial sources
• Kalman filter – Recursive linear quadratic estimator
– Combined measurements & physical models
6/2/2014 27
],,,,ˆ,[ gak bbvqpx
– Closely coupled KF – Solve it all at once
• Computationally expensive (Order n2 or n3)
– Loosely coupled KF
• Estimate visual delta-pose
• Estimate inertial delta-pose
• Combine with KF
• But loose cross-correlations
6/2/2014 28
– Recursive vs Batch solution
• Kalman Filters are “recursive”; only one frame deep
– Batch solution:
• Compute solution across multiple frames
• Bundle Adjustment with well selected keyframes
• Much more stable, but computationally expensive
• Asynchronous to frame updates; non-uniform keyframes
6/2/2014 30
– Consumer sensors are cheap
• Free: they are already in place
• Contextual & Motional
– Need additional constraints
• Contextual constraints
– Are you moving?
» Easy for robots
» Harder for humans
» Not only for robots anymore
• Motion constraints
– Wheel constraints help (only on flat ground with no slippage)
– Pedestrian constraints (track steps, distance & direction)
6/2/2014 31
Demonstration 1: Ideal Vision & PDR Scenario
● Vision-only: 5%, PDR-only: 3%, Fused: 1.5%
● Use case: head-mounted AR, vision mapping
Initial scale estimate
Sensor Platforms, Inc. Confidential and Proprietary
32
End Start
Demonstration 2: Non-Ideal Vision/Ideal PDR
● Vision-only: 15%, PDR-only: 2%, Fused: 1%
● Use case: AR over large spaces
Vision Outage Initial scale estimate
Vision Outage
Sensor Platforms, Inc. Confidential and Proprietary
33
Start
End
Demonstration 3: Non-Ideal Vision/Non-Ideal PDR
● Vision-only: 8%, PDR-only: 15%, Fused: 5%
● Use case: intensive gaming
Initial scale estimate
PDR Outage
Sensor Platforms, Inc. Confidential and Proprietary
34
Start End
• Optical Image Stabilization (OIS) – Optical (in-lens) with inertial attitude tracking; gyro based
• Super resolution – Across multiple frames stabilized with pose tracking
• Deblurring – Within frame pose-detection and deconvolution
6/2/2014 36
– Across multiple frames stabilized with pose tracking
– Inertial data stabilizes the solution
6/2/2014 37