Willem Mestdagh - Kevin Cordemans
Jun 26, 2015
Willem Mestdagh - Kevin Cordemans
PapersMS Research
PlayAnywhere:Compact Interactive Tabletop Projection-Vision System []
Andy Wilson
Thinsight:Integrated Optical Multi-touch Sensing through Thin Form-factor Displays []
Bill Buxton, Steve Hodges, ...
PlayAnywherePlayAnywhere:
A Compact Interactive Tabletop Projection-Vision System
Andy Wilson at Microsoft Research, Redmond
2005
Introduction
Looks great...
But... Calibration? Installation? Portability? Cost?
Mainstream Consumer AcceptanceMulti-touch: new sensing and display technologies
Ubiquitious computing
PortabilityEase of installation
Ease of use: any surface?Calibration
Cost
Devise an easy-to-use, low-cost multi-touch system...
PlayAnywhereFront-projected computer vision-based
interactive table system which uses a new projecting technology to obtain an exceptionnaly self-contained form factor
Requirements:Quick set up: no calibration beyond factoryNo “installation”Usable on any flat surfaceCompact but still displaying and
sensing over a large surface (form factor)General purpose(Low cost)
Future?Canesta projection keyboard
PlayAnywhereWhat is it?Known problems in existing systemsHow does it work?
PlayAnywhere configuration Projector Camera and IR Illuminant
Image Processing Image Rectification Touch and Hover Analysis of shadow PlayAnywhere Visual Code Flow Move
Computer Vision Pros and Cons
Known problems in existing systems
PlayAnywhereWhat is it?Known problems in existing systemsHow does it work?
PlayAnywhere configuration Projector Camera and IR Illuminant
Image Processing Image Rectification Touch and Hover Analysis of shadow PlayAnywhere Visual Code Flow Move
Computer Vision Pros and Cons
PlayAnywhere Configuration
All-in-One
IR illuminant
Projector (with mirrors)
Camera with IR-passing-filter
Prototype
ProjectorNEC WT600 DLP
40’’ diagonale afbeeldingVoordelen van de plaats van de projector
Gevaarlijke plaatsing projector is vermeden Wanneer camera en projector samen zijn geplaatst
geen nood om de camera te hercalibreren bij verplaatsing van het toestel
• Nadelen Beeldkwaliteit kan niet gegarandeerd worden,
want projectie-oppervlak is niet gekend
Camera en IR IlluminantDe projector belicht de scene met zichtbaar
stralingDe illuminant belicht de scene met infrarode
stralingCamera ontvangt alleen gereflecteerde
infrarode stralen dankij de infrared-passing-filter die aan de camera bevestigd is(Camera blind voor reflecties van de projectorstraling)
How does it work?Image Processing
Preparation: Image RectificationComputer Vision Techniques: Analysis of
ShadowHover and TouchFlow MovePlayAnywhere Visual Code
Image Rectification (1)
Remove distortion via standard bilinear interpolation techniques
Parameters for transformations need only be determined once (factory):Lens specs known Fixed topology device-plane (assumption)
Setup Good thing Bad thing
Wide lens Maximize usable surface Creates barrel distortion
Oblique position of camera
Optimizes form factor (compact device)
Produces projective distortion
Image Rectification (2)
Image Rectification (3)Input image and projected image are in a one-to-
one correspondence, meaning:A rectangular object on the surface appears as a
rectangular object in the input image at the same (scaled) coordinates.
Rectified image suited for computer vision techniques
Limitation: objects further away from the unit will appear at a lower resolution(Minimum effective resolution is less than that of the acquired image)
Hover and TouchDetect touch without relying on special
instrumentation of the surface, so that the device may operate on any available flat surface (goal)
PlayAnywhere: exploite change in appearance of shadows as an object approaches the surface
No tracking of fingers, ...
Hover and Touch
thresholding operation
HoverCandidate finger positions:highest on each of the distinct shadows which enter the scene from the bottom of the image
TouchAnalysis of shadow shapes:threshold width of finger shadow at a location slightly below the topmost point
Hover and TouchTo limit false positives, candidate finger must lie on
a shadow that extends to the bottom of the image
Objects on table can corrupt touch detection by mimicing arm shadows, meaning their shadows extends to the
bottom of the image
Presently, only one finger per hand(more fingers requires more spohisticated analysis)
Precision limited by image resolution
Flow MoveSupport direct manipulation of virtual objects
(doc’s, photo’s)Gestures that are natural to a specific activity
E.g. resizing and rotating photo’s as if they were lying on a table
PlayAnywhere: Optical flow techniquesBased on local texture on the movig objectNo absolute position-based tracker, no widgetsInstead simple statistics summarizing the flow
field, computing only relative motion information
Optical Flow (1)Optical Flow computed from the most recently
acquired rectified image I(t) and the previous image I(t-1)
Mask images with grid
Image (t-1) Image (t)
For each (x,y) on a regular grid, the integer vector quantity (dx, dy) is determined, such that: image patch P centered on (x, y) at time t-1 most closely matches image patch P’ centered on (x+dx, y+dy) at time t.
Image patches are compared by computing pixelwise absolute differences (low values close match).
If image patch P closely matches image patch P’, there is a good chance that the two patches are representing the same content, shifted to a different gridpoint in the interval.
(dx, dy) gives the direction of this shift.
Optical Flow (2)
Image(t)(x+dx, y+dy)Image patch P’
Image(t-1)(x, y)Image patch P
Resulting vector
Optical Flow (3)For a given patch in the image, select (dx, dy) that minimizes
Vector field can be used as an argument in a function to infer meaning from user’s
gesture
Resulting vector field
PlayAnywhere Visual CodesOok mogelijk te werken met infrarood
systemen in objecten mogelijkheid tot werken met barcodes (derde vb met barcode)
Resultaat http://youtube.com/watch?v=YRfWhUnYyY8
Nu : Thingsight
Beschrijving“Optical sensing system, fully integrated into
a thin form factor display”Scherm met multi-touch systeem2 mogelijkheden om multitouchsystemen uit
te werken : - een arrangement van electroden op het
oppervlak- werken met cameras : voor of achter het
scherm
VergelijkingBeide Touch-algoritme op basis van
schaduwinval van handenBeide Handig te gebruiken en te verplaatsenPrincipe werking ongeveer gelijk
verschil : thinsight werkt met elektroden op het scherm, playanywhere werkt met camera’s boven het scherm
Beiden hebben de mogelijkheid tot werken met externe objecten met een infrarode bron
Voordeel bij thinsight : geen oppervlakte voor apparaat nodig