4/27/16 1 How the Kinect Works Subhransu Maji Slides credit: Derek Hoiem, University of Illinois Photo frame-grabbed from: http://www.blisteredthumbs.net/2010/11/dance-central-angry-review T2 Kinect Device Kinect Device illustration source: primesense.com What the Kinect does Get Depth Image Estimate Body Pose Application (e.g., game)
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
4/27/16
1
HowtheKinectWorks
SubhransuMajiSlidescredit:DerekHoiem,UniversityofIllinois
Photo frame-grabbed from: http://www.blisteredthumbs.net/2010/11/dance-central-angry-review
T2
KinectDevice
KinectDevice
illustration source: primesense.com
WhattheKinectdoesGet Depth Image
Estimate Body Pose
Application (e.g., game)
4/27/16
2
HowKinectWorks:Overview
IRProjector
IRSensorProjected Light Pattern
Depth Image
Stereo Algorithm
Segmentation, Part Prediction
Body Pose
Part1:Stereofromprojecteddots
IRProjector
IRSensorProjected Light Pattern
Depth Image
Stereo Algorithm
Segmentation, Part Prediction
Body Pose
Part1:Stereofromprojecteddots
1. Overviewofdepthfromstereo
2. Howitworksforaprojector/sensorpair
3. StereoalgorithmusedbyPrimesense(Kinect)
DepthfromStereoImagesimage 1 image 2
Dense depth map
Some of following slides adapted from Steve Seitz and Lana Lazebnik
4/27/16
3
DepthfromStereoImages• Goal:recoverdepthbyfindingimagecoordinatex’thatcorrespondstox
f
x x’
Baseline B
z
C C’
X
f
X
x
x' Potential matches for x have to lie on the corresponding line l’.
Potential matches for x’ have to lie on the corresponding line l.
StereoandtheEpipolarconstraint
x x’
X
x’
X
x’
X
SimplestCase:Parallelimages• Imageplanesofcamerasare
paralleltoeachotherandtothebaseline
• Cameracentersareatsameheight
• Focallengthsarethesame• Then,epipolarlinesfallalong
thehorizontalscanlinesoftheimages
Basicstereomatchingalgorithm
• Foreachpixelinthefirstimage– Findcorrespondingepipolarlineintherightimage– Examineallpixelsontheepipolarlineandpickthebestmatch
– TriangulatethematchestogetdepthinformaXon
4/27/16
4
Depthfromdisparity
f
x’
Baseline B
z
O O’
X
f
zfBxxdisparity ⋅=′−=
Disparity is inversely proportional to depth.
x zf
OOxx =′−′−
Basicstereomatchingalgorithm
• Ifnecessary,recXfythetwostereoimagestotransformepipolarlinesintoscanlines
• Foreachpixelxinthefirstimage– Findcorrespondingepipolarscanlineintherightimage– Examineallpixelsonthescanlineandpickthebestmatchx’– Computedisparityx-x’andsetdepth(x)=fB/(x-x’)
Matching cost
disparity
Left Right
scanline
Correspondencesearch
• Slideawindowalongtherightscanlineandcomparecontentsofthatwindowwiththereferencewindowinthele[image
• Matchingcost:SSDornormalizedcorrelaXon
Left Right
scanline
Correspondencesearch
SSD
4/27/16
5
Left Right
scanline
Correspondencesearch
Norm. corr
Resultswithwindowsearch
Window-based matching Ground truth
Data
Addconstraintsandsolvewithgraphcuts
Graph cuts Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
Before
Failuresofcorrespondencesearch
Textureless surfaces Occlusions, repetition
Non-Lambertian surfaces, specularities
4/27/16
6
DotProjecXons
http://www.youtube.com/watch?v=28JwgxbQx8w
DepthfromProjector-SensorOnlyoneimage:Howisitpossibletogetdepth?
Projector Sensor
Scene Surface
Samestereoalgorithmsapply
Projector Sensor
Example:Bookvs.NoBookSource: http://www.futurepicture.org/?p=97
4/27/16
7
Example:Bookvs.NoBookSource: http://www.futurepicture.org/?p=97
Region-growingRandomDotMatching1. Detectdots(“speckles”)andlabelthemunknown2. Randomlyselectaregionanchor,adotwithunknown
deptha. WindowedsearchvianormalizedcrosscorrelaXonalong
scanline– Checkthatbestmatchscoreisgreaterthanthreshold;ifnot,
markas“invalid”andgoto2b. Regiongrowing
1. Neighboringpixelsareaddedtoaqueue2. Foreachpixelinqueue,iniXalizebyanchor’sshi[;thensearch
smalllocalneighborhood;ifmatched,addneighborstoqueue3. Stopwhennopixelsarele[inthequeue
3. Stopwhenalldotshaveknowndepthoraremarked“invalid”
http://www.wipo.int/patentscope/search/en/WO2007043036
ProjectedIRvs.NaturalLightStereo
• WhataretheadvantagesofIR?– WorksinlowlightcondiXons– Doesnotrelyonhavingtexturedobjects– Notconfusedbyrepeatedscenetextures– Cantailoralgorithmtoproducedpaeern
• Whatareadvantagesofnaturallight?– Worksoutside,anywherewithsufficientlight– Useslessenergy– ResoluXonlimitedonlybysensors,notprojector
• DifficulXeswithboth– Verydarksurfacesmaynotreflectenoughlight– SpecularreflecXoninmirrorsormetalcausestrouble
Part2:Posefromdepth
IRProjector
IRSensorProjected Light Pattern
Depth Image
Stereo Algorithm
Segmentation, Part Prediction
Body Pose
4/27/16
8
Goal:esXmateposefromdepthimage
Real-Time Human Pose Recognition in Parts from a Single Depth Image Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake CVPR 2011
Goal:esXmateposefromdepthimage
RGB Depth Part Label Map Joint Positions
http://research.microsoft.com/apps/video/default.aspx?id=144455
Challenges• LotsofvariaXoninbodies,orientaXon,poses• Needstobeveryfast(theiralgorithmrunsat200FPSontheXbox360GPU)
Pose Examples
Examples of one part
Extractbodypixelsbythresholdingdepth
4/27/16
9
Basiclearningapproach
• Verysimplefeatures
• Lotsofdata
• Flexibleclassifier
Getlotsoftrainingdata• Captureandsample500Kmocapframesofpeoplekicking,driving,dancing,etc.
• Get3Dmodelsfor15bodieswithavarietyofweight,height,etc.
• Synthesizemocapdataforall15bodytypes
Bodymodels Features• Differenceofdepthattwooffsets
– Offsetisscaledbydepthatcenter
4/27/16
10
PartpredicXonwithrandomforests• Randomizeddecisionforests:collecXonofindependentlytrainedtrees
• Eachtreeisaclassifierthatpredictsthelikelihoodofapixelbelongingtoeachpart– Nodecorrespondstoathresholdedfeature– TheleafnodethatanexamplefallsintocorrespondstoaconjuncXonofseveralfeatures
– Intraining,ateachnode,asubsetoffeaturesischosenrandomly,andthemostdiscriminaXveisselected
JointesXmaXon
• JointsareesXmatedusingmean-shi[(afastmode-findingalgorithm)
• Observedpartcenterisoffsetbypre-esXmatedvalue
Results
Ground Truth
Moreresults
4/27/16
11
Accuracyvs.NumberofTrainingExamples UsesofKinect• Mario:hep://www.youtube.com/watch?v=8CTJL5lUjHg• RobotControl:
hep://www.youtube.com/watch?v=w8BmgtMKFbY• Captureforholography:
hep://www.youtube.com/watch?v=4LW8wgmfpTE• Virtualdressingroom:
hep://www.youtube.com/watch?v=1jbvnk1T4vQ• Flywall:
hep://vimeo.com/user3445108/kiwibankinteracXvewall• 3DScanner:
hep://www.youtube.com/watch?v=V7LthXRoESw
Tolearnmore
• Warning:lotsofwronginfoonweb
• GreatsitebyDanielReetz:hep://www.futurepicture.org/?p=97
• Kinectpatents:hep://www.faqs.org/patents/app/20100118123hep://www.faqs.org/patents/app/20100020078hep://www.faqs.org/patents/app/20100007717
Nextweek
• Tues– ICESforms(important!)– Wrap-up,proj5results
• Normalofficehours+feelfreetostopbyotherXmesonTues,Thurs– Trytostopbyinsteadofe-mailexceptforone-lineanswerkindofthings
• FinalprojectreportsdueThursdayatmidnight
• Friday– FinalprojectpresentaXonsat1:30pm– Ifyou’reinajamforfinalproject,letmeknowearly
Related Documents
