Kinect & Leap Motion Overview Nicholas Gillian Responsive Environments, MIT Media Lab Thursday, September 5, 13
Kinect & Leap Motion Overview
Nicholas Gillian
Responsive Environments, MIT Media Lab
Thursday, September 5, 13
Leap Demo
Thursday, September 5, 13
Kinect Libraries, APIs & Tools
• OpenNI SDK
• Openframeworks
• Microsoft Official Kinect SDK
• Synapse
Thursday, September 5, 13
Microsoft Official Kinect SDK
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
Advantages
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
• Joint Orientation in addition to Joint Position
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
• Joint Orientation in addition to Joint Position
• Tracks up to 6 users, 2 users full skeleton tracking
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
• Joint Orientation in addition to Joint Position
• Speech Recognition
• Tracks up to 6 users, 2 users full skeleton tracking
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
• Joint Orientation in addition to Joint Position
• Speech Recognition
• Face tracking (feature tracking - i.e. mouth, nose, eyes, etc. and direction estimation)
• Tracks up to 6 users, 2 users full skeleton tracking
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Calibration-free skeleton tracking
Advantages
• Standing and seated tracking modes
• Joint Orientation in addition to Joint Position
• Speech Recognition
• Face tracking (feature tracking - i.e. mouth, nose, eyes, etc. and direction estimation)
• Good examples (c++, c#) and user documentation
• Tracks up to 6 users, 2 users full skeleton tracking
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
AdvantagesThursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
Disadvantages
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Only works on Windows
Disadvantages
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Only works on Windows
Disadvantages
• Very CPU intensive
Thursday, September 5, 13
Microsoft Official Kinect SDK
• http://www.microsoft.com/en-us/kinectforwindows/
• Only works on Windows
Disadvantages
• Very CPU intensive
• Skeleton tracking can fail drastically if not used in ideal scenarios
Thursday, September 5, 13
Open NI
Thursday, September 5, 13
Open NI
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Open NI
Advantages
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Open NI
• Cross Platform
Advantages
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Open NI
• Cross Platform
Advantages
• Works with a large number of depth sensors
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Open NI
• Cross Platform
Advantages
• Works with a large number of depth sensors
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
• Robust Skeleton Tracking
Thursday, September 5, 13
Open NI
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
• Cross Platform
Advantages
• Works with a large number of depth sensors
• Robust Skeleton Tracking
• Calibration-free skeleton tracking
Thursday, September 5, 13
Open NI
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
AdvantagesThursday, September 5, 13
Open NI
Disadvantages
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Open NI
Disadvantages
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
• Installation requires a few independent drivers, libraries, etc.
Thursday, September 5, 13
Open NI
Disadvantages
• Installation requires a few independent drivers, libraries, etc.
• API requires some time to digest
• http://openni.org/
• ...an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware.
Thursday, September 5, 13
Synapse
Thursday, September 5, 13
Synapse
• http://synapsekinect.tumblr.com/
Thursday, September 5, 13
Synapse
Advantages
• http://synapsekinect.tumblr.com/
Thursday, September 5, 13
Synapse
• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)
Advantages
• http://synapsekinect.tumblr.com/
Thursday, September 5, 13
Synapse
• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)
Advantages
• http://synapsekinect.tumblr.com/
• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed
Thursday, September 5, 13
Synapse
• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)
Advantages
• http://synapsekinect.tumblr.com/
• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed
Disadvantages
Thursday, September 5, 13
Synapse
• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)
Advantages
• http://synapsekinect.tumblr.com/
• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed
Disadvantages
• Skeleton tracking needs calibration pose
Thursday, September 5, 13
Synapse
• Packages up the Open NI tracking libraries for you and sends the tracked positions of a user’s joint positions out to another application via OSC (Open Sound Control)
Advantages
• http://synapsekinect.tumblr.com/
• Very easy to use - OS X version requires no setup, Windows version needs drivers to be installed
• Only works with “Kinect XBox360” hardware
Disadvantages
• Skeleton tracking needs calibration pose
Thursday, September 5, 13
Openframeworks
Thursday, September 5, 13
Openframeworks
• http://www.openframeworks.cc/
• https://github.com/ofTheo/ofxKinect
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• https://github.com/ofTheo/ofxKinect
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
• Supports multiple Kinects
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
• Supports multiple Kinects
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
• Supports multiple Kinects
• Works on OS X
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
Disadvantages
• Supports multiple Kinects
• Works on OS X
• Uses the libfreenect library
Thursday, September 5, 13
Openframeworks
• No skeleton tracking
Advantages
• http://www.openframeworks.cc/
• Easiest way to access the raw depth image from the Kinect
• https://github.com/ofTheo/ofxKinect
Disadvantages
• Supports multiple Kinects
• Works on OS X
• Uses the libfreenect library
Thursday, September 5, 13
Kinect Gesture Demo
Thursday, September 5, 13
Kinect Gesture Demo
Built Using:
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
Built Using:
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
• NITE library for calibration-free skeleton tracking
Built Using:
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
• NITE library for calibration-free skeleton tracking
• OSC (Open Sound Control) to send skeleton data to external apps
Built Using:
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
• NITE library for calibration-free skeleton tracking
• OSC (Open Sound Control) to send skeleton data to external apps
Built Using:
OSC
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
• NITE library for calibration-free skeleton tracking
• OSC (Open Sound Control) to send skeleton data to external apps
Built Using:
• Openframeworks as a wrapper for drawing and networking
OSC
Thursday, September 5, 13
Kinect Gesture Demo
• OpenNI API to interface with the Kinect
• NITE library for calibration-free skeleton tracking
• OSC (Open Sound Control) to send skeleton data to external apps
Built Using:
• Openframeworks as a wrapper for drawing and networking
• Gesture Recognition Toolkit: www.nickgillian.com/software/grt
OSC
Thursday, September 5, 13
www.nickgillian.com/09-05-13.html
Slides & Software Links
Thursday, September 5, 13