Web Motion: Motion Detection on the Web

Web MotionMotion Detection on the Web

Bill Fisher | @fisherwebdev | #webmotion

Monday, April 1, 13

People Get Ready• Slides: http://bill.hotstudiodev.com/webmotion.pdf

• Code: https://github.com/fisherwebdev/webmotion

• Me: @fisherwebdev

• Us: #webmotion

Monday, April 1, 13

Post-Touch

Monday, April 1, 13

Monday, April 1, 13

Monday, April 1, 13

Monday, April 1, 13

The Leap Motion Device

Monday, April 1, 13

• Very sensitive and high resolution motion detection

• Somewhat prone to noisy and inconsistent data

• 10,000 devices sent to developers

• APIs for many platforms, including the Web

Leap Motion

Monday, April 1, 13

• Current price is at $79.99

• App store called Airspace will include web apps

• Shipping May 13th

• Sold at BestBuy on May 19th

Leap Motion

Monday, April 1, 13

~ 10 - 20 mm

~ 400 - 600 mm

Monday, April 1, 13

Competitive Landscape

Monday, April 1, 13

Monday, April 1, 13

• XBox only. IE10 may be controlled with a XBox LIVE Gold account. Windows integration is coming.

• Current integration with IE10 is focused on voice control and is janky at best.

• Most use cases involve being far away from the sensor, and are based on gross motor movements.

• Complete loss of data within 16 cm

• Price has now dropped to $99

Kinect by Microsoft

Monday, April 1, 13

Monday, April 1, 13

• Leap Motion clone, but DIY friendly

• Cameras are PS3 Eyes, all parts are o!-the-shelf

• Company history shows dedication to open source

• Kickstarter project - $140 for a fully-assembled device

• Drivers and SDK not yet released, Windows will be first

• June 2013: estimated shipping date

• http://duo3d.com/

DUO by Code Laboratories

Monday, April 1, 13

Monday, April 1, 13

• Embedded tech, not currently a consumer device

• Uses time-of-flight instead of a camera-based solution

• Native APIs are provided by third party partners, but no Web API yet.

• http://www.pmdtec.com/

CamBoard Pico by PMD

Monday, April 1, 13

Monday, April 1, 13

Monday, April 1, 13

• PrimeSense technology is at the heart of the Kinect

• Relationship with Microsoft is apparently dissolving

• Now o!ering the tech as a standalone consumer device and as an embedded solution

• No apparent interest in directly supporting the Web. OpenNI is the only API.

• http://www.primesense.com/

Capri by PrimeSense

Monday, April 1, 13

Monday, April 1, 13

Monday, April 1, 13

• Worn as an armband, connected via Blue Tooth

• Detects electrical impulses of muscle movement

• Thalmic claims MYO will “just work” on with existing operating systems and the Web (mouse events???)

• Backed by Y Combinator - $149 consumer price

• Summer 2013: developer API

• Late 2013: shipping first 25,000 units

• https://getmyo.com/

MYO by Thalmic Labs

Monday, April 1, 13

Consistent Themes

Often Positioned as Embedded

Tiny or Discreet Form Factor

Often Capable of Depth

Games and Entertainment

Ignore the Web

Monday, April 1, 13

leapjs API

Monday, April 1, 13

How it works• Leap.app starts a local Node server on port 6437, and

leapjs connects to it via Web Sockets.

• leapjs continuous loop uses requestAnimationFrame for 60fps by default. You may opt to use a faster loop.

• Each cycle of the loop has a reference to the current frame, which contains most of the useful data.

Monday, April 1, 13

leapjs objects• Many objects can express the concept of validity.

• The high level objects of the API are Controllers, Frames, Pointables / Hands, and Gestures.

• Source code is highly readable and well commented with JSdoc annotations.

• Modular code, so you can trim it down.

Monday, April 1, 13

leapjs controller• Most central component of the API, but typically you

don’t use it much.

• Manages the loop. “this” inside the loop is the controller.

• Has references to other parts of the system, and this is the first place to look if you are looking for something not in the Frame.

Monday, April 1, 13

leapjs frames• Contains all pointables, fingers, tools, hands, gestures

for the current loop / rAF cycle.

• ID and timestamp

• Also contains global translation data, like scaleFactor and rotationMatrix

• CircularBu!er is a collection of the previous 200 frames

Monday, April 1, 13

leapjs pointables• Generic object that may be fingers or tools

• ID, validity, direction, length, width, velocity

• Tracking multiple pointables can be inconsistent. What happens when your pointable vanishes?

Monday, April 1, 13

leapjs hands• In some ways great, but even more inconsistent than

pointables. Multiple hands are particularly problematic.

• ID, validity, position, direction, palm vector, collection of pointables.

• sphereCenter and sphereRadius based on the cup of the hand.

• translations and matrix data

Monday, April 1, 13

leapjs hands

Monday, April 1, 13

leapjs gestures• The most high-level part of the API. This is a good

thing.

• Screen Tap, Key Tap, Swipe and Circle.

• Screen Tap and Key Tap feel silly, Swipe takes practice, but Circle is reliable.

• More gestures might be coming soon.

Monday, April 1, 13

Demos and Code

Monday, April 1, 13

Where are We Going?

Monday, April 1, 13

Pointer Events

Monday, April 1, 13

Web Standards• Pointer Events are a W3C submission from Microsoft.

Already in IE10, and they will be in WebKit soon.

• A replacement and abstraction of Mouse and Touch, with attention to stylus input.

• Independent events for each pointer, unlike Touch.

• Pointer Events Working Group has reached the “Last Call Working Draft”

• Could they be used for 3D input also?

Monday, April 1, 13

“It's unclear what the units would be for all this stu!. ... As soon as you're dealing with tracking real world stu!, it's in some di!erent coordinate system. If you can bring it back to screen space, you're doing it with some weird transform.”

-- Boris Smus at EdgeConf 2013

Monday, April 1, 13

“Considering this for the V1 spec would pose a risk to its current trajectory.”

“We should let a thousand flowers bloom.”

Monday, April 1, 13

Will They Bloom?

Monday, April 1, 13

Motion vs. Touch• Touch is extremely intuitive.

• Physical contact does matter.

• Motion tracking su!ers even more for avatar issues than Mouse. We will need a lot of extra feedback.

Monday, April 1, 13

Use Cases• Games

• Medical

• Gloved Interaction

• Kitchens, or other cases of wet or dirty hands

• Extremely casual or blind interaction: car stereos

• Jet Packs!

Monday, April 1, 13

Conclusions

Monday, April 1, 13

Leap Motion FTW• The data coming from the Leap Motion device is still

quite unstable in the leapjs API.

• A single pointable or hand is more stable than multiple ones.

• High level abstractions are a good idea.

• Consider using the leap as a controller for interaction engines, rather than directly mapping the data to the UI.

• It’s a lot of fun!

Monday, April 1, 13

Monday, April 1, 13

Links

Monday, April 1, 13

Demos and Documentation• https://github.com/leapmotion/leapjs

• http://leapmotion.github.com/leapjs/index.html

• https://developer.leapmotion.com/

• http://dewmap.com/experiment/gilt-leap/index.html

• https://github.com/rtilton1/leapjs/tree/master/examples/ironman2

• http://leapgamer.com/

Monday, April 1, 13

Don’t forget...

GE Party

6:30pm

111 Minna Street

beer!

Monday, April 1, 13

Thanks!

Bill Fisher | @fisherwebdev | #mowebapps

Monday, April 1, 13