USING VISION SENSORS FOR INNOVATIVE HCI - CHALLENGES FOR THE NEXT INTERACTIVE MEDIA - Hideki Koike Tokyo Institute of Technology
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
USING VISION SENSORS FORINNOVATIVE HCI
- CHALLENGES FOR THE NEXT INTERACTIVE MEDIA -
Hideki Koike Tokyo Institute of Technology
OUTLINE
➤ Vision-based HCI
➤ Challenges for the next interactive media
➤ Environment: Interactive Surface on the Water
➤ Contents: Spectating Ball Sports from Ball’s POV
➤ Advertisement: Unaware Gaze Navigation
➤ Lessons learned from our prototype development
VISION-BASED HCI
NEW I/O DEVICES HAVE BEEN CHANGING HCI
IBM PC Macintosh DataLand
CUI GUI PUI
I: keyboard
O: character display
I: keyboard, mouse
O: bitmap display
I: gesture recognition/ speech recognition
O: large display/wearable
ADVANCES IN COMPUTER VISION
➤ Hardware:
➤ Camera: smaller, higher resolution, and less expensive.
➤ GPU: faster and less expensive.
➤ Software:
➤ Open software (e.g. OpenCV, AR Toolkit)
wired unwired
SPECIAL USE CAMERAS
➤ Example:
➤ far/near infrared, polarization, high-speed, depth, mocap, action cam, eye tracking, light field
➤ Can see what human cannot
➤ Enable natural and intuitive UI
far infrared camera polarization camera depth camera
CHALLENGES FOR THE NEXT INTERACTIVE MEDIA
AQUATOP DISPLAY: INTERACTIVE SURFACE ON THE WATER
PROTOTYPE
WATER SPECIFIC GESTURES
Poking Stroking
Scooping Hitting
WATER SPECIFIC GESTURES
poking as selecting
stroking as swiping
scooping as dragging
COMPARING TO TRADITIONAL UI
selection page scroll transition exit app.
PC’s GUI mouse click slider click&drag win. close
button
Touch UI touch swipe touch&drag home button
AquaTop poke stroke scoop hit
IMMERSION BY FLUID SIMULATION
AQUATOP DISPLAY [2012]
EVALUATION: VISIBILITY
0.26 g/l
1.10 g/l
EVALUATION: POINTING ACCURACY
poking touching
AWARD
➤ Laval Virtual 2013: Grand Prix
➤ METI Innovative Technologies 2013: Special Award (Culture)
➤ TV Tokyo World Business Satellite 2013: Toretama Award & Kickstarter Award
➤ and more.
BALLCAM!: SPECTATING BALL SPORTS FROM
BALL’S POV
BALL’S POV
➤ 360 degree omnidirectional video streaming
➤ Ball’s Point of View
time
PROTOTYPE
➤ What is difficult?
➤ The ball is spinning, sometimes very fast (> 600rpm)!
➤ How to stabilize the video from spinning ball?
first prototype: The iBall [2011]
second prototype
BALLCAM! [2013]
OVERVIEW
Image Subsampling
View Expansiontriplet image sixlet image
Motion Interpolation
Distortion Correction
STEP 1: CORRECTING LENS DISTORTION AND ROLLING SHUTTER
P : back-projection
K: calibration matrixR(t): camera rotation
Gauss-Newton method
STEP 2: SELECTING IMAGES USING INTENSITY
STEP 3: EXTENDING VIEWS USING MULTIPLE FRAMES
frame t+1
frame t+2
triplet image N
sixlet image
triplet image N+1
frame t
STEP 4: IMAGE INTERPOLATION
imageinterpolation
Mf: forward mapping
Mb: backward mapping
MI: Identity mapping
BALLCAM! [2013]
OUR GOAL
➤ Rugby World Cup in Japan [2019]
➤ Tokyo Olympic [2020]
➤ World Cup Soccer [2022]
20162011 2012 2013 2014 2015
UNAWARE GAZE NAVIGATION
EXPERIMENTS
EXPERIMENT 1
no blur(σ=0) σ=5
experiment image
heat map
EXPERIMENT 1
no blur(σ=0) σ=5
experiment image
heat map
EXPERIMENT 1
blur strength and the time to fixation
EXPERIMENT 2
EXPERIMENT 2
USAGE SCENARIO
LESSONS LEARNED FROM OUR PROTOTYPE DEVELOPMENT
HOW IDEAS BECOME TO A UNIQUE PRODUCT
new(?) idea
rapid prototyping
advanced implementation
unique product
NEW IDEA(?)
➤ In most cases, it is not.
➤ Survey other work carefully.
➤ OK. It seems to be a new idea!
➤ However, 10,000 people in the world are inspiring the same idea at this moment.
RAPID PROTOTYPING
➤ Let’s make a prototype ASAP.
➤ The idea may be not so interesting as you thought.
➤ The idea may be more interesting than you thought.
➤ OK. It seems to be interesting!
➤ However, 100 people in the world are making the same prototype at this moment.
ADVANCED IMPLEMENTATION
➤ Let’s do serious implementation.
➤ Work harder.
➤ However, you may face very difficult technical issues.
➤ Use the wisdom of predecessors.
➤ You may be only one who finally develop the system!
CONCLUSION
➤ Vision-based HCI is a key to the next HCI.
➤ Effective use of special use cameras enables natural and intuitive interaction.
➤ Challenges for the next interactive media:
➤ Environment, Contents, and Advertisement.
➤ Rapid prototyping and advanced implementation make your ideas unique.
REFERENCES➤ AquaTop Display
➤ Y. Matoba et.al.: “AquaTop Display: A True Immersive Water Display System”, Proc. on ACM SIGGRAPH 2013, Emerging Technologies, Article 4, 2013.
➤ H. Koike et.al.: “AquaTop Display: Interactive Water Surface for Viewing and Manipulating Information in a Bathroom”, ACM Int’l Conf. on Interactive Tabletops and Surfaces (ITS’13), pp.155-164, 2013.
➤ BallCam
➤ R.Funakoshi et.al.: “Activity-Aware Video Stabilization for BallCam”, Proc. on ACM User Interface Software and Technology (UIST’16 Adjunct), pp.197-198, 2016.
➤ R. Funakoshi et.al.: “Synthesizing Pseudo Straight View form A Spinning Camera Ball”, 7th Augmented Human Int’l Conf. (AH’16), pp.30:1-30:4, 2016.
➤ K. Horita et.al.: “Experiencing the Ball’s POV for Ballistic Sports”, Proc. of the 4th Augmented Human Int’l Conf. (AH’13), pp.128-133, 2013.
➤ Unaware Blur
➤ H. Hata et.al.: “Visual Guidance with Unnoticed Blur Effect”, Int’l Working Conf. on Advanced Visual Interfaces (AVI2016), pp.28-35, 2016.
Related Documents
