This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Depth Aware Finger Tapping on Virtual Display
Ke Sun†, Wei Wang†, Alex X.Liu†‡, Haipeng Dai†Nanjing University†, Michigan State University‡
Mobisys’18 June 13, 2018
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 1 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Motivation
Traditional tapping-based interaction:Require physical devicesLimit the freedom of user hands
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 2 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Motivation
Tapping-in-the-air:Hands are free to interact with other objectsDepth measurements provide different levels feedback
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 3 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Limitation of Prior Arts
Customized depth-cameras
Low accuracy:Centimeter-level accuracy (without different levels feedback)High latency:Low frame rate and high computational requirements
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 4 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Problem Statement
Can we suppport tapping-in-the-air without depth-cameras?
and meet these design goalsHigh accuracy (mm-level)Low latency (< 20 ms)Different levels feedback (finger bending angle)Low computational cost (works on mobile devices)
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 5 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Basic Idea
Dolphin navigation:
Ultrasound + Vision
Use ultrasound based sensing, along with one COTS mono-camera,to enable 3D tracking of user fingers with high frame rate.
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 6 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Basic Idea
Use ultrasound based sensing, along with one COTS mono-camera,to enable 3D tracking of user fingers with high frame rate.
Light-weight computer vision algorithm to locate the fingertips in 2DAdaptive Skin Segmentation:Otsu’s method calculates the optimal thresholdHand detectionFind the centroid of the palm (Distance Transform)Fingertip Detection for tapping gestureExtreme-points-based scheme
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 10 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Ultrasound Signal Phase Extraction
Microphone
Speaker
Emit18~22kHzCWsoundsignal
Receivesoundsignal
Soundsignaldownconversion
Soundsignalphasechangemeasurement
Detectthestartofthefingertappingaction
Ultrasound Signal Phase Extraction Camera
Receiveframe Handdetection
Fingertipsdetection
Fingertip Localization
Tapping Detection and Tapping Depth Measurement
Confirmthetappingactionbasedonfinitestatemachine
TappingDepthMeasurement
Keystroke Localization
Keystrokelocalizationbasedonthedepth
measurement
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 11 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Ultrasound Signal Phase Extraction
Phase-based distance measurementMeasure phase changes caused by the movement16 single frequencies (17 ∼ 22 kHz) linear regression
I (normalized)700 900 1100 1300 1500 1700
Q (n
orm
aliz
ed)
200
400
600
800
1000
1200
1400
Pushing hand Tapping finger I (normalized)
1000 1050 1100 1150 1200
Q (n
orm
aliz
ed)
900
950
1000
1050
1100
Tapping finger
Tapping finger
Challenge:Phase changes caused by the finger movements is much smaller.Multipath interference in finger movements is much more significant.
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 12 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Ultrasound Signal Phase Extraction
Time (millisecond)0 500 1000 1500
I/Q (n
orm
aliz
ed)
800
850
900
950
1000
1050
1100
1150
IQExtreme PointFake Extreme Point
Peak and Valley EstimationFind the peak and valley
Avoid the error-prone step of static vector estimationExclude the fake extreme points:
”FingerInterval”: the magnitude gap of the finger”SpeedInterval”: the speed of the finger
Future: use modulated signal to locate the absolute distance andexclude other distance dynamic multipath
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 13 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Tapping Detection and Tapping Depth Measurement
Microphone
Speaker
Emit18~22kHzCWsoundsignal
Receivesoundsignal
Soundsignaldownconversion
Soundsignalphasechangemeasurement
Detectthestartofthefingertappingaction
Ultrasound Signal Phase Extraction Camera
Receiveframe Handdetection
Fingertipsdetection
Fingertip Localization
Tapping Detection and Tapping Depth Measurement
Confirmthetappingactionbasedonfinitestatemachine
TappingDepthMeasurement
Keystroke Localization
Keystrokelocalizationbasedonthedepth
measurement
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 14 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Finger Motion State
”Moving state”–Moves their finger to the keyAudio: Difficult to build the modelVideo: Easy to track the fingers
”Locating state”–Keeps their finger on the target key position brieflyVideo: Difficult to perceiveAudio: Easy to detect the short pause
”Tapping state”–”Tapping down state” & ”Tapping up state”Video: Difficult to measureAudio: Easy to measure the depth information
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 15 / 26
Tapping a neighboring key”Locating state” –> ”Tapping state”
Tapping the same key”Tapping state”
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 16 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Finger Tapping Detection
1 Audio to detect that the ”tapping state”Utilize the high sampling rate (48 kHz) –> Low latencyUtilize the sensitivity to the depth direction –> High accuracyUse only 1D information –> High false positive rates
2 Video to look back to the previous framesMeasure the duration of ”Moving state” and ”Locating state”Check the state machine to remove false alarms–> High robustnessMeasure the depth of finger tapping
3 Keystroke localizationCalculate the location of the fingertip during the ”Locating state”Determine the fingertip with the largest bending angle1-NN determine the pressed virtual key
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 17 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Measure the depth of finger tapping
Measure the bending angle of the fingerDeep finger tapping: camera-based model
Camera4(0,0,0) @(?) 2(0)
2(?) y
x
z
Gentle finger tapping: ultrasound-based model
Camera4(0,0,0)
x
z3@(?)
y
Microphone;(2), 0,0)
Speaker<(2*, 0,0)
0=(# 0 , & 0 , D 0 )0>(# ? , & ? , D ? )
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 18 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Implementation
Implemented on Android with NDKVideo: OpenCV C++Audio: C++
Camera
Speaker
Microphone
Microphone
Selfie stick
Video parameters used24 frame per second355 × 288 resolution
Audio parameters used48 kHz sampling rate512 samples per segment (10.7 ms)16 single frequencies (17 ∼ 22 kHz)
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 19 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Evaluation Setup
Three different use cases:Fix by selfie stickHold in handSet on the head by cardboard VR
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 20 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Result – Accuracy
0~20 20~40 40~60
Depth distance (mm)0
20
40
60
80
100
Tru
e Po
sitiv
e R
ate
(%)
Video+Audio Video
1 2 3 4 5 6 7 8
Users0
0.5
1
1.5
2
2.5
Fal
se n
egat
ive
rate
(%)
Fix by selfie stick Hold in hand Set on the head
Average movement distance error of 4.32mm (SD = 2.21mm)Average 98.4% accuracy with FPR of 1.6% and FNR of 1.4%Improve the gentle finger tappings accuracy from 58.2% to 97.6%
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 21 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Result – Latency
Time delay (millisecond)0 50 100 150
CD
F
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Average response latency of 18.08ms on commercial mobile phonesAverage response latency is 57.7ms smaller than the video-basedschemes
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 22 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Result – Case study
12.18 (SD=0.85) WPM for single-finger inputs13.1 (SD=1.2) WPM for multi-finger inputsAverage 95.0% TPR for 4-level feedbacks
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 23 / 26
Significant power consumption overhead of 48.4%More than 77% additional power consumption comes from speakerFuture: reduce the power consumption of the audio system
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 24 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Conclusion
Combining ultrasound sensing information andvision information to achieve tapping-in-the-air
Our system achieves design goalsHigh accuracy4.32 mm distance error, 98.4% accurayLow latency18.08 ms, 4x faster than video-based schemeDifferent levels feedbackbased on different bending angles of finger tappingsLow computational costworks on commercial mobile devices
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 25 / 26
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation System Design Experiments Conclusion
Q&A
Thank you!
Question?
Ke Sun, et al. Mobisys’18 June 13, 2018 Depth Aware Finger Tapping on Virtual Display 26 / 26