Snooping Keystrokes with mm- level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu † , Yan Wang † , Gorkem Kar # , Yingying Chen † , Jie Yang ‡ , Marco Gruteser # † Dept. of ECE, Stevens Institute of Technology, USA # Winlab, Rutgers University, USA ‡ Dept. of CS, Florida State University, USA DAISY Data Analysis and Information SecuritY Lab MobiCom 2015 Paris, France Sep. 9 – 11, 2015
23
Embed
Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Snooping Keystrokes with mm-level Audio Ranging on a Single Phone
Presenter: Jian Liu
Jian Liu†, Yan Wang†, Gorkem Kar #, Yingying Chen†, Jie Yang‡, Marco Gruteser#
†Dept. of ECE, Stevens Institute of Technology, USA# Winlab, Rutgers University, USA
‡ Dept. of CS, Florida State University, USA
DAISYData Analysis and Information SecuritY Lab
MobiCom 2015Paris, France
Sep. 9 – 11, 2015
2
Mobile Device Hardware Advancements
Stereo recording
High definition audio capabilities targeted at audiophiles Microphone arrays (stereo recording & noise canceling) 4x improvement in audio sampling rates
Such advancements have security concerns
Audio chipset: 192kHz playback and recording
Mic-1
Mic-2 Mic-3
3
The Results of the Advancements
Facilitating fine-grained localization based applications Tracking speakers in multiparty conversations Sensing touch interaction on surfaces around mobile devices
Eavesdropping keystrokes without suspicion Adding malware into the target user’s phone with microphone access Leaving a phone near a keyboard of the target user
Adding malware with Mics access Leaving a phone
Be careful of these nearby phone!They can hear your typing!
4
5
Related Work
typing has to satisfy English language pattern
require a-priori labeled training data
Label each key for training Multiple recording devices
Linguistic context Training with labeled dataMulti-phone to be placed
around
6
Our Approach
No involvement of multiple phones
No linguistic model
No labeled training (e.g., without any cooperation of the target user)
7
Available Audio Components in a Single Phone
Stereo recording of two microphones High sampling rate
Mic1
Mic2
Stereo recording
Mic3
Stereo 1
Stereo 2 Noise Cancellation
8
What can we obtain from the dual-Mic in a phone to snoop keystrokes?
9
`
Mic1Mic2t1=tt2=t+Δtt1=t’t2=t’+Δt’
Distance difference Δd1
Feature 1: Time Difference of Arrival (TDoA)
Most of the keys could be differentiated by the TDoAs
Theoretical TDoA
Measured TDoA
S L
Distance difference Δd2
Limits of Measured TDoA Dual-Microphone TDoA can only identify a group of keystrokes
Mic1
Mic2
Mic1
Mic2
r1 r2
d
Half hyperbola of constant TDoA
TDoA = Δtr1 – r2 = Δt·v
10
Measured TDoA has the Resolution Limited by Sampling Rate
Sampling by ADCSpeed of sound: 343m/s
Feature 2: Acoustic Signature Keystrokes of different keys sound different MFCCs (Mel-frequency Cepstral Coefficients) can be used to
discriminate sounds of different keys
11
MFCC of key ‘E’ MFCC of key ‘D’ MFCC of key ‘X’
12
We can combine TDoA and acoustic signatures to identify each keystroke!
System Overview
13
A Set of Keystrokes
Keystroke Detection & Segmentation
TDoA DerivationKey Groups Generation
Theoretical TDoA
Theoretical Key Groups
Grouping of Keystrokes
Acoustic Signature Extraction
MFCC-based Clustering with in a
Group
Cluster-based Letter Labeling
Identified Keystrokes
Theoretical Key Groups
14
A theoretical key group – keys having similar theoretical TDoAs
One theoretical key group
Q WA
E R T Y U I O PS D F G H J K L
Z X C V B N M
SortingLink any pair of keys whose
theoretical TDoAs are too similar
Keystroke Grouping
15
[sp − 5ms, sp + 100ms], where sp is starting point
A Set of Keystrokes
Keystroke Detection &
Segmentation
TDoA Derivation
Theoretical Key Groups
Grouping of Keystrokes
Cross-correlation approach
Theoretical key groups
g1 g2 g3 gn
Input keystrokes
Clustering within Each Group & Labeling
16
MFCC features: same key shows higher correlation, while different keys
present lower correlation
Theoretical TDoA
Acoustic Signatur
e Extractio
n
MFCC-based Clustering with
in a Group
Cluster-based Letter Labeling
Identified Keystrokes
A theoretical key group:keystrokes of multiple
keys with similar TDoAs
clustering
Each cluster contains keystrokes of the
same key
Keystroke clusters
1t 2t 3tMean TDoAs
Finding Minimum Distance
Theoretical TDoA
E D X Labeling
Evaluation
17
How robust is the system recovering keystrokes from different keyboards?
What is the performance with different sampling rates?
How does the placement of the phone influence the snooping accuracy?
Higher sampling rate improves the recognition accuracy
48 96 1920.5
0.6
0.7
0.8
0.9
1k=1 k=2 k=3
Top-k
Accu
racy
Sampling Rate (kHz)
22
ConclusionShow that a single phone can recover keystrokes by exploiting mm-level TDoA ranging and fine-grained acoustic features
Develop a training-free approach on a single phone that does not require a linguistic model to snoop keystrokes
Extensive experiments with different keyboards & microphones sampling rates demonstrate that our work could achieve sufficient accuracy for keystroke snooping