UbiTouch: Ubiquitous Smartphone TouchPads using Built-in … › foswiki › pub › Groups › WiNe › Research... · tempted to take advantage of built-in sensors to achieve the

UbiTouch: Ubiquitous Smartphone TouchPads usingBuilt-in Proximity and Ambient Light Sensors

Elliott Wen, Winston Seah, Bryan NgSchool of Engineering and Computer Science,

Victoria University of Wellingtonjiaqi.wen, winston.seah, [email protected]

Xuefeng Liu, Jiannong CaoDepartment of Computing,

The Hong Kong Polytechnic Univerisitycsxfliu, [email protected]

ABSTRACTSmart devices are increasingly shrinking in size, whichresults in new challenges for user-mobile interactionthrough minuscule touchscreens. Existing works to ex-plore alternative interaction technologies mainly rely onexternal devices which degrade portability. In this pa-per, we propose UbiTouch, a novel system that extendssmartphones with virtual touchpads on desktops usingbuilt-in smartphone sensors. It senses a user’s fingermovement with a proximity and ambient light sensorwhose raw sensory data from underlying hardware arestrongly dependent on the finger’s locations. UbiTouchmaps the raw data into the finger’s positions by utiliz-ing Curvilinear Component Analysis and improve track-ing accuracy via a particle filter. We have evaluate oursystem in three scenarios with different lighting condi-tions by five users. The results show that UbiTouchachieves centimetre-level localization accuracy and posesno significant impact on the battery life. We envisagethat UbiTouch could support applications such as text-writing and drawing.

INTRODUCTIONSmart devices provide various interaction interfaces,among which, touchscreens are the most prevalent one.Generally, a larger touch screen leads to better user ex-periences. Nevertheless, numerous smart devices areshrinking in sizes to improve portability, which resultsin new challenges for user-mobile interaction throughminuscule touchscreens. Particularly, it renders hand-writing input or drawing on those smart devices cumber-some. This difficulty motivated various researchers to ex-plore alternative interaction technologies. Notable sys-tems, e.g. RF-IDraw [24], TypingRing [17], Okuli [30],and OmniTouch [9], have been able to achieve very goodtracking accuracy. However, they rely heavily on periph-eral devices such as extra RFID tags and cameras, whichmay significantly limit the portability of smart devices.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected].

UbiComp ’16, September 12 - 16, 2016, Heidelberg, Germany

c© 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM.ISBN 978-1-4503-4461-6/16/09. . . $15.00

DOI: http://dx.doi.org/10.1145/2971648.2971678

To address the portability issue, researchers have at-tempted to take advantage of built-in sensors to achievethe same purpose. For example, Finger-in-Air [16] uti-lizes a smartphone camera to estimate users’ finger ges-tures. UbiK [25] leverages microphones on a smartphoneto detect keystroke locations, enabling text-input on asheet of paper where a keyboard outline is printed. UbiK[25] is solely designed for distinguishing different keysa user presses. It may not support applications thatrequire continuous finger tracking, for instance, hand-writing input or drawing. Despite the promising resultsobtained, vision-based systems [16] require the cameramodule to be continuously switched on. This could drainthe battery quickly and its performance may be nega-tively impacted by the low-light environment.

Nevertheless, these successful applications shed lighton our research and we pose the following question:“Is it possible to utilize built-in low-power sensors toconstantly localize and track a user’s finger, enablingtouchpad-like input experience?”. We answer this ques-tion by introducing UbiTouch, a novel system thatachieves the goal of augmenting smartphones with vir-tual touchpads by leveraging built-in smartphone sen-sors. Specifically, the smartphone’s proximity sensor(PS) and ambient light sensor (ALS) are utilized to sensethe movement. Meanwhile, the microphone and gyro-scope sensors are used to detect touch actions such astapping and dragging. These sensors are readily avail-able in almost every smart device and of low-power con-sumption.

The key challenge for UbiTouch lies in achieving fine-grained localization and tracking using sensors that areoriginally designed to provide coarse-grained informa-tion. An ALS is typically used to measure the luminanceof ambient lighting [13]. Readings from this sensor arenot directly related to a finger’s movement. Similarly, aPS is designed to detect the presence of nearby objectsand merely reports binary distance values representing“near” or “far” to smartphone operating systems [7],which does not benefits the tracking. In this paper, wetake a closer examination of the internal structures andunderlying mechanisms of the sensors [2] and uncoverthe potential for these sensors to be used to achieve ourgoal.

Specifically, we implement UbiTouch as a system withthree key components: finger tracking, touch action de-

tection, and run-time calibration. We introduce a fingerlocalization framework that adapts Curvilinear Compo-nent Analysis for mapping the raw sensor data to two-dimensional coordinates. It then estimates the fingermovement trajectories by utilizing a particle filter whichintegrates the historic finger locations to boost the track-ing accuracy. We also propose a touch action detectionalgorithm to capture touch tapping and dragging eventson ordinary surfaces by applying hypothesis testing tech-niques on audio signals from microphones. To providereliable results, we leverage the built-in gyroscope tocombat the environment noises. Additionally, we designa runtime calibration mechanism that takes advantageof extra run-time user feedback data to re-calibrate thetracking algorithm and combat minor variation of back-ground light conditions.

We consolidate the above techniques and implement aprototype system in an Android smartphone (LG Nexus5). We then carry out experiments in three scenarioscorresponding to different lighting conditions and by fiveindividuals. The results show that in general conditions,UbiTouch can detect and localize finger positions with amean localization error of 1.53 cm and a standard devi-ation of 0.42 cm. When UbiTouch works in conjunctionwith a hand-writing recognition application, it achieves acharacter recognition rate of 79%, comparable to a phys-ical touchpad. Another advantage of this system is thatit poses no significant impact on battery life of the smartphone.

The main contribution of this paper is to demonstratethe potential of built-in ambient light and proximity sen-sors to extend a smartphone touchscreen with virtualtouch input. The principles and methods explored inthis paper offer an alternative solution for interactingwith smart devices.

RELATED WORKIn this section, we review past work on exploring newinteraction technologies and position our contribution ofproviding a virtual touchpad without any extra sensors.

Extending smartphones’ input spaceOne problem with touchscreens is that the users’ fin-gers occupy valuable input space. Moreover, this prob-lem aggravates when devices are shrinking in size. Totackle this issue, a great number of research work shiftthe interaction away from the touchscreen to nearby ar-eas by augmenting smartphones with extra hardwaresuch as keyboards [11], touchpads [4], cameras [9], andother specially-designed sensors [29, 18]. Despite theremarkable achievement, these system inevitably de-grades devices’ portability. Considering that smart-phone platforms include increasingly sophisticated sen-sors, researchers start investigating the potential inputapproaches supported by the built-in smartphone sens-ing capabilities.

Built-in sensor based input approachesA great number of research works utilize the built-insmartphone sensors such as inertial, compass, micro-phone, and camera sensors to provide alternative inputapproaches. Research work [10] leverages inertial sen-sors and touch input to provide motion-enhanced touchgestures. MagiTact [12] utilizes built-in compass sensorto extend interaction space of small mobile devices. Sys-tems [20] detect users’ tap events by analyzing the au-dio signals captured by microphones. LucidTouch [26]enables users to control the applications by touchingthe back of the device using computer vision techniques.More recently, wireless technologies have drawn a greatamount attention of researchers. Some of them managedto extend smartphones input by utilizing wireless signalssuch as WiFi and RFID. For instance, Wigest [1] enablesusers’ gesture input to smart devices by sensing the WiFisignal strength.

Virtual touch surfaceAmong various interaction technologies, touching is stillmost straightforward and natural approach for smart de-vices. A number of researchers managed to augmentsmart devices with virtual touch surfaces. For instance,RF-IDraw [24] relies on a RFID ring to infer the fin-ger movement, allowing drawing in the air. TypingRing[17] leverages a ring equipped with motion sensors totrail a finger, enabling handwriting text-input experi-ence. Okuli [30] augments smartphones with a LEDlight array to capture the finger movement, serving asa virtual trackpad for the user. Besides, Canesta [21]and OmniTouch [9] achieve the same goal by utilizingexternal cameras and image process techniques.

Despite of the promising results achieved, these systemsinherit some limitations. Systems that rely on the ex-tra devices may severely degrade the portability of thesmart devices. More importantly, they also incur extracosts for the users. Recently, some researchers attemptto achieve finger tracking by purely leveraging the smart-phone built-in sensors. For instance, Finger-in-Air [16]leverages smartphone cameras to detect the movementof a finger. UbiK [25] takes advantage of microphones ona smartphone to detect keystroke locations, serving as avirtual keyboard. However, the vision-based system [16]continuously turning on cameras may drain the batteryswiftly and the performance may be negatively affectedby low-light environment. As for the UbiK [25], it issolely built to distinguish among different keys a userpresses, which may not be able to support applicationssuch as handwriting text-input and drawing that requirecontinuous finger movement tracking.

To address these issues, we propose a novel systemUbiTouch which uncovers the potential of the built-insmartphone sensors to extend smartphones with virtualtouchpads. Unlike existing works, it achieves continu-ous finger movement tracking while does not rely on anyexternal devices or cameras.

SYSTEM OVERVIEWUbiTouch extends touch-screen based devices with anexternal virtual touchpad on a common surface (e.g.,wooden desktop). Similar to a conventional touchpad,it accepts general touch actions including tapping anddragging.

The ALS and PS

X

Y

Figure 1: Typical system deployment.

Fig. 1 demonstrates the typical use case of UbiTouch.For the purpose of illustration, a 7 cm × 13 cm rectanglezone is marked on a desktop, serving as a virtual toucharea. An off-the-shelf Android smartphone (LG Nexus5) is then placed vertically on the top border of the zone,enabling the ALS and PS to sense the movement insidethe virtual touch area. Before running UbiTouch, an ini-tial training is needed, whereby the user moves his fingeralong some trajectories instructed by the system to gen-erate some training data. UbiTouch then learns a map-ping function from the training data through a neuralnetwork. It then uses the function to convert the subse-quent sensor readings into two-dimensional coordinates,enabling continuous tracking.

In this research, our design goal is for UbiTouch to workin a portable, accurate, and robust manner. UbiTouchshould only rely on smartphone built-in sensors toachieve centimetre-scale finger tracking and touch ac-tion detection. Meanwhile, UbiTouch should be resilientagainst minor changes of the ambient light. To achievethese goals, UbiTouch adopts a work-flow shown in Fig.2. It can be seen that the architecture of UbiTouch com-prises the following three major components: (i) fingertracking, (ii) touch action detection, and (iii) run-timecalibration and adaptation.

A PEEK AT TRAILING A FINGER USING ALS AND PSIn this section, we first provide some rationales abouthow raw readings from ALS and PS sensors could beutilized to track a user’s finger. Then we conduct somepreliminary experiments to determine the utility of theALS and PS information.

Figure. 3(b) demonstrates the underlying mechanismsof the ALS. It can be seen that the ambient light sen-sor consists of two photodiodes (CH0 and CH1) in twoseparate locations. The first photodiode is sensitive toboth visible and infrared light while the second photo-diode is primarily sensitive to infrared light. The fusionof the two readings is used to estimated luminance ofthe visible light. Assuming the settings of background

light sources (i.e., positions and brightness) remain un-changed, movement of an object or finger may alter thepath in which the light propagates. It then affects theintensity of the light received by the two photodiodes.Therefore, abundant information can be derived on howan object moves from the light intensity readings.

Meanwhile, the proximity sensor depicted in Fig. 3(c)also provides extra movement information. A LED emit-ter broadcasts a certain number of infrared pulses. Thepulses strike a nearby object (which is a user’s finger inour scenario) and get reflected to a receiver. The receivercounts the number of pulses and estimates the distancebased on a simple principle: the farther the object is, thefewer the number pulses received, since the infrared lighthas to travel in a longer path and loses more energy. Itis easy to see that variation of the pulse number has astrong link with the object movement, which may assistin our goal.

We start with some simple tests to determine the utilityof the ALS and PS information. We first setup the exper-iment environment which complies with the UbiTouch’stypical use case shown in Fig 1. All experiments areconducted in an ordinary office environment where thesettings of background light sources (i.e., positions andluminance) do not vary.

For the purpose of illustration, we specify a coordinatesystem for the area where the origin is located at thelower-left corner and two axes are marked by the dashlines in the figure. To understand how the movementof a finger affects the readings of the ALS and PS, weinstruct a user to move his finger along the trajectoriesin red. The trajectories are designed to be either parallelto the axis ‘X’ or axis ‘Y’ such that we can better observethe variation trend by altering one variable each time.

During the movement, we record five types of readings:(1) luminance (LUX); (2) distance measurement (a bi-nary value representing ‘far’ or ‘near’ ); (3) raw sensorreading from the photodiode CH0; (4) raw sensor read-ing from the photodiode CH1; (5) count of pulses re-ceived by the PS. The first two kinds of data can beeasily retrieved via the sensor API provided by the An-droid platform. However, the system blocks the accessto the remaining three types of data. To overcome thisconstraint, a patch is applied on the OS kernel, whichenables us to bypass the Android framework and accessthe raw readings from the hardware directly. Specif-ically, the patch first replaces the original proprietaryPS and ALS driver with an open-source one [3]. Thenew driver is designed to stream the data of registers inthe hardware to user-space applications through SYSFSinterfaces [15]. The UbiTouch application continuouslypolls the samples from the driver and process them. Inour implementation, we set the polling frequency for thePS and ALS to 100Hz, which is the maximum frequencythe hardware can support [2].

• PS• ALS

• Audio• Gyro

Mapping Function

Touch Action Detection

Tapping Detection Duration Estimation

Finger TrackingData

MappingParticle Filters

Data Preprocessing

Runtime Adaption

Runtime Calibration

ParameterUpdate

Initial TrainingParameter Learning

Training Data

TouchEvent

Figure 2: System Logic Flow for UbiTouch

Photodiode CH0 (Visible and Infra-red light)

Photodiode CH1(infra-red light)

Ambient LightSources

Infra-red emitter

Infra-red receiver

Ambient Light Sensor Proximity Sensor

(a) (b) (c)

ALS and PS

Figure 3: (a) shows the positions of the PS and ALS ona smart phone. (b) and (c) show the underlying mecha-nisms of the ALS and PS respectively.

We display these readings for the two trajectories in Fig.4(a) and Fig. 4(b) respectively. From top to bottom, thesub-figures demonstrate how the luminance, binary dis-tance measurements, CH0 readings, CH1 readings, andpulse counts change over the finger’s positions. It can beseen that, regardless of the moving directions, there is noobvious link between the movement and the luminance.We analyze the possible causes and find that the reso-lution for the noisy luminance readings is somewhat lowbecause the values are integers and only vary between20 to 26 in our experiments. It renders the luminancean unsuitable feature for our goal.

As for the distance measurements, the binary values willturn to one when the finger moves within a certain ra-dius (around 5cm) of the PS. Nevertheless the binaryinformation is still coarse-grained to realize accurate lo-calization and tracking.

We now turn our attention to the three remaining typesof data. The first observation we can easily identify isthat the readings for CH1 and CH0 have a strong cor-relation with the finger’s movement. Specifically, as thefinger moves closer to the ALS, the readings decrease.Another important observation is that the pulse countsalso have a strong link with the movement. When a fin-ger is getting further away from the PS, the pulse numberreceived significantly decreases.

212325

Lux

(a)

0

1

Dis

tance

9000100001100012000

CH

01600180020002200

CH

1

0 2 4 6 8 10 12 14

X-Coordinate (cm)

400

600

800

1000

Count

Num

ber

20

22

24

(b)

0

1

5900620065006800

11001150120012501300

0 1 2 3 4 5 6 7 8

Y-Coordinate (cm)

400600800

10001200

Figure 4: Recorded sensor readings during the move-ment along the aforementioned trajectories. The read-ings have been interpolated and outliers have been fil-tered out.

It can be seen that these data are location-dependentand have the potential to enable localization and track-ing. We explore the practical solutions to obtain fingerlocations in the next section.

OBTAINING FINGER LOCATIONS FROM RAW SENSORDATALet tuple st : (r0t , r

1t , ct) denote the readings for the pho-

todiode CH0, photodiode CH1, and pulse count at timet. Let lt : (xt, yt) denote the finger position, which is thecoordinate on the aforementioned virtual touch area. Tolocate the finger from the raw sensor data, we use a map-ping function f that maps the st into lt.

Ideally, a well-chosen deterministic mapping model thattakes the settings of the background light sources andplacement of the smartphone into consideration willachieve best performance. However, as the backgroundenvironment and smartphone deployment vary from timeto time, it would be fairly difficult and laborious for usersto decide the numerous parameters of the model whenthey are using the system. Thus, we seek a model thatautomatically tunes its parameters and adapts to differ-ent environments.

In this paper, we adopt a nonlinear mapping modelcalled Curvilinear Component Analysis (CCA) [6]. TheCCA is a self-organizing neural network which has theability to learn the mapping between a high-dimensionalspace to a low-dimensional one. Mathematically speak-ing, CCA attempts to train a network (i.e. a mappingfunction) to optimize the following criterion that explic-itly measures the preservation of the pairwise distances(denoted by E):

E =

N∑j=1

N∑i=1

(δ(si, sj)− d(li, lj))2F (dlilj , λ) (1)

where N denotes the total number of the training data.The δ(si, sj) is the curvilinear distance in the input spaceand d(li, lj) is the Euclidean distance in the output spacebetween the i-th and j-th data points. The factor Fweighs the contribution of each pair in the criterion,which is usually implemented as the Heaviside step func-tion:

F (d(li, lj), λ) =

{0, if d(li, lj)− λ < 01, if d(li, lj)− λ ≥ 0

(2)

where λ is a neighboring radius and set slightly largerthan the maximum curvilinear distance measured in theinput data set (i.e., max(δ(si, sj)|i < j ≤ n)).

The rationale behind this method is that if two read-ings are quite similar to each other, the points they aremapped into should also have a small distance. Con-versely, a huge difference between sensor readings indi-cates a large distance between the points. Thus, thismethod attempts to obtain a mapping function f by min-imizing the difference between the pairwise distances inthe input space and output space.

Please Move your finger and follow the cursor

Restart TrainingTraining

Trajectory

Cursor

Figure 5: The application for gathering training data.

To apply this approach, we collect some training dataas a series of st and the corresponding lt. Thus, a datagathering application shown in Fig. 5 is built to facilitatethis process. The application first requests the user toinput the size of the touch area and displays a trajectoryalong which a cursor gradually moves in a certain speed.The user is then requested to move his finger inside thetouching area on the desktop, imitating the movementindicated by the cursor.

The user may repeat the process for several times inorder to achieve a good synchronization between the fin-ger movement and the cursor motion. In this manner,

the coordinates of the cursor on the smartphone screencould be transformed to the coordinates of the fingeron the touch area by a simple scaling transformation.Compared with single point calibration, our calibrationprocess can obtain far more training data. The trainingprocess typically lasts one minute.

Before feeding the training data to the neural network,we notice that the training data contain many outliersand heavy noises. Three preprocessing tasks are used toclean the training data.

Outlier Removal. We find that there are some abruptchanges of the data that are obviously not caused bythe finger movement. Fig. 6(a) shows an example ofthe training data and it can be seen that there are somesignificant abrupt change in the l0 and l1 around thepositions 2, 3 and 6.5. There are likely to be outliersand should be eliminated.

Outliers

X-Coordinate (CM) X-Coordinate (CM)

Outliers

(a) (b)

Figure 6: (a) The original CH0/CH1 data. (b) TheCH0/CH1 data after the outliers are removed using theHampel filter.

To achieve that purpose, we first attempt the widely-used Standard Deviation (SD) method. However, thisapproach highly relies on the mean and standard devia-tion which are extremely sensitive to the presence of out-liers, hence does not perform well. Therefore, we utilize amore sophisticated outlier-removal algorithm called theHampel Identifier [14]. The algorithm declares any datapoints not within the range [µ−γ∗σ, µ+γ∗σ] as outliers,where u and σ denote the mean and the median abso-lute deviation of the sequence respectively, while γ is aconstant and the typical value is 3 for general cases. Fig.6(b) demonstrates the results for outlier removal. It canbe seen that the Hampel Identifier eliminates the outliersand preserves the desire CH0 and CH1 responses.

Interpolation. We program the application to poll thesensor data from the hardware every 10ms (i.e., 100Hz).However, we cannot guarantee that a reading is retrievedat the desired frequency, because the Linux kernel usedby typical smartphones does not provide real-time task

scheduling. A detailed look at the data shows that sam-pling jitters appear frequently and the sampling latencycan be up to 150 ms when the smartphone is havinga high computation load. Therefore, the data must beinterpolated before further processing. Here, we applylinear interpolation at the data to retrieve a re-sampledseries with the desired sampling frequency of 100Hz.

Noise Filtering. The final step of pre-processing is tosuppress the heavy noises in the data series. In our im-plementation, we utilize the wavelet filter proposed in[23] to smooth away the high-frequency noises. Specifi-cally, we apply 4-level ‘db4’ wavelet transform on sensordata and only use the approximation coefficients to ‘re-construct’ the filtered signal. Fig 7 demonstrates thedata sequences before and after filtering. It can be seenthat the noisy signal is getting much cleaner.

Figure 7: The effect of noise filtering.

The filtered data is fed to the CCA neural network. Inthe training phase, a tunable parameter called learn-ing rate (denoted by α ∈ [0, 1]) decides the convergencespeed of the learning process. Here, we adopt a linearlearning rate function which linearly decreases the ratewith time.

Improving Tracking Accuracy via a Hidden Markov ModelIn the previous section, we attempt to obtain the fingerlocations from the sensor readings by utilizing a CCAnetwork. It simply maps the sensor reading st into alocation lt, without taking previous readings into con-sideration.

This method does not take advantage of an importantobservation: a user typically moves his finger smoothly,thus the finger position at a consecutive moment is likelynot far away from the current location. The observa-tion indicates that the finger movement can be mod-eled by a Hidden Markov Model (HMM) shown in Fig.8. The model contains a set of hidden state variables(i.e., finger positions) and observable variables (i.e., sen-sor measurements). The current position lt depends onthe previous location lt−1 according to the probabilistictransition model p(lt|lt−1). The sensor measurement stcan be regarded as a stochastic projection of the hid-den state lt generated via the probabilistic observationmodel p(st|lt). We now want to sequentially estimatethe values of the hidden location lk, given the values ofthe observation process s0, · · · , sk, at any time step k,which follows the posterior density:

p(lk|s0, sk, ..., sk). (3)

It can be seen that this model enables us to further im-prove the tracking accuracy by incorporating the historicsensor measurements.

l1 l2 ……

s1 s2 s3 ……

Hidden States: Locations

Observable Variables: Sensor Readings

l3

State Transition Probabilities: p( ln | ln-1 )

Observation Probabilities: p( sn | ln )

Figure 8: HMM-based Movement Modeling.

To estimate the hidden states, we adopt an approachcalled Particle Filter [19] which can efficiently deal witha large number of possible states in our scenarios. Thealgorithm is described in three steps:

1. Initialization:

• For i = 1, ...,M , randomly initialize li0, and sett = 1.

2. Importance sampling step:

• For i = 1, ...,M , sample lit ∼ p(lt|lit−1).

• For i = 1, ...,M , evaluate the importance weights

by wit = p(st|lit).• Normalize the importance weights.

3. Selection step:

• Re-sample with replacement M particles (lit; i =

1, ...,M) from the (lit; i = 1, ...,M) according tothe importance weights.

• Set t← t+ 1 and return to the step 2.

This algorithm requires two probabilistic models: thetransition model p(lt|lt−1) and the observation model =p(st|lt).We first try to compute the transition model p(lt|lt−1).Based on the aforementioned observation, we can safelyassume the user moves his finger in a reasonable speednot exceeding a constant value Vmax. We now expressthe transition model as follow:

p(lt|lt−1) =

{0, if d(lt, lt−1)− Vmax ∗ T < 01

2πV 2maxT

2 , if d(lt, lt−1)− Vmax ∗ T ≥ 0,

(4)where T is the sampling period (10ms). Vmax here is setto 0.05m/s based on the previous finger touch research[5]. The intuition behind the transition model is that afinger located at lt−1 can uniformly move to any pointswithin the circle whose center and radius are lt−1 andVmax ∗ T respectively.

Next we have to figure out the observation model p(st|lt).Noting that we have obtained a mapping function f that

maps a sensor measurement st to a position lt in theprevious section, we could obtain the following equation:

st = f−1(lt) +W, (5)

where f−1 is the reverse mapping function that projectslt to st and can be obtained from the CCA network.W denotes the measurement noise and has a Gaussiandistribution, that is W ∼ N (0,Σ) and Σ = diag(σ2).The σ is a standard deviation and set to 10 which ismeasured from the real sensory data.

Therefore, the measurement model can be expressed as:

p(st|lt) = C∗exp((st−f−1(lt))

TΣ−1(st−f−1(lt))), (6)

where C is a constant and does not affect the final resultsdue to normalization in the importance sampling step.

It should be noted that, the algorithm’s time complex-ity is linear with respect to the number of samples M .Obviously, the more samples, the better the accuracy,so there is a trade-off between speed and accuracy. Tofind an optimal value of M for the system, we adopta strategy described in [22] which dynamically adjuststhe number of samples and is adaptive to the availablecomputational resources.

TOUCH ACTION DETECTIONThe system is now able to accurately track the fingermovement. However, to mimic a virtual touchpad, thesystem requires to detect two typical touch actions in-cluding tapping and dragging.

Our system realizes the detection by leveraging a smart-phone built-in microphone. Tapping on a desktop typ-ically generates audible stroke sounds. Similarly, drag-ging gestures request the user to rub on a desktop, pro-ducing constant audio noises as well. These signals couldbe received by the sensor and facilitate the touch actiondetection.

time(ms)0 50 100 150 200 250 300 350 400 450

ener

gy le

vel

0

0.05

0.1

0.15TappingDragging

Figure 9: Energy level of the audio signals for tappingand dragging. It is obtained via short-team average onthe audio signal with a moving window.

Fig 9 shows the energy level of the audio signals receivedwhen a user is tapping and dragging on a desktop. Theaudio energy generated by a tapping has a pattern simi-lar to an impulse response. It starts with a outstandingpeak in the beginning and fades away in a short period.Regarding the dragging action, it can be divided to twosub-actions: initial tapping and subsequent rubbing. Itincurs a peak at the very beginning and causes a longlasting disturbance to the audio signals. It can be seen

that the first step for our detection algorithm is to iden-tify the tapping event.

In our implementation, we utilize a hypothesis test-ing approach called the generalized likelihood ratio test(GLRT) [27] to determine whether there is a tappingevent. We store the incoming audio energy readings for100ms in a First-In-First-Out buffer and keep testing thefollowing two hypotheses on the data. The null hypoth-esis H0 is that there is no tapping on the desktop andhence the average of the energy level should remain sta-ble. Thus, we have

H0 : u(t) = µ0 + w(t) (7)

The variable u(t) is the mean of the energy level beforethe time t. The µ0 denotes the average energy and thevariable w(t) is the Gaussian noise with zero mean valueand unknown variance σ0.

The alternative hypothesis H1 is that there is a tappingsuch that the mean of the energy level will significantlyincrease due to the tapping impulse. Thus, we have

H1 =

{u(t) = µ0 + w(t) if t < tc

u(t) = µ1 + w(t) if t ≥ tc and µ1 6= µ0,(8)

where the tc represents the time when the tapping oc-curs, µ1 denotes the new average energy after the tap-ping. Note that u0, σ0, µ1, and tc are unknown param-eters.

Let θ0 denote {µ0, σ0}, and θ1 denote {µ0, σ0, µ1, tc}.GLRT tests the two hypotheses by

maxθ1 p(u | H1, θ1)

maxθ0 p(u | H0, θ0)

H0

≶H1

γ, (9)

where p(u | H1, θ1) is the probability density functionof u under hypothesis H1 and θ1. The p(u | H0, θ0) isdefined in a similar manner. The r is the threshold andset to 100 which works well in general cases.

However, the energy based detection system is likely tobe disturbed by ambient noises. According to the fieldtest, human voices or sudden random burst of soundnearby can frequently mis-trigger the detection system.To remove the false positives, we utilize a cross-checkingapproach that leverages a smartphone’s gyroscope.

The algorithm takes advantage of an observation that atapping on a desktop generates vibrations which couldbe easily captured by the gyroscope, while the ambientnoises do not. Thus, whenever a tapping event is de-tected by the audio signal, we further check the presenceof the vibrations in the gyroscope signals. To detect thevibrations, we adopt the GLRT algorithm in a similarmanner to the audio signal. This approach significantlysuppresses the false alarms caused by the ambient noises.

After a tapping event is detected, we further estimate theduration of the touch action in case there is a followingdragging event. The duration is estimated between thepoint when the tapping occurs and the point when the

power drops to below the threshold µ0 + (µmax − µ0) ∗10%. The µmax is the maximum power level that thetapping impulse can reach. Capping the duration at thepower level slightly above the normal noise energy helpsthe system combat the fluctuation of the noise energyfloor.

It should be noted that the user is suggested to per-form the touch actions gently, since a large movement ofthe finger may have negative impact on the finger track-ing accuracy. In practice, the detection algorithm canachieve great performance when the user gently hits orrubs the desk using his nail tip.

RUNTIME CALIBRATION AND ADAPTATIONWe assume that the smartphone is placed at an envi-ronment where the settings of ambient light includingpositions and luminance remain unchanged. However,in practice, the lighting conditions may vary slightly dueto various reasons (e.g., a bulb may slightly decreaseits brightness due to insufficient voltage), rendering themapping function retrieved from the initial training out-dated. In our system, rather than simply restarting thewhole training phrase, we adopt a run-time calibrationmechanism that enables the system to adapt to the mi-nor change of the environment with a tiny effort. Insome rare situations when the lighting conditions changedrastically, the user can still manually restart the entiretraining phrase.

The system executes runtime adaptation by allowing theuser to provide extra feedback on the localization results.When the user notice minor localization error, he couldpause the tracking algorithm for a short period and man-ually input the correct coordinate by clicking on the rightposition on the touchscreen. Through this run-time cal-ibration process, the system can harness extra trainingsamples to adapt to the slightly varying environment.Note that we should carefully adjust the learning rate αassociated with the new sample. A tiny α may renderthe update useless, while a huge α may amplify the noisein the sample and make the mapping function unstable.

In our implementation, we design the α based on thelocalization error. Specifically, if the original mapping

function f maps the sn into a coordinate ln, which hasa relatively small distance to the coordinate ln providedby the user, it means the function performs well and theupdate rate α should remain low. Conversely, a highupdate rate is selected when the distance is large. Basedon the principle, we set α as:

αt = L(||lt − f(st)||2) (10)

L is a shifted logistic function that can be expressed as:

L(x) =1

1 + exp(6− 3x)(11)

The logistic function converts the localization error intoa value between (0,1) and holds nice properties. First,when the localization error is pretty small (less then

0.5cm), the value remain relatively low. Second, as thelocalization error continuously increases, the alpha ratewill increase significantly. Finally, if the localization er-ror goes beyond a certain value (3cm), the function re-turns a α value close to 1.

SYSTEM EVALUATIONSIn this section, we provide detailed performance evalu-ation of UbiTouch in terms of finger tracking accuracy,touch gesture detection accuracy, and system energy effi-ciency. We conducted our experiments in three commonscenarios with dim, normal, and strong lighting condi-tion by five users. Then we showcase UbiTouch’s effec-tiveness when it works in conjunction with a handwritingrecognition system.

Accuracy of Finger TrackingThere are several underlying factors that may affect thetracking performance of UbiTouch. In this section, wefirst carried out a baseline test in a typical usage sce-nario. Then we consider the effect of two crucial fac-tors including (1) ambient lighting condition and (2) thetilted angle of the smartphone.

Accuracy for baseline experimentIn the baseline experiment, instead of running UbiTouchon an ordinary desktop, we deploy the system on top of aMacbook Pro’s physical touchpad whose size is 7.5 cm ×10.5 cm. It collects the finger locations and trajectorieswhich serve as the ground truth.

We then carried out a total of 100 static finger local-ization tests in two different setup: (1) tracking merelywith the CCA mapping function. (2) tracking with boththe mapping function and particle filter.

With only the mapping function, the average localizationerror is 1.69 cm with a standard deviation of 0.53 cm.The CDF shown in Fig 10 reveals that over 80% pointshave an error of less than 2.1 cm. When the mappingfunction works in conjunction with the particle filter, themean error reduces to 1.53 cm with a 0.42 cm standarddeviation. The corresponding CDF demonstrates that80% of the error values is now less than 1.8 cm. Theperformance improvement is attributed to the particlefilter which takes extra historic information into consid-eration.

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5Error (cm)

0.0

0.2

0.4

0.6

0.8

1.0

CD

F

Without Particle FilterWith Particle Filter

Figure 10: CDF for the localization error (cm).

Besides, the performance can also be observed throughthe spatial error distributions shown in Fig 11. An obvi-ous observation is that the remote spots far away from

the PS and ALS sensor tend to possess larger error thanthe area around the sensors does. We investigate thisphenomenon and discover that when the finger is notwithin the optimal detection range of the PS (10cm), thesensor readings are saturated with noises, which mightseverely degrade the localization performance. Noticingthat the transmission power of the PS is adjustable, wemay be able to increase the detection range by consum-ing more energy.

Figure 11: The spatial error distribution (interpolated).The position for the ALS and PS is around (12, 7.5).

Impact of ambient lighting conditionTo measure the impact of the ambient lighting condition,we deploy our system in three typical scenarios includinga storage room with a dim light bulb, an office room atcampus with moderate light level, and an outdoor placewith excessive sun light. Note that the settings (e.g.,positions and illumination) of background light sourcesfor these scenarios remain unchanged during our exper-iment. The experiments run on each scenarios, each re-peated 100 times. Each time we conduct an experiment,we re-calibrate the system to achieve accurate perfor-mance measurements. Fig. 12 plots the correspondinglocalization accuracy. The results show that UbiTouchcan maintain a localization error less than 1.54 cm acrossall the scenarios. It indicates that the system can per-form well regardless of the illumination of the ambientlighting.

Tilted AngleIn the previous experiments, the smartphone is deployedvertically on a desk. However, in practice, some usersprefer to place their phones in a slightly tilted angle fora better user experience or perspective. Hence, we car-ried out multiple tests under different inclined angles,which can be directly measured by the built-in orienta-tion sensors.

Fig. 13 demonstrates how the tracking accuracy changeswith the titled angle. It can be seen that the localiza-tion error mildly increases when the titled angle startsrising. It indicates that the system still performs wellwhen the smartphone is deployed at a slightly tilted an-gle (less than 10 degrees). However, the slope drasticallygrows starting from 25 degrees and the localization er-ror soars into xcm when the angle reaches 60 degrees.

Library Office PubError rate withgyro (FP/FN)

0.5%(0/0.5)

0.8%(0/0.8)

3.5%(0.3/3.2)

Error rate w/ogyro (FP/FN)

0.7%(0.3/0.4)

1.3%(0.5/0.8)

21%(15/6)

Table 1: Accuracy of tapping detection.

Library Office PubEst. Duration(sec) 9.7 10.2 10.5

Table 2: Accuracy of touch-event duration estimation.

It may be due to that when the smartphone is signifi-cantly inclined, the AS and PLS turn into a direction faraway from the finger, which has negative impact on thelocalization accuracy.

Accuracy of Touch Action DetectionWe now evaluate the effectiveness of the tapping anddragging detection component. We carried out the testin three scenarios: a library, an office at campus, anda pub. These situations represent environments withsmall, normal, and extreme background noise respec-tively.

First, we measured the accuracy of the tapping detec-tion. In every experiment, we instructed a user to tapon a desk to make 300 clicks, each on a randomly se-lected position. The tapping strength is maintained tobe a moderate level which is audible to the user.

Table 2 display the experiment results for false positive(false alarm) and false negative (mis-detection) rate. Wecan notice that the error rates remains relatively low(3.5% in the worst case). The results indicate the tap-ping detection algorithm is accurate and reliable.

We conduct the experiments again without using the gy-roscope. The results show that in a quiet environmentincluding the library and office, the system performsequally well even with the gyroscope disabled. However,in a noisy pub environment, the false positive rate soarsinto 15%, making the system unusable. It indicates thatthe gyroscope can greatly suppress the false positive ratein some noisy situation and improve the performance.

Next, we evaluate the effectiveness of the duration es-timation algorithm. We requested a user to perform a10-second dragging action after a tapping. The experi-ments were repeated for 50 times in the aforementionedthree situations. Table 2 shows the dragging durationestimation and demonstrates fair accuracy. Even in anoisy environment, the error is still below a reasonablevalue (0.5 seconds).

Power ConsumptionUbiTouch merely leverages built-in sensors including thePS, ALS, microphone, and gyroscope which are of lowpower consumption. Especially, the average operating

Loca

lizat

ion

Erro

r (cm

)

Figure 12: Localization accuracyunder differnt scenearios.

Angle (degree)

Loca

lizat

ion

erro

r

Figure 13: Localization error vs.smartphone inclined angle.

A B C D E0

20

40

60

80

100

Acc

ura

cy

PhysicalVirtual

Figure 14: Recognition rates for5 users.

current for the PS and ALS is only 176 µA [2]. We pro-file the power consumption of UbiTouch by utilizing aprofessional power monitor application called Powertu-tor [31]. Specifically, we measure the power cost in twosituations: (1) idle with screen on (2) running UbiTouchwith continuous touch input. Each situation lasts for 5minutes. The average power consumption for each stateis 6021 mW and 6930 mW respectively. Thus, UbiTouchincurs extra 15.1% power cost, that is slightly less thanthe power consumption of UbiK which is 18.5% [25].

Application TestsUbiTouch can support various types of applications suchas drawing and text-inputing. Here, we further evaluatethe system performance by replaying the user input intothe Google handwriting text-input application [8] andassessing the recognition rate. We carried out the ex-periments by requesting three users to input 100 randomEnglish alphabets. Fig. 15 shows two handwriting exam-ples retrieved from the physical and virtual touchpads.Note that instead of feeding the finger trajectories to theapplication, we conduct some pre-processing to increasethe recognition rate. Specifically, we apply a clusteringalgorithm called DBSCAN [28] to filter out some outliersand extract the skeleton of the input character, which isthen fed into the recognition software.

Fig. 14 demonstrates the recognition rates for five users.The average recognition rate is 79%. In the worst case,the recognition rate for UbiTouch is 72%, still compa-rable to the accuracy of the physical touchpad which is92%.

X (cm) X (cm)

Figure 15: Handwriting examples for English charactersL and C.

DISCUSSION

System GeneralizabilityCurrently, the system is only implemented on a specificphone model. We found that smart phones from differentvendors tend to equip different types of sensors, whichmeans our system may only be able to run in a specificmodel till now. However, since almost every sensor couldprovide raw distance and luminance readings. Thus, weenvision that the principles and methodologies exploredin the paper could be generalizable and applicable on anumber of devices.

Applicable scenariosThe prototype of UbiTouch is only applicable under twobasic conditions: (1) the smartphone is placed on a deskin a vertical or slightly inclined position and remainsstill, and (2) throughout the usage life-cycle, the ambientlighting conditions do not vary drastically.

Regarding the first condition, users have to deploy thesmartphone almost vertically, which is a somewhat un-natural position. The second condition not only requiresthat the positions and luminance of light sources basi-cally remain unchanged, but also demands that there isno significant movement in the neighboring area of theuser. We concede the rigidness of these conditions andenvision that more sophisticated signal processing tech-niques will be useful for eliminating these limitations.

Finger gesturesWhile interacting with the system, the user is suggestedto stay consistent in terms of finger gestures. A dras-tically change of the finger gesture (e.g., finger rota-tion/tilting) may negatively affect the tracking accuracy.We expect some pre-processing jobs to remove the im-pact of the gestures in the future.

CONCLUSIONIn this paper, we present UbiTouch, a prototype sys-tem that enhances smartphones with virtual touchpadsthrough the built-in smartphone sensors. We demon-strate that UbiTouch achieves centimeter-level localiza-tion accuracy and poses no significant impact on batterylife of a smartphone.

REFERENCES1. H. Abdelnasser, M. Youssef, and K. A. Harras,

“Wigest: A ubiquitous wifi-based gesturerecognition system,” in 2015 IEEE Conference onComputer Communications (INFOCOM). IEEE,2015, pp. 1472–1480.

2. Avago, “Avago 9930 integrated proximity andambient light sensor,”http://www.avagotech.com/products/.

3. Avago-Developer, “Avago 9930 linux driver,”https://github.com/CyanogenMod/android kernel lge hammerhead/blob/cm-13.0/drivers/misc/apds993x.c.

4. P. Baudisch and G. Chu, “Back-of-deviceinteraction allows creating very small touchdevices,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems. ACM,2009, pp. 1923–1932.

5. X. Bi, Y. Li, and S. Zhai, “Ffitts law: modelingfinger touch with fitts’ law,” in Proceedings of theSIGCHI Conference on Human Factors inComputing Systems. ACM, 2013, pp. 1363–1372.

6. P. Demartines and J. Herault, “Curvilinearcomponent analysis: A self-organizing neuralnetwork for nonlinear mapping of data sets,”Neural Networks, IEEE Transactions on, vol. 8,no. 1, pp. 148–154, 1997.

7. Google, “Environment sensor for android,”http://developer.android.com/guide/topics/sensors/sensors environment.html.

8. Google-input, “Google handwriting input,”https://play.google.com/store/apps/details?id=com.google.android.apps.handwriting.ime.

9. C. Harrison, H. Benko, and A. D. Wilson,“Omnitouch: wearable multitouch interactioneverywhere,” in Proceedings of the 24th annualACM symposium on User interface software andtechnology. ACM, 2011, pp. 441–450.

10. K. Hinckley and H. Song, “Sensor synaesthesia:touch in motion, and motion in touch,” inProceedings of the SIGCHI Conference on HumanFactors in Computing Systems. ACM, 2011, pp.801–810.

11. S. Hiraoka, I. Miyamoto, and K. Tomimatsu,“Behind touch, a text input method for mobilephones by the back and tactile sense interface,”Information Processing Society of Japan,Interaction 2003, pp. 131–138, 2003.

12. H. Ketabdar, K. A. Yuksel, and M. Roshandel,“Magitact: interaction with mobile devices basedon compass (magnetic) sensor,” in Proceedings ofthe 15th international conference on Intelligentuser interfaces. ACM, 2010, pp. 413–414.

13. N. D. Lane, E. Miluzzo, H. Lu, D. Peebles,T. Choudhury, and A. T. Campbell, “A survey ofmobile phone sensing,” Communications Magazine,IEEE, vol. 48, no. 9, pp. 140–150, 2010.

14. H. Liu, S. Shah, and W. Jiang, “On-line outlierdetection and data cleaning,” Computers &chemical engineering, vol. 28, no. 9, pp. 1635–1647,2004.

15. R. Love, Linux kernel development. PearsonEducation, 2010.

16. Z. Lv, A. Halawani, M. S. Lal Khan, S. U. Rehman,and H. Li, “Finger in air: touch-less interaction onsmartphone,” in Proceedings of the 12thInternational Conference on Mobile and UbiquitousMultimedia. ACM, 2013, p. 16.

17. S. Nirjon, J. Gummeson, D. Gelb, and K.-H. Kim,“TypingRing: A wearable ring platform for textinput,” in Proceedings of the 13th AnnualInternational Conference on Mobile Systems,Applications, and Services. ACM, 2015, pp.227–239.

18. I. Oakley and D. Lee, “Interaction on the edge:offset sensing for small devices,” in Proceedings ofthe SIGCHI Conference on Human Factors inComputing Systems. ACM, 2014, pp. 169–178.

19. B. Ristic, S. Arulampalam, and N. Gordon, Beyondthe Kalman filter: Particle filters for trackingapplications. Artech house Boston, 2004, vol. 685.

20. S. Robinson, N. Rajput, M. Jones, A. Jain,S. Sahay, and A. Nanavati, “Tapback: towardsricher mobile interfaces in impoverished contexts,”in Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems. ACM,2011, pp. 2733–2736.

21. H. Roeber, J. Bacus, and C. Tomasi, “Typing inthin air: the canesta projection keyboard-a newmethod of interaction with electronic devices,” inCHI’03 extended abstracts on Human factors incomputing systems. ACM, 2003, pp. 712–713.

22. A. Soto, “Self adaptive particle filter,” in IJCAI,2005, pp. 1398–1406.

23. J. D. Villasenor, B. Belzer, and J. Liao, “Waveletfilter evaluation for image compression,” ImageProcessing, IEEE Transactions on, vol. 4, no. 8, pp.1053–1060, 1995.

24. J. Wang, D. Vasisht, and D. Katabi, “RF-IDraw:Virtual Touch Screen in the Air using RF Signals,”in ACM SIGCOMM Computer CommunicationReview, vol. 44, no. 4. ACM, 2014, pp. 235–246.

25. J. Wang, K. Zhao, X. Zhang, and C. Peng,“Ubiquitous keyboard for small mobile devices:harnessing multipath fading for fine-grainedkeystroke localization,” in Proceedings of the 12thannual international conference on Mobile systems,applications, and services. ACM, 2014, pp. 14–27.

26. D. Wigdor, C. Forlines, P. Baudisch, J. Barnwell,and C. Shen, “Lucid touch: a see-through mobiledevice,” in Proceedings of the 20th annual ACMsymposium on User interface software andtechnology. ACM, 2007, pp. 269–278.

27. A. S. Willsky and H. L. Jones, “A generalizedlikelihood ratio approach to the detection andestimation of jumps in linear systems,” AutomaticControl, IEEE Transactions on, vol. 21, no. 1, pp.108–112, 1976.

28. I. H. Witten and E. Frank, Data Mining: Practicalmachine learning tools and techniques. MorganKaufmann, 2005.

29. N.-H. Yu, S.-S. Tsai, I.-C. Hsiao, D.-J. Tsai, M.-H.Lee, M. Y. Chen, Y.-P. Hung et al., “Clip-ongadgets: expanding multi-touch interaction areawith unpowered tactile controls,” in Proceedings ofthe 24th annual ACM symposium on User interfacesoftware and technology. ACM, 2011, pp. 367–372.

30. C. Zhang, J. Tabor, J. Zhang, and X. Zhang,“Extending mobile interaction through near-fieldvisible light sensing,” in Proceedings of the 21stAnnual International Conference on MobileComputing and Networking. ACM, 2015, pp.345–357.

31. L. Zhang, B. Tiwana, Z. Qian, Z. Wang, R. P.Dick, Z. M. Mao, and L. Yang, “Accurate onlinepower estimation and automatic battery behaviorbased power model generation for smartphones,” inProceedings of the eighth IEEE/ACM/IFIPinternational conference on Hardware/softwarecodesign and system synthesis. ACM, 2010, pp.105–114.