Natural and Intuitive Deformation Gestures for One-handed Landscape Mode Interaction
Abstract
The landscape orientation of smartphone offers a better
aspect ratio and extensive view for watching media and
photography. However, it presents challenges of
occlusion, reachability, and frequent re-gripping in one-
handed interactions. To address these issues we took
the opportunity of deformation gestures to interact with
future flexible smartphones. A preliminary survey was
conducted to understand one-handed landscape mode
usage patterns. Then, the 1st study was conducted to
identify 3 most preferred one-handed landscape mode
grips. In the 2nd study, we gathered unique user-
defined deformation gestures to identify the set of most
natural and intuitive gestures corresponding to each
grip. We also found 3 gestures that can be performed
in more than one grip. Finally, we discuss the influence
of the grips on performing gestures.
Author Keywords
Flexible display device; deformation gesture; one-
handed interaction; landscape mode interaction.
CSS Concepts
• Human-centered computing~Human computer
interaction (HCI) • Human-centered
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. TEI '19, March 17–20, 2019, Tempe, AZ, USA © 2019 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-6196-5/19/03. https://doi.org/10.1145/3294109.3300996
Pranjal Protim Borah
Embedded Interaction Lab,
Indian Institute of Technology
(IIT) Guwahati,
Guwahati, 781039, India
Keyur Sorathia
Embedded Interaction Lab,
Indian Institute of Technology
(IIT) Guwahati,
Guwahati, 781039, India
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
229
computing~User studies • Human-centered
computing~User centered design
Introduction
Spatial properties and arrangements of elements
influence people’s aesthetic responses to an image
irrespective of portrait or landscape aspect ratio [3]. As
an influence of predominant landscape orientation of
Television (TV) screens and computer monitors and for
extensive view with maximum screen real estate [9],
this horizontal orientation (Figure 1a) is often used in
smartphone context for watching videos [10, 9],
recording videos [12], and capturing photographs [12].
However, it presents a variety of challenges, when
interacted with one hand. These challenges include
finger occlusion [6, 15], unreachable target areas [6,
11, 15] on large screen devices and the tendency of
dropping the device due to frequent re-gripping [11, 5].
A few studies have been conducted for comfortable
one-handed landscape mode video browsing [11, 18]
and photo browsing [1]. However, these existing
researches are specific to particular applications and do
not provide a generic solution. Deformation gestures
(Figure 1b) as an input method for flexible handheld
devices present a potential alternative to address the
challenges of one-handed landscape mode interaction.
They have the potential to provide high resolution of
input through eye-free interaction that is independent
of Graphical User Interface (GUI) [17] and provides an
optimal mean for quick continuous bipolar input [2].
Extensive studies on flexible handheld devices were
conducted to identify bimanual deformation gestures
[13, 14, 16] and to address issues of reachability and
occlusion in one-handed portrait mode [6]. Audrey
Girouard et al. had identified a set of deformation
gestures that was easy to perform in a variety of hold
scenarios in one-handed portrait mode [6]. However, to
the best of our knowledge, no studies have investigated
suitable deformation gestures for one-handed
landscape mode usage. In this paper, we present a set
of user-defined deformation gestures for 3 different
user-preferred grips for one-handed usage in a
landscape mode.
Preliminary survey on one-handed use of
landscape mode
The preliminary survey (by means of a questionnaire)
was targeted to build an understanding of landscape
mode usage, handedness of using a smartphone in
one-handed landscape mode, and difficulties during
one-handed interaction with the applications. 50
university students (34 male and 16 female) responded
with a completed questionnaire, between the ages of
18-32 years (Mean = 21.7, SD = 3.4). 44 (88%) right-
handed and 6 (12%) left-handed users participated in
the survey. All the participants had more than 3 years
of experience in using smartphones.
Findings of the survey
Users prefer portrait orientation but often tend to adopt
landscape orientation for better representation of the
content of interest considering the adaptability of the
application. 52% of the participants indicated their
consent in often performing one-handed interaction in
landscape mode. Rest is equally divided into seldom
use (24%) and interacting with both hands (24%).
Majority of the participants (84.2%), who often perform
one-handed interaction preferred to use their right
hand, 13.2% of the participants preferred to use their
left hand and the rest (2.6%) had no preference.
Figure 1: (a) One-handed
landscape mode grip (b)
deformation gesture (Top-right
corner bend out) in one-handed
landscape mode.
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
230
The most preferred set of applications in landscape
mode include video players, camera applications,
games, and document viewers. We observed similar
challenges presented in literature including unreachable
touch target (81.6%), difficulties in re-gripping the
device during interaction (78.9%), interacting finger
occlusion (47.4%) and difficulty to perform multi-touch
input gesture (34.2%). We also observed that one-
handed landscape mode interaction is mostly preferred
in stationary situations (sitting or standing). In
addition, smartphones possessed by the participants
include 5″ (40%), 5.5″ (46%), 6″ (8%), and other
(6%) display sizes.
Study on User-Preferred Grip
The goal of this study was to identify the set of most
preferred one-handed grips for landscape mode
interaction. This study focuses on the right-handed
users due to the higher proportion of right-handed
landscape mode users (84.2% in the preliminary
survey) and the right-handedness in the global
population [8].
Task The task required participants to demonstrate two most
preferred grips to hold a rigid smartphone with the
right hand in the landscape mode. The grips required to
be the most natural, comfortable and safe (reluctant to
the tendency of dropping the device) grip with
minimum finger occlusion.
Participants 33 university students (22 male and 11 female),
between the ages of 18-30 (Mean = 22.5, SD = 3.2),
participated in the study. All the participants were
right-handed with more than 3 years of experience in
using a smartphone. We made a verbal announcement
in the classrooms to invite students to register their
participation in the study. No remuneration was
provided to the participants.
Apparatus The study was conducted using a “Lenovo K6 Power”
smartphone (5″ display size). The use of rigid
smartphone instead of a deformable prototype was to
avoid artificial grips [5] as users were not familiar with
the use of flexible smartphones.
Procedure
We performed a controlled experiment where each
session was video recorded with the participants’
consent. After the participants read the description of
the task, moderator verbally explained the same to the
participants. To ensure the demonstrated grips to be
natural and comfortable, participants were asked to
imagine and pretend using preferred applications for
which they had experienced the use of one-handed
landscape mode interaction for an extended period of
time (without dropping the device). They were also
asked to speak up the names of the applications and to
mimic similar inclination of the display for comfortable
view without occlusion. No content was displayed on
the screen to avoid any adverse effect of static content
(sometimes even irrelevant) on users’ imagination. The
participants were asked to keep one finger free so that
they can pretend to perform touch interaction on the
screen without any tendency of dropping the device.
This stands the safety of the grip and the notion that
future flexible devices will also continue to support the
traditional touch input method. Participants were asked
not to rest their right hand on an object (table or chair)
or any body part. This particular choice was made due
Figure 2: The most preferred
right-handed landscape mode
grips with at least one finger
available for interaction.
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
231
to its potential applicability to both sitting and standing
position. Finally, we noted the top two user preferences
for further analysis. All participants completed the task
within 6 minutes.
Results
A total of 57 grips were recorded. It includes 33 grips
as the first preference and 24 grips as the second
preference. As shown in Figure 2 and Table 1, four
unique grips were identified based on the location of
contact between the smartphone and the fingers and
the purpose (support, balance, and interact) of the
involved fingers. All the participants (100%) performed
Grip-A as the first preferred grip while 30%, 28%, and
15% participants performed Grip-B, Grip-C, and Grip-D
respectively as the second preferred grip (Figure 3). We
selected the 3 most preferred grips for our next study.
Grip Support Balance Interact
A Middle, Ring
and Little finger Index finger Thumb
B Thumb Middle and/or
Ring finger
Index
finger
C Index, Middle, Ring and Little
finger None Thumb
D Index, Middle, Ring and Little
finger
Thumb (Occasionally)
Thumb
Table 1: Purpose of the fingers involved in the grips
Study on User-defined Deformation Gesture
This study was aimed to identify the set of most natural
and intuitive deformation gestures with potential to
address the challenges of one-handed landscape mode
interaction.
Task The participants were asked to propose most naturally
recalled and intuitive deformation gestures for each
grip A, B, and C.
Participants
30 university students (20 male, 10 female), between
the ages of 18-32 (Mean = 22.4, SD = 3.6)
participated in the study. All the participants were
right-handed with no prior experience of using a flexible
handheld device. We made a verbal announcement in
the classrooms to invite students to register for their
participation in the study. There were no common
participants from the previous study. No remuneration
was provided for their participation.
Apparatus The study was conducted using a mockup prototype
(Figure 1b) made of flexible PVC sheet (1.5mm
thickness) sandwiched between two foam rubber sheets
(3mm thickness). Deformability introduces challenges
to the development of the mockup in terms of weight
and flexibility ratio. The dimension (14.2cm x 7.2cm x
0.8 cm) of the prototype was selected to resemble the
dimension of a state of art smartphone of display size
5″ (with bezel) and 5.5″ (thin bezel). However, the
weight resemblance was not incorporated to retain the
flexible property of the mockup. We were more
concerned about the flexibility of the prototype than its
weight for better agreement of the gestures [14].
Procedure
We conducted a controlled study where each session
was video recorded with the consent of participants for
further analysis. After the participants read the
description of the task, the moderator demonstrated
Figure 3: Percentage of users
performing the grips.
0 25 50 75 100
Grip-D
Grip-C
Grip-B
Grip-A
1st preference
2nd preference
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
232
the selected grips and two-handed deformation gesture
(folding the display vertically) using the mockup. We
devoted a minimum of 10 minutes to each participant
to experience the grips and interact with the mockup
until they were ready to start the study. All participants
completed the task within 20 minutes. To ensure
intuitiveness (relatively effortless [4] with reduced
cognitive load and deliberation [7]), participants were
asked to propose referent independent gestures. In
addition, the locations of deformation need to be easily
reachable without switching the grip or compromising
the safety of the grip. The participants were also asked
to perform the gestures without resting the right hand
on an object or any body part. There was no restriction
on the number of gestures proposed by the
participants. The sequence of the grips presented to the
participants was randomized in order to equalize the
effects of gestures proposed in the previous grip. We
considered the most spontaneously derived [7] and
possibly automatic [4] gesture to be the most intuitive
gesture. Accordingly, intuitiveness scores (maximum 7
and minimum 0) were assigned to the gestures based
on the order in which they were proposed. We defined
naturalness of a gesture as the frequency [7] of
participants proposing the same gesture.
Results
30 participants proposed a total of 367 gestures. All
participants proposed 1 to 7 gestures for each grip
except for one participant (p-13) who was unable to
offer any gesture for Grip-B and another participant (p-
27) proposed 8 gestures for Grip-C. Figure 4 and 5
shows the number of gestures proposed in each grip.
Later, we identified respectively 18, 15, and 20 unique
gestures proposed for Grip-A, Grip-B, and Grip-C.
Unique deformation gestures were identified based on
the location of initiating the deformation (top, bottom,
left, right, corner, side, diagonal- along a diagonal line,
horizontal- along the horizontal axis, and vertical- along
the vertical axis), type of deformation (bend, fold -
when touches the opposite surface, and roll) and
direction of deformation (in – movement of edges/
corners towards the user and out – movement of
edges/corners away from the user). This classification
of deformation gestures based on location and type of
deformation is similar to previous studies [6, 14, 16]
except for the direction of deformation. While proposing
the deformation gestures, most of the users referred
the directions as in/out (sometimes inward/outward).
This inspired us to use the words “in” and “out” (also
used in [2]) to match users’ mental model and to avoid
confusion as reported in the literature [6]. Accordingly,
the names of the deformation gestures also consist of
location, type, and direction of deformation. For
instance, in the gesture name “Top-right corner bend
in”, “top-right corner” is the location; “bend” is the type
of deformation and “in” is the direction of deformation
towards the user. We used the frequency and mean
intuitiveness score of gestures to confirm the final set
of most natural and intuitive user-defined deformation
gestures. The final set contains gestures with higher
frequency (F ≥ 15) and mean intuitiveness score (MI ≥
3). Hence, the final catalogue of gestures contains
respectively 4, 3, and 3 gestures for Grip-A, Grip-B,
and Grip-C (Figure 6). Here “Horizontal bend in” (Grip-
A, B and C), “Top-right corner bend in” (Grip-A and C)
and “Horizontal fold in” (Grip-A and B) gestures were
proposed for multiple grips.
We collected individual intuitiveness scores of the
participants corresponding to the gestures in each grip.
Since the collected data are not normally distributed,
Figure 4: Total gestures proposed
in each grip.
Figure 5: Mean of gestures
proposed in each grip.
131
97
139
0
50
100
150
Grip-A Grip-B Grip-C
0
2
4
6
8
Grip-A Grip-B Grip-C
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
233
we applied non-parametric tests (Friedman Test and
Wilcoxon Signed Ranks Test) to interpret the influence
of grips on the intuitiveness of gesture location, type,
and direction as shown in Table 2.
Grip
Significantly (p<.05) intuitive
Location Type of
deformation Direction
Corner Side
A Top-right - - in
B - Left - -
C Top-right - Bend in
Table 2: Intuitive location, type, and direction of deformation.
Discussion
Both the user studies produced useful insight about
how participants prefer to hold and interact with the
right hand in landscape mode. Observing the recorded
videos comprehensively, we noticed that participants
were familiar with all the grips but switching between
Grip-A and Grip-C seemed to be more effortless. We
found “Horizontal bend in” gesture to be the most
naturally occurring and intuitive gesture that can be
performed in all the three grips. “Top-right corner bend
in” and “Horizontal fold in” are relatively more natural
and intuitive to perform in more than one grip. These
gestures can be mapped to referents of equal priority in
respective grips. In Grip-A and C (Table 2), location
“Top-right corner” and direction “in” are more intuitive.
Similar findings were also reported in portrait mode in
[6]. In Grip-B, location “Left side” is more intuitive as
right corners are primarily involved in gripping the
device. In Grip-C, type of deformation “bend” is more
intuitive as holding the mockup on the palm of the
hand restricted the participants to perform fold gesture.
The range of distinct gestures proposed in each grip
was influenced by the control over the grip during
deformation. Control over the grip during interaction
can be described by the number of fingers involved in
support and balance (Table 1). Maximum control was
observed in Grip-C followed by Grip-A and then Grip-B
during deformation. This allowed users to propose
maximum (20 unique gestures) gestures in Grip-C with
comparatively richer naturalness and intuitiveness.
Conclusion
In this paper, we present a set of user-defined
deformation gestures for right-handed landscape mode
interaction. We conducted the 1st study to identify 3
most preferred right-handed landscape mode grips. In
the 2nd study, we gathered a set of user-defined
deformation gestures corresponding to the 3 grips.
Finally, we identified respectively 4, 3, and 3 unique
deformation gestures for Grip-A, B and C based on their
naturalness and intuitiveness scores. We also identified
that “Horizontal bend in”, “Top-right corner bend in”
and “Horizontal fold in” gestures can be performed in
more than one grip. We further discussed the effect of
the grips on the intuitiveness of performing
deformation gestures. We believe that this study will
provide valuable implications and reference in the
selection of deformation gestures for flexible handheld
devices. As succeeding work, we are investigating on
gesture-referent mappings for one-handed landscape
mode interactions considering the impact of occlusion.
References [1] Sung-Jung Cho, Roderick Murray-Smith, Yeun-Bae
Kim. 2007. Multi-context photo browsing on mobile devices based on tilt dynamics. In Proceedings of the 9th international conference on Human computer interaction with mobile devices
Figure 6: Final set of user-defined
gestures in (a) Grip-A, (b) Grip-B,
and (c) Grip-C (F represents
frequency and MI represents
mean intuitiveness score)
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
234
and services, 190-197. https://doi.org/10.1145/1377999.1378006
[2] Teemu T. Ahmaniemi, Johan Kildal, Merja Haveri. 2014. What is a device bend gesture really good for?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3503-3512. https://doi.org/10.1145/2556288.2557306
[3] Rudolf Arnheim. 1983. The power of the center: A study of composition in the visual arts. Univ of California Press.
[4] Nick Byrd. 2014. Intuitive and Reflective Responses in Philosophy.
[5] Matthew Ernst, Audrey Girouard. 2016. Bending
blindly: exploring bend gestures for the blind. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2088-2096. https://doi.org/10.1145/2851581.2892303
[6] Audrey Girouard, Jessica Lo, Md Riyadh, Farshad Daliri, Alexander Keith Eady, Jerome Pasquero. 2015. One-handed bend interactions with
deformable smartphones. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, 1509-1518. https://doi.org/10.1145/2702123.2702513
[7] Sukeshini A. Grandhi, Gina Joue, Irene Mittelberg. 2011. Understanding naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 821-824. https://doi.org/10.1145/1978942.1979061
[8] Curtis Hardyck, Lewis F. Petrinovich. 1977. Left-handedness. Psychological Bulletin, 84(3), 385-404. http://dx.doi.org/10.1037/0033-2909.84.3.385
[9] Jochen Huber, Jürgen Steimle, Max Mühlhäuser. 2010. Toward more efficient user interfaces for
mobile video browsing: an in-depth exploration of
the design space. In Proceedings of the 18th ACM international conference on Multimedia, 341-350. https://doi.org/10.1145/1873951.1873999
[10] Marco A. Hudelist, Klaus Schoeffmann, Laszlo Boeszoermenyi. 2013. Mobile video browsing with the thumbbrowser. In Proceedings of the 21st ACM international conference on Multimedia, 405-406. https://doi.org/10.1145/2502081.2502242
[11] Wolfgang Hürst, Philipp Merkle. 2008. One-handed
mobile video browsing. In Proceedings of the 1st international conference on Designing interactive user experiences for TV and video, 169-178. https://doi.org/10.1145/1453805.1453839
[12] Yuichiro Kota, Minoru Ikeda. 2006. Mobile phone and camera, U.S. Patent 7,003,318, Filed October 14, 2004, issued February 21, 2006.
[13] Byron Lahey, Audrey Girouard, Winslow Burleson, Roel Vertegaal. 2011. PaperPhone: understanding the use of bend gestures in mobile devices with
flexible electronic paper displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1303-1312. https://doi.org/10.1145/1978942.1979136
[14] Sang-Su Lee, Sohyun Kim, Bopil Jin, Eunji Choi, Boa Kim, Xu Jia, Daeeop Kim, Kun Pyo Lee. 2010. How users manipulate deformable displays as input devices. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 1647-1656. https://doi.org/10.1145/1753326.1753572
[15] Keyur Sorathia, Aditi Singh, Mayank Chhabra. 2017. BendSwipe: One Handed Target Zooming for Flexible Handheld Display. In IFIP Conference on Human-Computer Interaction, 431-435. https://doi.org/10.1007/978-3-319-68059-0_45
[16] Kristen Warren, Jessica Lo, Vaibhav Vadgama,
Audrey Girouard. 2013. Bending the rules: bend gesture classification for flexible displays. In Proceedings of the SIGCHI Conference on Human
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
235
Factors in Computing Systems, 607-610. https://doi.org/10.1145/2470654.2470740
[17] Doug Wightman, Tim Ginn, Roel Vertegaal. 2011. Bendflip: examining input techniques for electronic book readers with flexible form factors. In IFIP Conference on Human-Computer Interaction, 117-133. https://doi.org/10.1007/978-3-642-23765-2_9
[18] Yue Wu, Tao Mei, Nenghai Yu, Shipeng Li. 2012. Accelerometer-based single-handed video browsing on mobile devices: design and user studies. In Proceedings of the 4th International Conference on Internet Multimedia Computing and
Service, 157-160. https://doi.org/10.1145/2382336.2382381
WIP Session 1 TEI '19, March 17–20, 2019, Tempe, AZ, USA
236