International Journal of Human-Computer · International Journal of Human-Computer Studies ... Kochi University of Technology, 185 Miyanokuchi, ... Skinput (Harrison et al ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Int. J. Human-Computer Studies 109 (2018) 79–88
Contents lists available at ScienceDirect
International Journal of Human-Computer Studies
journal homepage: www.elsevier.com/locate/ijhcs
Designing motion marking menus for people with visual impairments
Nem Khan Dim
a , Kibum Kim
a , b , Xiangshi Ren
a , ∗
a The Center for Human-Engaged Computing, Kochi University of Technology, 185 Miyanokuchi, Tosayamada-Cho, Kami-Shi, Kochi, 782–8502, Japan b Department of Game and Mobile Engineering, Keimyung University, Daegu, 42601, South Korea
a r t i c l e i n f o
Keywords:
Marking menus
Motion gestures
Accessibility
People with visual impairments
a b s t r a c t
Current smartphone accessibility for people with visual impairments relies largely on screen readers and voice
commands. However, voice commands and screen readers are often not ideal because users with visual impair-
ments rely mostly on hearing ambient sound from the environment for their safety in mobile situations. Recent
research has shown that marking menus in mobile devices provide fast and eyes-free access for sighted users
Francone et al., 2010; Oakley and Park, 2007, 2009. However, the literature is lacking design implications and
adaptations that meet the needs of users with visual impairments. This paper investigates the capabilities of vi-
sually impaired people to invoke smartphone functions using marking menus via 3D motions. We explore and
present the optimal numbers of menu items (breadth) and menu levels (depth) for marking menus that people
with visual impairments can successfully adopt. We also compared a marking menu prototype to TalkBack TM
which is an accessibility menu system in Android smartphones. The experimental results show that our partici-
pants could perform menu selections using marking menus faster than when using TalkBack. Based on the study
results, we provide implications and guidelines for designing marking menus and motion gesture interfaces for
’clock origin. Each menu breadth had four depths from 1 to 4 levels.
he schematic diagram of menu breadth and levels is shown in Fig. 2 .
ur interview study informed us that people with visual impairments
ere more familiar with body-centric directions (left, right) than cardi-
al directions (east, west). According to these findings, we labeled all
enus in relation to the human body (left, right).
The rationale for selecting the number of menu items and menu lev-
ls was based on the experimental design from a previous marking menu
tudy ( Kurtenbach, 1993 ). We also added 6-item menus because we hy-
othesized that this menu layout could be a good option if menu selec-
ions in 8-item menus were found too error prone for our participants.
In each menu layout, three different menu selections were presented.
ach selection was presented three times. Three different menu selec-
ions were configured to include both easy and difficult target menus.
asy menus were those that existed along vertical and horizontal axes
i.e. left, right, up, down). Difficult menus were those that existed in
ff-axis positions (i.e. upper-left, upper-right, etc.). We paid attention
o ensure that menu selections included easy, moderately difficult and
ifficult menus. Each participant performed 432 trials in total (16 menu
ayouts × 3 menu selections × 3 repetitions × 3 blocks = 432). The
rder of the menu selections was counterbalanced using a Latin Square.
he occurrences of menu layouts were randomized among the partici-
ants.
.2. Participants
Twelve participants (2 females and 10 males), with ages ranging
rom 27 to 78 years, participated in the experiment. Seven of them par-
N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88
Fig. 2. Schematic diagram of menus used in the experiment. (a) menu items (breadth), (b) menu levels (depth). To select menus at more than one level, users perform continuous
marking gestures in the direction of the target menus (e.g., right, then downward). c) Tactile patterns for twelve directions. The participants were trained with regard to the directions
using tactile patterns at the parctice session.
t
v
a
o
s
t
i
4
n
l
t
o
2
×
T
t
C
v
c
s
d
g
t
V
n
s
4
U
c
e
f
Fig. 3. Participant performing menu selection.
w
f
u
w
t
t
a
icipated in the preliminary interviews. Two of the participants had light
isual perception and three of them could see objects, but none were
ble to distinguish between objects. The rest were totally blind. Two of
ur participants were late blind (onset at age 22 years and 48 years re-
pectively), the rest were early blind (onset from ages 0 to 3 years). All
he participants were right-handed. Each was paid $15 for their partic-
pation.
.3. Apparatus
Our study required a sound motion capturing system to avoid sensor
oise confusion. Thus, for capturing user hand movement in menu se-
ections, we used twelve Bonita 10 cameras (frame rate: 250 fps, resolu-
ion: 1 megapixel (1024 × 1024), lens operating range: up to 13m, angle
f view wide (4mm): 70.29° × 70.29°, angle of view narrow (12mm):
6.41° × 26.41°).
The participants held an Alcatel OneTouch Dimension - 5.37 × 2.74
0.30 in, Weight - 1.6g) smartphone for performing motion gestures.
en markers (14mm wide) were attached on the smartphone, the par-
icipant ’s wrist, forearm, elbow, arm and shoulder ( Fig. 3 ).
For the Vicon system, Nexus 2.2.1 motion capture software was used.
ustom software was also developed to track user hand movement data
ia the Vicon system. When a user started and ended a gesture, the
ustom software logged frame numbers from the Vicon cameras and po-
ition data from the markers which were attached to the body. Those
ata were later used for calculating errors and the response times of
estures. The software was also used to trigger real-time instructions to
he participant. Gesture recognition was stroke-by-stroke recognition.
elocity threshold of the movements (i.e., 4mm/s) was used to recog-
ize the final stroke. The motion tracking system ’s accuracy was tested
everal times to confirm that the desired movements were achievable.
.4. Building models
Each participant used their dominant hand to perform the gestures.
ser models were built for both left and right hands. Each model in-
luded four segments: shoulder, arm, forearm and a phone. Three mark-
rs were used for the shoulder segment, two for the arm, two for the
orearm and three for the phone.
82
One marker on the shoulder, one on the elbow and one on the phone
ere used to connect two segments, that is, connecting the hand and the
orearm, connecting the forearm and the upper arm, and connecting the
pper arm and the shoulder.
To enable the system to differentiate between the left and right hands
ith no errors, markers on the left arm were placed in different positions
o those on the right arm. Markers on the left arm were placed higher
han those on the right hand so that the two ratios would be different
nd the system would not confuse the left and the right arms.
N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88
4
t
s
t
e
a
t
t
a
d
a
t
t
c
e
l
i
s
f
s
p
W
(
t
t
t
w
f
s
p
r
O
p
t
t
fi
t
c
o
a
4
a
m
(
m
a
e
b
W
b
W
d
f
b
1
s
R
Table 1
Mean error rates. Standard errors are shown in paren-
theses.
Breadth Level
1 2 3 4
4 0.0 0.0 0.93 0.0
(0.0) (0.0) (0.93) (0.0)
6 0.93 2.78 10.04 20.34
(0.93) (2.46) (3.03) (4.01)
8 0.0 2.78 7.41 15.74
(0.0) (1.45) (3.16) (2.54)
12 35.18 24.07 36.33 37.96
(2.68) (4.90) (2.38) (3.73)
w
(
f
b
a
a
d
(
m
s
m
t
W
t
t
b
m
h
g
p
(
I
r
w
o
0
4
a
d
f
b
t
f
b
m
b
e
8
s
s
b
m
a
F
.5. Procedure
Participants ’ consent forms were gathered before the trials. The par-
icipants were then introduced to the purpose of the study and to the
tudy procedure. Four of our participants used their left hands to hold
he phone and to perform the gestures. Markers were put in place, and
ach participant was handed a smartphone on which three markers were
ttached.
The participants were then taught the menu layouts. To help the par-
icipants learn the menu layouts, we put tactile patterns of directions on
he wall ( Fig. 2 c). The participants were exposed to the tactile directions
nd allowed to practice until they felt that they were familiar with all
irections and direction labels. The participants were also trained with
ll menu layouts via verbal instructions. Then they were trained with
he experimental custom software to help them become familiar with
he experimental setup. To minimize the learning effect, we used 4 × 1
ondition which was the easiest layout for menu selections.
After training and practice sessions, the experiment started. Each
xperimental trial occurred as follows. The participant stood in a ‘re-
axed ’ state with the arms beside the body. Then the system ’s speaker
nstructed the participants to adopt a ‘ready ’ state. The participant re-
ponded by moving the dominant hand to any position in which they
elt comfortable to start the menu selection. After 500 milliseconds, the
ystem read out the menu name that the participant had to select. The
articipant responded by moving the dominant hand to select the menu.
hen the speed of the user ’s hand movements exceeded the threshold
i.e. 4mm/s) the system logged that point as the starting point of a ges-
ure. When the user slowed the speed down or stopped arm movement,
hat is, when the movement speed was less than the pre-established
hreshold and if the distance between the start point and the end point
as more than 20 mm, the system logged that point as the ending point
or the gesture. For menu selections with more than one menu level, the
ystem read out the next menu to be selected while the participant was
erforming the gesture for the current menu level. This procedure was
epeated for each gesture stroke until all menu levels were completed.
nce the participants finished menu selection, the system instructed the
articipant to return to the ‘relaxed ’ state. After three seconds, the next
rial started with the same procedure. No backward or reselecting of
argets was allowed. All the experimental instructions were made in the
rst language of each participant. We measured errors and menu selec-
ion times.
After the experiment, the participants were asked for their subjective
omments on each menu layout and direction that were particularly easy
r difficult for them to understand and to perform. Participants were also
sked for comments on the gestures.
.6. Results
We measured errors and menu selection times. Errors were defined
s errors in the angle of the participants ’ hand movements in respective
enu layouts. Angular errors were calculated for each gesture stroke
each menu level). Thus, if one of the gesture strokes with a particular
enu level had an angle error, that menu selection was regarded as
n erroneous menu selection. Response time was defined as the time
lapsed until the completion of the menu from the start of the gesture
y the participant.
Each participant performed three blocks of trials in the experiment.
e first checked the learning effect on menu selection over the three
locks of trials to see if the data collected had reached a level of stability.
e analyzed the error rates after each experiment block. Error rates
ecreased over blocks. The average error rates were 12.67 (SD = 1.31)
or block 1, 11.11 (SD = 0.87) for block 2, and 8.31 (SD = 0.83) for
lock 3. Significant differences in error rates were found between block
and block 3 (p < 0.05) and between block 2 and block 3 (p < 0.05). No
ignificant difference was found between block 1 and block 2 (p = 0.27).
esponse time also decreased over blocks. The average response times
83
ere 2.55 (SD = 0.33) for block 1, 2.43 (SD = 0.39) for block 2, and 2.03
SD = 0.14) for block 3. Significant differences in response times were
ound between block 1 and block 3 (p < 0.05) and between block 2 and
lock 3 (p < 0.05). No significant difference was found between block 1
epending on the consequences of errors and the purpose of use. Thus,
n table Table 3 , we showed menu layouts with error rates less than 10%
s suggested in ( Kurtenbach, 1993 ).
A previous marking menu study which was performed with sighted
eople recommended using menus with a breadth of 4 up to 4
evels (maximum error rate 5.10%, SD = 4.20) and menus with a
readth of 8 up to level 2 (maximum error rate 8.82%, SD = 4.62)
Kurtenbach, 1993 ). In our study, the nearest error rate to 8.82% was
ound when the participants performed gestures for 8-item menus up to
evel 3 (error rate 7.41%, SD = 3.16). Comparing results from our study
nd those from ( Kurtenbach, 1993 ), our results indicated lower error
ates in all menu layouts. Thus, it is questionable whether people with
isual impairments have any advantage over sighted people in spatial
bility when navigating hierarchic marking menus.
In addition to the differences between participants with and with-
ut visual impairments from our study and those in (Kurtenbach,
993), there is also the difference regarding input modalities. In
Kurtenbach, 1993 ), the participants drew marks on the screen using
pen or a mouse. In our study, the participants performed markings
sing motion gestures. Physical movement with motion gesture is an
xpressive channel which has six degrees of freedom that can be easily
pplied for proprioception. Moreover, menu direction in our study was
abeled in relation to the human body (left, right) which allowed the
articipants to rely on kinesthetic cues (awareness of object positions in
pace with respect to one ’s body).
In our experiments, the used menus were largely divided into three
roups: one group consisted of entirely on-axis menus (i.e. up, down,
eft and right), another group consisted of a mixture of on-axis and off-
xis menus, and the last group consisted entirely of off-axis menus (i.e.
pper-left, lower-right, etc). To find out whether these different layouts
f menu configuration effect error rates among menu breadth and depth,
e analyzed user performance in these three different menu configura-
N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88
Fig. 6. The menu systems used in Study 2. (a) TalkBack - linear gestures, (b) TalkBack - spatial localization. Four menus were arranged horizontally. Each menu is about 1.75cm wide
and about 1.5cm high, (c) Motion Marking Menus.
t
=
i
r
h
m
t
m
1
c
d
f
l
f
5
f
t
s
w
p
b
m
c
p
g
t
p
A
m
U
t
5
m
l
a
t
p
F
o
p
c
m
t
=
o
e
5
t
o
r
$
(
W
i
w
a
p
s
u
d
i
n
I
l
i
t
A
i
r
o
5
e
f
t
g
l
p
a
p
t
o
ions. As we expected, off-axis had a significant effect on errors (F(2, 22)
16.2, p < .05). We also found that axis and menu levels had significant
nteractions on errors (F(8, 88) = 30.56, p < .05). This was particularly
eflected in 6-item menus. In 6-item menus, errors become significantly
igher in level 3 (up to 10.04). As the number of menu levels increased,
ore off-axis items were included in the combination because most of
he menus existed off-axis (60, 120, 240 and 300 degrees) in 6-item
enus. Similarly, in 8-item menus, errors increased from 7.41 up to
5.74 in level 4.
To evaluate the most challenging strokes/directions of motion, we
ompared performance in pairs of menus (i.e. left/right, up/down,
own-left/down-right and up-left/up-right). In general, less errors were
ound for right and up-right directions when compared to left and up-
eft. However, no significant effect from direction on performance was
ound for any direction.
. Study 2: efficiency of motion marking menus in smartphones
In Study 1, we investigated usable menu layouts for marking menus
or people with visual impairments. Study 2 was intended to investigate
he feasibility of proposed interactions in terms of user performance and
ubjective assessment using the motion marking menu system. Study 2
as intended to answer Q3: How receptive are people with visual im-
airments to motion marking menu systems for mobile interactions? To
etter understand the relative efficiency and user assessments of the
arking menu system, we compared our prototype to TalkBack TM , a
ommercial menu system on Android devices for users with visual im-
airments.
In TalkBack, users can browse the menus either by directional flick
estures, that is, right for the next content and left for the previous con-
ent ( Fig. 6 a) or by localizing the menu contents (i.e. by estimating the
osition of the menu on the screen and directly pointing on it ( Fig. 6 b).
fter each gesture or spatial localization, the system reads out the active
enu name. Users gesture or localize until they find the desired menu.
sers confirm the menu by performing a double-tap on the screen. Users
hen perform the same procedure to find the submenu.
.1. Experimental design
We developed a proof-of-concept prototype for motion marking
enus with a breadth of four and a depth of two, i.e., 4 × 2 menu
ayout. Two mobile phone applications, phone book and music, were
ssigned for the high-level menu items. Four submenus were assigned
o each high-level menu. That is, four contact names were assigned in the
hone book menu and four submenus were assigned in the music menu
ig. (6) . Before the experimental tasks, a pilot study was conducted with
ne participant with visual impairments and two blind-folded partici-
ants. We chose four by four menu selections in order to minimize the
ognitive load of the participants when remembering the menu layout.
85
The participants selected each submenu three times. The same eight
enu items were assigned in the TalkBack system. Therefore, each par-
icipant performed 48 (2 menu systems × 8 menu selections × 3 blocks
48) menu selections in total. The design was within-subjects and the
rder of experiment was counter balanced. Task completion times and
rror rates were measured.
.2. Participants and apparatus
Twelve participants (3 female, 9 male) took part in Study 2. Nine of
hem had participated in Study 1. Ages ranged from 26 to 78 years. Four
f them were smartphone users and they were experienced with screen
eader functions on smartphones. All were right-handed. Each was paid
10 for their participation.
For both menu systems, an Alcatel OneTouch Idol 2S smartphone