Attention and selection behavior on "universal search" result pages based on proposed Google commitments of Oct. 21, 2013 Report about an eye tracking pilot study commissioned by ICOMP Initiative for a Competitive Online Marketplace
Introduction 2
Short report 4
Setup of the study 5
Detailed results of the study 14
References 23
Institute of Communication and Media Research
GERMAN SPORTS UNIVERSITY COLOGNE
- November 21st, 2013 -
2
1. Introduction
The pilot study as described in the following
illustrates an empirical approach regarding the
distribution of attention and the selection
behavior on Google "Universal Search" result pages.
The ISO Norm line 9241-11states as the three
criteria for usability: effectiveness, efficiency
and satisfaction. One tries to adjust the
interface design to these criteria for example by
user tests, interviews, heuristic evaluation by
experts or modeling the expected user behavior.
In markets with a large variety of offers and little
possibility of differentiation, providers can gain
a decisive competitive advantage by user
oriented interfaces. A precondition of this is that
relevant information can be obtained for entrepreneurial decisions to this regard.
3
Commissioned by the Initiative for a
Competitive Online Marketplace ICOMP, the
Institute of Communication and Media
Research (IKM) at the German Sports
University Cologne (DSHS) developed and realized a suitable procedure. In the following
• the main findings are briefly presented
• the methods are discussed
• individual results are briefly explained
A pilot study could be conducted within the time
and budget constraints, which already presents clear results.
4
2. Short report
(1) Google placed "Sponsored" Google-own page elements grab a pre-dominant amount of the total visual attention.
(2) The "alternative search sites" even with small logos do not evoke enough visual attention to stimulate the users to click on them.
(3) Map services can not compete against Google because the placed Google Map pane and Google Images thumbnails get visual attention of users more, earlier and longer than all other page elements.
(4) Google Ads presented as lateral skyscraper-shaped link lists are vastly irrelevant for visual attention as well as for mouse clicking behavior.
(5) The visual attention for organic links on the SERPs is negligible compared to those Google elements placed above them, enhanced with pictures. With browser windows not opened wide, organic links may go unseen.
5
3. Setup of the study
The study presented here represents a user
oriented study using a special kind of observation.
By interviewing, general usage habits regarding
the users' preferred web browsers, a specific
search engine or a specific online service can
be learned. Data on questionnaires cannot
represent the behavior retrospectively because
this is affected, for example, by memory effects
(Felix, Hink & Minor, 2001; Möller & Schierl,
2012). Therefore an eye tracker was used
additionally. This was a high resolution device
of the company Tobii, based on the cornea
reflex method (Duchowski, 2007). The use of
this elaborate method is necessary in order to
catch spontaneous user behavior that is not
adulterated by willful processes (Byrne et al.,
1999; Graham & Jefferey, 2011; Ferreira et al.,
6
2011). In respect to methodology Just and
Carpenter (1980) assumed that the users'
visual attention is focused on the object that is
also the object of the cognitive processing (eye-mind hypothesis). It is further assumed that
the time of fixation corresponds to the time of
cognitive processing (immediacy assumption).
35 persons were recruited as test subjects, 20
men and 15 women. The specification was to
measure eye movements on search engine
result pages (SERPs) given as examples in the
October 21, 2013 commitment proposal by
Google, Annex V, of the Case COMP/C-
3/39.740 – Foundem and others, and an
additional SERP depicting Google Maps. For
this, the following three SERPs were used as target pictures.
7
Fig.1: Target picture "iPod shopping" (from Annex V, P. 35)
8
Fig.2: Target picture "map London" (screenshot from actual Google web search engine)
9
Fig.3: Target picture "flight search" (from Annex V, p.39)
In order to have the subjects behave as they
would in normal, everyday Internet use, the
purpose of the test was not revealed to them
and to preserve their attention for the stimuli
10
used during the entire measuring cycle, a special test setting was created.
The subjects sat about 2 meters in front of a 46-
inch plasma screen monitor that showed
various images. The eye tracker stood about 70
centimeters (= 2,297 ft) in front of the subjects
and therefore was outside the regular sight field
of the participants. All instructions, stimuli and
questions were only shown on the screen to
avoid any distortions caused by the
experimenter. In order to get the subjects used
to the situation and gain their attention they
were informed that they were participating in a
perception experiment and were presented
several images and questions unrelated to the
SERPs to be examined; They were presented
with a Stroop test of word recognition and
thereafter with a geometrical-optical illusion
image. Then the subjects were asked to look at
an advertisement for eight seconds. In the
advertisement, a spokesperson with a visible
11
physical handicap was promoting a fictional
automotive brand. Besides the fictional brand,
we questioned the subjects about any
conspicuous issues concerning the
spokesperson. Then the subjects got written
instructions to click on a link they desired in an
on-line advertisement. Then they were asked
about any sponsors visible on this site. Next the
subjects were shown a picture of a table tennis
athlete in a tuxedo and asked to assign a name
to him from a list of choices. Then the
participants were shown a picture of the same
athlete performing his sport and were asked for
a name choice. After this subjects were asked if
they are iPod users, if they usually buy those
electronic devices in specialised shops,
discount shops, or online shops. After this
subjects were instructed to assume intention to
buy an iPod nano and select one of six search
queries by mouse click. The instruction
continued that after their choice a Google
12
search result page will be visible and that
subjects should click on it to proceed (cf. fig. 1,
taken from Annex V, p. 35) buying an iPod
nano. Next instruction was to memorise as
many details of an upcoming iPod ad. The ad
was visible for six seconds. Then subjects had
to answer how many apps were visible on the
display of iPod that was presented in the ad.
After that subjects were asked to use a SERP
to choose a map to find Apple Stores in London
(cf. fig.2, screenshot from actual Google web
search engine). Then they were asked if they
ever booked a flight on the web. The
information was given that web booking is very
usual in the US. The next SERP showed
different flight offers (cf. fig.2, taken from Annex
V, p. 39). Subjects should choose one of the
offerings, again by clicking as usual. After this
subjects were thanked for their participation and dismissed.
13
The eye movements and fixations were
recorded at a rate of 120 Hertz during the entire
measuring section. Video and audio of all
subjects was recorded with a webcam in
parallel and integrated with the stimuli
presented, the mouse clicks and the tracking data into an overall view.
For each target picture the following parameters were computed from the raw data:
1) Fixation count
2) Absolute duration of the fixations
3) Time to first fixation.
A fixation occurs when an eye movement rests
for at least 200 milliseconds on an area of 50 pixels.
The results arrived at are described in detail in the following chapter.
14
4. Detailed results of the study.
As the first analytical step, the left mouse clicks
of the subjects were marked in the three target
pictures as symbolized ( ) mouse-left-click.
At the same time these data were to be
complemented by the data of the eye
movement. Therefore corresponding heat maps
were generated. In these heat maps, a color
code indicates the diverse intensities of the
visual attention triggered by the SERPs. Analog
to a traffic light, the color red represents the
maximal viewing time and the color green the
minimal duration. In all areas without color
assignment there was no fixation. For optimal
comparability, each heat map is shown on a
single page followed by a table containing the
corresponding data. Results will be discussed below.
15
Fig.4: Mouse clicks and total visual attention on target picture "iPod shopping"
16
Table 1: Overview of the recorded parameters on target picture "iPod shopping"
Area of
interest (AOI)
Fixation
duration (FD)
Time to
First
Fixation (TFF)
Fixation
Counts (FCʼ)
Mouse
Clicks (MC)
ad on top of SERP
0.54 1.08 192 0
Googles
own
product
search results
1.10 0.27 715 18
“alternative sites”
0.67 3.11 311 1
organic blue links
1.60 0.35 459 13
17
ig.5: Mouse clicks and total visual attention on the target picture "map London"
18
Table 2: Overview of the recorded parameters on target picture "map London"
AOI FD TFF FC MC
Google map 0.77 0.46 286 15
“Google images” 0.98 0.62 348 12
organic blue links 1.39 2.87 322 6
19
Fig.6: Mouse clicks and total visual attention on the target picture "flight search"
20
Table 3: Overview of the recorded parameters on target picture "flight search"
AOI FD TFF FC MC
ad on top of SERP 0.85 1.50 143 2
Googles own product search results
2.03 0.10 556 15
“alternative sites” 0.91 5.54 110 4
organic blue links 1.16 3.35 196 11
Skyscraper-shaped ads 1.66 3.93 174 3
21
The SERP for the search term "iPod" reveals
that thumbnail product pictures guide the
spontaneous visual attention of users to the
"Google Shopping Results". For this area the
time to first fixation is least: 0.27 seconds. This
concurs with the amount of mouse clicks: 56%
of the participants clicked into this area with
Google's own product search. The average
amount of visible attention (the total fixation
duration) is 1.1 seconds. The maximum total
duration of fixations on the organic links is 1.6
seconds. 41% clicked one of the organic blue
links, mostly the first (Apple.com). One has to
keep in mind that the organic links were fixated
for the first time only after 3.5 seconds in
average! This is a long period, even for highly
involved users. To adhere to Google's example,
and to save time, the original mockup
screenshot from their commitments paper was
used, which contained Apple.com as first
ranked organic blue link. Further testing on
22
SERP examples without the very well known
brand Apple leading the organic blue links,
might show less attention and clicks to this position and even more in the product images.
The related ad at the top of the SERP had no
clicks and the "alternative search sites" were
scanned visually even less and only clicked
once. Both show a very small time in fixation
duration of round about half a second. One may
conclude that these areas are evoking no interest from recipients.
On the target SERP "map London" 46% of the
participants clicked on the Google map
whereas 36% clicked on the area of "Google
Images" which refers to another Google
service. Also the total duration of fixations for
both areas is comparable (0.77 seconds for the
Google map, 0.98 seconds for the "Google
images"). But the Google map gets visual
attention of the user faster (0.49 seconds)
23
compared to the "Google Image" (0.62
seconds). Number one organic link
mylondonmap.com is a site using the
syndicated Google Maps API and received two
clicks. The famous, official Transport for
London maps received no click. Further down
the SERP organic links again referring to Google Maps received four clicks together.
The "flight search" SERP in fig. 3 is the
visually most complex stimulus in the study
design. Similar to the SERP "iPod shopping"
only few mouse clicks are on the ad on the top
of the SERP (two clicks = 6%). Most clicks are
on the Google Flight Search "sponsored links"
area (43%), followed by the "organic links" area
(31%). The "alternative search sites" had little
visual attention with correspondingly only four
clicks (11%) and the ads on the right side just
three clicks (9%). Next to the amount of mouse
clicks, the total visual attention (2.03 seconds)
as well as the very small time to first fixation
24
(0.10 seconds) indicates that the Google Flight
Search area is the most attractive one of this
SERP. It is remarkable that even the organic
links at the bottom of the Page get more visual
attention (1.16 seconds) and faster (3.35
seconds) than the "alternative search sites"
(duration 0.91 seconds; time to first fixation = 5.54 seconds).
Finally all target pictures suggest that the
implemented "Sponsored (i)" labels do not
reach the beholder’s visual attention. This
finding is supported by the fact that those
labels were not clicked.
25
5. References
Byrne, M.D., Anderson, J.R., Douglass, S. and Matessa, M. (1999). Eye tracking the visual search of click-down menus. In Human Factors in Computing Systems: CHIʼ99 Conference Proceedings, ACM Press, New York, pp. 402-409.
Duchowski, A.(2007). Eye Tracking Methodology. Theory and Practice. Springer, London.
Felix, R., Hink, W. and Minor, M. (2001). Hereʼs looking at you kid: Eye-tracking applications in consumer information processing research. In Proceedings of the Annual Meeting of the Society for Marketing Advances, pp. 157-161.
Ferreira, P., Rita, P., Morais, P., Oliveira, J., Gamito, P., Santos, N., Soares, F. and Sottomayor, C. (2011). Grabbing attention while reading website pages: the influence of verbal emotional cues in advertising. Journal of Eyetracking, Visual Cognition and Emotion, 1, 1, pp. 64-68.
Graham, D.J. and Jefferey, R.W. (2011). Location, Location, Location: Eye-Tracking Evidence that Consumers Preferentially View Prominently Positioned Nutrition Information. Journal of the American Diet Association. 111, 11, 1704-1711. doi:
26
10.1016/j.jada.2011.08.005 Just, M.A. and Carpenter, P.A. (1980). A
theory of reading: from eye fixation to comprehension. Psychological Review, 87, 4, pp. 329–354.
Möller, C. and Schierl, T. (2012). Attention and selection behavior on "universal search" result pages. Journal of Eyetracking, Visual Cognition and Emotion, 2, 1, pp. 1-10.
27
Institute of Communication and Media Research German Sport University Cologne Am Sportpark Müngersdorf 6 50933 Cologne Germany Contact: Dr. Carsten Möller [email protected] +49(0)221 4982 6243 www.ikm-dshs.de