-
Alice in Warningland:A Large-Scale Field Study of Browser
Security Warning Effectiveness
Devdatta AkhaweUniversity of California, Berkeley\ast
[email protected]
Adrienne Porter FeltGoogle, Inc.
[email protected]
AbstractWe empirically assess whether browser security warn-
ings are as ineffective as suggested by popular opinionand
previous literature. We used Mozilla Firefox andGoogle Chromes
in-browser telemetry to observe over25 million warning impressions
in situ. During our fieldstudy, users continued through a tenth of
Mozilla Fire-foxs malware and phishing warnings, a quarter of
GoogleChromes malware and phishing warnings, and a third ofMozilla
Firefoxs SSL warnings. This demonstrates thatsecurity warnings can
be effective in practice; securityexperts and system architects
should not dismiss the goalof communicating security information to
end users. Wealso find that user behavior varies across warnings.
In con-trast to the other warnings, users continued through 70.2%of
Google Chromes SSL warnings. This indicates thatthe user experience
of a warning can have a significantimpact on user behavior. Based
on our findings, we makerecommendations for warning designers and
researchers.
1 Introduction
An oft-repeated maxim in the security community is thefutility
of relying on end users to make security decisions.Felten and
McGraw famously wrote, Given a choicebetween dancing pigs and
security, the user will pickdancing pigs every time [21]. Herley
elaborates [17],
Not only do users take no precautions againstelaborate attacks,
they appear to neglect evenbasic ones. For example, a growing body
ofmeasurement studies make clear that ...[users]are oblivious to
security cues [26], ignore cer-tificate error warnings [30] and
cannot tell legit-imate web-sites from phishing imitations
[11].1
\ast The Mozilla Firefox experiments were implemented while the
au-thor was an intern at Mozilla Corporation.
1Citations updated to match our bibliography.
The security communitys perception of the oblivioususer evolved
from the results of a number of laboratorystudies on browser
security indicators [5, 11, 13, 15, 26,30, 34]. However, these
studies are not necessarily rep-resentative of the current state of
browser warnings in2013. Most of the studies evaluated warnings
that havesince been deprecated or significantly modified, often
inresponse to criticisms in the aforementioned studies. Ourgoal is
to investigate whether modern browser securitywarnings protect
users in practice.
We performed a large-scale field study of user deci-sions after
seeing browser security warnings. Our studyencompassed 25,405,944
warning impressions in GoogleChrome and Mozilla Firefox in May and
June 2013. Wecollected the data using the browsers telemetry
frame-works, which are a mechanism for browser vendors tocollect
pseudonymous data from end users. Telemetryallowed us to
unobtrusively measure user behavior duringnormal browsing
activities. This design provides realism:our data reflects users
actual behavior when presentedwith security warnings.
In this paper, we present the rates at which users clickthrough
(i.e., bypass) malware, phishing, and SSL warn-ings. Low
clickthrough rates are desirable because theyindicate that users
notice and heed the warnings. Click-through rates for the two
browsers malware and phish-ing warnings ranged from 9% to 23%, and
users clickedthrough 33.0% of Mozilla Firefoxs SSL warnings.
Thisdemonstrates that browser security warnings can effec-tively
protect most users in practice.
Unfortunately, users clicked through Google ChromesSSL warning
70.2% of the time. This implies that theuser experience of a
warning can have a significant impacton user behavior. We discuss
several factors that mightcontribute to this warnings higher
clickthrough rates. Ourpositive findings for the other five
warnings suggest thatthe clickthrough rate for Google Chromes SSL
warningcan be improved.
-
We also consider user behaviors that are indicative ofattention
to warnings. We find that Google ChromesSSL clickthrough rates vary
by the specific type of error.In Mozilla Firefox, a fifth of users
who choose to clickthrough an SSL warning remove a default option,
showingthey are making cognitive choices while bypassing
thewarning. Together, these results contradict the stereotypeof the
wholly oblivious user with no interest in security.
We conclude that users can demonstrate agency whenconfronted
with browser security warnings. Users do notalways ignore security
warnings in favor of their desiredcontent. Consequently, security
experts and platformdesigners should not dismiss the role of the
user. Wefind that the user experience of warnings can have
anenormous impact on user behavior, justifying efforts tobuild
usable warnings.
Contributions. We make the following contributions:
To our knowledge, we present the first in-depth,large-scale
field study of browser security warnings.
We survey prior laboratory studies of browser secu-rity warnings
and discuss why our field study datadiffers from prior
research.
We analyze how demographics (operating systemand browser
channel), warning frequency, and warn-ing complexity affect users
decisions. Notably,we find evidence suggesting that technically
skilledusers ignore warnings more often, and warning fre-quency is
inversely correlated with user attention.
We provide suggestions for browser warning design-ers and make
recommendations for future studies.
2 Background
Web browsers show warnings to users when an attackmight be
occurring. If the browser is certain that an attackis occurring, it
will show an error page that the user cannotbypass. If there is a
chance that the perceived attack is afalse positive, the browser
will show a bypassable warningthat discourages the user from
continuing. We study onlybypassable warnings because we focus on
user decisions.
A user clicks through a warning to dismiss it and pro-ceed with
her original task. A user leaves the warningwhen she navigates away
and does not continue with heroriginal task. A clickthrough rate
describes the proportionof users who clicked through a warning
type. When auser clicks through a warning, the user has (1)
ignoredthe warning because she did not read or understand it or(2)
made an informed decision to proceed because she be-lieves that the
warning is a false positive or her computeris safe against these
attacks (e.g., due to an antivirus).
Figure 1: Malware warning for Google Chrome
Figure 2: Malware warning for Mozilla Firefox
Figure 3: SSL warning for Google Chrome. The first
paragraphchanges depending on the specific SSL error.
Figure 4: SSL warning for Mozilla Firefox
-
Figure 5: SSL Add Exception Dialog for Mozilla Firefox
We focus on three types of browser security warnings:malware,
phishing, and SSL warnings. At present, allthree types of warnings
are full-page, interstitial warningsthat discourage the user from
proceeding.
2.1 Malware and Phishing Warnings
Malware and phishing warnings aim to prevent users fromvisiting
websites that serve malicious executables or tryto trick users.
Google Chrome and Mozilla Firefox relyon the Google Safe Browsing
list [25] to identify mal-ware and phishing websites. The browsers
warn usersaway from the sites instead of blocking them because
theSafe Browsing service occasionally has false positives,although
the false positive rate is very low [25].
Clickthrough Rate. If a malware or phishing warningis a true
positive, clicking through exposes the user to adangerous
situation. Nearly all Safe Browsing warningsare true positives; the
false positive rate is low enoughto be negligible. The ideal
clickthrough rate for malwareand phishing warnings is therefore
close to 0%.
Warning Mechanisms. The browsers routinely fetch alist of
suspicious (i.e., malware or phishing) sites fromSafe Browsing
servers. If a user tries to visit a site thatis on the locally
cached list, the browser checks with theSafe Browsing service that
the URL is still on the malwareor phishing list. If the site is
still on one of the lists, thebrowser presents a warning.
The two browsers behave differently if a page loadsa third-party
resource (e.g., a script) from a URL on theSafe Browsing list.
Google Chrome stops the page loadand replaces the page with a
warning. Mozilla Firefoxblocks the third-party resource with no
warning. As aresult, Mozilla Firefox users can see fewer warnings
thanGoogle Chrome users, despite both browsers using thesame Safe
Browsing list.
Warning Design. Figures 1 and 2 show the GoogleChrome and
Mozilla Firefox warnings. Their phishingwarnings are similar to
their respective malware warn-ings. When a browser presents the
user with a malware orphishing warning, she has three options:
leave the pagevia the warnings escape button, leave the page by
closingthe window or typing a new URL, or click through thewarning
and proceed to the page. The warnings also allowthe user to seek
more information about the error.
Click Count. Mozilla Firefox users who want to bypassthe warning
need to click one button: the Ignore thiswarning link at the bottom
right. On the other hand,Chrome users who want to bypass the
warning need toclick twice: first on the Advanced link, and then
onProceed at your own risk.
2.2 SSL WarningsThe Secure Sockets Layer (SSL/TLS) protocol
providessecure channels between browsers and web servers, mak-ing
it fundamental to user security and privacy on theweb. As a
critical step, the browser verifies a serversidentity by validating
its public-key certificate against aset of trusted root
authorities. This validation will fail inthe presence of a
man-in-the-middle (MITM) attack.
Authentication failures can also occur in a wide vari-ety of
benign scenarios, such as server misconfigurations.Browsers usually
cannot distinguish these benign scenar-ios from real MITM attacks.
Instead, browsers presentusers with a warning; users have the
option to bypass thewarning, in case the warning is a false
positive.
Clickthrough Rate. We hope for a 0% clickthrough ratefor SSL
warnings shown during MITM attacks. However,many SSL warnings may
be false positives (e.g., servermisconfigurations). There are two
competing views re-garding SSL false positives. In the first,
warning textshould discourage users from clicking through both
trueand false positives, in order to incentivize developersto get
valid SSL certificates. In the other, warning textshould provide
users with enough information to correctlyidentify and dismiss
false positives. The desired click-through rates for false-positive
warnings would be 0%and 100%, respectively. In either case, false
positives areundesirable for the user experience because we do
notwant to annoy users with invalid warnings. Our goal istherefore
a 0% clickthrough rate for all SSL warnings:users should heed all
valid warnings, and the browsershould minimize the number of false
positives.
Warning Design. Figures 3 and 4 present Google Chromeand Mozilla
Firefoxs SSL warnings. The user can leavevia the warnings escape
button, manually navigate away,or click through the warning. In
Mozilla Firefox, theuser must also click through a second dialog
(Figure 5) tobypass the warning.
-
The browsers differ in their presentation of the techni-cal
details of the error. Google Chrome places informationabout the
specific error in the main warning (Figure 3, firstparagraph),
whereas Firefox puts the error informationin the hidden Technical
Details section and the secondAdd Exception dialog (Figure 5).
Click Count. Mozilla Firefoxs SSL warning requiresmore clicks to
bypass. Google Chrome users clickthrough a single warning button to
proceed. On the otherhand, Mozilla Firefoxs warning requires three
clicks:(1) click on I Understand The Risks, (2) click on theAdd
Exception button, which raises a second dialog,(3) click on Confirm
Security Exception in the seconddialog. By default, Firefox
permanently remembers theexception and will not show the warning
again if the userreencounters the same certificate for that
website. Incontrast, Chrome presents the warning every time anddoes
not remember the users past choices.
2.3 Browser Release ChannelsMozilla and Google both follow rapid
release cycles.They release official versions of their browsers
every sixor seven weeks, and both browsers update automatically.The
official, default version of a browser is referred to asstable
(Google Chrome) or release (Mozilla Firefox).
If users are interested in testing pre-release browserversions,
they can switch to a different channel. The sta-ble/release channel
is the recommended channel for endusers, but a minority of users
choose to use earlier chan-nels to test cutting-edge features. The
Beta channel isseveral weeks ahead of the stable/release channel.
Thedeveloper (Google Chrome) or Aurora (Mozilla Fire-fox) channel
is delivered even earlier. Both browsers alsooffer a nightly
(Mozilla Firefox) or Canary (GoogleChrome) release channel, which
updates every day andclosely follows the development
repository.
The pre-release channels are intended for advancedusers who want
to experience the latest-and-greatest fea-tures and improvements.
They give website, extension,and add-on developers time to test
their code on upcom-ing versions before they are deployed to end
users. Theearly channels are not recommended for typical end
usersbecause they can have stability issues, due to being un-der
active development. The rest of this paper assumesa positive
correlation between pre-release channels useand technical ability.
While this matches the intentionof browser developers, we did not
carry out any study tovalidate this assumption.
3 Prior Laboratory Studies
We survey prior laboratory studies of SSL and phishingwarnings.
The body of literature paints a grim pictureof browser security
warnings, but most of the warningshave since been deprecated or
modified. In some cases,warnings were changed in response to these
studies.
Only two studies evaluated warnings that are similarto the
modern (June 2013) browser warnings that westudy in this paper.
Sunshine et al. and Sotirakopouloset al. reported clickthrough
rates of 55% to 80% for theFirefox 3 and 3.5 SSL warnings, which
are similar butnot identical to the current Firefox SSL warning
[29, 30].However, Sotirakopoulos et al. concluded that
laboratorybiases had inflated both studies clickthrough rates
[29].
3.1 SSL Warnings
SSL warnings are the most studied type of browser warn-ing.
Usability researchers have evaluated SSL warningsin both
SSL-specific studies and phishing studies becauseSSL warnings and
passive indicators were once viewedas a way to identify phishing
attacks.2
Dhamija et al. performed the first laboratory study ofSSL
warnings in 2006. They challenged 22 study partic-ipants to
differentiate between phishing and legitimatewebsites in Mozilla
Firefox 1.0.1 [11]. In this version,the warning was a modal dialog
that allowed the user topermanently accept, temporarily accept, or
reject the cer-tificate. When viewing the last test website,
participantsencountered an SSL warning. The researchers
reportedthat 15 of their 22 subjects (68%) quickly clicked
throughthe warning without reading it. Only one user was laterable
to tell the researchers what the warning had said.The authors
considered the clickthrough rate of 68% aconservative lower bound
because participants knew thatthey should be looking for security
indicators.
In 2007, Schechter et al. studied user reactions to Inter-net
Explorer 7s SSL warning, which is the same one-clickinterstitial
that is present in all subsequent versions of In-ternet Explorer
[26]. Participants encountered the warningwhile logging into a bank
website to look up information.The researchers were aware of
ecological validity con-cerns with laboratory studies and split
their participantsinto three groups: participants who entered their
own cre-dentials, a role-playing group that entered fake
passwords,and a security-primed role-playing group that enteredfake
passwords. Overall, 53% of the total 57 participantsclicked
through. However, only 36% of the non-role-playing group clicked
through. The difference betweenthe role-playing participants and
non-role-playing partic-
2There is evidence that modern phishing sites can have valid
SSLcertificates [24].
-
ipants was statistically significant, illustrating one
chal-lenge of experiments in artificial environments.
Sunshine et al. performed multiple studies of SSL warn-ings in
2009 [30]. First, they conducted an online survey.They asked 409
people about Firefox 2, Firefox 3, andInternet Explorer 7 warnings.
Firefox 2 had a modal dia-log like Firefox 1.0.1, and Firefox 3s
warning is similarbut not identical to the current Firefox warning.
Lessthan half of respondents said they would continue to thewebsite
after seeing the warning. As a follow-up, Sun-shine et al. also
conducted a laboratory study that exposed100 participants to SSL
warnings while completing infor-mation lookup tasks. The
clickthrough rates were 90%,55%, and 90% when participants tried to
access their bankwebsites in Firefox 2, Firefox 3, and Internet
Explorer7, respectively. The clickthrough rates increased to
95%,60%, and 100% when participants saw an SSL warningwhile trying
to visit the university library website.
Sotirakopoulos et al. replicated Sunshines laboratorySSL study
with a more representative population sam-ple [29]. In their study,
80% of participants using Firefox3.5 and 72% of participants using
Internet Explorer 7clicked through an SSL warning on their bank
website.More than 40% of their participants said that the
labora-tory environment had influenced them to click throughthe
warnings, either because they felt safe in the studyenvironment or
were trying to complete the experimentaltask. Sotirakopoulos et al.
concluded that the laboratoryenvironment biased their results, and
they suspect thatthese biases are also present in similar
laboratory studies.
Bravo-Lillo et al. interviewed people about an SSLwarning from
an unspecified browser [5]. They asked 20participants about the
purpose of the warning, what wouldhappen if a friend were to click
through, and whether afriend should click through the warning.
Participantswere separated into advanced and novice browserusers.
Advanced participants said they would not clickthrough an SSL
warning on a bank website, but noviceparticipants said they
would.
Passive Indicators. Some studies focused on passiveSSL
indicators, which non-interruptively show the statusof the HTTP(S)
connection in the browser UI. Althoughbrowsers still have passive
SSL indicators, interruptiveSSL and phishing warnings are now the
primary tool forcommunicating security information to users.
Friedman et al. asked participants whether screenshotsof
websites depicted secure connections; many partici-pants could not
reliably determine whether a connectionwas secure [15]. Whalen and
Inkpen used eye-trackingsoftware to determine that none of their 16
participantslooked at the lock or key icon in the URL bar,
HTTP(S)status in the URL bar, or the SSL certificate when asked
tobrowse websites normally [33]. Some browsers modify
the lock icon or color of the URL bar to tell the user whena
website has an Extended Validation (EV) SSL certifi-cate. Jackson
et al. asked 27 study subjects to classify12 websites as either
phishing or legitimate sites, but theEV certificates did not help
subjects identify the phish-ing sites [19]. In a follow-up study,
Sobey et al. foundthat none of their 28 subjects clicked on the EV
indica-tors, and the presence of EV indicators did not
affectdecision-making [28]. Similarly, Biddle et al. found
thatstudy participants did not understand Internet
Explorerscertificate summaries [3].
In 2012, a Google Chrome engineer mentioned highclickthrough
rates for SSL warnings on his blog [20]. Weexpand on this with a
more accurate and detailed view ofSSL clickthrough rates in Google
Chrome.
3.2 Phishing Warnings
Phishing warnings in contemporary browsers are
active,interstitial warnings; in the past, they have been
passiveindicators in toolbars. Researchers have studied whetherthey
are effective at preventing people from entering theircredentials
into phishing websites.
Wu et al. studied both interstitial and passive phish-ing
warnings [34]. Neither of the warnings that theyevaluated are
currently in use in browsers. First, theylaunched phishing attacks
on 30 participants. The par-ticipants role-played during the
experiment while usingsecurity toolbars that display passive
phishing warnings.Despite the toolbars, at least one attack fooled
20 outof 30 participants. In their next experiment, they asked10
study participants to perform tasks on PayPal and ashopping wish
list website; they injected modal phishingwarnings into the
websites. None of the subjects enteredthe credentials into the
PayPal site, but the attack on thewish list site fooled 4 subjects.
The authors do not reportthe warning clickthrough rates.
Egelman et al. subjected 60 people to simulated phish-ing
attacks in Internet Explorer 7 or Mozilla Firefox2.0 [13]. Firefox
2.0 had a modal phishing dialog that isnot comparable to the
current Mozilla Firefox phishingdialog, and Internet Explorer had
both passive and activewarnings. Participants believed that they
were taking partin a laboratory study about shopping. The
researchersasked participants to check their e-mail, which
containedboth legitimate shopping confirmation e-mails and
similarspear phishing e-mails sent by the researchers. Users
whoclicked on the links in the phishing e-mails saw a
phishingwarning. Participants who saw Mozilla Firefoxs
activewarning, Internet Explorers active warning, or
InternetExplorers passive warning were phished 0%, 45%, and90% of
the time, respectively. The clickthrough rateswere an unspecified
superset of the rates at which peoplefell for the phishing
attacks.
-
3.3 Malware Download WarningsGoogle Chrome and Microsoft
Internet Explorer also dis-play non-blocking warning dialogs when
users attemptto download malicious executables. In a blog post, a
Mi-crosoft employee stated that the clickthrough rate for Inter-net
Explorers SmartScreen warning was under 5% [16].We did not study
this warning for Google Chrome, andMozilla Firefox does not have
this warning.
4 Methodology
We rely on the telemetry features implemented in MozillaFirefox
and Google Chrome to measure clickthrough ratesin situ. Telemetry
is a mechanism for browser vendors tocollect pseudonymous data from
end users who opt in tostatistics reporting. Google Chrome and
Mozilla Firefoxuse similar telemetry platforms.
4.1 Measuring Clickthrough RatesWe implemented metrics in both
browsers to count thenumber of times that a user sees, clicks
through, or leavesa malware, phishing, or SSL warning. Based on
this data,we can calculate clickthrough rates for each warning
type.As discussed in Section 2, we report only the
clickthroughrates for warnings that the user can bypass. We
measuredthe prevalence of non-bypassable warnings separately.
Tosupplement the clickthrough rates, we recorded whetherusers
clicked on links like Help me understand, View,or Technical
Details.
Bypassing some warnings takes multiple clicks, andour
clickthrough rates for these warnings represent thenumber of users
who completed all of the steps to proceedto the page. For Mozilla
Firefoxs SSL warning (whichtakes three clicks to proceed), we
recorded how oftenusers perform two intermediate clicks (on Add
Excep-tion or Confirm Security Exception) as well as theoverall
clickthrough rate.
We also measured how often users encounter and clickthrough
specific SSL errors. In addition to the overallclickthrough rates
for the warnings, we collected click-through data for each type of
Mozilla Firefox SSL errorand the three most common Google Chrome
SSL errors.
Our Mozilla Firefox data set does not allow us to trackspecific
telemetry participants. In Google Chrome, wecan correlate warning
impressions with psuedonymousbrowser client IDs; however, the
sample size for mostindividual users is too small to draw
conclusions. Wetherefore report the results of measurements
aggregatedacross all users unless otherwise specified. The
telemetryframeworks do not provide us with any personal or
demo-graphic information except for the operating system andbrowser
version for each warning impression.
4.2 Measuring Time Spent on WarningsWe also used the Google
Chrome telemetry framework toobserve how much time Google Chrome
users spent onSSL warnings. Timing began as soon as an SSL
warningcame to the foreground in a tab. In particular,
We recorded the time spent on a warning and associ-ated it with
the outcome (click through or leave).
We recorded the time spent on a warning and associ-ated it with
the error type, if it was one of the threemost common error types
(untrusted authority, namemismatch, and expired certificate).
Together, these correspond to five timing measurements(two for
outcome and three for error type). For scalability,the telemetry
mechanism in Google Chrome only allowstiming measurements in
discrete buckets. As a result, ouranalysis also treats time as a
discrete, ordinal variable.
We used log-scaled bucket sizes (e.g., the first bucketsize is
45ms but the last is 90,279ms) with 50 buckets,ranging from 0ms to
1,200,000ms, for the two outcomehistograms. The three error type
histograms had 75 buck-ets each, ranging from 0ms to 900,000ms. We
used morebuckets for the error histograms because we
anticipatedthat they would be more similar to each other.
4.3 EthicsWe collected data from users who participate in
theirbrowsers broad, unpaid user metrics programs. At firstrun of a
browser, the browser asks the user to share usagedata. If the user
consents, the browser collects data onperformance, features, and
stability. In some pre-releasedeveloper channels, data collection
is enabled by default.The browser periodically sends this
pseudonymous dataover SSL to the central Mozilla or Google servers
foranalysis. The servers see the IP addresses of clients
bynecessity, but they are not attached to telemetry data.
Alltelemetry data is subject to strict privacy policies
andparticipants can opt out by changing their settings [7,
23].Multiple Google Chrome committers and Mozilla
Firefoxcontributors reviewed the data collection code to ensurethat
the metrics did not collect any private data.
This work is not considered human subjects research byUC
Berkeley because the student did not have access todatabase
identifiers or personally identifying information.
4.4 Data Collection
Collection Period. Google Chromes malware and phish-ing
measurement code was in place in Chrome 24 priorto our work, and
our SSL measurement code was addedto Google Chrome 25. The Google
Chrome data in this
-
paper was collected April 28 - May 31, 2013. Our MozillaFirefox
measurement code was added to Firefox 17, anda bug in the SSL
measurement code was fixed in Firefox23. The data on the Firefox
malware warning, phishingwarning, and SSL Add Exception dialog was
collectedMay 1-31, 2013. The data on Firefox SSL warnings
wascollected June 1 - July 5, 2013, as the Firefox 23 fixprogressed
through the various release channels.
Sample Sizes. In Google Chrome, we recorded 6,040,082malware
warning impressions, 386,350 phishing warningimpressions, and
16,704,666 SSL warning impressions.In Mozilla Firefox, we recorded
2,163,866 malware warn-ing impressions, 100,004 phishing warning
impressions,and 10,976 SSL warning impressions. Appendix A fur-ther
breaks downs these sample sizes by OS and channel.
Number of Users. For Mozilla Firefox, we recordedwarning
impressions from the approximately 1% of Fire-fox users who opt in
to share data with Mozilla via teleme-try. In Google Chrome, we
observed malware, phishing,and SSL warning impressions on
2,148,026; 204,462; and4,491,767 clients (i.e., browser installs),
respectively.
4.5 Method Limitations
Private Data. Due to privacy constraints, we could notcollect
information about users personal demographicsor browsing habits.
Consequently, we cannot measurewhether user behavior differs based
on personal character-istics, the target site, or the source of the
link to the site.We also cannot identify SSL false positives due to
captiveportals, network proxies, or server misconfigurations.
Sampling Bias. The participants in our field study are nota
random population sample. Our study only representsusers who opt in
to browser telemetry programs. Thismight present a bias. The users
who volunteered might bemore likely to click through dialogs and
less concernedabout privacy. Thus, the clickthrough rates we
measurecould be higher than population-wide rates. Given thatmost
of our observed rates are low, this bias augments ourclaim that
clickthrough rates are lower than anticipated.
Overrepresentation. We present clickthrough ratesacross all
warnings shown to all users. A subset of userscould potentially be
overrepresented in our analysis.Within the Google Chrome data set,
we identified andremoved a small number of overrepresented clients
whowe believe are either crawlers or malware researchers.We were
unable to remove individual clients from theMozilla Firefox set,
but we do not believe this representsa bias because we know that
the overrepresented clientsin Chrome still contributed fewer than
1% of warningimpressions. Some clients experienced multiple typesof
warning impressions; we investigated this in Chrome
and found that the clickthrough rates do not differ ifwe remove
non-independent clients. Our large samplesizes and small critical
value (\alpha = 0.001) should furtherameliorate these concerns.
Frames. Our original measurement for Mozilla Firefoxdid not
differentiate between warnings shown in top-levelframes (i.e.,
warnings that fill the whole tab) and warningsshown in iframes. In
contrast, Google Chrome alwaysshows malware and phishing warnings
in the top-levelframe and does not render any warning type in
iframes.Since users might not notice warnings in iframes, the
twometrics are not necessarily directly comparable.
Upon discovering this issue, we modified our Firefoxmeasurement
implementation to take frame level intoaccount. Our new
implementation is not available to allFirefox users yet, but we
have data for recent pre-releasechannels. For malware and phishing
warning impressionscollected from the beta channel, the
clickthrough rate forthe top-level frame is within two percentage
points ofthe overall clickthrough rate. This is due to the
relativeinfrequency of malware and phishing warnings in iframesand
the low overall clickthrough rate. Since the framelevel does not
make a notable difference for malware andphishing warnings, we
present the overall rates (includingboth top-level frames and
iframes) for the full samplesizes in Section 5.1. The difference is
more importantfor SSL warnings: the clickthrough rate for
top-levelframes is 28.7 percentage points higher than the
overallclickthrough rate of 4.3%. Consequently, Section 5.2presents
only the top-level frame rate for SSL warnings,although it limits
our sample to pre-release users.
5 Clickthrough Rates
We present the clickthrough data from our measurementstudy.
Section 5.1 discusses malware and phishing warn-ings together
because they share a visual appearance. Wethen present rates for
SSL warnings in Section 5.2.
5.1 Malware and Phishing Warnings
The clickthrough rates for malware warnings were 7.2%and 23.2%
in stable versions of Mozilla Firefox andGoogle Chrome,
respectively. For phishing warnings,we found clickthrough rates of
9.1% and 18.0%. In thissection, we discuss the effects of warning
type, demo-graphics, and browser on the clickthrough rates.
5.1.1 Malware Rates by Date
The malware warning clickthrough rates for GoogleChrome vary
widely by date. We have observed click-through rates ranging from
11.2% to 24.9%, depending
-
Operating Malware PhishingSystem Firefox Chrome Firefox
Chrome
Windows 7.1% 23.5% 8.9% 17.9%MacOS 11.2% 16.6% 12.5% 17.0%Linux
18.2% 13.9% 34.8% 31.0%
Table 1: User operating system vs. clickthrough rates for
mal-ware and phishing warnings. The data comes from stable
(i.e.,release) versions.
Channel Malware PhishingFirefox Chrome Firefox ChromeStable 7.2%
23.2% 9.1% 18.0%Beta 8.7% 22.0% 11.2% 28.1%Dev 9.4% 28.1% 11.6%
22.0%Nightly 7.1% 54.8% 25.9% 20.4%
Table 2: Release channel vs. clickthrough rates for malware
andphishing warnings, for all operating systems.
on the week, since the current version of the warningwas
released in August 2012. In contrast, the MozillaFirefox malware
warning clickthrough rate across weeksstays within one percentage
point of the month-longaverage. We did not observe similar temporal
variationsfor phishing or SSL warnings.
Recall from Section 2.1 that Google Chrome andMozilla Firefoxs
malware warnings differ with respectto secondary resources: Google
Chrome shows aninterstitial malware warning if a website
includessecondary resources from a domain on the Safe Browsinglist,
whereas Mozilla Firefox silently blocks the resource.We believe
that this makes Google Chromes malwareclickthrough rates more
sensitive to the contents of theSafe Browsing list. For example,
consider the case wherea well-known website accidentally loads an
advertisementfrom a malicious domain. Google Chrome would showa
warning, which users might not believe because theytrust the
website. Mozilla Firefox users would not seeany warning.
Furthermore, Chrome phishing warningsare less likely to be due to
secondary resources, and thatwarnings clickthrough rates do not
vary much by time.
5.1.2 Malware/Phishing Rates by Warning Type
In Mozilla Firefox, we find a significantly higher click-through
rate for phishing warnings than malware warn-ings (\chi 2 test:
p(1) < 0.0001). This behavior is rational:a malware website can
infect the users computer withoutany action on the users part, but
a phishing website canonly cause harm by tricking the user at a
later point intime. Mozilla Firefox makes this priority ordering
explicitby choosing to display the malware warning if a website
is listed as both malware and phishing.3 However, thepractical
difference is small: 7.2% vs. 9.1%.
In Google Chrome, the average malware clickthroughrate is higher
than the phishing clickthrough rate. How-ever, the malware
clickthrough rate fluctuates widely (Sec-tion 5.1.1); the malware
clickthrough rate is sometimeslower than the phishing clickthrough
rate.
5.1.3 Malware/Phishing Rates by Demographics
We consider whether users of different operating systemsand
browser release channels react differently to warn-ings. As Table 1
depicts, Linux users have significantlyhigher clickthrough rates
than Mac and Windows userscombined for the Firefox malware warning,
Firefox phish-ing warning, and Chrome phishing warning (\chi 2
tests:p(1) < 0.0001). While the low prevalence of malwarefor
Linux could explain the higher clickthrough rates forthe Firefox
malware warning, use of Linux does not pro-vide any additional
protection against phishing attacks.The Chrome malware warning does
not follow the samepattern: Windows users have a significantly
higher click-through rate (\chi 2 tests: p(1) < 0.0001).
We also see differences between software releasechannels (Table
2). Nightly users click through GoogleChrome malware and Firefox
phishing warnings at muchhigher rates than stable users, although
they click throughFirefox malware and Google Chrome phishing
warningsat approximately the same rates.
In several cases, Linux users and early adopters clickthrough
malware and phishing warnings at higher rates.One possible
explanation is that a greater degree of tech-nical skill as
indicated by use of Linux or early-adopterversions of browsers
corresponds to reduced risk aver-sion and an increased willingness
to click through warn-ings. This does not hold true for all
categories and warn-ings (e.g., nightly and stable users click
through the Fire-fox malware warning at the same rate), suggesting
theneed for further study.
5.1.4 Malware/Phishing Rates by Browser
Google Chrome stable users click through phishing warn-ings more
often than Mozilla Firefox stable users. Thisholds true even when
we account for differences in howthe browsers treat iframes
(Section 4.5). Mozilla Fire-foxs beta channel users still click
through warnings at alower rate when we exclude iframes: 9.6% for
malwarewarnings, and 10.8% for phishing warnings.
One possibility is that Mozilla Firefoxs warnings aremore
frightening or more convincing. Another possi-
3Google Chrome will display both warnings. To preserve
inde-pendence, our measurement does not include any warnings with
bothphishing and malware error messages. Dual messages are
infrequent.
-
bility is that the browsers have different demographicswith
different levels of risk tolerance, which is reflectedin their
clickthrough rates. There might be differencesin technical
education, gender, socioeconomic status, orother factors that we
cannot account for in this study. Insupport of this theory, we find
that differences betweenthe browsers do not hold steady across
operating systemsor channels. The gap between the browsers narrows
orreverses for some categories of users, such as Linux usersand
nightly release users.
5.2 SSL WarningsThe clickthrough rates for SSL warnings were
33.0% and70.2% for Mozilla Firefox (beta channel) and GoogleChrome
(stable channel), respectively.
5.2.1 SSL Rates by Demographic
In Section 5.1, we observed that malware and
phishingclickthrough rates differed across operating systems
andchannels. For SSL, the differences are less pronounced.
As with the malware and phishing warnings, nightlyusers click
through SSL warnings at a higher rate for bothFirefox and Chrome
(\chi 2 tests: p < 0.0001).
The effect of users operating systems on SSL click-through rates
differs for the two browsers. In Firefox,Linux users are much more
likely to click through SSLwarnings than Windows and Mac users
combined (\chi 2 test:p < 0.0001), although it is worth noting
that the FirefoxLinux sample size is quite small (58). In Chrome,
Win-dows users are very slightly more likely to click throughSSL
warnings than Linux and Mac users combined (\chi 2
test: p < 0.0001).
5.2.2 SSL Rates by Browser
We find a large difference between the Mozilla Firefoxand Google
Chrome clickthrough rates: Google Chromeusers are 2.1 times more
likely to click through an SSLwarning than Mozilla Firefox users.
We explore fivepossible causes.
Number of Clicks. Google Chrome users click one but-ton to
dismiss an SSL warning, but Mozilla Firefox usersneed to click
three buttons. It is possible that the addi-tional clicks deter
people from clicking through. However,we do not believe this is the
cause of the rate gap.
First, the number of clicks does not appear to affectthe
clickthrough rates for malware and phishing warn-ings. Mozilla
Firefoxs malware and phishing warningsrequire one click to proceed,
whereas Google Chromesmalware and phishing warnings require two.
The GoogleChrome malware and phishing warnings with two clicksdo
not have lower clickthrough rates than the MozillaFirefox warnings
with one click. Second, as we discussin Section 5.2.3, 84% of users
who perform the first two
Operating SSL WarningsSystem Firefox Chrome
Windows 32.5% 71.1%MacOS 39.3% 68.8%Linux 58.7% 64.2%Android NC
64.6%
Table 3: User operating system vs. clickthrough rates for
SSLwarnings. The Google Chrome data is from the stable channel,and
the Mozilla Firefox data is from the beta channel.
Channel SSL WarningsFirefox ChromeRelease NC 70.2%Beta 32.2%
73.3%Dev 35.0% 75.9%Nightly 43.0% 74.0%
Table 4: Channel vs. clickthrough rates for SSL warnings.
clicks in Mozilla Firefox also perform the third. Thisindicates
that the extra click is not a determining deci-sion point.
Unfortunately, we do not have data on thedifference between the
first and second clicks.
Warning Appearance. The two warnings differ in sev-eral ways.
Mozilla Firefoxs warning includes an imageof a policeman and uses
the word untrusted in the title.These differences likely contribute
to the rate gap. How-ever, we do not think warning appearance is
the sole orprimary factor; the browsers malware and phishing
warn-ings also differ, but there is only about a 10%
differencebetween browsers for these warnings.
Certificate Pinning. Google Chrome ships with a listof pinned
certificates and preloaded HTTP Strict Trans-port Security (HSTS)
sites. Users cannot click throughSSL warnings on sites protected by
these features. Certifi-cate pinning and HSTS cover some websites
with impor-tant private data such as Google, PayPal, and Twitter
[8].In contrast, Mozilla Firefox does not come with manypreloaded
pinned certificates or any pre-specified HSTSsites. As a result,
Chrome shows more non-bypassablewarnings: our field study found
that 20% of all GoogleChrome SSL warning impressions are
non-bypassable, ascompared to 1% for Mozilla Firefox.
Based on this, we know that Mozilla Firefox users seemore
warnings for several critical websites. If we assumethat users are
less likely to click through SSL warningson these critical
websites, then it follows that MozillaFirefoxs clickthrough rate
will be lower. This potentialbias could account for up to 15 points
of the 37-point gapbetween the two clickthrough rates, if we were
to assumethat Google Chrome users would never click through
SSLerrors on critical websites if given the chance.
-
Remembering Exceptions. Due to the permanentlystore this
exception feature in Mozilla Firefox, MozillaFirefox users see SSL
warnings only for websites withoutsaved exceptions. This means that
Mozilla Firefox usersmight ultimately interact with websites with
SSL errorsat the same rate as Google Chrome users despite
havinglower clickthrough rates. For example, imagine a userthat
encounters two websites with erroneous SSL config-uration: she
leaves the first after seeing a warning, butvisits the second
website nine times despite the warning.This user would have a 50%
clickthrough rate in MozillaFirefox but a 90% clickthrough rate in
Google Chrome,despite visiting the second website at the same
rate.
We did not measure how often people revisit websiteswith SSL
errors. However, we suspect that people dorepeatedly visit sites
with warnings (e.g., a favorite sitewith a self-signed
certificate). If future work were toconfirm this, there could be
two implications. First, ifusers are repeatedly visiting the same
websites with errors,the errors are likely false positives; this
would mean thatthe lack of an exception-storing mechanism
noticablyraises the false positive rate in Google Chrome.
Second,warning fatigue could be a factor. If Google Chrome usersare
exposed to more SSL warnings because they cannotsave exceptions,
they might pay less attention to eachwarning that they
encounter.
Demographics. Its possible that the browsers have dif-ferent
demographics with different levels of risk toler-ance. However,
this factor likely only accounts for afew percentage points because
the same demographic ef-fect applies to malware and phishing
warnings, and thedifference between browsers for malware and
phishingwarnings is much smaller.
5.2.3 SSL Rates by Certificate Error Type
To gain insight into the factors that drive clickthroughrates,
we study whether the particular certificate erroraffects user
behavior.
Google Chrome. Google Chromes SSL warning in-cludes a short
explanation of the particular error, andclicking on Help me
understand will open a more-detailed explanation. In case a
certificate has multipleerrors, Google Chrome only shows the first
error out ofuntrusted issuer error, name mismatch error, and
certifi-cate expiration error, respectively.
Table 5 presents the clickthrough rates by error typesfor Google
Chrome. If Google Chrome users are payingattention to and
understanding the warnings, one wouldexpect different clickthrough
rates based on the warningtypes. We find a 24.4-point difference
between the click-through rates for untrusted issuer errors and
expired certifi-cate errors. One explanation could be that
untrusted issuer
Certificate Error Percentageof Total
ClickthroughRate
Untrusted Issuer 56.0% 81.8%Name Mismatch 25.0% 62.8%Expired
17.6% 57.4%Other Error 1.4% All Error Types 100.0% 70.2%
Table 5: Prevalence and clickthrough rates of error types for
theGoogle Chrome SSL warning. Google Chrome only displaysthe most
critical warning; we list the error types in order, withuntrusted
issuer errors as the most critical. Data is for the stablechannel
across all operating systems.
errors appear on unimportant sites, leading to higher
click-through rates without user attention or
comprehension;however, the Mozilla Firefox data suggests
otherwise.An alternative explanation could be that expired
certifi-cates, which often occur for websites with previouslyvalid
certificates [1], surprise the user. In contrast, un-trusted
certificate errors always occur for a website andconform with
expectations.
Mozilla Firefox. Mozilla Firefoxs SSL warning does notinform the
user about the particular SSL error by default.4
Instead, the secondary Add Exception dialog presentsall errors
in the SSL certificate. The user must confirmthis dialog to
proceed.
Table 6 presents the rates at which users confirm theAdd
Exception dialog in Mozilla Firefox. The errortypes do not greatly
influence the exception confirmationrate. This indicates that the
Add Exception dialog doesnot do an adequate job of explaining
particular error cate-gories and their meaning to the users. Thus,
users ignorethe categories and click through errors at the same
rate.This finding also suggests that the differences in
click-through rates across error types in Google Chrome cannotbe
attributed to untrusted issuer errors corresponding tounimportant
websites; if that were the case, we wouldexpect to see the same
phenomenon in Firefox.
Error Prevalence. The frequency of error typesencountered by
users in our field study also indicates thebase rate of SSL errors
on the web. Our Google Chromedata contradicts a previous network
telemetry study,which suggested that untrusted issuer errors
correspondto 80% of certificate errors seen on the wire [18].
Also,Google Chrome users see fewer untrusted issuer errorsthan
Mozilla Firefox users; this may be because MozillaFirefox users are
more likely to click on the AddException dialog for untrusted
issuer errors. Recall thatwe collect the Mozilla Firefox error type
statistics onlyafter a user clicks on the Add Exception button.
4This information is available under the Technical details link,
butour measurements indicate that it is rarely opened (Section
5.2.4).
-
Certificate Error Percentageof Total
ConfirmationRate
Untrusted Issuer 38% 87.1%Untrusted and NameMismatch
26.4% 87.9%
Name Mismatch 15.7% 80.3%Expired 10.2% 80.7%Expired, Untrusted
andName Mismatch
4.7% 87.6%
Expired and Untrusted 4.1% 83.6%Expired and Name Mis-match
0.7% 85.2%
None of the above
-
Date Invalid Name Invalid Authority Invalid
Untrusted Issuer
Name Mismatch
Expired
0400445495550611679755839933103711531282142515841761195721752418268829883321369141034560506856336261695977358597955610621118051312114584162101801720026222592474127499305653397337761419714665051851576326405871200791388796197768108669120785134252149220165857184349204903227748253140281363312733347601386356429432477311530528589678655423728498809721900000
29726 57841 181330 0.0195129206 0.0172819263 0.01866638871162
2575 8338 0.0008972521 0.0007693671 0.00072967581488 3179 10348
0.0011135482 0.0009498322 0.00093438692048 4330 13327 0.0014341184
0.0012937318 0.00128603792586 6129 17672 0.0019016838 0.001831243
0.00162387413679 9237 26943 0.0028993361 0.0027598616
0.00231022155722 16407 49754 0.0053540277 0.004902138
0.00359311979773 31289 98207 0.0105680549 0.0093486314
0.006136937917250 56722 189550 0.0203974747 0.0169475877
0.010832106728694 91402 318968 0.0343241452 0.0273093934
0.018018346141621 131258 468412 0.0504058134 0.0392177016
0.026135832756998 167760 594135 0.063934865 0.0501238905
0.035791792569097 190512 671696 0.0722812039 0.0569218087
0.043389337978215 199358 691696 0.0744334038 0.059564846
0.049114969883027 196592 667097 0.0717863055 0.0587384113
0.052136656682860 186615 604480 0.0650480904 0.0557574501
0.052031789380363 175240 536431 0.0577253378 0.0523587898
0.050463802676433 161351 471226 0.050708628 0.0482089882
0.047995966172087 146299 412341 0.0443720134 0.0437117016
0.045266903167063 132085 356626 0.0383765224 0.0394647954
0.042112091361199 118225 310620 0.033425817 0.0353236585
0.038429802955696 106502 271295 0.0291940539 0.0318210216
0.034974203950183 97773 237722 0.0255812635 0.029212942
0.031512325445687 87280 207645 0.0223446776 0.0260778086
0.028689070240904 81185 182721 0.0196626061 0.0242567242
0.025685593936318 75563 160877 0.0173119733 0.0225769643
0.022805823333073 70029 141367 0.0152125023 0.0209234974
0.020768131429324 64795 124313 0.0133773214 0.0193596655
0.018413953526578 60270 110218 0.0118605585 0.018007671
0.016689607724143 55668 97779 0.0105219978 0.0166326701
0.015160553821750 52553 87571 0.009423515 0.01570196
0.013657873719838 48189 78589 0.008456962 0.0143980696
0.012457236717971 44742 70654 0.0076030767 0.0133681635
0.011284857416648 40896 64167 0.0069050106 0.0122190428
0.010454081915652 37736 59229 0.0063736325 0.0112748875
0.009828645514865 34761 55095 0.0059287727 0.0103860071
0.009334450214425 32274 52055 0.0056016383 0.009642933
0.00905815313620 29410 50314 0.0054142893 0.0087872176
0.008552654713533 26744 47125 0.0050711211 0.0079906612
0.008498023213308 23974 41911 0.0045100425 0.0071630314
0.008356734913388 21045 38018 0.0040911168 0.0062878951
0.008406970713141 18499 34747 0.0037391245 0.0055271927
0.008251867513373 16232 32779 0.0035273481 0.0048498509
0.008397551513463 14308 30413 0.0032727428 0.0042749918
0.008454066813537 12399 28010 0.003014156 0.0037046144
0.00850053512989 11176 25100 0.0027010109 0.0033392024
0.008156419413014 9665 22125 0.0023808712 0.0028877408
0.008172118112640 8486 20483 0.0022041755 0.0025354753
0.007937265511783 7531 19297 0.0020765501 0.0022501372
0.007399113810720 6725 17605 0.001894474 0.0020093179
0.00673160499677 5985 15860 0.0017066945 0.0017882182
0.00607665498442 5512 14535 0.0015641113 0.0016468937
0.00530113887709 4859 13183 0.0014186226 0.0014517882
0.00484085287036 4433 12065 0.0012983146 0.0013245065
0.00441824376123 4100 10953 0.0011786523 0.0012250116
0.00384492695295 3751 10454 0.0011249549 0.0011207362
0.00332498584576 3417 9439 0.0010157308 0.0010209426
0.0028734914073 3169 8936 0.0009616029 0.0009468444
0.00255763313315 2866 8059 0.000867229 0.000856313 0.00208164832981
2658 7711 0.0008297807 0.0007941661 0.00187191362619 2626 7004
0.0007537004 0.000784605 0.00164459642275 2370 6777 0.0007292729
0.0007081165 0.00142858222101 2174 6307 0.0006786962 0.0006495549
0.00131931921817 2129 6049 0.0006509329 0.0006361097
0.00114098191567 2080 5759 0.000619726 0.0006214693
0.00098399491409 1840 5464 0.000587981 0.0005497613 0.0008847791258
1765 5189 0.0005583883 0.0005273526 0.00078995891131 1654 4816
0.0005182497 0.0004941876 0.00071020941085 1582 4793 0.0005157747
0.0004726752 0.00068132381089 1558 4535 0.0004880113 0.0004655044
0.0006838356885 1532 4442 0.0004780036 0.0004577361 0.0005557342851
1416 4200 0.000451962 0.0004230772 0.0005343839766 1322 4037
0.0004344216 0.0003949916 0.0004810083741 1293 3829 0.0004120387
0.0003863268 0.0004653096
11012 24265 81127 0.0087300761 0.0072499774 0.0069149658
0%1%2%3%4%5%6%7%8%
0
550
839
1282
1957
2988
4560
6959
1062
1
1621
0
2474
1
3776
1
Chart 2Untrusted IssuerName MismatchExpired
Figure 7: Google Chrome SSL clickthrough times (ms), by
errortype. The graph shows the percent of warning impressions
thatfall in each timing bucket. The x-axis increases
logarithmically,and we cut off the distribution at 90% due to the
long tail.
users who click through and proceed to the page. 47% ofusers who
clicked through the warning made the decisionwithin 1.5s, whereas
47% of users who left the page did sowithin 3.5s. We interpret this
to mean that users who clickthrough the warning often do so after
less consideration.
6.2 Time by Error TypeFigure 7 depicts the click times for three
error types (un-trusted authority, name mismatch, and expired
certificateerrors). Users clicked through 49% of untrusted
issuerwarning impressions within 1.7s, but clicked through 50%of
name and date errors within 2.2s and 2.7s, respectively.We believe
that this data is indicative of warning fatigue:users click through
more-frequent errors more quickly.The frequency and clickthrough
rate of each error type(as reported in Section 5.2) are inversely
correlated withthat error types timing variance and mode (Figure
7).
7 Implications
Our primary finding is that browser security warnings canbe
effective security mechanisms in practice, but theireffectiveness
varies widely. This should motivate moreattention to improving
security warnings. In this section,we summarize our findings and
their implications, presentsuggestions for warning designers, and
make recommen-dations for future warning studies.
7.1 Warning Effectiveness7.1.1 Clickthrough Rates
Popular opinion holds that browser security warnings
areineffective. However, our study demonstrates that browser
security warnings can be highly effective at preventingusers
from visiting websites: as few as a tenth of usersclick through
Firefoxs malware and phishing warnings.We consider these warnings
very successful.
We found clickthrough rates of 18.0% and 23.2% forGoogle Chromes
phishing and malware warnings, re-spectively, and 31.6% for
Firefoxs SSL warning. Thesewarnings prevent 70% (or more) of
attempted visits topotentially dangerous websites. Although these
warningscould be improved, we likewise consider these
warningssuccessful at persuading and protecting users.
Google Chromes SSL warning had a clickthrough rateof 70.2%. Such
a high clickthrough rate is undesirable:either users are not
heeding valid warnings, or the browseris annoying users with
invalid warnings and possibly caus-ing warning fatigue. Our
positive findings for the otherwarnings demonstrate that this
warning has the poten-tial for improvement. We hope that this study
motivatesfurther studies to determine and address the cause of
itshigher clickthrough rate. We plan to test an
exception-remembering feature to investigate the influence of
repeatexposures to warnings. At Google, we have also begun aseries
of A/B tests in the field to measure the impact of anumber of
improvements.
7.1.2 User Attention
Although we did not directly study user attention, tworesults of
our study suggest that at least a minority ofusers pay attention to
browser security warnings.
There is a 24.4-point difference between the click-through rates
for untrusted issuer errors (81.8%) andexpired certificate errors
(57.4%) in Google Chrome.
21.3% of the time that Mozilla Firefox users viewedthe Add
Exception dialog, they un-checked thedefault Permanently store this
exception option.
These results contradict the stereotype of wholly obliv-ious
users with no interest in security.
7.2 Comparison with Prior ResearchAs Bravo-Lillo et al. wrote
[5]:
Evidence from experimental studies indicatesthat most people
dont read computer warnings,dont understand them, or simply dont
heedthem, even when the situation is clearly haz-ardous.
In contrast, a majority of users heeded five of the sixtypes of
browser warnings that we studied. This sectionexplores why our
results differ from prior research.
Browser Changes. Most prior browser research was con-ducted
between 2002 and 2009. Browsers were rapidly
-
changing during this time period; some changes weredirectly
motivated by published user studies. Notably,passive indicators are
no longer considered primary secu-rity tools, and phishing toolbars
have been replaced withbrowser-provided, full-page interstitial
warnings. As a re-sult, studies of passive indicators and phishing
toolbars nolonger represent the state of modern browser
technology.
Two studies tested an older version of the Mozilla Fire-fox SSL
warning, in which the warning was a modal(instead of full-page)
dialog. Dhamija et al. observeda 68% clickthrough rate, and
Sunshine et al. recordedclickthrough rates of 90%-95% depending on
the type ofpage [11, 30]. The change in warning design could
beresponsible for our lower observed clickthrough rates.
Ecological Invalidity. Sunshine et al. and Sotirakopouloset al.
recorded 55%-60% and 80% clickthrough rates, re-spectively, for a
slightly outdated version of the MozillaFirefox SSL warning [29,
30]. They evaluated the Fire-fox 3 and 3.5 warnings, which had the
same layout andappearance as the current (Firefox 4+) warning but
withdifferent wording. Its possible that changes in word-ing caused
clickthrough rates to drop from 55%-80% to33.0%. However, during an
exit survey, 46% of Soti-rakopouloss subjects said they clicked
through the warn-ing because they either felt safe in the
laboratory envi-ronment or wanted to complete the task [29]. Since
theirstudy methodology was intentionally similar to the Sun-shine
study, Sotirakopoulos et al. concluded that bothstudies suffered
from biases that raised their clickthroughrates [29]. We therefore
attribute some of the discrepancybetween our field study data and
these two laboratorystudies to the difficulty of establishing
ecological validityin a laboratory environment.
In light of this, we recommend a renewed emphasison field
techniques for running and confirming user stud-ies of warnings.
Although we used in-browser telemetry,there are other ways of
obtaining field data. For exam-ple, experience sampling is a field
study methodologythat asks participants to periodically answer
questionsabout a topic [2, 6, 9, 27]. Researchers could install
abrowser extension on participants computers to observetheir
responses to normally occurring warnings and dis-play a survey
after each warning. This technique allowsresearchers to collect
data about participants emotions,comprehension, and demographics.
Participants may be-come more cautious or attentive to warnings if
the pur-pose of the study is apparent, so researchers could
obscurethe purpose by surveying subjects about other browsertopics.
Network-based field measurements also providean alternative
methodology with high ecological validity.A network monitor could
maintain its own copy of theSafe Browsing list and identify users
who click throughwarnings. If the monitor can associate network
flows
with specific demographics (e.g., students), it can
helpunderstand the impact of these factors on user behavior.Similar
studies could help understand SSL clickthroughrates; recent work
addressed how to reproduce certificatevalidation at the network
monitor [1].
7.3 DemographicsWe found that clickthrough rates differ by
operating sys-tem and browser channel. Our findings suggest that
highertechnical skill (as indicated by use of Linux and pre-release
channels) may predispose users to click throughsome types of
warnings. We recommend further inves-tigation of user demographics
and their impact on userbehavior. Large-scale demographic studies
might uncoveradditional demographic factors that we were unable
tostudy with our methodology. If so, can warning designaddress and
overcome those demographic differences?
Technically advanced users might feel more confidentin the
security of their computers, be more curious aboutblocked websites,
or feel patronized by warnings. Studiesof these users could help
improve their warning responses.
7.4 Number of ClicksOur data suggests that the amount of effort
(i.e., numberof clicks) required to bypass a warning does not
alwayshave a large impact on user behavior. To bypass GoogleChromes
malware and phishing warnings, the user mustclick twice: once on a
small Advanced link, and thenagain to proceed. Despite the hidden
button, users clickthrough Google Chromes malware/phishing warning
at ahigher rate than Mozilla Firefoxs simpler warning.
Fur-thermore, 84% of users who open Mozilla Firefoxs AddException
dialog proceed through it.
We find this result surprising. Common wisdom ine-commerce holds
that extra clicks decrease clickthroughrates (hence, one-click
shopping) [12, 31]. GoogleChromes warning designers introduced the
extra step inthe malware/phishing warning because they expected
itto serve as a strong deterrent. One possible explanation isthat
users make a single cognitive decision when facedwith a warning.
The decision might be based on the URL,warning appearance, or
warning message. Once the userhas decided to proceed, additional
clicks or informationis unlikely to change his or her decision.
Our data suggests that browser-warning designersshould not rely
on extra clicks to deter users. However,we did not explicitly
design our study to examine theeffects of multiple clicks. Future
studies on multi-clickwarnings could shed light on user decision
models andimpact security warning design. It is possible that
extraclicks do not serve as a deterrent until they reach
somethreshold of difficulty.
-
7.5 Warning Fatigue
We observed behavior that is consistent with the theory
ofwarning fatigue. In Google Chrome, users click throughthe most
common SSL error faster and more frequentlythan other errors. Our
findings support recent literaturethat has modeled user attention
to security warnings asa finite resource [4] and proposed warning
mechanismsbased on this constraint [14].
Based on this finding, we echo the recommendationthat security
practitioners should limit the number of warn-ings that users
encounter. Designers of new warningmechanisms should always perform
an analysis of thenumber of times the system is projected to raise
a warn-ing, and security practitioners should consider the
effectsthat warning architectures have on warning fatigue.
7.6 More Information
Users rarely click on the explanatory links such as
MoreInformation or Learn More (Section 5.2.4). Designerswho utilize
such links should ensure that they do not hidea detail that is
important to the decision-making process.
Mozilla Firefox places information about SSL errorsunder
Technical Details and in the Add Exceptiondialog instead of the
primary warning. Thus, the errortype has little impact on
clickthrough rates. In contrast,Google Chrome places error details
in the main text ofits SSL warning, and the error has a large
effect on userbehavior. It is possible that moving this information
intoMozilla Firefoxs primary warning could reduce theirclickthrough
rates even further for some errors.
8 Conclusion
We performed a field study with Google Chrome andMozilla
Firefoxs telemetry platforms, allowing us to col-lect data on
25,405,944 warning impressions. We findthat browser security
warnings can be successful: usersclicked through fewer than a
quarter of both browsersmalware and phishing warnings and a third
of MozillaFirefoxs SSL warnings. We also find clickthrough ratesas
high as 70.2% for Google Chrome SSL warnings, in-dicating that the
user experience of a warning can have atremendous impact on user
behavior. However, warningeffectiveness varies between demographic
groups. Ourfindings motivate more work on browser security
warn-ings, with particular attention paid to demographics.
AtGoogle, we have begun experimenting with new warningdesigns to
further improve our warnings.
Acknowledgements
We thank the participants in Google and Mozillas teleme-try
programs for providing us with valuable insight intoour warnings.
At Google, we would like to thank MattMueller for setting up the
malware and phishing measure-ments, Adam Langley for making
suggestions about howto implement SSL measurements, and many others
forproviding insightful feedback. At Mozilla, we would liketo thank
Sid Stamm for his mentorship and help collectingtelemetry data, Dan
Veditz for gathering data from Fire-fox 23, Brian Smith for
providing information about thetelemetry mechanisms, and the
Mozilla contributors whoreviewed our code and helped land this
telemetry [22].We also thank David Wagner, Vern Paxson, Serge
Egel-man, Stuart Schechter, and the anonymous reviewers
forproviding feedback on drafts of the paper.
References[1] AKHAWE, D., AMANN, B., VALLENTIN, M., AND
SOMMER,
R. Heres My Cert, So Trust Me, Maybe? Understanding TLSErrors on
the Web. In Proceedings of the 2013 World Wide WebConference
(2013).
[2] BEN ABDESSLEM, F., PARRIS, I., AND HENDERSON, T.
MobileExperience Sampling: Reaching the Parts of Facebook
OtherMethods Cannot Reach. In Privacy and Usability Methods Pow-wow
(2010).
[3] BIDDLE, R., VAN OORSCHOT, P. C., PATRICK, A. S., SOBEY,J.,
AND WHALEN, T. Browser interfaces and extended validationSSL
certificates: an empirical study. In Proceedings of the ACMWorkshop
on Cloud Computing Security (2009).
[4] BOHME, R., AND GROSSKLAGS, J. The Security Cost of CheapUser
Interaction. In Proceedings of the New Security ParadigmsWorkshop
(NSPW) (2011).
[5] BRAVO-LILLO, C., CRANOR, L. F., DOWNS, J. S., AND
KO-MANDURI, S. Bridging the Gap in Computer Security Warnings:A
Mental Model Approach. In IEEE Security and Privacy (March2011),
vol. 9.
[6] CHRISTENSEN, T., BARRETT, L., BLISS-MOREAU, E., LEBO,K., AND
KASCHUB, C. A Practical Guide to Experience-Sampling Procedures. In
Journal of Happiness Studies (2003),vol. 4.
[7] Google Chrome Privacy Notice.
http://www.google.com/chrome/intl/en/privacy.html.
[8] CHROMIUM AUTHORS. HSTS Preload and Certificate PinningList.
https://src.chromium.org/viewvc/chrome/trunk/src/net/base/transport_security_state_static.json.
[9] CONSOLVO, S., AND WALKER, M. Using the Experience Sam-pling
Method to Evaluate Ubicomp Applications. In PervasiveComputing
(2003).
[10] Convergence. http://www.convergence.io.
[11] DHAMIJA, R., TYGAR, J. D., AND HEARST, M. Why
phishingworks. In Proceedings of the SIGCHI Conference on
HumanFactors in Computing Systems (2006).
[12] DUTTA, R., JARVENPAA, S., AND TOMAK, K. Impact of Feed-back
and Usability of Online Payment Processes on ConsumerDecision
Making. In Proceedings of the International Conferenceon
Information Systems (2003).
-
[13] EGELMAN, S., CRANOR, L. F., AND HONG, J. Youve BeenWarned:
An Empirical Study of the Effectiveness of Web BrowserPhishing
Warnings. In Proceedings of the ACM CHI Conferenceon Human Factors
in Computing Systems (2008).
[14] FELT, A. P., EGELMAN, S., FINIFTER, M., AKHAWE, D.,
ANDWAGNER, D. How to ask for permission. In Proceedings of
theUSENIX Conference on Hot Topics in Security (HotSec) (2012).
[15] FRIEDMAN, B., HURLEY, D., HOWE, D. C., FELTEN, E.,
ANDNISSENBAUM, H. Users Conceptions of Web Security: A Com-parative
Study. In CHI Extended Abstracts on Human Factors inComputing
Systems (2002).
[16] HABER, J. Smartscreen application reputa-tion in ie9, May
2011.
http://blogs.msdn.com/b/ie/archive/2011/05/17/smartscreen-174-application-reputation-in-ie9.aspx.
[17] HERLEY, C. The plight of the targeted attacker in a world
of scale.In Proceedings of the Workshop on the Economics of
InformationSecurity (WEIS) (2010).
[18] HOLZ, R., BRAUN, L., KAMMENHUBER, N., AND CARLE, G.The ssl
landscape: a thorough analysis of the x.509 pki usingactive and
passive measurements. In Proceedings of the ACMSIGCOMM Internet
Measurement Conference (IMC) (2011).
[19] JACKSON, C., SIMON, D. R., TAN, D. S., AND BARTH, A.
Anevaluation of extended validation and picture-in-picture
phishingattacks. In Proceedings of the Workshop on Usable
Security(USEC) (2007).
[20] LANGLEY, A. SSL Interstitial Bypass Rates, February2012.
http://www.imperialviolet.org/2012/07/20/sslbypassrates.html.
[21] MCGRAW, G., FELTEN, E., AND MACMICHAEL, R. SecuringJava:
getting down to business with mobile code. Wiley ComputerPub.,
1999.
[22] MOZILLA BUGZILLA. Bug 767676: Implement Security
UITelemetry. https://bugzil.la/767676.
[23] Mozilla firefox privacy policy.
http://www.mozilla.org/en-US/legal/privacy/firefox.html#telemetry.
[24] NETCRAFT. Phishing on sites using ssl cer-tificates, August
2012.
http://news.netcraft.com/archives/2012/08/22/phishing-on-sites-using-ssl-certificates.html.
[25] PROVOS, N. Safe Browsing - Protecting Web Users for 5Years
and Counting. Google Online Security Blog.
http://googleonlinesecurity.blogspot.com/2012/06/safe-browsing-protecting-web-users-for.html,
June 2012.
[26] SCHECHTER, S. E., DHAMIJA, R., OZMENT, A., AND FISCHER,I.
The Emperors New Security Indicators. In Proceedings of theIEEE
Symposium on Security and Privacy (2007).
[27] SCOLLON, C. N., KIM-PRIETO, C., AND DIENER, E. Experi-ence
Sampling: Promises and Pitfalls, Strengths and Weaknesses.In
Journal of Happiness Studies (2003), vol. 4.
[28] SOBEY, J., BIDDLE, R., VAN OORSCHOT, P., AND PATRICK,A. S.
Exploring user reactions to new browser cues for extendedvalidation
certificates. In Proceedings of the European Symposiumon Research
in Computer Security (2008).
[29] SOTIRAKOPOULOS, A., HAWKEY, K., AND BEZNOSOV, K. Onthe
Challenges in Usable Security Lab Studies: Lessons Learnedfrom
Replicating a Study on SSL Warnings. In Proceedings of theSymposium
on Usable Privacy and Security (2011).
[30] SUNSHINE, J., EGELMAN, S., ALMUHIMEDI, H., ATRI, N.,AND
CRANOR, L. F. Crying Wolf: An Empirical Study of SSLWarning
Effectiveness. In Proceedings of the USENIX SecuritySymposium
(2009).
[31] TILSON, R., DONG, J., MARTIN, S., AND KIEKE, E. Factorsand
Principles Affecting the Usability of Four E-commerce Sites.In Our
Global Community Conference Proceedings (1998).
[32] WENDLANDT, D., ANDERSEN, D. G., AND PERRIG, A.
Perspec-tives: Improving SSH-style Host Authentication with
Multi-PathProbing. In USENIX Annual Technical Conference
(2008).
[33] WHALEN, T., AND INKPEN, K. M. Gathering evidence: Useof
visual security cues in web browsers. In Proceedings of theGraphics
Interface Conference (2005).
[34] WU, M., MILLER, R. C., AND GARFINKEL, S. L. Do
SecurityToolbars Actually Prevent Phishing Attacks? In Proceedings
ofthe SIGCHI Conference on Human Factors in Computing
Systems(2006).
A Sample Sizes
Malware Phish- SSL Adding Exception
Release 1,968,707 89,948 NC 1,805,928Beta 74,782 3,058 10,976
66,694Dev 61,588 2,759 15,560 53,001Nightly 58,789 4,239 18,617
64,725
Table 7: Warning impression sample sizes for Mozilla
Firefoxwarnings, by channel, for all operating systems.
Malware Phish- SSL Adding Exception
Mac 71,371 3,951 534 154,129Win 1,892,285 85,598 10384
1,634,193Linux 1,750 112 58 17,606
Table 8: Warning impression sample sizes for Mozilla
Firefoxwarnings, by operating system. The malware, phishing, and
theAdd Exception samples are from the release channel, whereasthe
SSL samples are from the beta channel. The frame issuedoes not
affect statistics that pertain only to the Add Exceptiondialog.
Malware Phishing SSLStable 5,946,057 381,027 16,363,048Beta
44,742 3,525 232,676Dev 14,022 1,186 66,922Canary 35,261 612
42,020
Table 9: Warning impression sample sizes for Google
Chromewarnings, by channel, for all operating systems.
In Google Chrome, we recorded 6,040,082 malwarewarning
impressions, 386,350 phishing warning impres-sions, and 16,704,666
SSL warning impressions. In
-
Malware Phishing SSLMac 598,680 20,623 947,971Windows 9,775,104
333,522 13,399,820Linux 15,456 577 515,319Android NC NC
1,499,938
Table 10: Warning impression sample sizes for Google
Chromewarnings, by operating system, for the stable channel.
Mozilla Firefox, we recorded 2,163,866 malware
warningimpressions, 100,004 phishing warning impressions, and45,153
SSL warning impressions. Tables 7, 8, 9, and 10further separate the
sample sizes based on OS and releasechannel.
IntroductionBackgroundMalware and Phishing WarningsSSL
WarningsBrowser Release Channels
Prior Laboratory StudiesSSL WarningsPhishing WarningsMalware
Download Warnings
MethodologyMeasuring Clickthrough RatesMeasuring Time Spent on
WarningsEthicsData CollectionMethod Limitations
Clickthrough RatesMalware and Phishing WarningsMalware Rates by
DateMalware/Phishing Rates by Warning TypeMalware/Phishing Rates by
DemographicsMalware/Phishing Rates by Browser
SSL WarningsSSL Rates by DemographicSSL Rates by BrowserSSL
Rates by Certificate Error TypeAdditional SSL Metrics
Time Spent On SSL WarningsTime by OutcomeTime by Error Type
ImplicationsWarning EffectivenessClickthrough RatesUser
Attention
Comparison with Prior ResearchDemographicsNumber of
ClicksWarning Fatigue``More Information''
ConclusionSample Sizes