Designing Affective Consumers: Emotion Analysis in Market Research

Kelly Gates, Ch 7: Designing Affective Consumers: Emotion Analysis in Market Research. In Toby Miller (ed.) Routledge Companion to Global Popular Culture. Routledge, 2014.

Abstract

This chapter looks at the resurgence of interest in emotion

measurement in the domain of market and media research, focusing

on efforts to market the nascent technology of automated facial

expression analysis (AFEA) for this purpose. Two start-up

companies are marketing prototype versions of AFEA as market

research tools: Affectiva and Emotient. Formed by researchers

from major universities, these companies market their AFEA

systems as either web-based platforms that process visual data

and serve up results from their servers, or as software

development kits that can be integrated with other developers’

devices and applications. This article situates these ventures in

the context of the history of market and media research, and

specifically the field’s long-standing interest in making emotion

a measurable phenomenon. It also looks at the resurgent interest

in emotion in more recent market research trends, namely

“neuromarketing” and “sentiment analysis.” The chapter lastly

1

examines the way Affectiva and Emotient conceptualize the uses

and users of their technologies, as suggested in their website

promotional materials. Two big promises made about AFEA as a

market research tool are (1) its ability to bypass people’s

conscious reflections about their mediated experiences and

instead provide “accurate” measures of their visceral, pre-verbal

responses, and (2) its “scalability,” or potential application

for large-scale visual sentiment analysis—mining visual data to

automatically gauge the affective tone of troves of visual media

content circulating online. This chapter argues that, in their

market research and product design applications, new emotion-

measurement technologies like AFEA are best understood as

technologies of media subjectivation—ways of both measuring and

modulating people’s embodied, affective engagements with media,

with each other, and with the world.

Introduction

In 2009, researchers from MIT launched a start-up company

called Affectiva to commercialize automated facial expression

analysis (AFEA)—their effort to program computers to parse out

2

meaningful expressions displayed on human faces captured in

video. The time was ripe, they decided, to migrate their work

from the lab to the marketplace and see what profitable uses

might take hold. At the time, the only commercial application of

automated facial expression analysis was Sony’s Smile Shutter™

app, a feature installed on its Cyber-shot cameras, designed to

automatically snap a photo when the person being photographed

smiles. (“Switch on Smile Shutter and let your Cyber-shot take

the photo for you!” proclaims an online promotion.) (Sony, “Smile

Shutter”). The founders of Affectiva had a different idea for

their version of the technology: “to measure the emotional

connection people have with advertising, brands, and media.” The

plan was to build a profitable company by marketing AFEA as a

market research tool, at the same time gathering video data from

the online world for further research and development of AFEA.

Since Affectiva’s launch, scientists from the Machine Perception

Lab at the University of California, San Diego formed a similar

venture called Emotient. At their website, Emotient claims to be

“the leading authority on facial expression recognition and

analysis technologies.” The company markets their FACET™ software

3

development kit (SDK) and FACET™ Vision products to “Fortune 500

companies, market research firms, and a growing number of

vertical markets” (Emotient, “About Emotient”).

Companies like Affectiva and Emotient are riding a wave of

newly intensified interest in emotion—in media and market

research, business and management consulting, security and

policing, and in a wide range of less applied disciplines. Market

research aims to capitalize on the development of new tools for

emotion measurement, like AFEA and functional magnetic resonance

imaging or fMRI, in order to gain greater purchase on the

internal drives that cause people to desire things, to form

attachments, to have particular kinds of emotional reactions to

media, and of course to choose certain brands over others and

spend money on products and services. These tools promise to

visualize and define people’s emotional fluctuations in more

sophisticated ways than existing techniques like surveys and

focus groups, giving market researchers direct access to people’s

pre-conscious and nonverbal sensory experience. In turn, emotion-

measurement data generated for market research serves the dual

purpose of furthering the research and development programs of

4

psychologists and computer scientists developing automated

techniques of emotion recognition and analysis. In fact, new

business ventures aimed at monetizing emotion measurement promise

to play a formative role in the development of artificial

intelligence—now conceived not as cold and calculating

information-processing machines, but as affectively engaged

systems that interact with humans in ways that attend to, and

also aim to interact with and even manipulate their emotional

states and fluctuations. In a broad sense, the computational

treatment of emotion promises to transform what Elizabeth Wilson

(2011) calls our “affective circuitry,” or the shape and

character of our emotional engagements—with machines, with each

other, and with the world.

This chapter provides a brief survey of emotion measurement

in the domain of market and media research, addressing key

critical questions and debates relevant to understanding the

application of automated facial expression analysis (AFEA),

functional magnetic resonance imaging (fMRI), and related

technologies “to measure the emotional connection people have

with advertising, brands, and media.” These market research

5

applications are significant in themselves, as well as in terms

of the contribution they make to the computational treatment of

emotion more broadly. New tools like AFEA are themselves products

of the attention that emotion has received since the 1990s in the

intersecting fields of computer science, human-computer

interaction, machine learning and artificial intelligence

research. But it would be wrong to suggest that emotion was

irrelevant to these fields before then; the seemingly recent

upturn in attention to emotion in the artificial intelligence

domain belies its long-standing importance, even if its relevance

was denigrated much of the time (Wilson 2011). Precisely what is

new about the current wave of emotion research is a central

question addressed here—how to best make sense of the complex

conjuncture of disciplines, ideas, interests, aims, methods and

technologies that define emotion-based market research and the

contribution it promises to make to the conditions of human

affective experience.

Some Background on Emotion in Media and Market Research

6

Emotion was an object of special interest for market and

media research well before the development of technologies like

AFEA or fMRI. “Since strategies of mobilizing emotional

commitment have been around for a long time,” writes Mark

Andrejevic (2013), “the newness of this discourse seems to hinge

more on its urgency in a multiplatform, multi-outlet era than its

originality” (p. 50). Marketing in general, according to the

editors of a recent anthology titled The Rise of Marketing and Market

Research, “is about reconciling the imperatives of production with

the needs and desires of consumers” (Berghoff, Scranton, &

Spiekermann 2012: 1-2). Packing emotional force into media

content has long been one of the main goals of the commercial

creative industry, with the attachment of emotions to products

and services for sale in the marketplace the central aim of

modern advertising appeals. A dialectic relationship between

advertising strategies and people’s emotional needs and desires

lies at the heart of consumer capitalism and its ideological

diffusion (Lears 1983, 1994). The methods devised historically to

inform this process are central to the historical evolution of

market research as such.

7

The question of what role psychology and its theories should

play in the development of advertising strategies was much

debated in the early twentieth century—as suggested, for example,

in applied psychologist Walter D. Scott’s famous article, “The

Psychology of Advertising,” published in The Atlantic Monthly in 1904.

But there is some dispute about what direct role applied

psychology actually played in efforts to rationalize consumption.

The marketing guru Hans Domizlaff apparently thought market

research in general—informed by psychology or otherwise—was

“utterly useless,” while his contemporary, the famous Eric

Dichter, had a doctorate in psychology and used in-depth

interviews designed “to trace un-conscious, mainly sexual motives

behind purchasing decisions” (Berghoff, Scranton, & Spiekermann

2012: 6-7). Dichter founded the Institute for Motivational

Research in 1946 and promoted himself as “an expert who possessed

the key to the hidden secrets of consumers’ psyches” (ibid: 8).

Even before Dicther’s well-known rise to prominence in U.S.

marketing field, the Austrian sociologist Paul Lazarsfeld (1934),

founder of Columbia University’s Bureau of Applied Social

Research, advocated for the use of psychological analysis to

8

uncover the unconscious motivations behind the act of buying. He

suggested that psycholinguistics, for example, could be used to

interpret what was beneath the surface of subjects’ responses to

interview questions (Lazarsfeld 1937). Lazarsfeld’s media effects

research easily crossed over from academic to business

applications, more or less by design.

Of course, to reduce the history of the relationship between

psychology and market research to the differing approaches of

celebrated marketing and ad men would mean falling prey to their

astute self-promotion. But what we can glean from the historical

emphasis on these professional figures is that psychology was a

part of the conversation among marketing strategists throughout

the early formation of the modern market research industry. Early

twentieth-century advertising strategies were informed, without a

doubt, by the belief that emotional appeals would be more

effective than rational ones at assimilating people to a culture

of consumption. Writing in the 1980s, the historian T.J. Jackson

Lears argued that what oriented people toward commodity

consumption in early twentieth-century America was an emerging

“therapeutic ethos,” or “a shift from a Protestant ethos of

9

salvation through self-denial toward a therapeutic ethos

stressing self-realization in this world” (Lears 1983: 4).

Advertisers both capitalized on and helped to construct this

secular therapeutic ethos, Lears argued. To do so, they hired

psychological consultants to help them devise ways to “arouse

consumer demand by associating products with imaginary states of

well-being” (Lears 1983: 19). A therapeutic approach that

targeted the human psyche would allow advertising to depart from

rational appeals and “speak more directly to consumers’ desires

for sensuous enjoyment” (ibid: 19).

Making a different case about historical changes in

advertising strategies, Michael Schudson (1986) insisted that the

shift toward emotional appeals in the first half of the twentieth

century had less to do with advertisers applying psychology and

behavioral science and more to do with marketplace changes that

altered how advertising agencies operated. These changes included

assumptions about the emotional vulnerability of valuable female

consumers, the growing need to compete with an ever-increasing

amount of advertising clutter, and the rising prominence of

visual media, seen as especially amenable to emotional appeals.

10

In Schudson’s view, advertising did not need psychological

theories and methods to recognize the power of emotion as a

source of human motivation.

There was also attention to emotion among media researchers

on the other side of the aisle—that is, among those concerned

with the potentially dangerous effects of new media on the

emotional well being of individuals and society. Such a concern

manifested in the Payne Fund Motion Picture studies in the 1930s

—“the first systematic study of media effects” (Malin 2009: 389).

Media historian Brent Malin focuses on one such study, led by

Christian Ruckmick at the University of Iowa’s Department of

Psychology and resulting in a 1933 report titled The Emotional

Response of Children to the Motion Picture Situation. Ruckmick’s study examined

the emotional responses of children to movies by hooking them up

to “psycho-galvanometers and pneumo-cardiographs that monitored

perspiration, respiration, and heart rate” (Malin 2009: 368). An

early study of emotional response to media, Ruckmick’s research

already made apparent the suspicion researches felt about the

conscious, introspective accounts respondents gave to describe

their feelings. Respondents’ subjective descriptions of their

11

emotions seemed inherently flawed and unscientific. In contrast,

instruments for measuring embodied physiological fluctuations

promised “to reach a deeper emotional truth of the body” (Malin

2009: 382). Malin argues that the study of emotion as a

physiological phenomenon was tied to the perceived need to

establish psychology’s scientific legitimacy, aligning it with

the biological sciences and disarticulating it from philosophy.

“By mediating a subject’s emotions through a constant flow of

empirical data,” writes Malin, “the psycho-galvanometer promised

to remove the human scientist from the process of emotional

interpretation, even as it provided intimate access to a

subject’s innermost feelings” (Malin 2009: 375).

These early research efforts of Payne Fund psychologists to

gauge the emotional effects of movies by measuring physiological

processes like perspiration, respiration and heart rate, were not

aimed at designing more effective techniques of emotional

manipulation for commercial purposes. Instead, the Payne Fund

motion picture studies were motivated by a paternalistic concern

about the potentially dangerous emotional impact new media were

having on a vulnerable population, and a perceived need to

12

encourage greater emotional restraint among what were seen as

easily excitable audiences (Malin 2009). Still, it is not hard to

envision how the techniques that the researchers devised to

measure the apparent physiological manifestations of emotion

could be repurposed to serve affect-oriented strategies of

persuasion and selling, to devise ways of encouraging people to

deeply engage with a culture of consumption and tie their

identities and desires to a more consumerist way of being.

For its part, the film industry made significant innovations

in market and audience research throughout the twentieth century.

It did not necessarily adopt the kind of emotional response

measurement methods that the Payne Fund psychologists used, at

least not on a significant scale. However, beginning in the 1930s

and ’40s the industry did move in the direction of more

“scientific” forms of market research, namely random sampling and

statistical analyses of survey responses and ticket sales, in

contrast to the more intuitive forms of trial-and-error testing

of movie popularity that took place during the earlier days of

film (Bakker 2003). In addition, new market research firms like

George Gallup’s Audience Research pioneered techniques like the

13

“Preview Jury System,” test-marketing new films by using devices

that gauged a sample of viewers’ reactions, scene-by-scene. While

this sort of research measured viewer taste, not their emotional

responses per se, it did suggest the extent to which the film

industry aspired to design and redesign film content in order to

achieve desired audience reactions. More so and earlier than

other industries, the big studios incorporated market research at

the design stage of their product development, rather than after

the products went to market, tweaking movie content in order to

best fit consumer preferences, with the aim of generating as much

profit as possible. In fact, more specialized techniques of test-

marketing in the film industry emerged in tandem with the rising

sunk costs associated with filmmaking (Bakker 2003). Return on

investment became the top priority along with the high costs of

such developments as the full-length feature film, the

incorporation of sound, the integration of production and

distribution, and other capital-intensive industry practices that

took shape throughout the twentieth century. The more money that

went into filmmaking, the more attention the film industry gave

to gauging viewer response.

14

There is a great deal of additional literature on the

history of advertising, market and media research that could be

consulted for insights about the field’s important shifts and

developments; here the focus has been narrowly on attention to

emotion, in order to provide some historical context for

understanding the recent resurgence of interest in emotion in the

new media landscape. This very partial background on attention to

emotion in the domain of media and market research suggests that

it was well recognized that tapping into emotional lives would be

a highly effective way of persuading people to become active

consumers and brand loyalists. Many of the well-known figures in

marketing and advertising certainly fancied themselves experts in

human psychology and were celebrated as such. However, there is

no clear historical record showing that media and market

researchers engaged in the directed study of emotional response

on any consistent, systematic basis. Surveys were a key consumer-

response technology, but they were limited to enumerating what

people were willing and able to articulate about their feelings,

or what researchers could interpret from those responses, and

consumer self-disclosures were almost always viewed with some

15

suspicion. Despite its limitations, the survey was by far the

dominant mode of market research throughout much of its history,

along with focus groups and other forms of consumer self-

reporting. Perhaps market researchers doubted the usefulness or

reliability of the available emotion-measurement methods, or

perhaps ways of measuring emotional response were not possible or

cost-effective on a large enough scale to provide useful results.

In any case, a central problem for advertisers and marketers has

long concerned how to understand consumers and their motivations

in more depth and specificity. The question of how to increase

the emotional valence of commercial media was ever-present, and

an adequate means of effectively measuring and analyzing people’s

inner emotional lives seemed always out of reach. The lack of

means to more effectively access the depths of people’s psyches

consistently dogged advertisers’ ability to fully exploit the

gaping hole in the individual and collective sense of well-being,

brought on to a great extent by the consumer culture itself.

Emotion Analysis in Market and Media Research Today

16

The more recent turn to the close study and measurement of

emotion in market research is evident in a slate of trade books

that began appearing on the topic of in the late 1990s and early

2000s. These works included titles like The Marketing Power of Emotion

(O’Shaughnessy and O’Shaughnessy 2003); Emotions, Advertising and

Consumer Choice (Hansen and Christensen 2007); and Emotional Branding:

The New Paradigm for Connecting Brands to People (Marc Gobé 2001/2010). In

The Marketing Power of Emotions, the authors explain the prevailing

view in the field that strong brand loyalties depend on the

emotional connections consumers establish with brands, and that

effective marketing techniques require a better understanding of

how consumer choices are guided by emotions. What they call

“NERS” scores, or “net emotional response strengths,” are

estimated by drawing on a variety of emotion measurement methods,

including skin conductivity tests, heart rate measurement, gaze

tracking, brain scans, and the analysis of facial expressions

(O’Shaughnessy and O’Shaughnessy 2003). Combining a variety of

emotion-measurement techniques promises to empower market

researchers to determine the right emotional response, of the right

17

intensity, and the right time to produce the sought-after emotional

engagement with a product, company or brand.

Apparent in this popular trade literature, and in marketing

industry press more broadly, is a resurgent interest in devising

ways of bypassing consumers’ introspective verbal explanations of

emotions and targeting their pre-verbal sensations and

unconscious desires. This preoccupation with probing the depth of

the consumer psyche is especially evident in the rise of

“neuromarketing” – the application of neuroscience to marketing,

using brain imaging technologies to analyze consumer responses to

products, packaging, advertising, and other marketing techniques

(Schneider and Woolgar 2012: 169). The ratings firm Nielsen, for

example, is now in the business of neuro-market research, moving

beyond people meters to monitoring brain activity. Brain imaging

technologies promise to give market researchers more accurate and

in-depth assessments of people’s visceral, pre-verbal responses

to commercial messages than their conscious responses to survey

questions or focus group discussions. Much like the physiological

measures employed by the Payne Fund researchers, the aim of

neuromarketing is “to reach a deeper emotional truth of the

18

body,” in this case by visualizing and interpreting the

neurological activity that takes place along with people’s

exposure to media.

The neuromarketing phenomenon, and its aim of bypassing

conscious consumer response, has caught the attention of scholars

in science and technology studies (STS) investigating the

“neuroscientific turn” in the human sciences (Littlefield and

Johnson 2012). Tanja Schneider and Stephen Woolgar (2012) have

examined neuromarketing from the perspective of STS, questioning

how “the consumer” is conceptualized in the new methods and

practices of neuromarketing. They examine academic and popular

accounts of neuromarketing, finding a consistent tendency to

conceptualize the consumer as “an unknowing, unreliable entity,”

a passive and secondary object of attention who is unaware of her

own true motives (Schneider and Woolgar 2012: 185). The claim in

the domain of neuromarketing is that identifying consumer

motivations requires expert interpretation of brain activity:

“This depiction of the consumer as non-knowledgeable is premised

on the [notion] that consumers do not know why they buy

something, whereas consumers’ brains can provide objective

19

answers” (ibid: 181). Neuromarketing promises to create new

knowledge about consumers through specialized analysis of their

brains, revealing the hidden causes of buying behavior by

entrusting brain-interpretation to a new community of experts

(neuromarketers) empowered with new technologies (like fMRI).

Thus neuromarketing redistributes the “accountability relations”

that animate the market research scenario: “Accountability for

subjects’ motives passes from the subjects to the technology and

its operatives” (ibid: 184).

Schneider and Woolgar’s attention to the reconceptualization

of “the consumer” in neuromarketing is consistent in many ways

with the argument that what needed to be produced en mass, to

fully usher in commodity capitalism, was not only consumer goods

but consumer-subjects themselves. The rise of neuromarketing and

its particular ways of conceptualizing the consumer suggests that

reproduction of the ethos of consumption—getting people to

identify with a consumerist way of being at a deep psychic level—

remains central to the reproduction of the capitalist system. New

technologies like fMRI and automated facial expression analysis

promise to be useful not only for accessing, measuring and

20

interpreting the unaware consumer’s embodied affective activity,

but also for developing and testing ways of targeting and

stimulating that activity—likewise in ways that new neuro-

consumers need not consciously be aware of.

It is not only “the consumer” that is being reproduced and

re-conceptualized in current methods of market and media

research. The resurgence of interest in emotion-based market

research methods corresponds with the ways “the audience” has

been rethought and reconstituted along with the rise of

interactive media. “Audience research,” whether referring to the

ratings industry or the academic sub-discipline of audience

studies, no longer adequately captures the range of approaches

being developed for studying how people engage with networked,

digital media. Applied or market-based studies of media reception

have moved beyond the conventional ratings industry practice of

counting the number of viewers of a television show, or even

segmenting those viewers into different niches and analyzing the

ways they choose and interpret media products differently. Now,

online social media and streaming content providers double as

massive user monitoring systems, gathering volumes of detailed

21

personal data that can be mined to construct a virtually infinite

variety of niche user categories. In addition, “usability

studies,” administered at the design stage, test how people

interact with media products like video games, software apps, and

websites, measuring test-subjects fluctuating levels of emotional

intensity as they interact with these media products. The

expressed aim of usability studies is typically to make

interactive media more “user-friendly” or engaging, but

ultimately the goal is to make them more monetize-able, and in

the case of video games, harder to stop playing. For video games

in particular, preference for the notion of “user” seems apt,

given the reportedly addictive qualities of commercially

successful video games, and the efforts of more highly

capitalized game developers to design the games in ways that

keeps users hooked on playing (closely analyzing users’ responses

to game mechanics, for example). Video games share much in common

with slot machines—designed, both algorithmically and

ergonomically, to lure players into a trancelike state that

gamblers call the “machine zone,” as Natasha Dow Schüll (2013)

documents in her great book, Addiction by Design.

22

While it may be the case that market-based media research is

responding to changes in media forms and uses, in fact the

relationship between user studies, media forms, and uses is more

reciprocal than linear, more cybernetic feedback loop than one-

way, cause-effect relationship. We might say that the

relationship between new emotion-oriented market research

methods, on the one hand, and our affective engagement with media

on the other, is one of coproduction. Market-based media research

itself has a central role to play in delimiting the range of

possible media experiences and engagements that developers

envision and attempt to design into media products. And in fact

one of the promises of new approaches like neuromarketing is to

provide greater potential for incorporating product testing in

the design stage (Ariely and Burns 2010). We might even say that

neuromarketing aims to more tightly integrate brain activity with

commercial media design, challenging the notions of both finished

media product and fully formed consumer subject.

Here it is useful to consider Jack Bratich’s (2013)

suggestion that media studies ought to re-examine what is meant

by “the audience,” as well as other ways of conceptualizing

23

collective and individual engagements with media, including

consumers, publics, masses, recipients, fans, users, and so on.

For Bratich, these individual and collective engagements with

media are better conceptualized as different media subjectivities. In

other words, things like “the audience” and “the consumer” are

not pre-constituted people or objects-in-themselves that already

exist as individual or collective recipients of media. Instead,

they are and always have been subjects-in-perpetual-formation, as

well as “a convergence of discursive problematizations” (Bratich

2013: 425)—different ways of envisioning subjectivities, making

them knowable and bringing them into being. “Every participant in

a communicative act has an imagined audience,” write Alice Marwick

and danah boyd (2011: 115), and how an audience is imagined can

be very different in different communicative contexts. Further,

in online social media especially, actual readers or viewers can

be very different than what the producer-user imagines her

audience to be—a fact that is especially true of Twitter, given

the variety of ways people consume and spread tweets, as Marwick

and boyd point out. The study of audiences reaches its limit in

the emergence of interactive media, Bratich argues, allowing us

24

to better understand “the audience” for what it always was: a

form of media subjectivation, one way among others in which

collective and individual engagements with media are

conceptualized and constituted (Bratich 2013). In their market

research and product design applications, new emotion-measurement

technologies like fMRI and AFEA can best be understood as

technologies of media subjectivation.

The resurgence of emotion measurement methods also coincides

with another significant development in the field of market

research: namely, the rise of predictive analytics, or the use of

statistical analysis, data mining and machine learning techniques

to make predictions about the future. The subtitle of a trade

book titled Predictive Analytics defines it as “The Power to Predict

Who Will Click, Buy, Lie, or Die” (Siegel and Davenport 2013),

suggesting some of its applications: for media and market

research, fraud and crime prediction, insurance and medicine. In

market research, these two coexisting trends—emotion analysis

techniques and predictive analytics—correspond to what are,

roughly speaking, two long-standing approaches or “schools of

thought”: “One favors psychology and science whereas the other

25

privileges statistics by analyzing demographic and other data”

(Berghoff, Scranton, & Spiekermann 2012: 11). In addition, both

of these approaches share the common aim of “utilizing prediction

to overcome market uncertainties”—persistent efforts, at the

heart of market research, to deal with the “fundamental

unpredictability of the future” (ibid: 11). In fact, it could be

argued that “predictive analytics” is simply a new buzzword or

way of describing forms of market analysis that, while much more

data-intensive today, are nonetheless consistent with more long-

standing ways of applying statistical analysis to predict market

trends. Still, it would not be accurate to describe current

predictive marketing techniques, or the digital media landscape

it aims to make sense of, as simply more data-intensive. As one

marketing blogger puts it, “marketing is undergoing an

existential change” (Lyons 2014), upended by new forms of

quantitative analysis afforded by, and demanded of, vast

quantities of data generated from a proliferation of sources—

credit-card transactions, social media platforms, internet

browsing and search queries, streaming media services, smart

26

phone apps, GPS and cellular location data, text messaging, and

more.

Where predictive analytics meets the resurgent interest in

emotion in market research we find the new field of “sentiment

analysis.” Computer scientist Bing Liu defines sentiment

analysis, “also called opinion mining,” as a field of study that

analyzes people's opinions, sentiments, attitudes, and emotions

towards “products, services, organizations, individuals, issues,

events, topics, and their attributes,” using the computational

techniques of natural language processing (Liu 2012: 1). In his

recent book Infoglut, Mark Andrejevic (2013) offers a critique of

sentiment analysis in the context of his broader discussion of

the crisis of representation associated with current conditions

of information overabundance. The marketing industry is at the

forefront of efforts to mine the realm of sentiment and emotion

online, he explains, especially as it manifests in the vast

troves of data generated through users’ everyday activities on

social media platforms. Andrejevic suggests that this interest in

sentiment data fits well with at least two trends in the

information-era economy: “first, the increasing importance

27

attached to emotional response as a means of navigating a

landscape of information glut; and second, the role of

information about preferences, opinions, and emotional response”

in facilitating the mass customization economy (Andrejevic 2013:

44). Sentiment analysis opens up the realm of emotion to mass

quantification, data collection and mining, holding out the

measurement of emotional response as “the key to cutting through

the clutter of available information” (Andrejevic 2013: 43-44).

It promises to give businesses a means of gauging the Internet’s

background feeling tone, making order out of Internet chaos and

gaining predictive power. Predictive analytics provides the

knowledge base for experimental interventions, devising new

management strategies, not only in marketing but in politics and

other domains, “to minimize negative sentiment and maximize

emotional investment”; the aim is “not merely to record sentiment

as a given but to modulate is as a variable” (Andrejevic 2013:

46).

Automated Facial Expression Analysis as Market Research

Technology

28

This brings us to the companies introduced at the outset of

this chapter, Affectiva and Emotient. These new ventures are

attempting to transition automated facial expression analysis

(AFEA) from the lab to the marketplace, marketing their versions

of the technology as market research tools that can stand-alone

or be integrated with other devices and platforms. One of the big

promises of AFEA is its potential application for visual

sentiment analysis—mining visual data to automatically gauge the

affective tone of individual content or collections of images and

video. Whether the technology will fulfill this promise, beyond

limited applications that can measure a small number of faces in

constrained settings, remains uncertain. Automated facial

expression analysis is still an unproven technology, and it is

too soon to say whether these particular ventures—the first of

their kind—will be successful, or what direction their business

might lead. In fact, companies like Affectiva and Emotient are

arguably more important for their research activities than for

their business viability. There is always a possibility that one

or both of these companies will fold, or more likely, be acquired

by larger companies to be enfolded into separate business

29

ventures with different or related research and development

priorities.

This means that we can consider here only a version of how

these emotion-measurement products are designed to work, and the

sort of promises that the companies make about their products, at

a particular moment in time (circa 2014). (For an analysis of the

social construction of AFEA before the formation of these

companies, see Gates 2011). The companies’ product descriptions

and promotional material cannot tell us how these technologies

are actually used in practice. However, while one might argue

that how the technologies actually work in practice is more

important than product descriptions or claims made in promotional

material, in fact the descriptions and promises are likewise

important. They tend toward an ideal vision, or how the

technologies would ideally function, if they actually delivered

what the companies envision is most needed and desired in the

market research domain. Of course, the companies would be foolish

to make lofty promises that their products have no chance of

delivering. As with much of the promotional rhetoric associated

with tech products and companies, the descriptions that Affectiva

30

and Emotient offer negotiate a balance between what is currently

possible and what developers hope the technology might be capable

of in the future, based on their perceptions of potential needs

of those users they deem relevant.

So how do these companies define their emotion-measurement

technologies? Not surprisingly, the Affectiva and Emotient

product descriptions are suffused with the “digital sublime”

(Mosco 2005) and the language of “technological solutionism”

(Morozov 2013). At the company website, Affectiva explains its

mission “to digitize emotion, so it can enrich our technology,

for work, play and life” (Affectiva, “About Affectiva”). Here

they suggest that the importance of “digitizing emotion,” and

hence the value of its product, extends across different spheres

of social life, in fact to all of life. Similarly, Emotient

suggests in a promotional video that its “industry-leading

emotion-aware system will enable a revolution in device and

application personalization” (Emotient, “Emotion-Aware

Computing”). The promise is that Emotient’s product can allow

device and software developers to design forms of emotional

engagement into their devices and apps. An Emotient promotional

31

video, “Enabling Human-Aware Devices,” begins with the question,

“What if your devices could read your emotions, and respond to

them?” (ibid.). It proceeds to show images of people engaged in

various activities while making facial expressions, each face

overlaid with a graphical measuring device that identifies the

specific emotions and their temporal intensities. The video

suggests a number of potential applications of the Emotient API,

including organizing personal photo collections by emotion, real-

time measurement of videogames to maximize player engagement, and

the possibility of “adapt[ing] your service to changes in the

user’s temperament, adding a new dimension to the user’s

experience,” in this case suggesting an in-car application by

depicting an image of a driver expressing “frustration” (ibid.).

Affectiva offers its product, called “Affdex,” in two

different forms: either as a cloud-based platform, where video

captured from webcams is processed on Affectiva’s servers via an

Internet connection, or as a software development kit, allowing

software developers to integrate Affdex into their own apps.

Emotient also describes two types of applications, one that can

operate in real-time using a webcam over an internet connection,

32

and another that can analyze sets of images or recorded videos

“in batch mode,” for “non time-sensitive requirements” (Emotient,

“About Emotient”). One possibility for the latter app, Emotient

explains, is for “aggregate customer sentiment analysis”: “Major

retailers, brands and retail technology providers can use

Emotient’s technology for aggregate customer sentiment analysis

at point of sale, point of entry, or on the shelf” (Emotient,

“Markets”).

Visitors to the Affectiva website can try a demo of Affdex

that takes the form of cloud-based platform. The demo records and

analyzes viewers’ facial expressions in response to a selection

of advertisements. The demo asks users to “Please click the

‘Allow’ button in the video window to grant us access to your

webcam for recording” (Affectiva, “Affdex Demo”). Willing

participants then watch an advertisement as their computer’s

webcam records their reactions. When the ad is finished, a brief

survey asks for basic demographic information (age and gender),

whether viewers have seen the ad before, and whether they will

allow Affectiva to use their facial expressions “to spread the

word about Affdex” (ibid.). (Note that viewers of the demo have

33

already consented to allowing Affectiva to use the video for

internal research purposes by clicking on the “allow” button to

enable the software app to access their webcam.) After the brief

survey is complete, the results are processed and then displayed,

overlaid with “Tips for Using the Affdex Dashboard.” A graph

shows the fluctuations in the viewer’s facial expressions over

the course of the ad, as well as the way his or her results

compare to other viewers who watched the ad, along the dimensions

of “surprise,” “smile,” “concentration,” “dislike,” “valence,”

“attention,” and “expression.” The graphical results are not

exactly intuitive, suggesting that some amount of additional

expertise is required to interpret them.

As a web-based platform that records video images of willing

visitors to the Affectiva website, the Affdex demo provides a

data-gathering mechanism to build video archives for further

experimentation and tech development, collecting data on facial

expressions “in the wild” from Internet users. The use of the

demo as a facial data-gathering mechanism is significant, as it

gives developers more visual data needed to make advances in

developing computer vision algorithms for the automation of

34

facial expression analysis. In return for doing “the work of

being watched” (Andrejevic 2002), visitors who opt in to having

their faces recorded while viewing ads earn the benefit of seeing

the results of their own expression analysis, with the suggestion

that they can learn more about themselves and their emotive

responses by examining line graphs of their expressive responses

to ads.

Emotient does not offer a demo of its product for visitors

to try; instead there are a number of videos at the website that

demonstrate the different uses of the Emotient application

programming interface (API). One depicts Dr. Marian Bartlett,

Emotient’s Lead Scientist and one of the company founders, posing

a series of facial expressions of varying intensity, as an

overlaid square frames her face and an image to her right

displays the graphical measurements of her facial movements. In

other Emotient product videos, the people having their faces

analyzed are not depicted as scientists or developers of the

technology, but regular people of various ages (mostly young),

genders, and ethnicities (mostly white), using different kinds of

35

technologies and having visible, seemingly spontaneous emotional

reactions.

Given the issues raised in this chapter, it is important to

consider the way these companies conceptualize “the consumer”

whose emotions they are trying to measure. What assumptions are

made about people and their emotions and engagements with media

in Affectiva and Emotient’s company literature? It is not

surprising to find that these companies carry on the aim of

bypassing consumers’ conscious self-reporting of emotional

response, promoting their facial expression analysis systems as

capable of accessing and deciphering objective measures of

people’s feelings by reading them directly off the body. “People

struggle to accurately describe their emotional experience,”

notes the Affectiva website, and the “traditional survey self

report, while powerful, is … hampered by cognitive bias.

Consumers either can’t or won’t provide the level of detail

needed to really understand the effectiveness of the

creative”—“the creative” here referring to the media products

people are exposed to (Affectiva, “Media Measurement”). To remedy

the problem of vague respondents, unaware of or unwilling to

36

reveal their own true emotions, Affdex “bypasses ‘cognitive

editing’ and deliver[s] scalable, authentic emotional insight

across key emotional measures” (ibid.). For its part, Emotient

promises to provide a means of accessing nonverbal, “unedited”

emotional response via their product’s unique ability to target

and identify “microexpressions,” or “very rapid flashes of

involuntary facial muscle movement that are easily overlooked by

the human eye” (Emotient, “Markets”). They insist that “there is

a wealth of information contained within microexpressions,”

beyond what subjects would explicitly reveal about their feelings

(ibid.). As with neuromarketing, and even much earlier approaches

to studying emotional response, the conscious and self-aware

subject is bypassed, dismissed as providing incomplete and

unreliable about self-motivations. Instead, trust is placed in

technologies to provide more accurate and objective measures by

targeting the body’s physiological processes. AFEA and fMRI

promise to bring into existence even more infinitesimal levels of

embodiment, making visible and knowable small spaces and times

that were previously imperceptible and beyond the threshold of

knowledge or intervention (Thrift 2008).

37

However, the question of how these companies conceptualize

users is a bit more complicated than simply reproducing the non-

knowledgeable consumer so common to field of market research. In

the efforts of these companies to define both the uses and the

users of their products, we find a sort of subject-object

slippage, so that it is not always easy to determine who the

preferred users are. This slippage or confusion stems in part

from the fact that, in addition to defining the range of subjects

whose emotions are open to examination, these companies are also

in the business of delineating and speaking to their own

customer--not the end-user of a device or software platform whose

emotions are being measured, but instead “the researcher” or

expert-professional subject invested in emotional measurement

output. There is a clear concern for this particular type of user

—an actor with an interest in measuring the emotions of others.

There are instances when Affectiva and Emotient are clear

about precisely who the envisioned users of the technology are,

the specific actors whom they intend to “empower” with their

technologies: “The Emotient API offers facial expression and

emotion detection and analysis tools to empower companies and market

38

research firms to create new levels of customer engagement,

research, and analysis” (Emotient, “Products,” emphasis added).

However, at other times the subject who is target of emotion-

measurement and the researcher or emotion-data analyst are more

conflated. The Affdex demo, for example, suggests that anyone and

everyone can and should be concerned with emotional self-

awareness, and even use emotion-measurement technologies to

evaluate and modulate themselves. Even “Lead Scientists” subject

their own facial expressions to measurement, as does Dr. Marian

Bartlett of Emotient (even if only to demonstrate how well the

technology works). This subject-object confusion no doubt stems

in part from the ongoing “interpretive flexibility” of these

technologies, even as those developing and marketing the

technologies strive toward “rhetorical closure” (to use the

language of STS) (Pinch and Bijker 1989). But the effect is a

sort of flattening out of the application of emotion analysis, a

democratization of the technology’s suggested uses, such that

everyone is both a potential user and a potential subject to be

analyzed. This so-called democratization of users and uses is

partly the result of the effort to portray emotion analysis as

39

beneficial to everyone, from consumers or end-users to market

researchers and tech developers. The promotional material

suggests a kind of all-encompassing emotion-measuring system, one

that reaches everyone equally, benefitting all by “empowering,”

“enabling,” and “enriching” everyone’s emotional life, through the

sublime solution of digitization. If in the past an adequate

means of effectively measuring and analyzing people’s inner

emotional lives seemed always out of reach, the computational

analysis of emotion promises to leave no emotional being or

experience unexposed or unexamined.

The way that Affectiva and Emotient conceptualize their

envisioned users is not exactly the same as found in

neuromarketing discourse, at least in terms of Schneider and

Woolgar’s (2012) findings. To be sure, companies marketing

emotion-sensing systems for market research applications often

try to piggyback on the cultural fascination with and currency of

neuroscience by associating their products with neuromarketing.

Affectiva, for example, describes Affdex as a “neuromarketing

tool that reads emotional states such as liking and attention

from facial expressions using an ordinary webcam...to give

40

marketers faster, more accurate insight into consumer response to

brands, advertising and media” (Affectiva, “Affdex Facial

Coding”). However, they also make a concerted effort to

differentiate their product from neuromarketing techniques,

suggesting that it has certain relative advantages for market

researchers. After describing Affdex as a “neuromarketing tool,”

the company then suggests that it offers unique benefits over

neuroscience: while neuromarketing techniques “have been gaining

in popularity,” they involve “in-lab methods that require complex

hardware and black-box analysis” (Affectiva, “Solutions”). In

contrast, Affectiva claims, “Faces are easy to understand, and

the spontaneous reactions we see on faces are unfiltered and

unbiased” (ibid.). The implication here is that neuromarketing

involves cumbersome technologies and analytical results that are

not transparent or legible to non-experts. The difference with

Affdex, they suggest, is that it is easy for anyone to use and

produce readily readable results in the form of direct

measurements of facial motion, rather than esoteric output that

requires those who would use such methods to place their faith in

the specialized knowledge and interpretive skills of experts.

41

Here the suggestion is that reading emotion off the surface of

the body is more viable approach for market researchers, given

their knowledge and area of expertise, than attempting to

visualize and make sense of the invisible interior activity of

the brain. In other words, market researchers and tech developers

are not neuroscientists, and transferring agency and

accountability to the latter may not be in the former’s best

interest.

Another key emphasis in the promotional discourse and

business models of companies like Affectiva and Emotient is the

“scalability” of their emotion-sensing technologies. The promise

of “scalability” is to extend emotion-measurement to the level of

populations, creating “big data” systems for sentiment analysis,

adaptable to the aims of predictive analytics. Distributing

automated emotion measurement technologies across networks and

integrating them with networked devices and platforms promises to

break down the conventional distinction in market research

between the analysis of emotional response on the small scale of

the laboratory or focus group setting, and large-scale

statistical analysis of mass markets. In other words, making AFEA

42

apps “scalable” is one way of merging the two “schools of

thought” that have long defined market research, the one favoring

psychology and science, and the other privileging statistics and

data analysis. Both of these approaches—psychological methods on

a small scale and statistical analysis on a large scale—are

future-oriented, aiming to make future predictions in order to

overcome market uncertainties. This future orientation is very

much tied to the issue of scalability in the way these companies

define and promote the value of their products. For example,

appealing to the needs of marketers to deal with the

unpredictability of the future, Emotient emphasizes both

scalability and future-orientation: “We built our architecture

with the future in mind; it is highly scalable and extensible to

adapt quickly with changing market and customer needs” (Emotient,

“About Emotient”).

This promise of future scalability, as well as better

functionality, is akin to a point that Schneider and Woolgar make

about neuromarketing: in neuromarketing discourse, there is some

acknowledgment of the limitations, with the promise of perfected

techniques deferred to a future time when “what turns out to have

43

been temporary technical problems have been overcome” (Schneider

and Woolgar 2012: 184). In other words, if current technologies

and forms of expert interpretation prove far from perfect, not to

worry—their continuous improvement is in process and eventual

perfection assured. Whether in fact AFEA technologies are ready

for the “scalable” demands of big data and predictive analytics,

or if they ever will be, is yet to be determined. But for

certain, the likelihood of getting there requires transitioning

emotion-measurement prototypes out of the lab and into devices,

platforms and distributed digital networks. Regardless of whether

two companies called Affectiva and Emotient are market contenders

with a viable future, they already have a place in the history of

emotion analysis in market research, as well as in the

“technological trajectory” of efforts to digitize and automate

emotion analysis. As I have argued elsewhere, the very

possibility of creating automated forms of facial expression

analysis suggests that certain assumptions about human affective

relations are already in circulation—assumptions that posit

affect as a physiological process capable of being not only coded

and analyzed, but also engineered (Gates 2011). The belief that

44

this is possible, and the perceived need and demand to make it

so, suggest that companies like Affectiva and Emotient are just

the beginning of efforts to monetize new emotion measurement

technologies.

Conclusion

The marketization of new emotion measurement techniques as

market research tools is a significant development, and not just

for businesses determined to tap into and manipulate our

unconscious drives and desires. The market orientation of this

early application of new affect-sensing technologies shapes their

“technological trajectory” (MacKenzie 1993), designing the

priorities of the market and monetization into these

technologies. This is not to deny that these technologies have

other potential applications, less inflected with market values,

such as autism therapy or basic research in psychology. But the

digitization of emotion for market research purposes promises to

scale up these technologies, broadening their reach and making

them more widely applicable for a range of institutional

applications that promise to have more far-reaching effects.

45

Ultimately, the underlying aim of emotion analysis in market

research is to inform the design of media and emotional

subjectivation, ways of measuring and modulating embodied,

affective engagements to bring emotional subjectivities into

being. The automation of emotion sensing, and its “scalability,”

promises to extend experimental techniques out of the lab and

across distributed networks and their user populations. By

integrating automated facial expression analysis into distributed

digital networks, market-research ventures like Affectiva and

Emotient offer prototypes for more dispersed and broad-based

applications of automated emotion-sensing, building out networked

“emotion-aware” systems that aim to modulate of our affective

relations—with one another, with machines, and with the world we

inhabit.

There are plenty of questions in need of further

consideration. What are the connections, collaborations, and

points of differentiation between academic research fields of

psychology and neuroscience, on the one hand, and neuromarketing

and sentiment analysis on the other? How do more recent

intersections between these fields follow or differ from the

46

historical relationships between academic psychology and the

market research industry? How closely allied are these fields, in

other words, and what is the nature of their exchange in

concepts, methods, and research? What historical connections

might be found between media and market research (and emotion

research in particular) and the history of computing and

computerization, artificial intelligence research, and

cybernetics? How did computerization in media and market research

change the field’s methods and approaches, or how might the

history of the field be rethought through the lens of the current

conjuncture of market research methods and data science? In the

domain of emotion research, in what ways are digitization and

monetization correlated, and in what sense are they discrete

phenomenon or processes? And how does the priority of media

monetization shape the forms that “emotionally aware” digital

technologies end up taking, if not also the forms of emotional

awareness that users of these technologies are able or encouraged

to learn and identify with? We might also ask questions about how

to intervene on the technological trajectory pushing the build-

out of emotion-sensing systems to serve market demands. How can

47

media subjectivities subvert those forms of media subjectivation

that would reproduce the unsustainable imperatives of

consumption? In short, how do we reinvent ourselves and our

“affective circuitry” in a manner consistent with a more just,

ethical and ecologically viable existence?

Bibliography

Affectiva (2014) “Affdex Demo,” http://www.affdex.com/technology/affdex-demo/

Affectiva (2014) “Affdex Facial Coding,” http://www.affdex.com/technology/affdex-facial-coding/

Affectiva (2014) “Media Measurement,” http://www.affdex.com/solutions/media-measurement/

Affectiva (2014) “Solutions,” http://www.affdex.com/solutions/

Andrejevic, M. (2013) Infoglut: How Too Much Information is Changing the WayWe Think and Know. New York: Routledge.

------ (2002) “The Work of Being Watched: Interactive Media and the Exploitation of Self-Disclosure,” Critical Studies in Media Communication 19(2): 230-248.

Ariely, D. and Burns, G. S. (2010) “Neuromarketing: The Hope and Hype of the Neuroimaging Business,” Nature 11 (April): 284-292.

Bakker, G. (2003) “Building Knowledge about the Consumer: The Emergence of Market Research in the Motion Picture Industry,” Business History 45(1): 101-127.

48

Berghoff, H.; Scranton, P.; and Spiekermann, U. (2012) The Rise of Marketing and Market Research. New York: Palgrave Macmillan.

Bratich, J. (2013) “From Audiences to Media Subjectivities: Mutants in the Interregnum,” in K. Gates (volume ed.), Valdivia, A. (general ed.) The International Encyclopedia of Media Studies. Vol VI: Media Studies Futures, Oxford: Wiley-Blackwell, pp. 425-454.

Emotient (2014) “About Emotient,” http://www.emotient.com/about

Emotient (2014) “Emotion-Aware Computing,” [video] http://www.emotient.com

Emotient (2014) “Markets,” http://www.emotient.com/markets

Emotient (2014) “Products,” http://www.emotient.com/products

Fullerton, R. A. (2007) “‘Mr. MASS Motivations Himself’: Explaining Dr. Ernest Dichter,” Journal of Consumer Behavior 6: 369-382. Gates, K. (2011) Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance, New York: NYU Press.

Gobé, M. (2001/2010) Emotional Branding: The New Paradigm for Connecting Brands to People. New York: Allworth Press.

Hansen, F. and Christensen, S. R. (2007) Emotions, Advertising and Consumer Choice. Copenhagen: Copenhagen Business School Press.

Lazarsfeld, P. F. (1934) The psychological aspect of market research. Harvard Business Review 13(1): 54-71.

Lazarsfeld, P. F. (1937) The use of detailed interviews in marketresearch. The Journal of Marketing 2: 3-8.

49

Littlefield, M. M. and Johnson, J. M. (Eds.) (2012) The Neuroscientific Turn: Transdisciplinarity in the Age of the Brain. Ann Arbor: University of Michigan Press.

Lears, T.J. Jackson. (1983) “From Salvation to Self-realization: Advertising and the Therapeutic Roots of Consumer Culture, 1880-1930,” in R. Wightman Fox and T.J. Jackson Lears (eds.) The Culture of Consumption: Critical Essays in American History, 1880-1980, New York: Pantheon, pp. 1-38.

Lears, T.J. Jackson. (1994) Fables of Abundance: A Cultural History of Advertising in America. New York: Basic Books.

Liu, B. (2014) Sentiment Analysis and Opinion Mining. San Rafael: Morgan &Claypool.

Lyons, D. (2014 March 26) “Digital Distress: Quants are Taking Over, and People are Freaking Out” [weblog posting] Hubspot.com, http://blog.hubspot.com/opinion/digital-distress

MacKenzie, D. (1993) Inventing Accuracy: A Historical Sociology of Nuclear Missile Guidance. Cambridge: MIT Press.

Malin, B. (2009) “Mediating Emotion: Technology, Social Science, and Emotion in the Payne Fund Motion-Picture Studies,” Technology and Culture 50(2): 366-390.

Marwick, A. and boyd, d. (2011) “I tweet honestly, I tweet passionately: Twitter users, context collapse, and the imagined audience,” New Media & Society 13(1): 114-133.

Morozov, E. (2013) To Save Everything, Click Here: The Folly of Technological Solutionism. New York: Public Affairs.

Mosco, V. (2005) The Digital Sublime: Myth, Power, and Cyberspace. Cambridge,MA: MIT Press.

50

IBM (n.d.) “Predictive analytics,” http://www-03.ibm.com/software/products/en/category/predictive-analytics

O’Shaughnessy, J. and O’Shaughnessy, N. J. (2003) The Marketing Power of Emotion. Oxford: Oxford University Press.

Pinch, T. J. & Bijker, W. E. (1989) “The Social Construction of Facts and Artifacts: Or How the Sociology of Science and the Sociology of Technology Might Benefit Each Other,” In W. E. Bijker, T. Hughes, and T. J. Pinch (eds.), The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology, Cambridge: MIT Press, pp. 17-50.

Scott, W. D. (1904 January 1) “The Psychology of Advertising,” The Atlantic Monthly. http://www.theatlantic.com/magazine/archive/1904/01/the-psychology-of-advertising/303465/.

Schneider, T. and Woolgar, S. (2012) “Technologies of Ironic Revelation: Enacting Consumers in Neuromarkets,” Consumption, Markets, Culture 15(2): 169-189.

Schudson, M. (1986) Advertising, the Uneasy Persuasion: Its Dubious Impact on American Society. New York: Basic Books.

Schüll, N. Dow. (2013) Addiction by Design: Machine Gambling in Las Vegas, Princeton: Princeton University Press.

Siegel, E. and Davenport, T. H. (2013) Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die. New York: Wiley.

Sony (n.d.) “Smile Shutter,” http://www.sony.co.uk/hub/learnandenjoy/2/2.

Thrift, N. (2008) Non-representational theory: Space, politics, affect. New York: Routledge.

51

Wilson, E. H. (2011) Affect and Artificial Intelligence. Seattle: Universityof Washington Press.

52