NetBase Social Media

Can Social Media MeasureCustomer Satisfaction?

Elliot Bricker, Director, Product Management, NetBase MARCH 201 1

Reviewed by an independent team from the Walter A. Haas School of Business at the University of California at Berkeley. The team was led by Vito A. Sciaraffia – PhD candidate in business economics and strategy.

Can Social Media Measure Customer Satisfaction?

2

Executive SummaryExecutive SummaryExecutive SummaryExecutive Summary

Because customer satisfaction is a key predictor of financial performance, businesses have

invested a lot of money and resources in tracking the satisfaction of customers through

surveys and benchmarking. Social media moved into the mainstream in 2010; savvy

executives soon recognized that the millions of blog entries, micro-blogs, status updates, and

comments that consumers post every day could become a new source of customer

satisfaction data – one that is significantly faster and less expensive than traditional survey

methods. The question is, does sentiment expressed in social media—that is, whether online

posts are positive, negative, or neutral—correlate with established customer satisfaction

metrics?

To answer that question, NetBase compared the Net Sentiment Score in the NetBase Insight

Scorecard to published scores from the American Consumer Satisfaction Index (ACSI) and

found a high correlation (Pearson Product Moment Coefficient r=.773). In this white paper, we

discuss our analysis and also share research findings that will help you benchmark your online

sentiment scores with industry peers. We also discuss the roles that Passion Intensity and

Share of Buzz play as social metrics that measure distinct facets of the customer experience.


3

Table of Contents

Customer Satisfaction Matters – at the Bottom Line .....................................................................................4

Social Media: A New Source of Customer Satisfaction Data ......................................................................4

Does Online Sentiment Correlate With Customer Satisfaction? ...............................................................4

NetBase Insight Scorecards Track Three Key Metrics of Online Brand Equity ................................... 5

Measuring Sentiment and Passion With the Science of Language .......................................................... 5

NetBase’s Net Sentiment Score ............................................................................................................................... 7

Correlating Net Sentiment to ACSI ........................................................................................................................ 8

NetBase’s Passion Intensity Score ........................................................................................................................ 10

Is Passion Intensity Really Different From Net Sentiment and Buzz? .................................................... 11

Bringing it All Together: The Brand Passion Index ........................................................................................ 12

Conclusion ....................................................................................................................................................................... 13

Methodology .................................................................................................................................................................. 13

Appendix ......................................................................................................................................................................... 14

Related Reading ........................................................................................................................................................... 15

About the Author ......................................................................................................................................................... 15


4

Customer Satisfaction Matters – at the Bottom Line

Businesses know that customer satisfaction is a predictor of financial performance. Satisfied

customers generate more revenue. They are also more profitable because

it costs much less to market to existing customers than to acquire new

ones.

As a result, businesses have invested a lot of money in tracking the

satisfaction of their customers through a variety of customer satisfaction

(CSAT) and voice-of-customer (VoC) surveys. Email surveys are the most common data

source, followed by telephone incident follow-up surveys, and finally self-service follow-up

using VoC tools that “intercept” users during visits to a web site.

There are a variety of established measurement methodologies such as the Net Promoter

Score (NPS®) or SERPVAL/SERVQUAL and benchmarking services such as the American

Customer Satisfaction Index that serve to make CSAT survey data more meaningful and

actionable. However, all survey-based tracking methods have a common set of shortcomings:

• They rely on limited samples of customers

• Data collection takes time, slowing down responses to potential customer satisfaction

issues

• Because of the costs involved, most companies can only afford to track their own

brands

Social Media: A New Source of Customer Satisfaction Data

Forward-looking businesses have recognized that a new source of customer satisfaction has

emerged: social media. Every hour, consumers make more than 500,000 new blog and

micro-blog entries, status updates, and comments in social media. They are raving about

companies and brands that they like and spreading the word about the bad experiences they

have had. In fact, social media is where new impressions propagate first and fastest today.

Does Online Sentiment Correlate With Customer

Satisfaction?

Social media offers a fast, inexpensive way to measure customer satisfaction for both your

brand and your competitors’ brands. As executives evaluate it as a strategic data source,

many are asking the key question: How can social media help us to measure customer

satisfaction? In theory, an aggregate measure of online sentiment—that is, whether online

posts are positive, negative, or neutral—should correlate with established customer

satisfaction metrics.

In this white paper, we will explore that question. We will look at how the Net Sentiment

Score in the NetBase Insight Scorecard compares to scores calculated with popular CSAT


5

methodologies. Our findings demonstrate a strong correlation between the American

Consumer Satisfaction Index (ACSI) and NetBase Net Sentiment Scores derived from social

media opinions. We will also share research findings that will help you benchmark your CSAT

scores with industry peers. In addition, we will look at the roles that Passion Intensity and

Share of Buzz play in understanding customer satisfaction. Finally, we will discuss the role

that social media can play in helping executives to understand the underlying root causes of

customer satisfaction issues, so that they can make smarter business decisions faster.

NetBase Insight Scorecards Track Three Key Metrics of

Online Brand Equity

NetBase Insight Scorecards were designed to give

businesses a reliable way to measure, enhance, and

protect brand equity in social media. They deliver

up-to-date metrics on three key aspects of your

customers’ experience:

• Share of Buzz - How much people are talking about your brand

• Net Sentiment - How positively they perceive your brand

• Passion Intensity - How emotionally charged their feelings are

These three metrics, along with measurements of

key conversation drivers that you specify, are

displayed with 12 month historical trending and

benchmarked against your key competitors.

Because they do not require expensive consulting

fees or time-consuming data collection and

assembly, NetBase Insight Scorecards give you a

fast way to stay on top of your consumers’

experiences with your brand.

Measuring Sentiment and Passion With the Science of

Language

ConsumerBase, the data source accessed via Scorecard, is a social intelligence warehouse

containing a full year’s worth of social media commentary across more than 95 million


6

sources, from forums and blogs to Facebook and Twitter. We use the most advanced natural

language processing (NLP) engine to read, understand, and categorize every posting in

ConsumerBase according to the sentiments, emotions, and key ideas that your consumers

have expressed. Then, we aggregate these sentiments, emotions, and ideas into the metrics

that you see in your Scorecard: Share of Buzz, Net Sentiment, and Passion Intensity.

NetBase’s NLP engine represents a big leap in accurately analyzing content from the social media universe. Unlike tools that infer sentiment based on statistical keyword matching, NetBase understands sentence grammar at a deep level and delivers over 80 percent accuracy.

This approach does not involve counting words or analyzing text; NetBase reads sentences, evaluates grammatical sentence patterns, and organizes results to be fully searchable on a wide variety of attributes. We analyze social media content in two steps: parsing and normalization.

First, the NLP engine parses each sentence it captures from social media at a very deep level. This process is similar to the sentence diagramming that students do in a high school English class—it identifies and links the subjects, objects, verbs, adjectives, and other linguistic patterns in the sentence to extract deep and accurate understanding of what is being said. By analyzing this “connective tissue” within each sentence, our NLP engine can account for the complexities in language that make keyword-matching algorithms inaccurate.

Anaphora resolution is a key part of the parsing process that ensures that sentiment-rich associations are not missed.

Anaphora are grammatical substitutes that refer back to your brand name in sentences that may otherwise be missed in most social media analysis tools. Typically, a pronoun (e.g. it, her, him, their, they), anaphora refer back to another unit, as the use of "they're" and "they", and "them" refer to M&Ms in the sentences: M&Ms are sweet and delicious. I remember always hearing that they[M&Ms] melt in your mouth and not in your hands. We're going to buy more[M&Ms] today down at the store because we love them[M&Ms]. So we see that the anaphors obey what linguistics call binding conditions. Anaphora resolution requires semantic understanding that is even today undergoing active academic and business research in the field of natural language processing.


7

Next, our NLP engine normalizes all the parsed sentences to make them easy to aggregate into the metrics shown in our Scorecards. It takes sentences (what we call “sound bites”) and stores them, based on the type of insight they reveal, in a single, consistent format, regardless of the structure of the underlying sentences. Normalization is a fundamental part of NetBase’s unique value because it allows our solutions to expose not just positives and negatives but also deeper-level insights such as passion.

Working with many of the most sophisticated brands in the world, we have optimized the

NetBase NLP engine specifically for understanding social media and the Web. In addition to

standard English, our extensive lexicon includes a wide variety of “urban words” and phrases,

alternative spellings, and abbreviations common in social media, as well as common

misspellings. We are constantly incorporating new rules into this lexicon based on the work of

our internal linguistics experts, ongoing testing that we do using “crowdsourced” human

evaluators, and feedback from customers.

NetBase’s Net Sentiment Score

Automated sentiment analysis focuses on analyzing the content of online posts, determining

whether they are positive, negative, or neutral, and aggregating the sentiments detected into

a single generic score. The Net Sentiment Score computes a ratio of positive and negative

mentions of a topic. Our analysis is localized to individual sentences, our “sound bites;” we do

not view an entire post as a single sentiment.

The formula for Net Sentiment is:

A Net Sentiment of 100 means that all mentions of the topic are positive, and -100 means all

mentions of the topic are negative.

The average Net Sentiment Score across the thousands of brand names, companies, people,

and other product names mentioned in ConsumerBase is 32323232. This indicates that overall, the

chatter about brands and experiences is somewhat positive.

The NetBase Insight Scorecard charts the Net Sentiment Score in a time series for the brands

that you track. This view makes sentiment data more actionable because you can easily see

its directionality. You can also see a count of positivepositivepositivepositive sound bites (green area in chart on the

next page) and negativenegativenegativenegative sound bites (red area). The y-axis secondary scale (far right)

displays a scale for the volume of sentiment polar sound bites.


8

Below are some examples of a few industry- or category-specific Net Sentiment Scores. They

have been computed over the course of a single year and represent a comprehensive

selection of companies, their related products, and brands for each industry.

Additional analysis was conducted to prove that the distribution of Net Sentiment Scores

does not cluster for a specific industry. In other words, Net Sentiment Scores follow a typical

“bell curve” distribution where values are distributed fairly evenly around their mean and tail

off at the extremes of high and low scores. Please see the Appendix for more details.

Correlating Net Sentiment to ACSI

In order to look at how Net Sentiment compares

to established measures of customer

satisfaction, we computed Net Sentiment Scores

for 12 retail businesses (retail, wholesale, or

department stores) and compared them against

the academic methodology of the ACSI,

developed at the University of Michigan. These

scores are the aggregate of 12 months of data

that span February 1, 2010 until mid-February,

2011. The ACSI releases industry results

monthly, and the scores reflect the most recent

installment – February 2011.

The American Customer Satisfaction Index (ACSI) is a uniform, national, cross-industry measure of satisfaction. The distinguishing feature of the ACSI methodology is its patented cause-and-effect approach to customer satisfaction measurement. Three questions comprise the assessment of satisfaction:

• What is your overall satisfaction with our product or service?

• To what extent has our product or service met your expectations?

• How well did our product or service compare with the ideal?

Organizations normalize (weight) and average the three ratings to produce a score from 0 to 100.


9

The comparison between the NetBase Net Sentiment Score and the scores reported by the

ACSI organization for the same time period showed a high (r=.773) Pearson Product Moment

Coefficient –meaning there is a good correlation between the two scores.

Store

Net Sentiment Score

Target 78 51.3

Kohl's 81 70.1

J.C. Penney 80 65.3

Sam's Club 78 75.0

Lowe's 77 59.4

Walgreens 77 52.5

Best Buy 77 66.6

Macy's 76 66.2

Home Depot 75 58.0

Rite Aid 75 51.8

CVS 74 50.5

Wal-Mart 73 40.6

To assure that the correlations of Net Sentiment and other customer satisfaction

methodologies didn’t reflect the state of one industry, we also looked at a cross-industry

spectrum of profiles. This included companies from the automotive, airline, financial, Internet

retail, Internet travel, CPG, and grocery store verticals.1 The correlation approached a similar

coefficient to that measured for the single retail store industry. As can be seen in the chart

on the next page, that number was .714. This again implies that NetBase’s Net Sentiment

Score serves as a strong indicator of CSAT.

1 The companies included: Bank of America, BMW, Charles Schwab, Delta Airlines, eBay,E*TRADE, Expedia, Hershey, JP Morgan Chase, Kellogg's, Kia, Kraft Foods, Kroger, Lincoln Mercury (Ford), Nestlé, Netflix, Newegg Orbitz, Publix, Safeway, Southwest Airlines, TD Ameritrade, US Airways, Wells Fargo, and Whole Foods.


10

NetBase’s Passion Intensity Score

Marketers know that passion intensity matters. It drives word-of-mouth – both good and bad.

It generates brand loyalty and repurchase. But until now, companies did not have a way to

quantifyquantifyquantifyquantify how strongly consumers feel about particular brands.

The NetBase Insight Scorecards include a Passion Intensity ScorePassion Intensity ScorePassion Intensity ScorePassion Intensity Score, a number that ranges

from 0 to 100. This score complements the Net Sentiment Score by adding another

dimension of the customer experience: emotion. The formula is:

To calculate the number of mentions in each category, the NetBase NLP engine identifies

comments and posts that use a specific set of emotions and qualities. Measurements of the

"Love" emotion aggregate linguistic associations to feelings such as love, adore, fan, luv, love, adore, fan, luv, love, adore, fan, luv, love, adore, fan, luv,

thrilled thrilled thrilled thrilled and many more. Additionally, the "Love" emotion accounts for positive subjective

qualities such as brand associations to terms like incredible, fantastic, tremendous, incredible, fantastic, tremendous, incredible, fantastic, tremendous, incredible, fantastic, tremendous,

amazingamazingamazingamazing,,,, and so on.

Measurements of the "Hate" emotion cover expressions such as hate, despise, loathe, hate, despise, loathe, hate, despise, loathe, hate, despise, loathe,

detestdetestdetestdetest,,,, and disgustdisgustdisgustdisgust, , , , with the added complement of negative subjective qualities that include

horrific, appalling, shockihorrific, appalling, shockihorrific, appalling, shockihorrific, appalling, shocking, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck,ng, horrible, awful, terrible, suck, stinkstinkstinkstink,,,, and so on.

The denominator of the formula combines the total number of these “emotion-laden” love

and hate mentions with a measure of other non-emotional but subjective qualities such as

best, worst, suck, rocks, good, best, worst, suck, rocks, good, best, worst, suck, rocks, good, best, worst, suck, rocks, good, difficult, difficult, difficult, difficult, or awesomeawesomeawesomeawesome.


11

As with Net Sentiment, we calculated an average Passion Intensity Score of 30303030 across

thousands of brands, products, people names, and issues. To give you an idea of the range of

passion intensity, here are some measurements that show scores at both ends of the

spectrum:

Lower PassioLower PassioLower PassioLower Passion Intensityn Intensityn Intensityn Intensity Higher Passion IntensityHigher Passion IntensityHigher Passion IntensityHigher Passion Intensity J.P. Morgan Chase 8.4 Windows XP 62 iShares 6.2 Volkswagen 63 Nexium 14 Grand Theft Auto 70 Marmite 81

For the 12 stores we looked at in the ACSI comparison, we calculated the Passion Intensity

Scores seen below. The scores aggregate over 12 months of analysis for individual mentions

expressing emotions and subjective-quality insights:

StoreStoreStoreStore Passion Passion Passion Passion IntensityIntensityIntensityIntensity

Target 26.2

Kohl's 41.9

J.C. Penney 15.0

Sam's Club 41.3

Lowe's 31.8

Walgreens 37.9

Best Buy 90.0

Macy's 43.6

Home Depot 30.9

Rite Aid 46.0

CVS 34.4

Wal-Mart 35.6

Is Passion Intensity Really Different From Net Sentiment

and Buzz?

NetBase believes that there is no single score that can capture all facets of online brand

equity. That’s why our Scorecards contain three metrics—Share of Buzz, Net Sentiment, and

Passion Intensity—with in-line comparisons to competitors and historical values.

Our research has shown that each of the metrics we calculate and display differs greatly from

the metric of sentiment. For example, the Passion intensity metric is very low in correlation

(r=0.10048 Pearson Product Moment Coefficient) to the Net Sentiment Score.


12

Affective discussion in social media is not always an associative measure of sentiment. For example, where sentiment can be very positive, ardor and fervency can be said to be rather blasé.

The low levels of correlation between our Net Sentiment and Passion Intensity Scores attests

to the fact that each metric stands on its own measuring an aspect of your brand’s consumer

satisfaction, with its own distinctive implication in your analysis.

Similarly, Buzz Volume—the number of occurrences of sound bites for the topic of interest—is

a facet of the customer experience that is distinct from Net Sentiment. Consumers may be

very chatty or relatively silent about a brand, but the data still proves that no correlation

exists in the volume of chatter to their overall expressions of polarity in sentiment. A

correlation analysis showed that the direction and degree (closeness) of linear relations

between buzz and sentiment was quite low at 0.01159.

Bringing it All Together: The Brand Passion Index

In order to have one crisp

visualization of the three metrics

of Buzz, Passion Intensity, and

Sentiment, NetBase provides a

Brand Passion Index.

The Brand Passion Index is an

intuitive visualization that lets

you quickly analyze consumer

sentiment, buzz, and passion

intensity across multiple brands

and view historical change in a

single chart. The map has four

quadrants: Like, Dislike, Hate,

and Love. You and your

competitors are placed on one

of these quadrants based on the


13

Passion Intensity that consumers express (Like versus Love) and their Net Sentiments (Like

versus Dislike). The size of the competitor’s bubble indicates the volume of posts for that

competitor.

Conclusion

Social media is the next strategic source of consumer insights – and of competitive

advantage. In this white paper, we have shown that consumer satisfaction expressed in social

media is both important and measurable. We have looked at how NetBase’s Net Sentiment

Score correlates with established measures of customer satisfaction (CSAT). We have also

examined Passion Intensity and Share of Buzz as distinct facets of a consumer’s experience

with a brand.

Social media insight and analysis can play an important role in the next wave of customer-

centric businesses. The businesses that will get ahead of the competition are the ones that are

starting down this path today.

Methodology

The data and statistical methodologies have been reviewed by an independent team from the

Walter A. Haas School of Business at the University of California at Berkeley. The team was

led by Vito A. Sciaraffia, PhD candidate in business economics and strategy. Mr. Sciaraffia

holds an MS in business administration from UC Berkeley, an MBA with concentration in

finance and statistics, an MA in finance from the University of Chile, and a BS in economics

and management from the Pontifical Catholic University of Chile. Additionally, he holds

several academic and professional certifications in statistics. If you would like additional

information about this white paper’s supporting data, please contact [email protected].


14

Appendix

In evaluating Net Sentiment Scores as indicators of customer satisfaction, we were concerned

about “clustering” in specific industries, where a majority of scores fall close to the mean

rather than following a typical “bell curve” distribution.

To test our hypothesis that the distribution of Net Sentiment Scores does not cluster for a

specific industry, we took over 125 women’s and men’s fashion store brands and computed

the normal distribution for each of their scores. We found that the variance, which describes

the spread of the distribution about the mean value, indicates that there is good scattering.

Additionally, there is enough mass (observations) to both sides of the mean. The lower and

upper tails of the distribution are as expected (gradually more unlikely to observe low scores

and high scores) with values nicely distributed between Net Sentiment scores of 28 to 80.

This says that the Net Sentiment Scores for each of the stores is not closely clustered around

the mean value for all brands. The 2D Histogram of the mass function show other “good”

distribution characteristics: A mean of 57.99 for Net Sentiment Score and a standard

deviation of 13.25 shows that one standard deviation on either side of the mean approaches

68%, two standard deviations - 95%, and three - 99%.

Some representative Net Sentiment Scores for both the low end and high end of the

spectrum of the 125 data points are:

Guess Jeans and Accessories 28.4

Rainbow Shops 29.0

Tous 29.9

Champs Sports 31.5

Armani A/X 80.0

GapMaternity 81.5

Heritage 1981 81.8

Tag Heuer Boutique 82.2

Boss Hugo Boss 82.8


15

Related Reading

“Net Promoter® Industry Report 2010,” Satmetrix, 2010.

“Social Media: Are You Listening to the Voice of the Customer?” A Joint Research Paper from

Verint Systems and TSIA, 2011.

“How Does NetBase Achieve the Best Accuracy for Understanding Consumers Online?”

NetBase, September 2010.

“The World’s Most Valuable Brands. Who’s Most Engaged?” Wetpaint and Altimeter Group,

July 2009.

“ACSI Score & Its Calculation,” Jeffrey Henning Mar. 11, 2009,

<http://blog.vovici.com/blog/bid/18135/ACSI-American-Customer-Satisfaction-Index-Score-

Its-Calculation>.

About the Author

Elliot Bricker participates in new feature design, pre- and post-sale

customer support, and other product success areas at NetBase. Elliot

began his career at Information Builders working in the areas of business

intelligence, data connectivity, expert systems software, and data

warehousing. Within numerous start-up companies, Elliot has focused on

applying machine learning as well as semantic search and categorization

of enterprise content to solve business problems. Most recently, his

interests have centered on opinion mining, specifically bringing together

social intelligence with business intelligence. He also has expertise in text

mining, analytics, data visualization, data warehousing, and optimization

algorithms. Elliot has contributed to numerous patent applications, both in the United States

and internationally. He holds BS and MS degrees in Computer Science.

Additional Acknowledgments

NetBase would also like to thank Linda Sonne-Harrison of Giant Stride Marketing Group for

her helpful review, feedback, and editing in regards to the content of this paper.

NetBase Social Media Insight & Analysis helps marketing teams make smarter business decisions faster. We

deliver tools and Scorecards that give market researchers and brand managers a reliable way to understand

online brand equity, analyze and compare consumer passion, and generate deep insights that answer their “why”

questions. Serving hundreds of corporate customers, our products were developed in partnership with five of the

top 10 CPG companies, including Coca-Cola and Kraft, and are used by four of the top 10 market research firms,

including J. D. Power & Associates. Based in the heart of Silicon Valley, NetBase is a privately held company.

For more information, visit: www.netbase.com @Net_Base NetBaseInc

©2011 NetBase Solutions, Inc.

NetBase Solutions, Inc.

2087 Landings Drive

Mountain View | CA 94043

P 650.810.2100

F 650.968.4872

www.netbase.com

About NetBase

NetBase Social Media

Documents

online brand

online sentiment

brand passion

net sentiment

net sentiment

customer satisfaction

measure customer

share research