ACSI American Customer Satisfaction Index TM Does Interviewing Method Matter? Comparing Consumer Satisfaction Results across Internet and RDD Telephone.
Post on 26-Mar-2015
215 Views
Preview:
Transcript
ACSIAmerican Customer Satisfaction Index TM
Does Interviewing Method Matter? Comparing Consumer Satisfaction Results across Internet
and RDD Telephone Samples
Forrest V. Morgeson III, Ph.D.
Director of Research, American Customer Satisfaction Index
Barbara Everitt Bryant, Ph.D.
Research Scientist-Emerita, University of Michigan
Reg Baker
President, Market Strategies International
Presented at the 66th Annual American Association for Public Opinion Research Conference
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Research Questions and Findings
• Research Questions: Does interview method matter? Do the results produced in a multi-industry consumer satisfaction study differ significantly across a sample collected through RDD/probability sampling and telephone interviewing, and one collected via online panel/nonprobability sampling and Internet interviewing?
• Research Design: We utilize a multi-method sample of consumer satisfaction data, structural equation modeling techniques, and two tests of difference to investigate the significance of differences in survey responses across samples drawn and interviewed using these two methods
• Findings: While some differences are observed, interview method only marginally impacts the means of the survey responses or the parameter estimates from the structural models. Overall, the findings suggest that mixed-method interviewing is feasible and reliable for consumer-oriented survey research projects
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Overview of the ACSI
• Established in 1994, ACSI is the only standardized measure of customer satisfaction in the U.S. economy, covering approximately 225 companies in 45 industries and 10 economic sectors; companies measured account for roughly one-third of the U.S. GDP
• 100+ departments and agencies of the U.S. federal government also measured on an annual basis, along with local and state government measures
• Results from all surveys are published monthly in various media and on the ACSI website, www.theacsi.org
Structure of the ACSI
Hotels Limited-ServiceRestaurants Full-Service Restaurants
NewspapersMotion PicturesBroadcasting TV NewsSoftwareFixed LineTelephone ServiceWireless TelephoneServiceCable & Satellite TV
EnergyUtilities
SupermarketsGasoline StationsDepartment &Discount StoresSpecialty Retail StoresHealth & Personal Care Stores
BanksLife InsuranceHealthInsuranceProperty &
Casualty Insurance
AirlinesU.S.Postal ServiceExpressDelivery
Local GovernmentFederal Government
Accommodation &Food Services
InformationUtilities Finance &Insurance
Transportation &Warehousing
PublicAdministration/Government
RetailBrokerageTravel
E-Commerce
Hospitals
Health Care & Social Assistance
NationalACSI
Manufacturing/Durable Goods
E-Business
Personal ComputersElectronics(TV/VCR/DVD)Major AppliancesAutomobiles& Light
VehiclesCellular Telephones
News &InformationPortals/SearchEnginesSocial Networking
Manufacturing/Nondurable Goods
Food ManufacturingPet FoodSoft DrinksBreweriesCigarettesApparelAthletic ShoesPersonal Care& CleaningProducts
Retail Trade
The ACSI Model and Methodology
Customer Expectations
Customer Expectations
• Satisfaction• Comparison w/ Ideal• Confirm/Disconfirm
Expectations
PerceivedQuality
PerceivedQuality
PerceivedValue
PerceivedValue
CustomerComplaints
CustomerComplaints
CustomerLoyalty
CustomerLoyalty
• Repurchase Likelihood• Price Tolerance
(Reservation Price)
• Overall• Customization• Reliability
• Overall• Customization• Reliability
• Price Given Quality• Quality Given Price
• Complaint Behavior
Customer Satisfaction
Customer Satisfaction
• In ACSI methodology, customer satisfaction is imbedded in a system of relationships, and analyzed as part of a structural equation model. The model produces two critical pieces of data useful to researchers and firms/agencies:
• The model provides mean scores (on a 0-100 scale) for each measured composite or latent variable
• The model provides parameter estimates (or path coefficients) indicating what most strongly influences satisfaction, and in turn how satisfaction influences future consumer behaviors
ACSI Data Collection
• Each year, including all private sector, public sector and custom research projects, ACSI collects approximately 125,000 interviews of consumers
• From 1994 through 2009, nearly all of this data (with a few exceptions for e-commerce companies) was collected over the telephone using random-digit-dial probability sampling and CATI
• Beginning in 2010, and following pilot testing that produced promising results, ACSI moved to a multi-method interviewing approach, with roughly half the data for any measured company/government agency collected using RDD probability sampling and CATI, and the other half collected using a nonprobability panel of double opt-in respondents interviewed online
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Extant Research
• While a handful of studies comparing results for samples interviewed online to samples interviewed over the telephone exist,* these studies have focused almost exclusively on political opinions, voter preference, etc.
• There remains very little research into what differences (if any) are likely to be observed across these two interviewing methods for consumer-oriented data, where a significant portion of data collection and survey research is focused
*Chang, L. and J.A. Krosnick (2009). “National Surveys via RDD Telephone Interviewing Versus The Internet: Comparing Sample Representativeness and Response Quality,” Public Opinion Quarterly, 73(4), 641–678.
Fricker, S., M. Galesic, R. Tourangeau and T. Yan (2005). “An Experimental Comparison of Web and Telephone Surveys,” Public Opinion Quarterly, 69(3), 370-392.
Vannieuwenhuyze, J., G. Loosveldt and G. Molenberghs (2010). “A Method for Evaluating Mode Effects inMixed-Mode Surveys,” Public Opinion Quarterly, 74(5), 1027-1045.
Findings from the AAPOR Online Task Force
• Findings from the AAPOR Online Task Force* suggest that there is no theoretical basis for assuming that samples drawn from nonprobability online panels are representative of a larger population, and that therefore results may differ when compared to an RDD probability sample interviewed over the telephone
• However, this research also concludes there may be instances in which online panels are useful and reliable, and we conduct a series of empirical tests to see if customer satisfaction data (ACSI) is such a case
*Baker, R. et al. (2010). “Research Synthesis: AAPOR Report on Online Panels,” Public Opinion Quarterly, 74(4), 711–781.
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Research Questions
• From the perspective of the ACSI project and its methodology, two questions regarding multi-method interviewing are most relevant and important:
• Do mean scores exhibit significant differences between a sample interviewed online when compared to a sample interviewed using RDD/CATI?
• Do model parameter estimates exhibit significant differences between a sample interviewed online when compared to a sample interviewed using RDD/CATI?
Data
• To seek answers to our research questions, we utilize a sample of data consisting of approximately 9000 interviews
• Roughly half of these cases were collected via Internet interviewing (from a sample balanced to Census demographics from a large online panel (the Research Now panel)), and the other half collected using RDD and CATI, allowing us to test the similarities/differences produced by these two interviewing methods
• The ACSI model (shown earlier) was estimated independently for each industry and each interviewing method, producing distinct mean scores and estimates (path coefficients) facilitating these comparisons
Data
• The data represent consumer responses to questions measuring satisfaction (and the other modeled variables) with companies and industries in six NAICS sectors (for more information on the companies included in the sample, see Appendix A): – Apparel manufacturing (Manufacturing/nondurable goods)
– Personal computers (Manufacturing/durable goods)
– Fast food restaurants (Food services)
– Insurance (Finance and insurance)
– Supermarkets (Retail)
– Wireless phone service (Information)
Tests of Difference
• To test for significant differences in mean scores across the two interviewing methods for each ACSI variable in each of the industries included in the sample, independent sample t-tests were utilized
• To test for significant differences in parameter estimates for the structural model for each of the industries included in the sample, chi-square difference tests were utilized, with parameters constrained to equality and significant chi-square statistics indicative of significant parameter estimate differences
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Results and Findings
• Across all of the tests – which included comparisons of 36 sets of mean scores across the two interviewing methods, and 54 sets of model parameter estimates – some significant differences were observed
• In total, 36% of the mean scores (13 of 36) compared across the two modes exhibited significant differences. Scores skewed higher on the Internet, with 9 of 13 significant differences reflecting “better” ratings among Internet respondents (i.e. higher ratings, fewer complaints)
• Moreover, 39% of the model parameter estimates (21 of 54) from the structural models compared across the two methods exhibited significant differences
• (Two industry examples follow. All test results provided in Appendix A)
Example 1: Supermarket Industry Results
Supermarket Industry
Telephone Internet Sig. Diff. Variable N Mean N Mean Expectations 784 79.08 790 80.24 Quality 784 80.43 790 79.43 Value 783 76.54 790 77.34 Satisfaction 784 76.38 790 75.59 Comp. (%) 782 10.87 788 10.53 Loyalty 782 76.37 786 82.60 ***
Supermarket Industry
Path Coefficient Tele. Internet Sig. Diff.
Expect. → Quality 0.776 0.833
Quality → Value 0.528 0.629
Expect. → Value 0.196 0.111
Value → Sat. 0.444 0.481
Quality → Sat. 0.372 0.505 **
Expect. → Sat. 0.195 0.051 **
Sat. → Comp. -0.286 -0.308
Comp. → Loyalty 0.045 -0.016
Sat. → Loyalty 0.616 0.638
• For the tests for this industry, one variable mean score of the six tested was significantly different across the two samples, while two of nine parameter estimates were significantly different
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Example 2: Wireless Industry Results
Wireless Industry
Telephone Internet Sig. Diff.
Variable N Mean N Mean
Expectations 475 75.69 490 80.94 ***Quality 478 75.88 493 78.80 *
Value 470 72.70 488 71.54
Satisfaction 478 71.17 492 71.20
Comp. (%) 475 30.95 485 21.65 **Loyalty 473 69.31 462 74.14 *
Wireless Industry
Path Coefficient Tele. Internet Sig. Diff.
Expect. → Quality 0.775 0.56 **
Quality → Value 0.85 0.998 **
Expect. → Value 0.042 -0.058 Value → Sat. 0.457 0.529 Quality → Sat. 0.48 0.476
Expect. → Sat. 0.053 0.005 Sat. → Comp. -0.621 -0.601 Comp. → Loyalty -0.033 -0.037 Sat. → Loyalty 0.942 0.96
• For the tests for this industry, four of the variable mean scores exhibited significant differences, with scores skewing higher (and complaint rate lower), and two of the parameter estimates exhibited significant differences
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Results and Findings
• The above are “hard tests” of multi-method interviewing. As many projects (including ACSI) have not traded telephone-only for Internet-only interviewing, a “fairer” test is to compare the telephone interview results to the mixed-method, mixed-sample results
• For these tests, the results are more promising. Looking only at differences in mean scores, of the 36 sets of means compared only 11% (4 of 36) exhibited significant differences
• (Two industry examples follow. Full results for these tests are included in Appendix A)
Example 3: Mixed-Sample vs. Telephone-Only
Mixed-Sample Telephone Sig. Diff. N Mean N Mean Apparel Industry
Expectations 957 83.99 475 84.14
Quality 957 85.12 475 86.33
Value 958 82.28 475 84.05
Satisfaction 957 81.26 475 83.16 *Comp. (%) 955 1.47 475 0.63 Loyalty 950 79.78 473 79.52
PC Industry
Expectations 1156 83.51 556 82.94
Quality 1157 82.40 556 81.44
Value 1153 82.49 553 82.35
Satisfaction 1157 78.81 556 77.78
Comp. (%) 1147 12.64 553 15.91
Loyalty 1158 74.05 557 71.76
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
• Overview: Research Questions and Findings
• The American Customer Satisfaction Index (ACSI)
• Extant Research on Interviewing Method Differences
• Data and Analysis Methods
• Results and Findings
• Conclusions and Implications
Discussion Agenda
Conclusions
• While some differences in both mean scores and model parameter estimates are exhibited when comparing telephone-only interviewing to Internet-only interviewing, the differences account for a minority in both cases
• The results are even more promising when comparing mean scores for telephone-only and mixed-method interviewing; only a small fraction of the comparisons are significantly different in this case
Implications and Future Research
• These tests provide evidence for the feasibility and reliability of mixed-method sampling for consumer-oriented survey research projects
• For projects working with this kind of data, both means scores and model estimates appear to be relatively stable across interviewing methods
• However, because we examine only consumer-oriented data, those working with dissimilar types of data should perform tests similar to ours to examine the reliability of mixed-method interviewing, as results may vary
• Research expanding the types of data tested should help market researchers determine the feasibility of multi-method interviewing for particular client engagements
Appendix A: Supplemental Results and Information
Interview Data by Industry/Company
Industry Companies
Apparel Liz Claiborne; VF Corporation; Levi Strauss; Jones Apparel Group; Hanesbrands
Personal Computers Compaq; Apple; Hewlett Packard; Dell; Acer
Fast Food Wendy’s; KFC; Little Caesar Enterprises; Domino’s; Taco Bell; Pizza Hut; Burger King; McDonald’s; Papa John’s; Starbucks
Insurance Farmer’s Group; Allstate; State Farm; Geico; Progressive; MetLife; Prudential; New York Life; Northwestern Mutual Life
Supermarkets Publix; Winn-Dixie; Supervalu; Safeway; Wal-Mart; Kroger; Whole Foods
Wireless Service Verizon; AT&T; Sprint Nextel; T-Mobile
Apparel and PC Industries Results
Telephone Internet Sig. Diff.
N Mean N Mean
Apparel Industry
Expectations 475 84.14 482 83.83
Quality 475 86.33 482 83.93 *
Value 475 84.05 483 80.54 **
Satisfaction 475 83.16 482 79.39 **
Comp. (%) 475 0.63 480 2.29 *
Loyalty 473 79.52 477 80.05
PC Industry
Expectations 556 82.94 600 84.03
Quality 556 81.44 601 83.28
Value 553 82.35 600 82.63
Satisfaction 556 77.78 601 79.76
Comp. (%) 553 15.91 594 9.60 **
Loyalty 557 71.76 601 76.17 **
Path Coefficient Tele. Internet Sig. Diff. Apparel Industry
Expect. → Quality 0.625 0.778 **Quality → Value 0.721 0.847 Expect. → Value -0.031 -0.020 Value → Sat. 0.449 0.350 *Quality → Sat. 0.415 0.553 *Expect. → Sat. 0.069 0.051 Sat. → Comp. -0.034 -0.057 Comp. → Loyalty -0.216 0.014 Sat. → Loyalty 0.772 0.908 *PC Industry
Expect. → Quality 0.690 0.636 Quality → Value 0.772 0.924 **Expect. → Value 0.027 -0.104 *Value → Sat. 0.397 0.419 Quality → Sat. 0.551 0.630 Expect. → Sat. 0.074 -0.041 **Sat. → Comp. -0.547 -0.475 Comp. → Loyalty -0.016 -0.002 Sat. → Loyalty 0.971 1.155 **
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Fast Food and Insurance Industries Results
Telephone Internet Sig. Diff.
N Mean N Mean
Fast Food Industry Expectations 1150 78.45 1169 79.98 *
Quality 1150 80.32 1169 80.59
Value 1149 80.48 1170 80.25
Satisfaction 1150 75.65 1170 75.24
Comp. (%) 1149 8.09 1162 8.00
Loyalty 1145 74.44 1156 78.21 ***
Insurance Industry
Expectations 970 81.72 1047 82.47
Quality 973 83.81 1046 82.93
Value 966 79.83 1039 78.40
Satisfaction 971 79.68 1047 77.97
Comp. (%) 976 7.48 1048 6.20
Loyalty 942 76.41 1006 78.11
Path Coefficient Tele. Internet Sig. Diff.
Fast Food Industry
Expect. → Quality 0.748 0.876 ***
Quality → Value 0.694 0.647
Expect. → Value 0.115 0.197
Value → Sat. 0.350 0.425 **
Quality → Sat. 0.630 0.511 **
Expect. → Sat. 0.051 0.116
Sat. → Comp. -0.347 -0.376
Comp. → Loyalty 0.049 -0.019 *
Sat. → Loyalty 0.825 0.797
Insurance Industry
Expect. → Quality 0.672 0.710
Quality → Value 0.703 0.820 **
Expect. → Value 0.212 0.120 *
Value → Sat. 0.379 0.521 ***
Quality → Sat. 0.527 0.481
Expect. → Sat. 0.076 0.007 *
Sat. → Comp. -0.386 -0.308
Comp. → Loyalty -0.083 -0.037
Sat. → Loyalty 0.813 0.929 **
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Mixed-Method vs. Telephone-Only Means Tests (1)
Mixed-Method Telephone Sig. Diff.
N Mean N Mean
Fast Food Industry
Expectations 2319 79.22 1150 78.45
Quality 2319 80.46 1150 80.32
Value 2319 80.36 1149 80.48
Satisfaction 2320 75.44 1150 75.65
Comp. (%) 2311 8.05 1149 8.09
Loyalty 2301 76.33 1145 74.44 *
Insurance Industry
Expectations 2017 82.11 970 81.72
Quality 2019 83.35 973 83.81
Value 2005 79.09 966 79.83
Satisfaction 2018 78.79 971 79.68
Comp. (%) 2024 6.82 976 7.48
Loyalty 1948 77.29 942 76.41
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
Mixed-Method vs. Telephone-Only Means Tests (2)
Mixed-Method Telephone Sig. Diff.
N Mean N Mean
Supermarket Industry
Expectations 1574 79.66 784 79.08
Quality 1574 79.93 784 80.43
Value 1573 76.94 783 76.54
Satisfaction 1574 75.98 784 76.38
Comp. (%) 1570 10.70 782 10.87
Loyalty 1568 79.49 782 76.37 **
Wireless Industry
Expectations 965 78.36 475 75.69 *
Quality 971 77.36 478 75.88
Value 958 72.11 470 72.70
Satisfaction 970 71.19 478 71.17
Comp. (%) 960 26.25 475 30.95
Loyalty 935 71.70 473 69.31
*All variables scaled 0-100, worse to better rating; “Sig. Diff.” column reports significant difference between the Telephone and Internet interview samples ; * = p<.05; ** = p<.01; ***p<.001.
top related