3/17/2015 1 Agree-Disagree Response Formats: Problems and Alternatives Sowmya Anand Survey Research Laboratory University of Illinois at Chicago GENERAL INFORMATION 2 • Please hold questions until the end of the presentation • Webinar slides available at http://www.srl.uic.edu/SEMINARS/semnotes.htm • Please raise your hand so that I can see that you can hear me
23
Embed
Agree-Disagree Response Formats: Problems and … Spring 2015...3/17/2015 2 Agree-Disagree Questions Simplest form –ask respondents whether they agree or disagree with a statement
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
3/17/2015
1
Agree-Disagree Response Formats:
Problems and AlternativesSowmya Anand
Survey Research Laboratory
University of Illinois at Chicago
GENERAL INFORMATION
2
• Please hold questions until the end of the presentation
• Webinar slides available at
http://www.srl.uic.edu/SEMINARS/semnotes.htm
• Please raise your hand so that I can see that you can hear me
3/17/2015
2
Agree-Disagree Questions
� Simplest form – ask respondents whether they agree or disagree with a statement
� Do you agree or disagree with the following statement?
My health is good.
Agree
Disagree
3
Agree-Disagree Questions
� Can include intensity (agree strongly or agree somewhat)
� How much do you agree or disagree with the following statement?
My health is good.
Strongly agree
Somewhat agree
Strongly disagree
Strongly disagree
4
3/17/2015
3
Agree-Disagree Questions
� Can include a midpoint (neither agree nor disagree)
� How much do you agree or disagree with the following statement?
My health is good.
Strongly agree
Somewhat agree
Neither agree nor disagree
Strongly disagree
Strongly disagree
5
Agree-Disagree Questions
� In self-administered surveys, often part of a scale or a set of items that uses the same response options.
Strongly
agree
Somewhat
agree
Somewhat
disagree
Strongly
disagree
People around here are willing
to help their neighbors.
People in this neighborhood
share the same values
This is a close-knit
neighborhood
6
3/17/2015
4
If you have used agree-disagree
survey questions…
� You are not alone.
� These questions are ubiquitous in survey research today…
7
American National Election Studies – 2012
Survey (In-Person and Web)
� Political efficacy
� 'Sometimes, politics and government seem so complicated that a person like me can't really understand what's going on.'
� Do you AGREE STRONGLY, AGREE SOMEWHAT, NEITHER AGREE NOR DISAGREE, DISAGREE SOMEWHAT, or DISAGREE STRONGLY with this statement?
8
3/17/2015
5
National Health and Nutrition Examination
Survey (In-Person)
� I am going to read a statement and I want you to let me know if you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree with the statement.
I enjoy participating in PE or gym class.
9
2013 Quinnipiac University Poll (Web
survey)
� "In general, do you agree or disagree with the 1973 Roe v. Wade Supreme Court decision that established a woman's right to an abortion?"
10
3/17/2015
6
Handbook of marketing scales (2010)
� Ten-Item and Five-Item Personality Inventories (Gosling, Rentfrow, and Swann 2003)
� Long-Term Orientation: LTO (Bearden, Money, and Nevins 2006)
� Maximization (Schwartz et al. 2002; Nenkov et al. 2008)
� Need for Cognition: NFC (Cacioppo and Petty 1982)
� Consumer’s Need for Uniqueness: CNFU (Tian, Bearden, and Hunter 2001)
� ….and many others
11
Why are Agree-Disagree Questions so
Popular?
� They’re easy to write.
� It’s easy to ask about several unrelated topics using the same response scale.
� They seem efficient
� Space in self-administered questionnaires
� Time in both self-administered and interviewer-administered surveys
� Sometimes one needs to re-use items and scales used in previous surveys.
12
3/17/2015
7
Issues with the use of A-D items
1. A-D Questions more of a cognitive burden for respondents.
2. Acquiescence response bias
3. Difficulty in interpreting the meaning of a “disagree” response
4. Respondents often have to think through double negatives
13
1. A-D items present more cognitive burden
� Cognitive necessary to answer a question carefully and completely (Tourangeau, Rips, and Rasinski, 2000)
� Understand the question
� Retrieve relevant information from memory
� Integrate the information into a judgment
� Map that judgment onto the response options provided
14
3/17/2015
8
Do you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree with the statement.
“I am usually happy.”
15
For example:
� A respondent answering this question has to:
� Understand that the question is asking about the number of times he or she experiences the emotion of happiness
� Determine the underlying dimension:
� How often do I feel happy (frequency)
� Retrieve relevant information
� I laugh a lot, everyone says I smile a lot…
� Form a judgment – I am usually happy.
� Mapping that judgment into the A-D format
� AD dimension different from that of the underlying question
� Mapping more difficult
16
For example:
3/17/2015
9
� Map that judgment into the A-D format requires extra step
� Take judgment = I am usually happy
� Compare to statement = I am usually happy
� Link judgment to A-D response option
� strongly agree, agree, neither agree nor disagree, disagree or strongly disagree
17
For example:
One goal in writing survey questions:
� To make the respondent’s cognitive task as simple as possible
� If the task of answering a survey question is difficult
� Some respondents will look for ways to make the task easier
� Either go through the cognitive steps necessary to answer a survey question incompletely or not at all � survey satisficing (Krosnick, 1991)
� One easy way to do that is to agree with statements made by the interviewer
18
3/17/2015
10
Problems with individual A-D items
1. A-D Questions more of a cognitive burden for respondents
2. Acquiescence response bias
3. Difficulty in interpreting the meaning of a “disagree” response
4. Respondents often have to think through double negatives
19
� Definition: Agreeing with assertion made regardless of content� Response option selected does not reflect ‘true’ underlying attitude
� Example: “I like candy” � Acquiescing respondents as likely to ‘agree’ as with, “I dislike candy” (e.g., Gendall & Hoek, 1998)
� ARB is more likely when� Ability is low: e.g., Cognitive skills: Education
� Motivation is low: e.g., Interest in the survey or question topic
� Task difficulty is high: e.g., Location of the question in the survey
20
2. Acquiescence Response Bias (ARB)
3/17/2015
11
Form A (agree I’s version):Individuals are more to blame than social conditions for crime and lawlessness in this country.
Form A (Agree I’s)Agree (I): 59.6%Disagree (SC): 40.4
100(473)
Form B (agree SCs version):Social conditions are more to blame than individuals for crime and lawlessness in this country.
2. Acquiescence Response Bias (ARB)
Form B (Agree SCs)Agree (SC): 56.8%Disagree (I): 43.2
100(472)
� Deference norms � greater ARB
� Cultures high in power distance
� Cultures low in individualism.
� Evidence that ARB is greater among
� Latino respondents (relative to non-Hispanic Whites)
� Respondents from collectivistic countries (e.g., east asiancountries) than those from individualistic countries (e.g., western countries)
� Individuals who score low on measures of individualism and high on measures of collectivism
22
Cultural Differences in ARB
3/17/2015
12
Example:
� Members of four racial/ethnic groups rated agreement with the following two statements:
� I tend to put off making important decisions until the last possible moment.
� I usually make important decisions quickly and confidently.
� Percent who agreed with both statements:
� Non-Hispanic White = 15%
� Non-Hispanic Black = 26%
� Mexican-American = 31%
� Korean American = 31%
23
Possible Results of ARB
� Failure to find relationships between variables that should be related (hidden by differences in ARB)
� Including actual differences between groups
� Finding spurious relationships between variables where there should be none (actually due to ARB)
� Including spurious appearance of systematic group differences
24
3/17/2015
13
Problems with individual A-D items
1. A-D Questions more of a cognitive burden for respondents
2. Acquiescence response bias
3. Difficulty in interpreting the meaning of a “disagree” response
4. Respondents often have to think through double negatives
25
3. Difficulty in Interpreting the
meaning of a “Disagree” Response
� Researchers often choose statements that are moderate
� “I am usually happy”
� Use of words like “some” “mostly” “often” “usually”
26
3/17/2015
14
� Consider the following survey question:
“Do you agree or disagree with the statement: My health is good?”
� Does a respondent who answers “disagree somewhat” think that:
� His or her health is poor
� His or her health is excellent
27
3. Difficulty in Interpreting the
meaning of a “Disagree” Response
Problems with individual A-D items
1. A-D Questions more of a cognitive burden for respondents
2. Acquiescence response bias
3. Difficulty in interpreting the meaning of a “disagree” response
4. Respondents often have to think through double negatives
28
3/17/2015
15
4. Respondents Often have to think through
double negatives.
� 'Public officials don't care much what people like me think.‘
Do you agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly with this statement?
29
4. Respondents Often have to think through
double negatives.
� Disagree = Public officials DO care
� Requires thinking through a double negative
� Extra cognitive step
� Double negatives are understood and used in other languages differently
30
3/17/2015
16
� Use item-specific response formats
� Decide what the underlying dimension of interest is
� Write a question using item-specific responses to measure that underlying dimension
� Questions with item-specific responses provide higher quality data than A-D items
� (Saris, Revilla, Krosnick, & Shaeffer, 2010 Survey Research Methods)
� Smaller number of higher quality items
31
So What To Do?
� 'Sometimes, politics and government seem so complicated that a person like me can't really understand what's going on.' Do you agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly with this statement?
� Decide what the underlying dimension of interest is:� How often politics and government are too complicated to understand
� How complicated politics and government are
� How well the R feels s/he understands politics and government
32
Example: 2012 ANES
3/17/2015
17
� How often do politics and government seem so complicated that you can't really understand what's going on? always, most of the time, about half the time, some of the time, or never?
33
Example: 2012 ANES
� “I enjoy participating in PE or gym class.”
� Underlying dimension is
� how much does the respondent like PE
34
Example: National Health and Nutrition
Examination Survey
3/17/2015
18
� Do you like or dislike PE or gym class?
� [IF LIKE]: Do you like it a lot or somewhat?
� [IF DISLIKE]: Do you dislike it a lot or somewhat?
OR
� How much of the time do you like PE or gym class? Always, most of the time, about half of the time, some of the time, or never?
35
Example: National Health and Nutrition
Examination Survey
� "In general, do you agree or disagree with the 1973 Roe v. Wade Supreme Court decision that established a woman's right to an abortion?"
� Underlying dimension is presumably opinions about legalized abortion
� Support for Supreme Court decision
� Trust in Government?
36
Example: 2013 Quinnipiac University Poll
3/17/2015
19
� Alternative item-specific question:
� Which of the following best represents your opinion about abortion: 1) abortion should not be legal under any circumstances, 2) abortion should be legal only in case of rape, incest, or when the woman’s life is in danger, 3) abortion should be legal for reasons other than rape, incest, or danger to the woman or 4) abortion should be legal under all circumstances.
37
Example: 2013 Quinnipiac University Poll
But, But what about…
� Comparing results to previous studies
� If data comparability is absolutely a must, use old items, but understand and try to estimate and control for potential error.
� Include ARB experiments or
� Include items to measure ARB to use as a covariate
� *NOT* ideal solution
� Doesn’t account for many of the problems with A-D questions (double negatives; uncertainty in interpreting disagree responses)
38
3/17/2015
20
But, But what about…
� Longitudinal data collection
� Include new and old questions as a means to transition.
39
And what about…
� Length of time (or amount of space) for revised items
� Study by Anand, Parsons, and Owens (2010)
� Respondents randomly assigned to receive item-specific response options or yes/no response options
� Did not significantly affect interview length
� Using item-specific items does not appear to substantially increase administration time
� Trade-off with data quality
� Minimizes needed sample size and items
40
3/17/2015
21
� Agree-disagree questions:
� Yes-no questions and True-False questions come under
this heading
� Avoid!!!!!
� Recommended alternative: questions with item-specific
response options
41
Final conclusions:
Final conclusions:
� Particularly problematic:
� Culturally diverse populations
� Group differences
� Circumstances or individuals where ARB is most likely
� Negative statements
� Statements that use moderate qualifiers (e.g., most, usually,
sometimes)
42
3/17/2015
22
Final conclusions:
� Very limited set of circumstances where you may consider using A-D items:
� Direct comparison with previous data in which they were
used
� Longitudinal data collections
� Even in these cases, there may be ways to move away from
A-D items.
43
Thank You!
Questions?
44
3/17/2015
23
References:
� Anand, S., Owens, L. K., & Parsons, J. A. (2010). Forced-choice vs. yes-no questions: Data quality and administrative effort. Paper presented at the annual conference of the World Association for Public Opinion Research. Chicago, IL.
� Bearden, W. O., Netemeyer, R. G., & Haws, K. L. (2010). Handbook of Marketing Scales: Multi-Item Measures for Marketing and Consumer Behavior Research. Thousand Oaks, CA: Sage Publications.
� Gendall, P., & Hoek, J. (1990). A question of wording. Marketing Bulletin, 1, 25-36.
� Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (205). The relation between culture and response styles. Journal of Cross-Cultural Psychology, 36, 264-277.
� Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236.
� Saris, W., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disgree response options to questions with item-specific response options. Survey Research Methods, 4, 61-79.
� Schuman, H., & Presser, S. (1981). Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York, NY: Academic Press.
� Smith, P. B. (2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35, 50-61.
� Tourangeau, R., Rips, L., & Rasinski, K. (2000). The Psychology of Survey Response. New York, NY: Cambridge University Press.