Asking People About Themselves: Survey Research …...124 Chapter 7 • Asking People About Themselves: Survey Research desirability can be a problem in many research áreas, but it

Cozby, P. (2008). Meihods in Behavioral Research. Boston : McGraw-Hill Higher Education.

Asking People About Themselves: Survey Research

LEARNING OBJECTIVES a Discuss reasons for conduct ing survey research. m Ident i fy factors to consider when w r i t i n g questions for interviews and questionnaires,

i n c l u d i n g def ining research objectives and question w o r d i n g . m Describe different ways to construct questionnaire responses, i n c l u d i n g closed-ended

questions, open-ended questions, and rat ing scales. • Compare the two ways to administer surveys: w r i t t e n questionnaires and oral interviews. • Define interviewer bias. a Describe a panel study.

m Dis t inguish between probabi l i ty and nonprobabiHty sampling techniques. m Describe simple random sampling, stratified r a n d o m sampling., and cluster sampling. • Describe haphazard sampling^ purposive sampling, and quota sampling. i Describe the ways that samples are evaluated for potent ia l bias, i n c l u d i n g sampling frame

and response rate.

Survey research employs questionnaires and interviews to ask people to provide i n f o r m a t i o n about themselves—their att itudes and beliefs, demo-graphics (age, gender, income, m ar i ta l status, and so on) and other facts,

and past or intended future behaviors. I n this chapter we w i l l explore methods o f designing and conduct ing surveys, i n c l u d i n g sampl ing techniques.

W H Y C O N D U C T SURVEYS?

A m u l t i t u d e o f surveys are being conducted all the t ime. Just l o o k at your daily newspaper. The Centers for Disease C o n t r o l and Prevention is r e p o r t i n g results o f a survey o f new mothers asking about breast feeding. A college survey center is repor t ing the results o f a telephone survey asking about po l i t i ca l attitudes. I f y o u look around your campus, y o u w i l l find academic departments conduct ing surveys o f seniors or recent graduates. I f y o u make a major purchase, y o u w i l l l ikely receive a request to complete a survey that asks about your satisfaction. I recently visited the American Psychological Association Web site and read a re-p o r t called Stress in America that presented the results o f an Internet survey o f over 1,800 adults that was conducted i n 2007. Surveys are clearly a c o m m o n and i m p o r t a n t m e t h o d o f s tudying behavior.

Surveys provide us w i t h a methodology for asking people to tel l us about themselves. They have become extremely i m p o r t a n t as society demands data about issues rather than only i n t u i t i o n and anecdotes. M y department needs data f r o m graduates to help determine changes that s h o u l d be made to the cur­riculum. A u t o companies want data f r o m buyers to assess and improve product qual i ty and customer satisfaction. W i t h o u t collecting such data, we are total ly dependent u p o n stories we m i g h t hear or letters that a gradúate or customer m i g h t write . Other surveys can be i m p o r t a n t for m a k i n g publ ic pol icy decisions by lawmakers and publ ic agencies. I n basic research, many i m p o r t a n t variables, i n c l u d i n g attitudes, current emot ional states, and self-reports o f behaviors, are most easily studied using questionnaires or interviews.

We often t h i n k o f survey data p r o v i d i n g a "snapshot" o f h o w people t h i n k and behave at a given p o i n t i n t ime. However, the survey m e t h o d is also an i m ­p o r t a n t way for researchers to study relationships amo ng variables and ways that attitudes and behaviors change over t ime. For example, Steinberg and Dornbusch (1991) examined the relationship between the n u m b e r o f hours that h i g h school students w o r k and variables such as grade p o i n t average, d r u g and alcohol use, and psychosomatic distress. The sample consisted o f 3,989 students i n grades 10-12 at nine h igh schools i n Cal i fornia and Wisconsin. The re­searchers f o u n d that " l o n g w o r k hours d u r i n g the school year are associated w i t h lower investment and performance i n school, greater psychological and so-matic distress, d r u g and alcohol use, delinquency, and a u t o n o m y f r o m parents" (Steinberg & Dornbusch, 1991, p. 304). Figure 7.1 shows a typical finding: There are some positive aspects o f w o r k i n g fewer t han 10 hours per week (as opposed

Why Conduct Surveys? 123

Not Employed

1-10 11-15

Work Hours

16-20 21 +

FIGURE 7.1 Relationship between hours of work and grade point average Source: "Negative Correlates o f Part-cime Employment D u r i n g Adolescence," by L. Steinberg and S. M . Dornbusch, 1991, Developmental Psychology, 27, pp. 303-313. Copyright © 1991 by the American Psychological Association. Reprinted by permission.

to n o t being employed); however, increasingly negative effects are associated w i t h longer w o r k hours.

Survey research is also i m p o r t a n t as a complement to experimental research findings. Recall f r o m Chapter 2 tha^Winograd-and Soloway (1986) conducted experiments o n the condit ions that lead to forget t ing where we place some-t h i n g . To study this topic using survey methods, B r o w n and Rahhal (1994) asked b o t h younger and older adults about their actual experiences when they h i d something and later forgot its locat ion. They reported that older adults take longer t h a n younger adults to find the object and that older adults hide objects f r o m potent ia l thieves, whereas younger people hide things f r o m friends and relatives. Interestingly, most lost objects are eventually f o u n d , usu-ally by accident i n a locat ion that had been searched previously. This research il lustrates a p o i n t made i n previous chapters tha t múltiple methods are needed to understand any behavior.

A n assumption that underlies the use o f questionnaires and interviews is that people are w i l l i n g and able to provide t r u t h f u l and accurate answers. Researchers have addressed this issue by s tudying possible biases i n the way peo­ple respond. A response set is a tendency to respond to all questions f r o m a par­t icular perspective rather t h a n to provide answers that are directly related to the questions. Thus, response sets can affect the usefulness o f data obtained f r o m self-reports. The most c o m m o n response set is called social desirability, or " f a k i n g good." The social desirability response set leads the i n d i v i d u a l to answer i n the most socially acceptable way—the way that "most people" are perceived to respond or the way that w o u l d reflect most favorably o n the person. Social

124 Chapter 7 • Asking People About Themselves: Survey Research

desirability can be a prob lem i n many research áreas, b u t i t is probably most acute when the question concerns a sensitive topic such as violent or aggressive behavior, substance abuse, or sexual practices. However, i t s h o u l d n o t be as-sumed that people consistently misrepresent themselves. Jourard (1969) sug-gested that people are most l ikely to lie when they don ' t t rust the researcher. I f the researcher openly and honestly communicates the purposes and uses o f the research, promises to provide feedback about the results, and assures confi-dentiality, then the participants can reasonably be expected to give honest responses.

We t u r n n o w to the major considerations i n survey research: construct ing the questions that are asked, choosing the methods for presenting the questions, and sampl ing the individuáis tak ing part i n the research.

C O N S T R U C T I N G QUESTIONS T O ASK

A great deal o f t h o u g h t m u s t be given to w r i t i n g questions for questionnaires and interviews. This section describes some o f the most i m p o r t a n t factors to consider when construct ing questions.

Defining the Research Objectives When construct ing questions for a survey, the first t h i n g the researcher m u s t do is explicitly determine the research objectives: W h a t is i t that he or she wishes to know? The survey questions must be t ied to the research questions that are being addressed. Too often, surveys get out o f h a n d when researchers begin to ask any question that comes to m i n d about a topic w i t h o u t considering exactly what useful i n f o r m a t i o n w i l l be gained by do ing so. This process w i l l usually require the researcher to decide o n the type o f questions to ask. As noted previ-ously, there are three general types o f survey questions (Judd, Smi t h , & Kidder, 1991).

Attitudes and Beliefs Questions about attitudes and beliefs focus o n the ways that people evalúate and t h i n k about issues. Should more money be spent o n menta l health services? Are y o u satisfied w i t h the way tha t pólice responded to your cali? H o w do y o u evalúate this instructor?

FactS and DemographlCS Factual questions ask people to indícate things they k n o w about themselves and their s i tuat ion. I n most studies, asking some demographic i n f o r m a t i o n is necessary to adequately describe your sample. Age and gender are typically asked. Depending on the topic o f the study, ques­tions o n such i n f o r m a t i o n as ethnicity, income, m ar i ta l status, employment sta­tus, and number o f chi ldren m i g h t be included. Obviously, i f y o u are interested

Constructing Questions to Ask 125

i n m a k i n g comparisons among groups, such as males and females, you m u s t ask the relevant i n f o r m a t i o n about group membership. I t is unwise to ask such questions i f y o u have no real reason to use the i n f o r m a t i o n , however.

Other factual i n f o r m a t i o n y o u m i g h t ask w i l l depend o n the topic o f your survey. Each year, Consumer Reports magazine asks me to tel l t h e m about the re-pairs that have been necessary o n many o f the products that I own, such as m y car and dishwasher. Factual questions about illnesses and other medical in for ­m a t i o n w o u l d be asked i n a survey o f health and qual i ty o f life.

Behaviors Other survey questions can focus o n past behaviors or intended future behaviors. H o w many times last week d i d y o u exercise for 20 minutes or longer? H o w many chi ldren do y o u p lan to have? Have you ever been so depressed that y o u called i n sick to work?

Question Wording A great deal o f care is necessary to wri te the very best questions for a survey. Cog-nitive psychologists have ident i f ied a number o f potent ia l problems w i t h ques­t i o n w o r d i n g (see Graesser, Kennedy, Wiemer-Hastings, & O t t a t i , 1999). M a n y o f the problems stem f r o m a d i f f i cu l ty w i t h unders tanding the question, i n c l u d i n g (a) unfami l ia r technical terms, (b) vague or imprecise terms, (c) ungrammat ica l sentence structure, (d) phrasing that overloads w o r k i n g memory, and (e) embed-d i n g the question w i t h misleading i n f o r m a t i o n . Here is a question that i l lus-trates some o f the problems ident i f ied by Graesser et al.:

D i d your mother, father, full-blooded sisters, full-blooded brothers, daughters, or sons ever have a heart attack or myocardial infarction?

There is m e m o r y overload because o f the length o f the question and the need to keep track o f al l those relatives whi le reading the question, and the re-spondent m u s t w o r r y about two different diagnoses w i t h regard to each relative. Further, the t e r m myocardial infarction may be u n f a m i l i a r to most people. H o w do y o u wri te questions to avoid such problems? The f o l l o w i n g Ítems are i m p o r t a n t to consider when you are w r i t i n g questions.

Simplicity The questions asked i n a survey should be relatively simple. People should be able to easily understand and respond to the questions. A v o i d jargon and technical terms that people w o n ' t understand. Sometimes, however, y o u have to make the question a b i t more complex to make i t easier to under­stand. Usually this occurs when y o u need to define a term or describe an issue pr ior to asking the question. Thus, before asking whether someone approves o f Proposi t ion J, you w i l l probably want to provide a br ie f description o f the content o f this bal lot measure.

126 Chapter 7 • Asking People About Themselves: Survey Research

Double-Barreled Questions Avoid "double-barreled" questions that ask two things at once. A question such as "Should sénior citizens be given more money for recreation centers and f o o d assistance programs?" is d i f f i c u l t to answer because i t taps two potential ly very different attitudes. I f y o u are inter-ested i n b o t h issues, ask two questions.

Loaded Questions A loaded question is w r i t t e n to lead people to re­spond i n one way. For example, the questions " D o y o u favor e l i m i n a t i n g the wasteful excesses i n the publ ic school budget?" and " D o you favor reducing the publ ic school budget?" w i l l l ikely elicit different answers. Or consider that m e n are less l ikely to say they have "raped" someone t h a n that they have "forced sex"; similarly, women are less l ikely to say they have been raped t h a n forced to have unwanted sex (Koss, 1992). Questions that include emotional ly charged words such as rape, waste, immoral, ungodly, or dangerous may influence the way that people respond and thus lead to biased conclusions.

Negative WÓrding Avoid phrasing questions w i t h negatives. This ques­t i o n is phrased negatively: " D o you feel that the city should n o t approve the pro-posed women's shelter?" Agreement w i t h this question means disagreement w i t h the proposal. This phrasing can confuse people and result i n inaccurate answers. A better f o r m a t w o u l d be: " D o you believe that the city s h o u l d approve the proposed women's shelter?"

"Yea-Sayíng" and "Nay-Saying" W h e n y o u ask several questions about a topic, there is a possibil i ty that a respondent w i l l employ a response set to agree or disagree w i t h al l the questions. Such a tendency is referred to as "yea-saying*' or "nay-saying." The problem here is that the respondent may i n fact be expressing true agreement, b u t alternatively may s imply be agreeing w i t h anything y o u say. One way to detect this response set is to w o r d the questions so that consistent agreement is unlikely. For example, a study o f fami ly c o m m u m -cation patterns m i g h t ask people how m u c h they agree w i t h the f o l l o w i n g statements: "The members o f my fami ly spend a l o t o f t ime together" and " I spend most o f m y weekends w i t h friends." Similarly, a measure o f loneliness (e.g., Russell, Peplau, & Cutrona, 1980) w i l l phrase some questions so that agree­ment means the respondent is lonely ( " I feel isolated f r o m others") and others w i t h the meaning reversed so that disagreement indicates loneliness (e.g., " I feel par t o f a group o f friends") . A l t h o u g h i t is possible that someone could legit i -mately agree w i t h b o t h items, consistently agreeing or disagreeing w i t h a set o f related questions phrased i n b o t h standard and reversed formats is an indicator that the i n d i v i d u a l is "yea-saying" or "nay-saying."

Graesser and his colleagues have developed a computer p r o g r a m called QUAID (Question Understanding Aid) that analyzes question w o r d i n g . Researchers can try o u t their questions online at the Q U A I D Web site (http:// mnemosyne.csl.psyc.memphis.edu/quaid). You should also review the question w o r d i n g examples i n Table 7.1.

Responses to Questions 127

TABLE 7.1 Question wording: W h a t is the problem?

Read each of the foi lowing questions and identify the problems for each.

Negative wording Simplicity

Double-barreled Loaded

Professors should not be requíred to take daily attendance.

1 = (Strongly Disagree) and 5 = (Strongly Agree)

I enjoy studying and spending time w i t h friends o n weekends.

Do you support the legislation that would unfairly tax hard-working farmers?

I would describe myself as attractive and intelligent.

Do you believe the relationship between cell phone behavior and consumption of fast food is orthogonal?

Restaurants should not have to be inspected each month .

Are you i n favor o f the boss's w h i m to cut lunchtime to 30 minutes?

Answers are provided at the end o f the chapter.

RESPONSES T O QUESTIONS Closed- Versus Open-Ended Questions Questions may be either closed- or open-ended. W i t h closed-ended questions, a l i m i t e d number o f response alternatives are given; w i t h open-ended questions, respondents are free to answer i n any way they like. Thus, you could ask a person: " W h a t is the most i m p o r t a n t t h i n g chi ldren should learn to prepare t h e m for life?" fo l lowed by a l ist o f answers f r o m w h i c h to choose (a closed-ended question) or y o u could leave this question open-ended for the person to provide the answer.

Using closed-ended questions is a more s tructured approach; they are easier to code and the response alternatives are the same for everyone. Open-ended questions require t i m e to categorize and code the responses and are therefore more costly. Sometimes a respóndenos response cannot be categotized at al l be-cause the response doesn't make sense or the person couldn ' t t h i n k o f an answer.

128 Chapter 7 • Asking People About Themselves: Survey Research

Sti l l , an open-ended question can yield valuable insighcs i n t o what people are t h i n k i n g . Open-ended questions are most useful when the researcher needs to k n o w what people are t h i n k i n g and h ow they natural ly view their w o r l d ; closed-ended questions are more l ikely to be used when the dimensions o f the variables are well defined.

Schwarz (1999) points o u t that the two approaches can sometimes lead to different conclusions. He cites the results o f a survey question about prepanng children for l ife. W h e n "To t h i n k for themselves" was one alternative i n a" closed-ended list, 62% chose this o p t i o n ; however, only 5% gave this answer w h e n the open-ended formar was used. This finding points to the need to have a good understandmg o f the topic when asking closed-ended questions.

Number of Response Alternatives W i t h closed-ended questions, there are a fixed number o f response alternatives. I n public opinión surveys, a simple "yes or n o " or "agree or disagree" d i c h o t o m y is often sufficient. I n more basic research, i t is of ten preferable to have a suff i -cient number o f alternatives to allow people to express themselves—for example, a 5- or 7-point scale ranging f r o m "strongly agree to strongly disagree" or "very positive to very negative." Such a scale m i g h t appear as follows:

Strongly agree , , Strongly disagree

Rating S cales Rating scales such as the one jus t shown are very c o m m o n i n many áreas o f research. Rat ing scales ask people to provide " h o w m u c h " judgments o n any number o f dimensions—amount o f agreement, l i k i n g , or confidence, for example. Rat ing scales can have many different formats. The f o r m a t that is used depends o n factors such as the topic being investigated. Perhaps the best way to gain an unders tanding o f the variety o f formats is s imply to l o o k at a few exam-ples. The simplest and most direct scale presents people w i t h five or seven response alternatives w i t h the endpoints o n the scale labeled to define the extremes. For example,

Students at the university should be'required to pass a comprehensive examination to gradúate.

Strongly agtee Strongly disagree

H o w confident are y o u that the defendant is gui l ty o f a t tempted murder?

N o t at all conf ident Very confident

Responses to Questions 129

Graphic Ratíng Scale A graphic ra t ing scale requires a m a r k along a con-t inuous 100-millimeter l ine that is anchored w i t h descriptions at each end.

H o w w o u l d y o u rate the movie you jus t saw?

N o t very enjoyable Very enjoyable

A ruler is then placed o n the line to obta in the score on a scale that tanges f r o m 0 to 100.

Semantic DífFerential Scale The semantic dif ferential scale is a mea-sure o f the meaning o f concepts that was developed by Osgood a n d his associates (Osgood, Suci, & Tannenbaum, 1957). Respondents rate any concept—persons, objects, behaviors, ideas—on a series o f b ipolar adjectives using 7-point scales.

Smoking cigarettes

Good .

Strong

Active

Bad

VVeak

Passive

Research o n the semantic dif ferential shows that v i r tua l ly anything can be meas-ured using this technique. Ratings o f specific things (marijuana), places (the stu-dent center), people (the governor, accountants), ideas (abortion, tax reduct ion) , and behaviors (attending church, using publ ic transit) can be obtained. A large body o f research shows that the concepts are rated along three basic dimensions: the first and most i m p o r t a n t is evaluation (e.g., adjectives such as good-bad, wise-foolish, k ind-crue l ) ; the second is activity (active-passive, slow-fast, excitable-calm); and the t h i r d is potency (weak-strong, hard-soft , large-small) .

Nonverbal Scale for Chíldren Young chi ldren may n o t understand the types o f scales weVe just described, b u t they are able to give tatings. For example, you could ask chi ldren to "Point to the face that shows h o w y o u feel about the toy."

Labeling Response Alternatives The examples thus far have labeled only the endpoints on the r a t i n g scale. Respondents decide the meaning o f the other response alternatives. This is a

130 Chapter 7 • Asking People About Themselves: Survey Research

reasonable approach, and people are usually able to use such scales w i t h o u t di f f icul ty . Sometimes researchers need to provide labels to more clearly define the meaning o f each alternative. Here is a fair ly standard alternative to the agree-disagree scale shown previously:

Strongly Agree Undecided Disagree Strongly agree disagree

This type o f scale assumes that the middle alternative is a " n e u t r a l " p o i n t halfway between the endpoints. Sometimes, however, a perfectly balanced scale may n o t be possible or desirable. Consider a scale asking a college professor to rate a student for a job or gradúate program. This part icular scale asks for com-parative ratings o f students:

I n comparison w i t h other graduates, h o w w o u l d y o u rate this student's potent ia l for success?

Lower Upper Upper Upper Upper 5 0 % 50% 25% 10% 5%

Notice that most o f the alternatives are asking people to make a ra t ing i n terms o f the top 25% o f students. This is done because students w h o apply for such programs tend to be very b r i g h t and motivated, and so professors rate them favorably. The w o r d i n g o f the alternatives attempts to forcé the raters to make finer dist inctions among genetally very good students.

Labeling alternatives is part icular ly interesting when asking about the fre-quency o f a behaviot. For example, y o u m i g h t ask, " H o w of ten do y o u exercise for at least 20 minutes?" What k i n d o f scale should y o u use to let people answer this question? You could list (1) never, (2) rarely, (3) sometimes, (4) frequently. These terms convey your meaning b u t they are vague. Here is another set o f alternatives, s imilar to ones described by Schwarz (1999):

less than twice a week

about twice a week

about f o u r times a week

about six times a week

at least once each day

A different scale m i g h t be:

less than once per m o n t h

about once a m o n t h

Finaüzing the Questionnaire 131

about once every two weeks

_____ about once a week

more t h a n once per week

Schwarz (1999) calis the first scale a high-frequency scale because m o s t alternatives indícate a h i g h frequency o f exercise. The other scale is referred to as l o w frequency. Schwarz points o u t tha t the labels s h o u l d be chosen carefully be-cause people may interpret the meaning o f the scale differently, depending o n the labels used. I f y o u were actually asking the exercise question, y o u m i g h t de­cide o n alternatives di f ferent f r o m the ones described here. Moreover, your choice s h o u l d be inf luenced by factors such as the p o p u l a t i o n y o u are s tudying . I f y o u are s tudyin g people w h o generally exercise a lo t , y o u w i l l be more l ike ly to use a higher-frequency scale t h a n y o u w o u l d i f y o u were s tudying people w h o generally d o n ' t exercise a great deal.

FINAÜZING T H E Q U E S T I O N N A I R E Formatting the Questionnaire The p r i n t e d questionnaire s h o u l d appear attractive a n d professional. I t s h o u l d be neatly typed and free o f spel l ing errors. Respondents s h o u l d find i t easy to iden-t i f y the questions and the response alternatives to the questions. Leave enough space between questions so people don ' t become confused when reading the questionnaire. I f y o u have a part icular scale formar , such as a 5 -point r a t i n g scale, use i t consistently. D o n ' t change f r o m 5- to 4- to 7-point scales, for example.

I t is also a good idea to carefully consider the sequence i n w h i c h y o u w i l l ask your questions. I n general, i t is best to ask the most interest ing and i m p o r t a n t questions first to capture the a t t e n t i o n o f your respondents and motivate t h e m to complete the survey. Roberson and S u n d s t r o m (1990) obtained the highest r e t u r n rates i n an employee a t t i tude survey w h e n i m p o r t a n t questions were pre-sented first a n d demographic questions were asked last. I n addi t ion , i t is a g o o d idea to group questions together w h e n they address a s imilar theme or topic . D o i n g so w i l l make your survey appear more professional, and your respondents w i l l be more l ikely to take i t seriously.

Refining Questions Before actually administer ing the survey, i t is a good idea to give the questions to a small g t o u p o f people and have t h e m " t h i n k a l o u d " while answering them. The par-ticipants m i g h t be chosen f r o m the p o p u l a t i o n being studied, or they could be friends or colleagues w h o can give reasonable responses to the questions. For the " t h i n k a l o u d " procedure, y o u w i l l need to ask the individuáis to tel l y o u h o w they interpret each question and h o w they respond to the response alternatives. This pro­cedure can provide valuable i n f o r m a t i o n that y o u can use to improve the questions. (The importance o f p i l o t studies such as this is discussed further i n Chapter 9.)