Research Methods

© 2012 by W. W. Norton & Company

Chapter 6

Tools for Interrogating Frequency Claims:Surveys and Polls

Detailed Learning Objectives: Question Wording and Observations

1. Give examples of how question wording can change the results of a survey or poll. 2. Describe the different ways questions can be worded: open-ended, forced- choice, and using rating scales. 3. Explain how to increase the construct validity of questions by wording them carefully and by avoiding leading questions, double-barreled questions, and double negatives. 4. Explain how question order can change the meaning (and validity) of a question.5. Explain ways to increase the construct validity of questions by preventing respondent shortcuts (such as nay-saying), biases (such as trying to look good), or simple inability to report. 6. Describe how observational techniques for measurement are different from survey techniques. 7. Explain ways to improve the construct validity of observations by reducing observer bias and target reactivity.

Detailed Learning Objectives: Sampling

8. Indicate the population that a given sample might represent. 9. Define three sampling problems that lead to biased samples. 10. Understand why a random sample is more likely to be a representative sample and why representative samples have external validity to a particular population. 11. Explain five techniques for random sampling: simple random, multistage, cluster, stratified random sampling, and oversampling. 12. Describe three techniques of nonrandom sampling: purposive, convenience, and snowball sampling. 13. Explain why representative samples may be especially important for many frequency claims. 14. Consider times when an unrepresentative sample may be appropriate for a frequency claim. 15. Explain why it is more important to ask how a sample was collected rather than how large the sample is.

1. A response set can be in the form ofa) Open ended questionsb) Leading questionsc) Yay-saying answersd) A Likert scale

2. A survey question that has you rate a response from agree to disagree is an example of

a) A Likert scaleb) A double barreled questionc) The semantic differential format

3. Which of the following is NOT a way to prevent or avoid observer biasa) Masked study designb) Inclusion of filler questionsc) Measuring traces that the behavior leaves behindd) Make unobtrusive observations

4. Choose the following that are correctly matched (can be more than one)a) Random sampling—external validityb) Random assignment—internal validityc) Random sampling—internal validityd) Random assignment—external validity

Question Wording

Construct Validity of Surveys and Polls

• Choosing question formats• Writing well-worded questions• Encouraging accurate responses

Choosing Question Formats

Open-ended questionsForced-choice formatLikert scaleSemantic differential format (example shown)

Writing Well-Worded Questions

• Leading questions• Double-barreled questions• Double negatives (vs. negatively

worded items)• Question order

How can you write items that avoid these problems?

Encouraging Accurate Responses

• Response sets (three types)– Yea-saying/ nay-saying– Fence sitting

• Faking good or bad• Saying more than we can know• Measuring subjectivity vs.

objectivity

What are some solutions to these problems?

Behavioral Observations

Construct Validity of Behavioral Observations

• Examples of frequency claims based on observational data

• Are observations better than self-reports?• Good observations are both reliable and valid

Some Examples of Frequency Claims Based on Observational Data

Are Observations Better than Self-Reports?

Observers may see what they expect to seeObservers can affect what they see

Masked research designs

Observer effectsUnobtrusive observationsWait it outMeasure results of a behavior

Observe ethically

What are some solutions to these problems?

Figure 6.6 Unobtrusive observations This one-way mirror allows researchers to unobtrusively record the behaviors of children in a preschool classroom.

Good Observations Are Both Reliable and Valid

Good Observations Are Both Reliable and Valid

Figure 6.8 Table 1 from Campos and colleagues’ (2009) study of family interactions. The table depicts the degree of interrater reliability for each of the behaviors coded.

Sampling

Generalizing to Others: Sampling Participants

• Populations and samples• How to get a representative sample• When does external validity matter?• When a representative sample is not the top

priority• Are bigger samples better samples?

www.everydayresearchmethods.com

Key Questions

a. Read the first three paragraphs carefully. What do you think of the sampling techniques used in this poll? To what population can we generalize the results from this poll? b. Your friend comments, “wow! 375,000 people responded to this survey! We can definitely generalize this to the American population.” What do you say?

Populations and Samples

Population is ?Sample is ?

Representative Samples

Probability samplingSimple random samplingCluster sampling

Multistage sampling

Stratified random samplingOversamplingSystematic sampling

Unrepresentative Sampling

Self-selectedPurposiveConvenience

Homework

• What do you think of the construct validity of the polls your group brought in?

• What do you think about the external validity of the polls your group brought in?

Working with your own examples

More PracticeImagine that you are planning to estimate the price of the average book at the UD college bookstore. The bookstore carries 13,000 titles, but you plan to sample only 200 books. You will select a sample of 200 books, record the price of each book, and use the average of the 200 books to estimate the average price of the 13,000 books in the bookstore.

Based on this information, answer the following questions:a. What is the sample in this study, and what is the population?b. How might you collect a simple random sample of books?c. How might you collect a stratified random sample?d. How might you collect a convenience sample?e. How might you collect a systematic random sample?f. How might you collect a cluster sample of 200 books?