CTSC Topics in Managing Data Seminar Seriescci.case.edu › cci › images › 1 › 15 › Q_Design_Layout_2010-03-17.pdf · 2014-02-03 · Topics in Managing Data: Questionnaire

CTSC Topics in Managing Data Seminar Series

Questionnaire Design: Layout & Administration

Thank you for attending today’s seminar!

Please make sure to sign the registration sheet

Please complete an evaluation form and turn it in at the end of the seminar

Next Seminar: Wednesday, April 14, 2010, noon to 1 pm, Wolstein 6136

Topic: Data Cleaning Part 1: Ranges and Distributions of Numeric Variables

Questionnaire Design: Layout and Administration

Topics in Managing Data: A CTSC Seminar Series for Research Personnel

Sue Caban [email protected], Data Management CoreCenter for Clinical Investigation March 17, 2010

mailto:[email protected]

3

Topics in Managing Data: Questionnaire Design: Layout & Administration

During This Seminar:

• Think of yourself and the role you play in your research group

• Think of one or more of your research projects / studies that you support

• Think of ways that you work with questionnaires, and how this information applies to you

• Please ask questions!

4


Overview: Today’s Material

1) Layout

2) Administrationa) Methodsb) Phone Scriptc) Pretestingd) Improving Response Ratese) Translation

5


Layout: Tips to Improve Readability• Visually pleasing to read• Appropriate font size for the intended audience• Include sufficient “white space” on page / screen• Clear directions on filling out form or answering

questions on web site• Directions and section headings should be given in a

different type face, e.g., ALL CAPITAL LETTERS, bold, italics

• Number EVERY question• Keep numbering simple – keep sublevels to a minimum• Make sure page breaks do not occur in the middle of a

question

6


Layout: Tips to Improve Readability• Have one column of questions down the left side of the

page• Avoid having multiple columns of questions across the

page• Avoid printing two-sided forms• Use alternating grey and white rectangles behind items

when there are many items• Use bold outline boxes to group items into sections• Clear skip patterns• Use arrows to show skips if possible• Proofread your document!

7


Layout: Before

8


Layout: After

9


Layout: Question Order

• Introduction• Respondent Selection (Screener

Questions)• Substantive Questions• Background Questions (Demographics)

10


Layout: Question Order

• Logical• Organize into Sections• Place “Easy” Questions at Beginning• Place Most Sensitive Questions at the End• Remind Participant about Confidentiality

11


Layout: First Questions

• Most Relevant to the Research• Easy to Answer, to Ease Participant into

the Interview• Closed-Ended Format• Non-Personal, Non-Threatening Questions• Interesting Questions to Capture Attention• Applicable and Answerable by Most

Respondents

12


Layout: Screener Question• May need a “Screener” Question in order

to Determine Eligibility• Word Screener Question Carefully

“How many children under the age of 18 live in this household?”

“Of the people who live in this household, how many are adults age 18 and over, and how many are children under age 18?”

13


Layout: Post-Interview Questions• Interviewer Perceptions – Ratings of:

– Participant’s Understanding of Questions– Participant’s Quality of Responses to

Questions– Participant’s Honesty in Answering Questions

14


Administration: Methods• Self-Administered

– Mail– Internet– In-Person with Study Personnel Nearby

• Administered by Interviewer– In-Person– Phone– Observational

• Anonymous versus Confidential– Anonymous surveys often used when questions are

about very sensitive topics or illegal behaviors

15


Administration: Methods• Administered by Interviewer

– Interview-based Survey: Pre-set Closed Q’s & Order– Fully Structured: Pre-set Open Q’s & Order– Semi-Structured: Question Order can be Modified– Unstructured: Informal, Conversational

16


Administration: Tips on Rapport• Listen more than you speak• Ask questions in a clear and non-threatening

manner• Do not give cues as to the “correct” answer• Vary voice and facial expressions to indicate

interest• Probes to get participants to expand an answer:

– Silence– Inquiring glance– Repeat all or a portion of respondent’s answer

17


Administration: Completing Forms• Use blue or black ink• Do not use red ink• Neatness counts!• Make changes using a single line drawn through the

original response, mark new response, initial, and date• Be careful to complete date questions using format

indicated• Be careful to select only the number of responses

permitted (only one, all that apply)• On scannable forms (TeleForms) be careful to mark

answer in area provided• Be careful to follow indicated skip patterns

18


Administration: Coding Open-Ended Q’sPurpose:• Simplify many individual responses that are similar in

content in a consistent manner to permit statistical analysis of repsonses

Process:• Audiotape interview to capture verbatim wording of

responses• Transcribe responses verbatim• Combine details of verbatim responses into a limited

number of response categories (codes) after a sizable number of forms have been completed (~50 or more)

• Create a Q-by-Q Manual for reference when coding

19


Administration:• Pretest your instrument with a few people

outside your research field (spouses, friends)• Focus on clarity of questions, length of

questionnaire• Ask yourself the purpose of every question;

Minimize participant burden by keeping questionnaire length short

• Interviewer Debriefing: Examine all questions for awkwardness, logic in skip patterns, that read differently out loud than intended

20


Administration:

• Post-Interviews with Respondents:– Open-ended questions to learn perceptions of

respondents, such as– “Were there any questions that you were not

sure how to answer? Is so, which ones? Why were you not sure how to answer?”

– “Were there any things related to these issues that we did not cover?”

• Pilot test the revised instrument with intended target audience

21


Administration:

• Mail a cover letter to announce upcoming phone call interview

• Avoid a December mailing of cover letter and mailed survey

• Mailing a follow-up letter for non- respondents will increase response rate; Include another survey and postage paid return envelope

22


Cover Letter Should Include Answers To:• What is the study about?• Why is the study important?• How will the study results be used?• Why is the study pertinent to me?• How was I selected?• Will my responses be confidential?• Who do I contact if I have questions about the

study?• What is the contact person’s phone number?

23


Interview Phone Call Script: Exercise

Close your eyes and listen to the first script

24


Interview Phone Call ScriptHello, my name is _____ . I’m calling from the University of

Maryland Survey Research Center. The Survey Research Center is conducting a survey for the governor’s 1991 commission on violent street crime. The survey results will be used to help plan programs to reduce crime through prevention, education programs for young people, and better law enforcement. Your household was chosen through a random-selection process. Your participation is completely voluntary but is very important to the representatives of the survey. All your answers will be kept in strictest confidence.

Designing Surveys, pp. 79-80

25


Interview Phone Call Script: RevisedHello, I’m calling from the University of Maryland. My name

is _____ . We are doing a study about crime for the State of Maryland’s Summit on Violent Street Crime. The results will be used to help plan ways to reduce crime. Your household was chosen through a random- selection process. Your participation is completely voluntary. All your answers will be kept in strictest confidence. For this study, I need to speak with the adult in your household who is 18 or older and will have the NEXT birthday. Who would that be?

Designing Surveys, p. 80

26


Interview Phone Call Script: FinalHello, I’m calling from the University of Maryland.

My name is _____ . We are doing a study about the State of Maryland’s Summit on Violent Street Crime. I need to speak with the adult in your household, who is 18 or older, and will have the next birthday. Who would that be?


27


Tactics to Convert Reluctant Participants:• Keep the conversation going

• Identify reason for reluctance

• Try to address reason for reluctance

28


Tactics to Convert Reluctant Participants:Bad Timing:• Don’t push for the interview• Show understanding of the bad timing• Use judgment in asking for a callback time

Suspicion of a Sales Call:• Repeat sponsor and purpose of study• Give 800 number or ask respondent to call

collect to verify legitimacy of the study


29


Tactics to Convert Reluctant Participants:No Time / Not of Interest:• Give actual length of time• Explain how quickly the interview goes

Respondent Does Not Know about Topic:• Focus on opinion aspect of survey• Downplay knowledge questions• Explain that all opinions are valuable

Gatekeepers:• Ask that the respondent hear what the survey is about• Call back another time to try to avoid the gatekeeper


30


Translating Questionnaires

Two Questions to Consider:• Should We Translate Questionnaires?

– Are the concepts applicable in both the source and target cultures

• Can We Translate Questionnaires?– Semantic Equivalence: Language Nuances– Conceptual Equivalence– Normative Equivalence: Variations in Social

Norms and Conventions

31


Translating QuestionnairesNormative Equivalence:• Willingness to discuss certain topics with

strangers (political opinions, family matters)• Manner in which ideas are expressed: Variations

in the degree to which individuals show assertiveness vs. acquiescence, objectivity vs. social desirability, etc.

• Treatment of strangers: Some societies have a social norm of suspicion, others have a social norm of hospitality

32


Translating QuestionnairesInformation Obtained in Questionnaires:• Demographics• Behavioral Reports (“Did you vote in the last

election?”)• Attitudes and Opinions (“On a scale of 1 to 10,

where 1 equals very dissatisfied and 10 equals very satisfied, …”)

• Knowledge (“What is the name of your representative in Congress?”

• Intentions, Expectations, and Aspirations (“If the election were held today, for whom would you vote for President?”)

33


Translating Questionnaires• In general, asking single questions that seek

factual information create the fewest problems when translating

• Translating multi-item scales that deal with abstract concepts are the most problematic to translate

• Include input of translator as much as possible in questionnaire development process

• Avoid using a translator of opportunity (bilingual and available)

34


Translating Questionnaires

The Translated Instrument Must Possess:• Validity• Reliability• Legality: What is considered “sensitive

information” or “legal” differs from society to society

• Utility (cost-effectiveness)• Equivalence

35


Sampling of Participants:

• Select a sample that is generalizable

• Speak with your friendly Statistician about sampling

36


ReferencesAsking Questions: The Definitive Guide to Questionnaire Design – For

Market Research Political Polls, and Social and Health Questionnaires, Revised Edition. Norman M. Bradburn, Seymour Sudman, and Brian Wansink. Jossey-Bass John Wiley and Sons, Inc., San Francisco CA. 2004.

Designing Surveys: A Guide to Decisions and Procedures. Ronald Czaja and Johnny Blair. Pine Forge Press, Thousand Oaks, CA. 1996.

Mental Measurements Yearbook: Online version available through CWRU Libraries. http://library.case.edu/ksl/index.aspx. Keyword search on “Mental Measurements Yearbook”; Select “Mental Measurements Yearbook (Online)”.

Questionnaire Design: How to Plan, Structure, and Write Survey Material for Effective Market Research, second edition. Ian Brace. Kogan Page, London. 2008.

http://library.case.edu/ksl/index.aspx

37


ReferencesReal World Research: A Resource for Social Scientists and

Practitioner-Researchers, second edition. Colin Robson. Blackwell Publishers, Oxford UK. 2002.

Translating Questionnaires and Other Research Instruments: Problems and Solutions. Orlando Behling and Kenneth S. Law. Sage Publications. 2000.

Using Web and Paper Questionnaires for Data-Based Decision Making: From Design to Interpretation of the Results. Susan J. Thomas. Corwin Press, Thousand Oaks, California. 2004.

CTSC Topics in Managing Data Seminar Series

Questionnaire Design: Layout & Administration

Thank you for attending today’s seminar!

Please make sure to sign the registration sheet

Please complete an evaluation form and turn it in at the end of the seminar

Next Seminar: Wednesday, April 14, 2010, noon to 1 pm, Wolstein 6136

Topic: Data Cleaning Part 1: Ranges and Distributions of Numeric Variables

CTSC Topics in Managing Data Seminar Seriescci.case.edu › cci › images › 1 › 15 › Q_Design_Layout_2010-03-17.pdf · 2014-02-03 · Topics in Managing Data: Questionnaire

Documents