Questionnaire Design & Survey Methodology for Medical Education Research Proposals ADE Medical Education Research Faculty Development Program February 14, 2006
Questionnaire Design & Survey Methodology for Medical Education Research Proposals
ADE Medical Education Research
Faculty Development Program
February 14, 2006
Faculty Introductions
Wendy Cohn, Ph.D. Jim Martindale, Ph.D. Natalie May, Ph. D. Lisa Rollins, Ph.D.
Goals of Session I
How to write a questionnaire Best ways to administer a survey Measurement and Analysis Q & A Prepare for Session II
What You Must Consider
What is the question? Has the question been addressed previously? If so,
how? What is the appropriate methodology? Whom do you need to sample and how? What do you need to ask? How should you ask it? Is there an existing survey/validation? How should the survey data be analyzed? How should the results be reported?
Finding Your Question
Begin with a purpose! For example: “The purpose of this study is to
test (the theory) by relating (the independent variable) to (the dependent variable) for (participant group) at (the research site).
Finding Your Question (cont’d) Pose a question Begin with “how,” “what,” or “why” Specify your variables Use words to describe the connection
between/among variables (describe, compare, relate, etc.)
Indicate the participants and setting
Conceptual Survey Outline
A conceptual map of your questionnaire. Helps guide the development of your
questionnaire so that you include relevant questions and exclude irrelevant questions.
Key elements: Concept; term; operational definition;
measurement; priority
Types of Questions
Dichotomous Questions Nominal Questions Ordinal Questions Interval/Ratio Questions Filter or Contingency Questions
Question Formats
Structured Fill-in the blank Rating Likert Scale Check all that apply
Unstructured Open ended question
I oppose electronic prescribing and electronic medical records for
patients.
SA A N D SD
Which of these five statements best describes your chairman?
Innovative but lacking in leadership qualities
About the same on innovation and leadership qualities
Stronger on leadership than innovation
A born leader
A real innovator
How do you feel about healthcare in the United
States?
About how many books have you read for leisure during the past year?
___________ Number of books
Don’t you agree that the new resident work hour regulations are
limiting to your education?
SA A N D SD
How supportive is your spouse about your efforts to quit smoking?
Very Supportive Somewhat Not at allsupportive supportive supportive
If you fixed dinner at home last night, did you eat meat as part of that meal?
Yes
No
Please identify your insurance carrier:
Aetna Blue Cross Health South Self Pay
Do you incorporate obstetrics/ gynecology into your practice?
SA A N D SD
Do you favor or oppose not allowing the state to raise taxes without approval of 60% of the voters?
Favor
Oppose
Summary of Typical Problems Double-barreled questions Vague questions Leading questions Premature assumptions/bias Insufficient alternatives with forced-choice options Answer options don’t fit with the question Double negatives
Issues to Consider Regarding Content Is the question necessary/useful? Are several questions needed? Do respondents have the needed info? Does the question need to be more specific? Is the question biased or loaded? Is the question asking about sensitive
information?
Layout Issues
Self-explanatory Visually clear and uncluttered Group tasks/types of questions Consistent and clear response options
Layout Issues (cont’d)
Consider question placement Skip patterns kept to a minimum (if needed,
use arrows and boxes) Beware of using double-sided pages There is elegance in simplicity!
Tips to Reduce Non-Response Professional, personalized, attractive, easy to
complete Tasks/directions should be clear Easy to read, uncluttered, visually appealing Response task should be easy Length of form
Survey Implementation
A well designed questionnaire is critical…however: Implementation procedures have a greater
influence on response rates.
Experimental research has identified the factors that influence survey response.
Choices for implementation
Telephone Costly; need expertise
Web Efficient; cheap; limited range of question types;
hard to apply full range of implementation procedures
Mail Very well studied; moderately costly; can do well
Multi-method Can be done but complicated; get help
Implementation procedures that affect response Multiple contacts Contents of letters Appearance of envelopes Incentives Personalization Sponsorship & its explanation
The five most important elements for achieving high response rates Respondent-friendly questionnaire Four contacts by first class mail
Pre-notice letter; questionnaire; thank you postcard; replacement questionnaire; final
Return envelopes with real first class stamps Personalization of correspondence Token prepaid financial incentives
First Contact: Pre-Notice Letter Provides positive notice that the recipient will
be receiving a questionnaire Important characteristics:
Brief Personalized Positively worded Aimed at building anticipation vs. providing too
many details Sent about 1 week in advance
Second Contact: The Questionnaire Mail Out
• Cover letter (1 page; date; purpose of letter; why request is important; confidentiality; voluntary participation; enclosures of stamped envelop and incentives; who to contact with questions
• Questionnaire• Return envelope with stamp• Assembling the packet
Third Contact: The Postcard Thank You/Reminder Written to remind participants that a
questionnaire was sent to them Elements:
Reminder that questionnaire was sent Thank you to those who have returned; request
for others to do so. Invitation to ask for a replacement questionnaire.
Fourth Contact: The First Replacement Questionnaire Elements:
Haven’t heard from you Others have responded and answers are
important Eligibility Confidentiality Voluntary
The Fifth Contact: Special Procedures Less frequently done in practice Last attempt; most intense
Certified mail Telephone
Sampling
The survey population consists of all units (e.g. households, individuals) to which one desires to generalize survey results.
The sampling frame is the list from which a sample is to be drawn to represent the population.
A sample are all units of the population that are drawn for inclusion in the survey.
Probability Sampling
Get help! How large of a sample do you need?
How precise do you need your estimates? Size of the population How varied the population is with respect to your
characteristic of interest Amount of confidence you wish to have in the
estimates of the entire population.
What Am I Measuring? A Look At Reliability Reliability has to do with the quality of
measurement. Practically speaking, reliability is the "consistency" or "repeatability" of your measures.
Internal Consistency is one type of reliability measure. Cronbach's alpha measures internal consistency by how well a set of items (or variables) measures a single uni-dimensional latent construct.
What Am I Measuring?A Look at Validity Construct-the degree to which inferences can
legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based.
External-the degree to which the conclusions in your study would hold for other persons in other places and at other times.
Summary Points
There are a lot of things to consider when developing surveys
Leave enough time for the process See if it has been done before Get help if you need it No matter what, another set of eyes can be
helpful Length and simplicity
Next Session (Feb. 21)
You bring: Questionnaire drafts Research questions Ideas Questions
We’ll bring: List of resources IRB contacts & information
Other Ideas for Final Session?
Contact Information
Wendy Cohn [email protected] Jim Martindale [email protected] Natalie May [email protected] Lisa Rollins [email protected]