EMR 6500: Survey Research Dr. Chris L. S. Coryn Lyssa N. Wilson Spring 2015.

EMR 6500:Survey Research

Dr. Chris L. S. CorynLyssa N. Wilson

Spring 2015

Agenda

• Stratified random sampling for proportions

• From questions to a questionnaire• Budgeting survey research

Stratified Random Sampling

Estimate of Population Proportion

L

iiiLLst pN

NpNpNpN

Np

12211 ˆ

1ˆˆˆ

1ˆ

Example for a Population Proportion

N n n+

Town A 155 20 16 0.80

Town B 62 8 2 0.25

Rural 93 12 6 0.50

Example for a Population Proportion

Example for a Population Proportion

Selecting the Sample Size for Estimating Population Proportions

Sample Size for Estimating Population Proportions

• As with allocation methods for means and totals, allocation methods for stratified random samples for proportions can be equal, proportional, or optimal depending on the study’s purpose

Sample Size for Estimating Population Proportions

From Questions to a Questionnaire

General Premises

• The design of a questionnaire should consider how to motivate the recipient to respond

• It should also avoid measurement errors, ranging from order effects to item nonresponse

Guidelines for Ordering Questions

Ordering Questions

• General guidelines1. Group related questions that cover

similar topics, and begin with questions likely to be salient to nearly all respondents

2. Choose the first question carefully3. Place sensitive or potentially

objectionable questions near the end4. Ask questions about events in the order

that they occurred5. Avoid unintended order effects

Guidelines for Creating a Common Visual Stimulus

Visual Stimulus

• General guidelines1. Establish consistency in the visual presentation of

questions (across pages and screens) and use alignment and vertical spacing to help respondents organize information on the page

2. Use color and contrast to help respondents recognize the components of the questions and the navigational path through the survey

3. Visually group related information in regions through the use of contrast and enclosure

4. Use visual elements and properties consistently across questions to emphasize or deemphasize certain types of information

5. Avoid visual clutter6. Minimize the use of matrixes and their complexity

Yes

No

First name/initials(optional) (name)

Relative 1

(name)

Relative 2

(name)

Relative 3

Does this relative live in Kalamazoo? . . . . . . .

If no: about how far away from Kalamazoo do they live? . . . . . . . . Miles Miles Miles

About how old is this relative? . . . . . . . . . . . .

Years Years Years

Every day

Every week

Approximately, how often do you communicate with this relative? . . . . . . . . . . . .

Once a month

Less than once a month

Every day

Every week

Once a month

Less than once a month

Every day

Every week

Once a month

Less than once a month

Guidelines for Mail Questionnaires

Mail Questionnaires

• General guidelines1. Determine whether keypunching or optical imaging and

scanning will be used, and assess the limitations imposed on designing and processing questionnaires

2. Construct paper questionnaires in booklet formats, and choose physical dimensions based upon printing and mailing considerations

3. Decide question layout and how questions will be arranged on each page

4. Use symbols, contrast, size, proximity, and pagination effectively when designing branching instructions to help respondents correctly execute them

5. Create interesting and informative front and back pages that will have wide appeal to respondents

6. Avoid placing questions side-by-side on a page so that respondents are asked to answer two questions at once

Guidelines for Web Questionnaires

Web Questionnaires

• General guidelines1. Decide whether an electronic alternative is

appropriate2. Choose how the survey will be programmed and

hosted, commensurate with needs, skills, and sponsorship

3. Take steps to ensure that questions display similarity across platforms, browsers, and user settings

4. Decide how many questions will be presented on each page and how questions will be presented

5. Develop a screen format that emphasizes the respondent rather than the sponsor

Web Questionnaires

• General guidelines6. Use a consistent page layout across screens and

visually emphasize questions information that respondents will need to complete the survey while deemphasizing information not essential to the task

7. Do not require responses unless absolutely necessary

8. Design survey-specific and item-specific error messages to help respondents troubleshoot

9. Evaluate carefully the use of interactive features, balancing improvements in measurement with the impact on respondent burden and the implications with mixed-mode surveys

Web Questionnaires

• General guidelines10. Use audiovisual capabilities sparingly, and evaluate

the differential effect they have on respondents11. Allow respondents to stop the survey and finish

completing it at another time12. Whenever possible, collect paradata that provide

feedback on how respondents interact with questionnaire

13. Test the survey using a variety of platforms, connection speeds, browsers, and user-controlled settings, and test the database to ensure that items are collected and coded accurately

14. Take screenshots of each page of the final questionnaire for testing and documentation

Guidelines for Pretesting Questionnaires

Pretesting Questionnaires

• General guidelines1. Obtain feedback on the draft of the questionnaire

from a number of people, each of whom has specialized of some aspect of questionnaire quality

2. Conduct cognitive interviews of the complete questionnaire in order to identify wording, question order, visual design, and navigational problems

3. When the stakes are high, consider experimental evaluations of questionnaire components

4. Conduct a small pilot study with a subsample of the population in order to evaluate interconnections among questions, the questionnaire, and implementation procedures

Budgeting Survey Research

General Cost Frameworks for Survey Research

• Fixed-price– All costs are subsumed under a single

dollar value, no matter time, effort, and other expenditures

• Cost-reimbursable– Reimbursement for actual expenditures

Direct Costs

• Personnel– Hourly staff (+7.65% FICA)– Salaried staff (+ 48.75% for full benefits)– Academic staff (+ 22.25% for full benefits)

• Travel– Local transportation, flights, per diem, lodging

• Other costs– Materials and supplies (e.g., office and operating supplies,

telephone and conference calls, postage and shipping)– Publication and dissemination (e.g., standard printing,

copying, production)– Participant costs and/or incentives– Consultants or subawards– Computer services

Indirect Costs

• F&A (facilities and administration)– 49% is WMU’s current HHS approved

rate– 24% is WMU’s current off-campus rate

(requires rental space elsewhere)– Other rates can be negotiated based on

institutional policies (e.g., existing policy of funders)

Typical Budgeting Methods

• Per task• Per day (e.g., %FTE)• In budgeting, both are often

necessary

Consider…

• A mail survey to 500 randomly selected recipients– Printing of surveys = $2.00/survey– Postage = $0.45/survey for delivery +

$0.45/survey for return– Processing of surveys = 1 hour/survey @

$18/hour– $2/recipient incentive

• What is the total cost for just the above, which does not include other costs?