Technological Advances in Survey Data Collection Methods Survey Research Laboratory University of Illinois at Chicago September 2011 Traditional Survey Methods • Mail/self-administered questionnaires Census questionnaires go back centuries • Face-to-face interviews Employed in the U.S. since 1930 • Telephone interviews In common use since the 1970s Some New Survey Technologies For face-to-face surveys: •ACASI (audio computer-assisted self-interviews) •Hand-held devices (PDAs, Smartphones, Tablets) •MiFi to create mobile hot-spots in rural areas For telephone surveys: •IVR (interactive voice response), aka: T-ACASI •VOIP For self-administered surveys: •Web surveys •Also hand-held devices New computer technologies: • Combine power of computer administration Automated question routing Checking for bad or inconsistent answers Data encryption Wireless transmission • With the advantages of self- administration Elimination of interviewer variance (but replaced by device/browser variance) Improved reporting of sensitive information ACASI Development • Developed at the Research Triangle Institute in early-to-mid-1990’s • Adapted for the National Household Survey on Drug Abuse in 1997-99 • Now understood that most respondents do not take advantage of audio features (Couper et al., POQ, 2009) IVR Development • First reported in late 1990s • Two approaches: Initial contact by recording & IVR survey Human contact, then IVR survey • Developmental work centered on market research applications • So far, less commonly applied in academic research
7
Embed
Technological Advances in Traditional Survey Methods ...Other Advantages/Disadvantages • Cost Effectiveness ACASI most cost-effective for complex, large-sample, sensitive surveys
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Technological Advances in Survey Data Collection Methods
Survey Research Laboratory
University of Illinois at Chicago
September 2011
Traditional Survey Methods
• Mail/self-administered questionnaires
� Census questionnaires go back centuries
• Face-to-face interviews
� Employed in the U.S. since 1930
• Telephone interviews
� In common use since the 1970s
Some New Survey Technologies
For face-to-face surveys:
•ACASI (audio computer-assisted self-interviews)
•Hand-held devices (PDAs, Smartphones, Tablets)
•MiFi to create mobile hot-spots in rural areas
For telephone surveys:
•IVR (interactive voice response),
aka: T-ACASI
•VOIP
For self-administered surveys:
•Web surveys
•Also hand-held devices
New computer technologies:
• Combine power of computer administration� Automated question routing
� Checking for bad or inconsistent answers
� Data encryption
� Wireless transmission
• With the advantages of self-administration� Elimination of interviewer variance (but replaced by device/browser variance)
� Improved reporting of sensitive information
ACASI Development
• Developed at the Research Triangle Institute in early-to-mid-1990’s
• Adapted for the National Household Survey on Drug Abuse in 1997-99
• Now understood that most respondents do not take advantage of audio features (Couper et al., POQ, 2009)
IVR Development
• First reported in late 1990s
• Two approaches:� Initial contact by recording & IVR survey
� Human contact, then IVR survey
• Developmental work centered on market research applications
• So far, less commonly applied in academic research
Web Surveys
• First reported use in early 1990’s
• Dramatic increase in use over the past decade
• Hundreds of web survey software packages now available
• Knowledge Networks panel
Typology of Web Surveys
With thanks to Mick Couper, University of Michigan Survey Research Center(see: Couper, Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly 2000;
64: 464-494)
Nonprobability Methods
1. Polls as entertainment
2. Unrestricted self-selected surveys
3. Volunteer opt-in panels
Probability Methods
4. Intercept surveys
5. List-based samples
6. Web option in mixed-mode surveys
7. Pre-recruited panels of internet users
8. Pre-recruited panels of full population
Sources of Survey Error
1. Sampling error
2. Coverage error
3. Nonresponse error
4. Measurement error
5. Processing error
Coverage Error
• No real effect of ACASI and IVR on coverage error
• Coverage error is a big problem for web surveys� Poor frame availability for general population
� Around 75% of U.S. population are ever internet users
� Digital divide: differences in accessibility by race/ethnic minorities
� Solutions: (1) weighting (2) web-enabled panels
European HH Internet Access: 2008Data Source: http://epp.eurostat.ec.europa.eu/
Darker colors = higher internet access
Nonresponse Error
• Unit nonresponse
� Little effect of ACASI
� IVR – breakoffs a big problem
� Low response rates in web surveys
• Item nonresponse
� In general, less missing data in ACASI, IVR and web surveys (when programmed correctly)
Measurement Error
• General belief that self-administered survey data of higher quality� More privacy = more reporting of sensitive behaviors
� ACASI, IVR and Web surveys are all enhancements to self-administered survey research
� More non-differentiation in web surveys
� Computer methods employ different sensory channels: visual vs. aural
Processing Error
• ACASI, IVR and Web surveys can all be used to program data consistency checks
• Skip patterns can be automated
• Out-of-range values prohibited
• Some data can be generated by the device directly (date, time, length)
• Programming errors can, however, be fatal
Other Advantages/Disadvantages
• Cost
� ACASI/IVR more expensive
� To purchase software & recording systems
� To develop programming expertise
� ACASI requires expensive hardware for each interviewer
� Requires availability of timely technical support
� Web surveys – relatively inexpensive
� Software systems are affordable for most organizations
� Additional hardware required for hosting web surveys� And hosting is more expensive
� Surrender control if hosting survey with outside vendor� IRBs may impose restrictions on use of outside vendors
Other Advantages/Disadvantages
• Cost Effectiveness
� ACASI most cost-effective for complex, large-sample, sensitive surveys of the general population
� Web surveys most cost-effective for relatively brief and less complex surveys of special populations
� But this is changing
Other Advantages/Disadvantages
• Time efficiency
� ACASI/IVR can require considerable time to program and test
� It can take months/years to develop skills with these systems
� Web surveys – can be done very quickly in comparison to most other methods
� Are now relatively simple to program
� And complex programming functions are becoming more available
Other Advantages/Disadvantages• Computer familiarity
� ACASI, IVR and Web surveys are very applicable with young, educated and professional samples
� These technologies can be a barrier to survey participation and data quality for some populations who are less familiar and/or comfortable with these technologies, or who have limited access
• Still some gap in technical competence and information literacy among several groups:
• Older citizens
• Minorities
• Low SES groups
Other Advantages/Disadvantages
• Data Complexity� ACASI can be used to collect lengthy and highly complex data sets
� IVR can also be used to collect complex data but data complexity may contribute to break-off problem
� Web surveys have historically been thought to be useful only for collecting relatively simple data and brief sets � this is not SRL’s experience
� ACASI/IVR/Web remove the interviewer. It is difficult to replace an interviewer’s ability to identify respondent fatigue and respond in ways that maintain respondent engagement.
Other Advantages/Disadvantages
• Questionnaire Design
� These methods facilitate question tailoring
� Complex skip patterns, contingent on responses to multiple questions
� Personalize/customize question wording based on earlier or pre-coded or pre-loaded data (‘fills’)
� Examples: use terminology most comfortable with, specific dates, avoid awkward-looking text (eg., his/her/yours)
Other Advantages/Disadvantages
• Data Quality
� These methods reduce interviewer error
� These methods can also improve the quality of open-ended responses, although:
� The typing skills of respondents are another consideration when asking open-ended questions
� Can be used to detect interviewer falsification
� Random respondent selection can be automated to avoid errors
Other Advantages/Disadvantages
• Administration
� ACASI, IVR will take longer, on average, to administer to each respondent
� In the field, use of computers can sometimes be ‘clunky’ or awkward
� but things are improving as computers become lighter and more mobile
� Web surveys can be made very convenient and user-friendly
� and respondents are becoming more comfortable with web surveys
• Use of video-clips in ACASI & web surveys• ACASI digital recording of open-ended responses• Touch screen systems• Digital pens• Improvements in voice recognition systems• Daily uploading of encrypted, completed field
interviews via remote wireless connections• Data encryption
Continuing technical challenges:
•Wireless-only households
•Data security & IRB issues
State-level Estimates of the Percentage of Adults Living in Wireless-only Households: United States, July 2009-June 2010
Source: Wireless Substitution: Early Release of Estimates From the National Health Interview Survey, July - December 2008. by Stephen J. Blumberg, Ph.D., and Julian V. Luke, Division of Health Interview Statistics, National Center for Health Statistics
United States Flu Activity
• Google Flu Trends estimate• United States Data
Smart Phones
Relatively untapped in Survey Research
Technology being adopted at fast rate
Potentially very flexible tool, multiple modes on a single device