1 Session 20: Data Collection in the Field Ina F. Wallace RTI International IES/NCER Summer Research Training Institute, June 2007.

1

Session 20: Data Collection in the Field

Ina F. Wallace

RTI International

IES/NCER Summer Research Training Institute, June 2007

2

Data Collection in the Field

Topics Covered

• Methods of Data Collection

• Use of State-of-the-Art Data Collection Procedures

• Ensuring Data Quality

• Conducting Small-Scale Validation Studies

• Ethical Issues in Data Collection

3

Methods of Data Collection• Interviewer Administered

– In Person• Surveys• Assessments

– Telephone

• Self-Administered– Paper and Pencil

• Mail-in• In-person

– Computer Assisted Self Interview– Web procedures

4

Data Collection Mode Characteristics

High Data Collector

Involvement

Low Data Collector Involvement

Paper Computer Paper Computer

Direct Contact with Respondent

Face-to-Face Paper and Pencil Interviewing/

Assessment (PAPI)

Face-to-Face

Computer Assisted Personal Interview/

Assessment (CAPI)

(Audio) Computer Assisted Self-Interview (CASI,

A-CASI)

Indirect Contact with Respondent

Telephone

PAPI

Telephone Computer Assisted Telephone Interview (CATI)

Mail, Fax, E-mail questionnare

Touch-tone Data Entry (TDE), Web, Electronic Mail Survey (EMS)

No Contact with Respondent

Direct Observation

Computer Assisted Data Entry (CADE)

Administrative

Records

Electronic Data Interchange (EDI)

Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

5

Face-to-Face InterviewingAdvantages• Maximum degree of communication and interaction

• Can utilize PAPI or CAI• Allows for long, complex interviews

• Variety of tactics can be used to gain cooperation

• Response rates usually high

Disadvantages•High cost due to travel•Can generate social desirability for sensitive questions

•Interviewers can affect respondents

•Can generate interviewer variance

•Presence of others can affect responses

•Falsification of interview

6

Telephone InterviewingAdvantages• Cost lower than face to face

• More easy to provide training, supervision, and feedback

• Social desirability bias less than with face to face

• Interviewer variance less

Disadvantages• Less flexible• No ability to use visual aids

• Must be shorter• Questions must be less complicated

• Less capacity for motivating the respondent

• No coverage of non-telephone units

• Response rates lower

7

Mail SurveysAdvantages• Lowest cost• Low social desirability • Can utilize visual aids • Allows respondent to take time and look up information (if necessary)

• No interviewer variance

Disadvantages• Little control over who completes the survey

• Item non-response high• Response rates lower• Require a long field period to obtain an acceptable response rate

• Respondents must have good literacy skills

• Limit to closed-format questions

8

Computerized Procedures• Computer Assisted Personal Interview

(CAPI)• Computer Assisted Telephone Interview

(CATI)• Computer Assisted Self Interview (CASI)• Audio Assisted Self Interview (ACASI)• Telephone Audio Assisted Self Interview

(T-ACASI)• Computer Assisted Data Entry (CADE)• Web Surveys

9

Computer Assisted ProceduresAdvantages• Complex skip patterns and

branching are easier to follow

• Previously acquired information can be programmed

• Ranges, inconsistencies, and illegitimate values can be checked

• Help screens can be included

• Data are available for analysis more quickly

Disadvantages• Time consuming to develop

and program computer • Must be adequately tested• Can’t change things easily

once fielded• Risk of computer crashing• Costly

10

Direct Assessment

• Used to obtain psychological and educational information about an individual

• Individually administered

• Research driven NOT for clinical or diagnostic purposes

• Follows a standardized protocol

11

Observational Techniques• Involve a researcher making observations

within a natural setting• Direct Observation - individuals know they

are being observed– Continuous Monitoring recording as much

behavior as possible– Time Allocation randomly visiting the sample

to observe at different times

• Unobtrusive Observation – individuals do not know they are being observed

12

Observational Techniques

Advantages• No or limited participation needed

• May be good for generating hypotheses

• Sensitive social issues can be obtained more accurately

• High in validity because of the depth of information collected

Disadvantages• Variance associated with

observers is similar to that found with interviewers

• Can be very costly as they are often labor intensive

• Time consuming• Reliability can be low• Generalizability can be

low

13

Summary: Issues in Choosing a Strategy

• Type of Population

• Nature of Research Issue

• Question Form

• Question Content

• Response Rates

• Costs

• Length of Data Collection

14

Methods of Administration

• Standardized Interviewing Approach– A protocol in which interviewers interact with

the respondent in a manner which is consistent across all interviewers

• Conversational Interviewing– A protocol in which interviewers modify and

adapt questions to the respondent’s situation

• Hybrid Interviewing – Combination of the two approaches

15

Factors that Can Explain Interviewer Variance

Interviewer•Characteristics•Appearance•Motives•Beliefs/attitudes•Perceptions•Expectations•Skills/Knowledge

Respondent•Characteristics•Knowledge•Interest/motivation•Confidence•Strength of convictions•Expectations

Interviewer EffectsQuestionnaire

•Definition clarity•Terminology/jargon•Question form•Instructions•Question wording•Question topic

Survey Conditions and Setting

•Mode of interview•Standardization •Interviewer training•Interviewer supervision•Monitoring/observation

From Biemer & Lyberg (2003). Introduction to Survey Quality

16

Factors that Can Explain Interviewer Variance

Interviewer•Characteristics•Appearance•Motives•Beliefs/attitudes•Perceptions•Expectations•Skills/Knowledge

Respondent•Characteristics•Knowledge•Interest/motivation•Confidence•Strength of convictions•Expectations

Interviewer EffectsQuestionnaire

•Definition clarity•Terminology/jargon•Question form•Instructions•Question wording•Question topic

Survey Conditions and Setting

•Mode of interview•Standardization •Interviewer training•Interviewer supervision•Monitoring/observation

From Biemer & Lyberg (2003). Introduction to Survey Quality

17

Role of the Interviewer/Assessor

• Gaining Cooperation– Converting reluctant participants

– Recognizing and dealing with refusals

• Training and Motivating Respondents– Interviewers implicitly and explicitly teach

respondents how to behave

• Obtaining Quality Data

• Ensuring Confidentiality

18

Assessing Children• Assessor Characteristics

– Flexibility– Vigilance– Self-awareness– Eye contact

• Establishing Rapport– Allaying apprehensions– Helping children feel at ease– Maintaining limits on permissible behavior

• Arranging the Physical Environment

19

Ensuring Quality Data

• Recruiting interviewers/assessors– Characteristics of all data collectors– Characteristics of interviewers– Characteristics of assessors

• Training on standardized procedures

• Certification as data collectors

• Monitoring performance

20

Training

Content of Training• Presenting the study • Contacting participants• Administering the

questionnaire/ assessment• Probing• Recording responses• Confidentiality• Interpersonal relations• Administrative procedures

Procedures for Training• Written materials• Lecture and presentation• Computer-based tutorials• Planned exercises• Role Playing• Observation• Certification

21

Interviewer Training Agenda• Introduction to the Study• Role of the Interviewer• Confidentiality Procedures• Review of Questionnaire and Protocols• Standardized Interviewing Techniques• Round Robin Interview• Gaining Cooperation• Presentation of Scenarios• Round Robin Demonstrations• Paired Mocks• Certification

22

Assessor Training Agenda

Day 1:

Large group • Introduction to Project• Role as an Assessor• Best practices with

children

Small groups • Pre-training quiz• Introduction to all

materials• Learn each

assessment–Observation–Round Robins

Day 2:

Small groups • Individual practice with

assessments• Paired practice with

assessments

Large group• Building rapport and

gaining cooperation• Administrative

responsibilities • Data security • Informed Consent

Day 3:

Small groups • Review of

assessments• Paired practice• Post-training quiz

Large group • Concluding remarks• Certification

23

Monitoring PerformanceWhat to Monitor• Detection and prevention

of falsified information• Compliance with rules

and guidelines about administering the questionnaire or assessment

• Performance on non-interview/assessment tasks

How to Monitor• Telephone verification

with a sample of respondents

• Direct observation – in-person or by telephone

• Audio-recording – CARI• Review of materials• Performance and

production measures

24

Human Subjects Issues• Institutional Review Boards • Informed Consent

– Who is carrying out research under whose sponsorship– Description of purposes and procedures– Duration of involvement– Risks and Benefits– Confidentiality– Voluntary nature of research including stopping early

and skipping questions– Compensation– Contact information

• Confidentiality Issues

25

Human Subjects Issues• Institutional Review Boards • Informed Consent

– Who is carrying out research under whose sponsorship– Description of purposes and procedures– Duration of involvement– Risks and Benefits– Confidentiality– Voluntary nature of research including stopping early

and skipping questions– Compensation– Contact information

• Confidentiality Issues

26

Data Collection EvaluationStage of Data

CollectionEvaluation Method Purpose

Design Expert review of instrument Identify problems with instrument

Pretest Behavior coding

Cognitive interviewing

Evaluate one or more stages of the response process

Pretest/Post-Data Collection

Respondent focus groups

Interviewer debriefings

Evaluate instrument and data collection procedures

Pretest/Data Collection

Supervisor observation

Telephone monitoring

CARI

Evaluate performance of data collector

Post-Data Collection

Experimentation

Internal Consistency

External validation

Re-interview/retest

Compare alternative methods of data collection

Evaluate components of MSE

Validate estimates

Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

27

Post-Data Collection Evaluations

• Experiments

• Observational Studies

• Internal Consistency Studies

• External Validation Studies

• Administrative Record Check Studies

• Re-interview/Re-Test Studies

28

Post-Data Collection Evaluations

• Experiments

• Observational Studies

• Internal Consistency Studies

• External Validation Studies

• Administrative Record Check Studies

• Re-interview/Re-Test Studies