Using Audit Trail Data for Interviewer Data Quality Management · 2015. 1. 28. · Survey Research Operations Survey Research Center Institute for Social Research 1 Using Audit Trail

Survey Research OperationsSurvey Research CenterInstitute for Social Research

1

Using Audit Trail Data forInterviewer Data Quality

Management

Haoyu Gu, Mick Couper, Nicole Kirgis, Sharon Parker, Sarrah Buageila

University of MichiganSurvey Research Center

May 18, 2013

Survey Research OperationsSurvey Research Center ▪ Institute for Social Research

Outline• National Survey of Family Growth

(NSFG) and the use of Paradata andDashboard

• Dashboard for interviewer data quality▪ Audit trail data▪ Performance indicators▪ Dashboard layout▪ Use of interviewer data quality dashboard

• Summary2


National Survey of Family Growth (NSFG)

• Nationally representative, in person, cross-sectionalstudy

• women and men 15-44 years of age• Collects data on pregnancy, childbearing, health,

and parenting.• Quarterly data collection design

• ~40 Interviewers▪ work consistently during a data collection year

3

Year 1 (Sep 2011- Sep 2012) 2 (Sep 2012-Sep 2013) 3…

Quarter 1 2 3 4 1 2 3 4 1…Week 1 -

121-12

1-12

1-12

1-12

1-12

1-12

1-12 …

*This NSFG is carried out under a contract with the CDC’s National Center for Health Statistics,Contract # 200-2000-070012006 (for 2006-2010) and Contract # 200-2010-33976 (for 2010-).

The NSFG DashboardEffort

Interviewer’s WorkingDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and

Screener HPI by Day of Survey

0

5

10

15

20

25

30

35

1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI

Phase 1 Phase 2

Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main andScreener HPI by Day of Survey

0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

Hours

% Productivity

Calls/Day

Calls/Hour

% Peak Calls

Screener/Main Calls

Active SampleDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

% Occupied

% Eligible

% Nonworked

Noncontacts

Mean Calls

% 8+ Calls

% Locked Bldgs

% Resistant

% Hard Appt.

Propensity

ProductivityDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

Interviews

Cumulative Interviews

Hours/Interview

Calls/Interview

Data Set BalanceDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

Response Rate

% With Kids

% Sexually Active

Group Rates

CV Group Rates


CAPI and Audit Trail Data• CAPI – Computer-Assisted Personal Interviewing• ACASI – Audio Computer-Assisted Self-Interviewing

• Audit Trail data▪ Key strokes▪ Timing between key strokes▪ Can be played back


Data Sources• Audit Trail data collected during NSFG previous

cycle and current cycle

6

Interviewer 1

Field1 Visit 1Field2 Visit 1Field3 Visit 1

…Field1 Visit 2Field x Visit y

….

Interviewer 2

Interview 1

Interview n

Interview 1

Interview m

Field1 Visit 1Field2 Visit 1Field6 Visit 1

…Field2 Visit 2Field j Visit k

….

Interview 2…

Interview 2…


Interviewer Level Indicators

7

• Nine individual indicators▪ Field time▪ Error escapes▪ Error suppressions▪ Error jumps▪ Backups▪ Don’t know responses▪ Refused responses▪ Help key used▪ Remarks used

Error Checks


Variable Reduction

8

• Principle Component Analysis

• Based on 22,565 completed interviews collectedduring 2006-2010

• Based on seven indicators• Use oblique rotation• Keep three factors


Variable ReductionFactors Loading Name in dashboard

Factor1 Field Time (High)

Backup (High)

Error Jumps (Medium )

Error Escape (Medium)

Too Fast

Factor2 Error Suppression (High)

Error Escape (Medium)

Error Jump (Medium)

Many Error Checks

Factor3 Don’t Know (High)

Refusal (High)

Many DK/RF

9


Dashboard Layout

• Three factors• Z-score for 9 individual indicators• ~40 Interviewers• Every 2 weeks

10


Dashboard Layout

• Highlight outliers

• Comparison across interviewers and over time

11


Dashboard Layout

12


Interviewer Evaluation andIntervention

13


Interviewer Evaluation andIntervention

14


Interviewer Evaluation and Intervention

15


Summary• The data quality dashboard has been used to

▪ Identify outlying interviewer▪ Monitor the change of the interviewer performance▪ Identify problematic cases

• Future work▪ Improve usability of the dashboard▪ Separate ACASI section▪ Monitor section level indicators▪ Monitor case level indicators

16


References• Groves, R. M., Mosher, W. D., Lepkowski , J. M., Kirgis, N. G., 2009. Planning and

Development of the Continuous National Survey of Family Growth., Vital and healthstatistics, Series 1, Programs and Collection Procedures, Number 48, 09/2009

• Lepkowski, J. M. , Mosher, W. D. , Davis Karen E , Robert M Groves, John Van Hoewyk,2010. The 2006-2010 National Survey of Family Growth: sample design and analysis of acontinuous survey. Vital and health statistics. Series 2, Data evaluation and methodsresearch, Number 150, 06/2010;

• Couper, M.P., Sadosky, S.A., Hansen, S.E., 1993. “Measuring Interviewer Performanceusing CAPI”, p.845-850 in Proceedings of the Joint Statistical Meeting of the AmericanStatistical Association, Section on Survey Research Methods. Alexandria, VA: AmericanStatistical Association.

• Couper, M.P. Hansen, S.E. and Sadosky, S.A., 1997. “Evaluating Interviewer Performance ina CAPI Survey.” In Survey Measuerment and Process Quality, edited by L.E. Lyberg, P.P.Bierer, M. Collins, E.D. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin. New York: JohnWiley

• Hansen, S. E., Marvin, Theresa, 2001. “Reporting on Item Times and Keystrokes fromBlaise Audit Trails”, 7th International Blaise Users Conference ,Washington DC.

• Ho P., Smith L., Chan W., Kang S., Tan L., 2005, SASifying CAPI Audit Trails for Analysis,Federal Committee on Statistical Methodology Conference Papers

• Mockovak W. , Powers R., 2008, The Use of Paradata for Evaluating Interviewer Trainingand Performance, Proceedings of the Joint Statistical Meeting of the American StatisticalAssociation, Section on Survey Research Methods

17


Thank You

guh @ umich.edu

18

Using Audit Trail Data for Interviewer Data Quality Management · 2015. 1. 28. · Survey Research Operations Survey Research Center Institute for Social Research 1 Using Audit Trail

Documents