Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK.

Measuring Coverage:Post Enumeration Surveys

Owen AbbottOffice for National Statistics, UK

Agenda

• Introduction• Why have a PES?• Essential features of a PES

– Survey Design– Fieldwork

• Analysing the data– Matching– Estimation

• Results from 2001 UK Census• Discussion

Why do we need a PES?

• Census won’t count every household or person• Undercount causes bias in estimates• In the UK in 2001, we estimated that 3 million

persons (6%) did not fill in the form• Increasing problem from 1981 to 1991 to 2001• The undercount is not evenly spread

– Inner Cities – Deprived areas – Young persons

Why do we need a PES?

• Census counts alone not good enough• UK Users demand robust census population

estimates– Central Government resource allocation– Yearly demographic population estimates– Government Policy

• So we need to measure how many households and persons the census misses, and work out:

– where they are missed from– their characteristics

Basic Methodology

• PES - Census Coverage Survey (CCS) in UK– In the UK approx 1% population

• Match the PES to the Census• Use the people the PES sees that the census didn’t

to estimate how many missed– where and characteristics

• Add to the Census counts (either at aggregate level or impute (UK))

2001 UK ‘One Number Census’ frameworkCENSUS

CENSUS +CCS

DESIGN GROUPESTIMATE BYAGE AND SEX

LADESTIMATES

ADJUSTEDINDIVIDUAL AND

HOUSEHOLD DATAAND TABLES

NATIONALPOPULATIONESTIMATE

MATCHING

QualityAssurance

Dual System andregression estimation

CCS

Synthetic estimation

Imputationcontrolled to LAD

estimates

Sum

Post Enumeration Survey

Key features:

A - Design– Sample survey– Sample size dependent on accuracy (and geographic

level) requirements

B - Fieldwork– Conducted after the census has finished– Independent re-enumeration– Area based– Door to door interview– Focused on measuring coverage

Post Enumeration Survey - Design

• Multi-stage Stratified sample • Select a sample of (small) geographical areas that

can be re-enumerated– UK uses Postcodes (about 20 hhs)– US uses blocks (about ????100 hhs)

• Sample stratified by:– Geography– Area type– Demography

2001 UK PES Design

Geographical Strata:• Local Authorities (mean pop 120k) grouped into

contiguous groups called Estimation areas (EAs), each having 500k pop

Area Type and Demographic strata:• Within every EA a sample of 1991 Enumeration

Districts was selected, stratified using a hard-to-count index and the 1991 age-sex structure

– (1991 EDs have about 200 households)

2001 UK PES Design

• Hard to count index was a national stratification using a combination of variables associated with undercount e.g:

– Unemployed– Multi-occupied– Private rented– Language difficulty

• 3 level index, split into 40%, 40%, 20% nationally

• Within each selected ED a sample of 3, 4 or 5 postcodes was selected

Post Enumeration Survey - Field

• Aim: enumerate all the people and households in the sampled areas

• Carry out the survey after the Census– Census fieldwork finished

• Independence critical (see later)– Interview based– Independent re-enumeration– Separate fieldforce and management– No address list (UK have address list for Census)– Difficult if doing quality at same time, as not independent

Post Enumeration Survey - Field

• In UK, focused on measuring coverage– Previously measured quality as well– Found that separate surveys more effective– Can focus on getting maximal response in sampled areas

• UK 2001 PES used very short interview– key household and demographic questions only

• Accommodation type

• Tenure

• Name

• Gender

• Date of Birth (or Age)

• Student

• Ethnicity

• Activity last week

Post Enumeration Survey - Field

• Other initiatives to maximise response:– Pairwork and teamwork– Refusal avoidance training – Calling strategy– Up to 10 attempts to interview– Last attempt deliver form to return in post

Post Enumeration Survey

• Interviewer Duties:– Establish the postcode boundaries

– Conduct independent listing of all residential and non-residential addresses

– Seek out obscure accommodation

– Deliver advance notification cards

– Identify/probe for all households at an address

– Make contact with householders

– Conduct doorstep interviews

– Persuade potential refusals

– Report Progress

Post Enumeration Survey

• Map

Post Enumeration Survey

• Property Listing Listing Sheet for Postcode …………………….. Sheet ….…. of ……….

Interviewer No. Address/Household Advance Leaflet Delivered

Form No.

Date 1 Date 2 Date 3 Date 4 Date 5 Date 6 Date 7 Date 8 Date 9 Date 10 Time 1 Time 2 Time 3 Time 4 Time 5 Time 6 Time 7 Time 8 Time 9 Time 10 Interview Refusal Non-residential Vacant Visitor only Communal

Notes:

Analysing the data - Matching

• Match Census returns to CCS returns• Require very high quality

– Minimise false negative matches (missed matches, see later)

• In 2001, we used hierarchical nature of data to help match

– Match within sampled areas (geographical blocking)– First match household– Then match persons within households

Analysing the data - Matching

• Used a five stage strategy, designed to minimise false negative matches:

– Exact matching– High probability matching– Clerical assisted probability matching– Clerical matching– Final expert review of non-matches

• Developed our own in-house system• Allowed access to scanned form images (this was

crucial)

PO155RR

ERIC

SMITH

13

MALE

SINGLE

ERIC

SMITH

13

MALE

SINGLE

PO155RR

29 29

Analysing the data - Matching

• Output:– Match between Census and CCS– Census only– CCS only

Analysing the data – Estimation

• Dual System Estimation (DSE)– Capture-recapture as used for wildlife

• Simple example: How many fish in a lake?– Catch as many as possible on day 1

• Count them (N1)

• Mark with a red dot• Return them to the lake

– Catch as many as possible on day 2• Count them (N2)

• Count how many have red dots (N12)

– Number of fish in lake= (N1 * N2)/N12

Analysing the data - Estimation

• Use matched Census+CCS data• DSE estimates adjustment for those missed in both

Census and CCSCounted By CCSYes No

Counted Yes n11 n10 n1+

By Census No n01 n00 n0+

n+1 n+0 n++

DSE count (for a postcode):

n++ = n1+ x n+1 n11

Analysing the data - Estimation

• DSE assumptions– Independence– Homogeneity of capture probabilities– Perfect matching– Closure– No list inflation

• Violation of these assumptions leads to bias (in both directions)

• Lots of literature on DSE

Analysing the data – Estimation

• DSE can only be used within the sample• Need additional step to get to population totals• In 2001, we used DSE at postcode level• Then used a ratio estimator to predict for non-

sampled postcodes (again lots of literature)

Census

DSE

Analysis – Getting to small areas

• Ratio estimator produced estimates for 500k population blocks

• Needed estimates for Local Authorities (about 120k population)

• Sample size not sufficient to do directly• So used small area estimation techniques

– these borrow strength across areas– We used a fixed effect to model LA differences

• LA population estimates from the model then constrained to EA totals

Quick summary of 2001 UK method

• In 2001, One Number Census methodology was developed

– Large CCS (320,000 households)– Matching– Capture Recapture– Modified ratio estimator– Small area estimation to get LA totals– Imputation

• Estimated 1.5 million households missed• 3 million persons missed (most from the missing

households but some from counted households)

Results

• England and Wales population about 50m individuals in 20m households

• Estimated 1.5 million households missed• 3 million persons missed (most from the missing

households but some from counted households)

Underenumeration in 2001Underenumeration of Census by agegroup

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

0 1-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85+

Agegroup

ON

C/C

en

su

s

Males Females

Response Rates in 2001

Summary

• Fundamental that the census is good– This does not make a bad census good, it makes a

good census better!

• US, Australia, NZ, Canada, UK all measure coverage (and most use a PES)

– All aim at measuring coverage for assessing census quality, most do not fully adjust the outputs

– Coverage for most is around 96-98%– Increasing problems of overcoverage

• The design and fieldwork of the PES are important to get right

More info

• Brown, J.J., Diamond, I.D., Chambers, R.L., Buckner, L.J., and Teague, A.D. (1999), “A methodological strategy for a one-number census in the UK,” Journal of the Royal Statistical Society A, 162, 247-267.

• www.statistics.gov.uk/census2001/onc.asp

• owen.abbott@ons.gov.uk