Confidentiality Protection of Social Science Micro Data: Synthetic Data and Related Methods John M. Abowd Cornell University and Census Bureau January.
Post on 15-Jan-2016
214 Views
Preview:
Transcript
Confidentiality Protection of Social Science Micro Data: Synthetic Data and Related Methods
John M. AbowdCornell University and Census Bureau
January 30, 2006UCLA Institute for Digital Research and EducationPresentation
Acknowledgements
• Many current and past LEHD staff and senior research fellows contributed to the development of the LEHD infrastructure system and the Quarterly Workforce Indicators. Kevin McKinney, Bryce Stephens and Lars Vilhuber were particularly responsible for the confidentiality protection system.
• Fredrik Andersson and Marc Roemer at LEHD did the data analysis and implementation of the On the Map package. John Carpenter of Excensus, Inc. developed the mapping application.
• Gary Benedetto, Lisa Dragoset, Martha Stinson and Bryan Ricchetti did the synthesis programming for the SIPP-PUF application.
Overview
• What is the problem?• What are synthetic data?• How can the research community benefit from
synthetic data?• The NSF-ITR synthetic data grant• The Census Bureau’s synthetic data and related
products:– QWI Online– On the Map– The new SIPP-SSA-IRS Public Use File
• Tools
Information Release and Data Protection are Competing Objectives
• Statisticians call this the Risk-Utility tradeoff
• Economists prefer to distinguish between technological trade-offs and preference trade-offs
• Information release and data protection are technological tradeoffs
A Simple Example of the Technological Trade-off
• There are two outputs: information released and data protection
• Consider a census with sampling as the release technology
• The PPF measures the amount of information that must be sacrificed to get additional protection
• The information measure is Shannon’s H (or the Kullback-Liebler difference between the census and the sample)
• The protection measure is the maximum probability of an exact disclosure
Information Gain-Protection PPF
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00
Protection
Info
rmat
ion
Marginal Cost of Protection
0.00
0.50
1.00
1.50
2.00
2.50
3.00
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00
Protection
Pri
ce
What Are Synthetic Data?
• Public use micro data products that reproduce essential features of confidential micro data products
• Essential features include:– Univariate distributions overall and in
subpopulations– Multivariate relations among the variables
Some History
• Original fully synthetic data idea was due to Rubin (JOS, 1993)– Synthesize the Decennial Census long form responses for the
short form households, then release samples that do not include any actual long form records
• Original partially synthetic data idea was due to Little (JOS, 1993)– Synthesize the sensitive values on the public use file
• Critical refinement (Fienberg, 1994)– Use a parametric posterior predictive distribution (instead of a
Bayes bootstrap) to do the sampling• Other authors, particularly Raghunathan, Reiter, Rubin, Abowd,
Woodcock– Partially synthetic data with missing data (Reiter)– Sequential Regression Multivariate Imputation (Raghunathan,
Reither, and Rubin; Abowd and Woodcock)
How Can You Preserve Confidentiality and Multivariate Relations?
• Fundamental trade-off:– better protection v. better data quality
• Protection results from summarizing the data with a complicated multivariate distribution, then sampling that distribution instead of the original data
• The synthetic data are not any respondent’s actual data
• But, for some techniques, it may still be possible to re-identify the source record in the confidential data
• New techniques address this problem
How Can the Research Community Benefit from Synthetic Data?
• Sophisticated research users must help develop the synthesizers in order to promote and improve analytic validity
• Many more users will have access to the information because there is a public use micro data product.
The Research – Synthetic Data Feedback Cycle
ScientificModeling
ScientificModeling
DataSynthesis
DataSynthesis
ConfidentialityProtection
ConfidentialityProtection
AnalyticValidity
AnalyticValidity
The Multi-layer System
Basic confidential data– Fundamental product of virtually all Census
programs– Leads to the publication of public-use products
(summary data, micro data, narrative data)
Gold-standard confidential data– Edited, documented and archived research
versions of confidential data– Used in internal Census research and at
Research Data Centers
More Layers
Partially-synthetic micro data– Preserves the record structure or sampling frame
of the gold standard micro data– Replaces the data elements with synthetic values
sampled from an appropriate probability model
Fully-synthetic micro data– Uses only the population or record linkage
structure of the gold standard micro data– Generates synthetic entities and data elements
from appropriate probability models
The NSF Information Technologies Research Grant
• A program that encourages innovative, high-payoff IT research and education
• Our grant proposal cited the many research studies and data products created by previous NSF support for the Research Data Center network and the Longitudinal Employer-Household Dynamics Program
What Is It?
• $2.9 million 3-year grant to the RDC network (Cornell is the coordinating institution)
• Provides core support for scientific activities at the RDCs
• To develop public use, analytically valid synthetic data from many of the RDC-accessible data sets
• To facilitate collaboration with RDC projects that help design and test these products
The Quarterly Workforce Indicators
• QWI was the LEHD Program’s first public use data product
• QWI Online
• Detailed labor force information by sub-state geography, detailed industry, ownership class, sex and age group.
The Confidentiality Protection System
• All QWI protections are done by noise infusion of the micro-data
• All micro-data items are distorted at least minimal percentage up to a maximal percentage
• Only the distorted items are used in the production of the release product
Protection and Validity Principles
• Cells with few businesses contributing or with few individuals contributing have been distorted in the cross-section but not the time-series
• Bias in the cross-section is controlled and random, no analyst knows its sign
• More information
Theoretical Distribution of the QWI Distortion Factor
Theoretical Distribution of the QWI Distortion Factor
Actual Confidentiality Protection Distortion: Employment, Beginning-of-Quarter
Table 8: Distribution of Error in First Order Serial Correlation
Graph: Distribution of Error in First Order Serial Correlation
Enhancements
• The current product has suppressions for cells too small to protect by noise infusion
• The enhanced product replaces these suppressions with synthetic data
Percentage of Data Items in County Level Release File
Sector Sub-sectorIndustry
GroupReleased 86.45 75.43 70.08 Not significantly distorted 70.06 58.96 57.22 Significantly distorted 16.39 16.47 12.86Suppressed 13.54 24.57 29.91
Released 100.00 100.00 100.00 Not significantly distorted 70.07 58.96 57.23 Significantly distorted* 29.93 41.04 42.77Suppressed 0.00 0.00 0.00*approximate
Employment (Beginning-of-quarter)
Percentage of Data Items in QWI County-level Release File NAICS IL 2001:1-2004:1
Beginning of Period Employment in NAICS Sector 62
Beginning Period Employment in Naics Sector 52Men and Women Ages 19-21
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
2001-1 2001-2 2001-3 2001-4 2002-1 2002-2 2002-3 2002-4 2003-1 2003-2 2003-3 2003-4 2004-1
Time
Co
un
ts Current QWI
Improved QWI
Full Quarter New Hires in NAICS4 3259
Full-Quarter New Hires in Naics4 3259 Women Aged 55-64
0
2
4
6
8
10
12
14
2001-1 2001-2 2001-3 2001-4 2002-1 2002-2 2002-3 2002-4 2003-1 2003-2 2003-3 2003-4 2004-1
Time
Co
un
ts Current QWI
Improved QWI
The Census Bureau’s First Public Use Synthetic Data Application
• LEHD On-the-map application• Shows commuting patterns at the Census
Block level with characteristics of the origin and destination block groups
• Origin block data are synthetic– Sampled from the posterior predictive distribution
of origin blocks and origin characteristics given destination block, destination block characteristics.
• On-the-map
Where people living in the selected area (Mobile’s neighboring communities of Daphne and Fairhope) work
Source: “On the Map” beta application, Longitudinal Employer-Household Dynamics Program, U.S. Census Bureau September 23, 2005
DRAFT – Beta Test Document OnlyDRAFT – Beta Test Document Only
Where people working in the selected area (downtown Mobile) live
Source: “On the Map” beta application, Longitudinal Employer-Household Dynamics Program, U.S. Census Bureau September 23, 2005
DRAFT – Beta Test Document OnlyDRAFT – Beta Test Document Only
Synthetic Data Model
• yijk are the counts for residence block i, work place block j and characteristics k.
• Characteristics are age groups, earnings groups, industry (NAICS sector), ownership sector.
I
i
yjkijkijkiijkyp
1||| )|(
IjkjkIjkjk yy |1|1 ,...,Dirichlet~
Complications
• Informative prior “shape”
• Prior “sample size”
• Work place counts must be compatible with the protection system used by Quarterly Workforce Indicators (QWI)– Dynamically consistent noise infusion
W1 W2 …. WJR1 2 5 … … 50R2 3 … … 400R3 … … 50R4 90 … … 200R5 … … 100R6 … … 20R7 … … 20R8 … … 20R9 … … 40R10 … … 100
Total 5 95 … … 1000
Residence Block (i)
Work Block (j) Total
Residence Block (i) Prior distribution Likelihood Posterior Expected Counts
Posterior Probabilities Synthetic Data
(Aggregated Work Block Distribution)
(Original Work Block Distribution)
R1 0.050 0.400 2.350 0.196 1R2 0.400 0.600 5.800 0.483 2R3 0.050 0.350 0.029R4 0.200 1.400 0.117R5 0.100 0.700 0.058 1R6 0.020 0.140 0.012R7 0.020 0.140 0.012R8 0.020 0.140 0.012 1R9 0.040 0.280 0.023
R10 0.100 0.700 0.058 1Total 1.000 1.000 12.000 1.000 6
Work block 5Prior 7QWI estimate 6
Analytic Validity
• Assess the bias
• Assess the incremental variation
Census Tract (1)
Workers
(2) Average commute
distance in true data (in miles)
(3) Average commute
distance in synthetic data (in miles)
(4) Difference in miles
(5) Standard deviation
across 10 implicates over (1)
1 6,747 17.9 17.9 0.0 0.019
2 4,535 14.6 14.8 0.1 0.013
3 2,251 18.5 19.3 0.9 0.018
4 1,932 12.0 13.2 1.3 0.043
5 1,996 15.0 15.0 -0.1 0.028
6 2,135 14.3 15.7 1.3 0.036
7 1,809 12.8 13.9 1.1 0.036
8 2,004 8.5 8.5 0.0 0.039
9 1,515 11.8 12.1 0.3 0.021
10 1,365 21.1 23.2 2.0 0.040
11 1,233 16.3 17.4 1.1 0.031
12 879 15.1 16.8 1.8 0.067
13 811 11.3 11.3 0.0 0.072
14 634 10.4 10.4 -0.1 0.051
15 618 9.6 9.6 0.0 0.046
16 526 11.4 10.1 -1.3 0.088
17 531 17.1 18.4 1.3 0.045
18 541 14.4 14.5 0.2 0.063
19 378 15.0 14.4 -0.6 0.069
20 372 7.7 7.2 -0.5 0.069
21 138 7.8 8.1 0.3 0.064
Total 32,951
Size weighted average of absolute difference in commute distance in confidential and synthetic data
Population in Work Block Mean P20 P40 P60 P80
1-5 9.33 4.72 5.72 8.98 13.95
6-10 5.89 1.88 3.13 4.69 8.71
11-20 3.82 1.76 2.42 2.68 4.24
21-50 3.34 1.19 1.76 2.27 3.58
51-100 2.21 0.69 1.55 1.44 2.36
101-250 1.38 0.40 0.65 1.92 2.12
250-500 0.96 0.16 0.38 0.72 1.64
501-high 0.27 0.05 0.12 0.15 0.13
Confidentiality Protection
• The reclassification index is a measure of how many workers were geographically relocated by the synthetic data.
j
I
iijij yyy /)~abs(
1
Panel 1: Reclassification Index for County Residence Patterns
Population in Work Block Mean P25 P50 P75
1-5 0.59 0.00 0.50 1.00
6-10 0.46 0.29 0.42 0.57
11-20 0.35 0.23 0.35 0.44
21-50 0.24 0.17 0.21 0.33
51-100 0.19 0.12 0.16 0.24
101-250 0.11 0.08 0.11 0.14
250-500 0.10 0.04 0.09 0.12
501-high 0.06 0.03 0.04 0.08
Panel 2: Reclassification Index for Census Tract Residence Patterns
Population in Work Block Mean P25 P50 P75
1-5 0.72 0.50 0.67 1.00
6-10 0.49 0.33 0.50 0.63
11-20 0.46 0.33 0.43 0.58
21-50 0.35 0.27 0.33 0.42
51-100 0.29 0.24 0.28 0.33
101-250 0.22 0.16 0.22 0.27
250-500 0.18 0.11 0.17 0.25
501-high 0.14 0.11 0.14 0.17
Panel 3: Reclassification Index for Block Residence Patterns
Population in Work Block Mean P25 P50 P75
1-5 0.85 0.50 1.00 1.00
6-10 0.57 0.42 0.57 0.67
11-20 0.47 0.35 0.47 0.57
21-50 0.37 0.29 0.35 0.43
51-100 0.33 0.28 0.31 0.39
101-250 0.26 0.22 0.25 0.32
250-500 0.25 0.23 0.26 0.27
501-high 0.21 0.19 0.20 0.22
SIPP-SSA-IRS Public Use File
• Links IRS detailed earnings records and Social Security benefit data to public use SIPP data
• Basic confidential data: SIPP (1990-1993, 1996); W-2 earnings data; SSA benefit data
• Gold standard: completely linked, edited version of the data with variables drawn from all of the sources
• Partially-synthetic data: created using the record structure of the existing SIPP panels with all data elements synthesized using Bayesian bootstrap and sequential regression multivariate imputation methods
Multiple Imputation Confidentiality Protection
• Denote confidential data by Y and disclosable data by X.
• Both Y and X may contain missing data, so that Y = (Yobs , Ymis) and X = (Xobs, Xmis).
• Assume database can be represented by joint density p(Y,X,θ).
Sequential Regression Multivariate Imputation Method
• Synthetic data values Y are draws from the posterior predictive density:
• In practice, use a two-step procedure: 1) draw m completed datasets using SRMI (imputes values for all missing data)2) draw r synthetic datasets for each completed dataset from predictive density given the completed data.
dXYpXYYpXYYp obsobsobsobsobsobs ,|,,|~
,|~
Confidentiality Protection
• Protection is based on the inability of PUF users to re-identify the SIPP record upon which the PUF record is based.
• This prevents wholesale addition of SIPP data to the IRS and SSA data in the PUF
• Goal: re-identification of SIPP records from the PUF should result in true matches and false matches with equal probability
Disclosure Analysis
• Uses probabilistic record linking
• Each synthetic implicate is matched to the gold standard
• All unsynthesized variables are used as blocking variables
• Different matching variable sets are used
Percentage of Non-matches, False Matches, and True MatchesTotals for Group1-Group7
70 710 2273 5586 1246335935
108205
10 64 264 665 13636292
25121
3 55 179 462 947 378220746
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 to 9 11 to 61 65 to 208 242 to 501 607 to 918 1149 to 3966 4095 to 31256
cell size categories
False Matches
True Matches
Non-matches
Testing Analytic Validity
• Run analyses on each synthetic implicate.– Average coefficients– Combine standard errors using formulae that take account of
average variance of estimates (within implicate variance) and differences in variance across estimates (between implicate variance).
• Run analyses on gold standard data.• Compare average synthetic coefficient and standard
error to the same quantities for the gold standard.• Analytic validity is measured by the overlap in the
coverage of the synthetic and gold standard confidence intervals for a parameter.
Log Annual Earnings Amount
Synthetic Completed Synthetic Completed Synthetic CompletedIndependent Variables Avg. Coeff. Avg. Coeff. Tratio Tratio DF DFIntercept 8.4372 7.8519 9.7979 7.0765 7.9269 7.7770 14.8082 200.2576 2.8961 6.6134college_only 0.7618 0.7154 1.0119 0.5118 0.8012 0.6295 55.6671 18.4204 0.6308 3.5572disab_nowork -0.8474 -0.8595 0.2248 -1.9196 -0.6510 -1.0679 -1.8829 -9.2725 2.9124 3.3828divorced -0.1131 -0.1327 0.0108 -0.2371 -0.1139 -0.1516 -2.2564 -11.6721 2.6819 121.5500femaleblack -0.4383 -0.4969 -0.3716 -0.5050 -0.4627 -0.5311 -14.0983 -26.7033 3.9099 8.8353femalewhite -0.4384 -0.4359 -0.3645 -0.5122 -0.4225 -0.4492 -14.6838 -56.1920 2.6776 21.7965graduate 0.8479 0.8293 0.9778 0.7180 0.9179 0.7407 75.3638 20.4927 0.7458 3.6625highschool_only 0.2767 0.2344 0.3208 0.2326 0.2638 0.2049 19.5239 15.2380 1.8294 6.5118hispanic 0.0317 0.0784 0.0706 -0.0072 0.0950 0.0617 3.2362 7.7969 1.4043 129.2977maleblack -0.2369 -0.3163 -0.1868 -0.2870 -0.2915 -0.3411 -10.0306 -21.3101 4.0804 58.8818married -0.1007 -0.1027 0.0403 -0.2418 -0.0879 -0.1175 -1.7267 -11.6605 2.8145 41.8875ser_totyears2_2000 -0.008388 -0.013910 0.004399 -0.021174 -0.012314 -0.015506 -1.5559 -17.1739 2.9438 5.5757ser_totyears3_2000 0.000213 0.000346 0.000585 -0.000160 0.000387 0.000306 1.3521 16.2383 2.9609 6.9507ser_totyears4_2000 -0.000002 -0.000003 0.000001 -0.000006 -0.000003 -0.000004 -1.3938 -17.2223 2.9741 8.3544ser_totyears_2000 0.170864 0.270394 0.358130 -0.016402 0.295549 0.245240 2.1708 22.0735 2.9221 4.6084somecollege 0.449987 0.407803 0.925822 -0.025847 0.470358 0.345247 23.9657 14.2315 0.5699 3.6969widowed -0.7116 -0.5511 0.2288 -1.6520 -0.4862 -0.6160 -1.8221 -15.2869 2.8415 10.7620
Table 3: Log SER Earnings in 2000 for all individuals with positive earnings in this year
Confidence Interval Confidence IntervalSynthetic Completed
Log Annual Benefit Amount
Synthetic Completed Synthetic Completed Synthetic CompletedIndependent Variables Avg. Coeff. Avg. Coeff. Tratio Tratio DF DFIntercept 5.2392 5.4950 5.3505 5.1280 5.9115 5.0784 110.6907 30.3282 3.0110 3.1841age_first_benefit 0.0163 0.0119 0.0188 0.0139 0.0190 0.0048 28.2442 3.8653 1.3292 3.1370birthdate 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 -3.4365 -7.5960 3.4749 17.3979black_nonhisp -0.1381 -0.1485 -0.0999 -0.1763 -0.1338 -0.1632 -7.2851 -16.5883 5.0031 808.8552college_only 0.1737 0.1620 0.2139 0.1335 0.1786 0.1455 8.5725 16.2717 5.3961 97.3222disab_nowork -0.0641 -0.0610 -0.0339 -0.0942 -0.0375 -0.0844 -4.3107 -4.7709 4.8339 8.9261divorced 0.0508 0.0995 0.0800 0.0216 0.1238 0.0753 2.9333 6.7859 37.7125 197.0986graduate 0.1953 0.1752 0.2383 0.1522 0.1971 0.1533 9.1176 13.9156 5.0602 16.7543highschool_only 0.0922 0.0903 0.1076 0.0769 0.1092 0.0713 10.9656 9.0304 9.3056 6.9655hispanic -0.1035 -0.1601 -0.0848 -0.1223 -0.1307 -0.1895 -9.7891 -10.0587 12.9980 8.5134male 0.3544 0.3321 0.3648 0.3440 0.3402 0.3239 58.0446 66.8254 29.5818 3339.4344married 0.0063 0.0682 0.0581 -0.0455 0.0902 0.0462 0.2367 5.1394 5.7554 101.4151somecollege 0.1429 0.1372 0.1602 0.1257 0.1539 0.1205 14.5992 14.4611 14.1280 13.8189widowed 0.2047 0.2517 0.2714 0.1379 0.2775 0.2260 6.2358 16.4587 4.8026 37.9058
Table 4: Log Monthly Benefit Amount in December, 2001 for individuals 62 and older in 2000
Confidence Interval Confidence IntervalSynthetic Completed
Tools
• NSF sponsored supercomputer
• Virtual RDC
• Cornell INFO 747
The NSF-sponsored Supercomputer on the RDC Network
• NSF01 is a 64-processor (384GB memory) supercomputer
• Installed and optimized for complex data synthesizing and simulation
• Projects related to the ITR grant have access and priority
The Virtual RDC
• Virtual RDC (news server)• The virtual RDC environment contains
multiple servers that closely approximate an RDC compute server (e.g., NSF01)
• Disclosure-proofed metadata and synthetic data
• Now fully operational• Any current or potential RDC user can have
an account
Cornell Information Science 747
• INFO 747
• Course available to any potential RDC user, on DVD and via internet feed
• Training for using RDC-based data products
• Training for creating and testing synthetic data
Conclusions
• An important and challenging area that social scientists must be part of
• Use of confidential data collected by a public agency carries with it an obligation to disseminate enough data to permit scientific discourse
• Synthetic data is an important tool for this dissemination
top related