NEW DIRECTIONS in DATA COLLECTION and ANALYSIS

1

NEW DIRECTIONS in DATA COLLECTION and ANALYSIS

2014 IAOS ConferenceDa Nang, Vietnam

Hermann Habermann

The Role of Official Statistics

2

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity. . .”

3

Current Model - Strengths

Survey Sampling Is a Well-Established Scientific Method

• Probability Based/Generalizable• Error Structure• Time Series

4

Current Model - Strengths (Cont.)

• Founded on Idea that Information is Owned by Individual/Permission Needed

• Publicly Developed Rules for Privacy and Confidentiality

• Accepted as Gold Standard

• Enabled International Comparisons

5

Current Model - Weaknesses

• Sampling is a Mature Technology– Seminal paper on Theory of Estimation Based on

Confidence Intervals by Neyman in 1934– Turning Theory into Practice by Hanson and

Hurvitz in 1943 – multi-stage sampling• Government Budgets Are Under Attack– Official statistics are often viewed as too costly– Not timely– Difficult to recruit

6

Current Model – Weaknesses (Cont.)

• Administrative Records – Public and Private Are Appealing

• Census Becoming Increasingly Problematic

• Production of Small Area Data Costly and Complex

• Response Rates Are Falling

7

Falling Response Rates

8

Summary of Current Model

• While Still the Gold Standard

• There Are Continuing Demands for

– Data collection that is cheaper and more timely

– Data analysis that is more relevant and at smaller

areas

• Transformative Movements Exist

9

Three Transformative Movements Are Changing the Future

• Control of Data by Corporations

• Rising Influence of NGOs

• Population Movement Toward Cities

10

Control of Data by Corporations

• Corporations Always Interested in Collecting and Analyzing Data about Individuals

• Were Limited by Computer Power, Storage Costs, and Analytic Techniques

• Technological Change – Allows for greater data collection– Greatly reduces the cost of storage– Increases modalities of data collection

• Wearables, smart phone integration, use of nanotechnology

11

Examples

• Reduction in Storage Cost Perhaps Most Important– $450,000 per gigabyte in 1980– $.05 in 2013

• If air travel improved at same rate since 1980, a 16 hour flight would now take 5 milliseconds

• Google Corporation Estimated to Store About 10 E or One Million Times the Text Stored at the US Library of Congress

12

Examples (Cont.)

• Collecting, Aggregating, and Brokering Personal Data

• Second Largest US Company in this Field, Acxiom– 23,000 computer servers – 50 trillion data transactions per year– 1,500 pieces of data per consumer

• Data from Publicly Available Records• Online Behavior Tracked by Cookies, Browser

Advertising• Survey Data

13

Examples (Cont.)

• Over 1.2 billion Facebook Monthly Active Users1

• Facebook Makes Its Own Rules– Recent manipulation

• Weibo Has over 500 Million Users

http://www.statista.com

14

Second Transformative MovementRising Influence of NGOs

• NGOs Still Controversial but Significant in Large Parts of the World

• NGOs Are More Nimble – Do Not Have to Satisfy a Government Constituency

• More Easily Integrate Data Estimates and Analysis

• Computing Power and Software Tools Available

15

Alternative Producers of Official Statistics?

• Pew Foundation

• Gates Foundation - Financial Inclusion Tracker

. Surveys in Uganda, Tanzania and Pakistan

• Monster.com/Google/MIT - Billion $ Price Project

• ADP Employment Estimates

16

Alternative Producers of Official Statistics? (Cont.)

• Alternatives Are– Cheaper– More timely– Small area data– Deemed “Fit for Use”

• Are Official Statistics Still Official?

17

Third Transformative MovementPopulation Shifts to Cities

• In 1960 34% of Population Lived in Cities

• In 2014 it was 54% and

• In 2050 projected to be 66%

UNFPA/Forbes

18

Why Concentrate on Cities

• Policy Officials Focused on Building Smart Cities– Singapore/India project

• Engines of Economic and Social Growth

• Have the Critical Density to Facilitate the Investments in Multiple Sensors

19

New Cities Will Be Smart

• Creating Transport Systems to Avoid Congestion

• Construction Using Low Energy Housing Materials

• Extensive Use of IT Systems to Run Various Urban Functions and the Provision of E-Government Services to its Citizens

20

Example City Projects

• New York City uses data to remedy a wide range of issues affecting short and long term issues — from commuting times to emergency preparedness to air quality.

• City Projects Bring Together Computer Scientists, Social Scientists, Modelers, and Statisticians

Center for Urban Science and Progress cusp.nyu.edu

21

Research Directions

• Developing the Science of Big Data –Integrate Mobile Technology, Embedded Sensors, Blogs, Social Media, and Location-Based Tools, with Traditional Sources of Data

• Future Cities Laboratory Focuses on Sustainable Urbanisation

The Social and Decision Analytics Laboratory at Virginia Tech, Singapore-ETH Zurich Center

22

Challenges for NSOs

• NSOs Are National, Corporations International• Working with Users on “Fitness for Use”

• Integrating Corporate and NGO data

• Incorporating Terabytes of City Data

• Integrating Computer Scientists, Social Scientists, and Statisticians

23

Initiatives

• Examples– Indonesia Using Twitter to predict commodity prices– Australia looking for evidence of the use of tax havens, and

data-matching to identify small online retailers that are not meeting their compliance obligations

– Universities building consortiums to develop big data science– Use of commercial credit card data in GDP– Estimation using unstructured data e.g. Facebook

• International Meeting Programs Address Many of These Issues

24

Challenges for International Agencies and Societies

• Creating a Neutral Big Tent• Minimizing the Gap between Developed and

Developing Countries • Engaging With Existing Efforts to Define

Privacy/Confidentiality

• Supporting a Unifying Approach

25

Suggestions for IAOS

• Support and Disseminate Good Practices• Develop Programmes for UN Statistical

Commission • Insure Representation at International

Activities– Develop privacy and confidentiality policies

• Develop a Continuing Presence

– Create a semi permanent position

26

“It is a far, far better thing that I do, than I have ever done…”