Combining data through standards and metrics by Mike Thacker

COMBINING DATA THROUGH STANDARDS AND METRICS

Mike Thacker – Porism Limited

Open Data

Combining data through standards and metrics

2

WHAT WE MEAN BY STANDARDS

Combining data through standards and metrics 3

Making local data useful

• Municipalities need to be compare with one another

• Innovators need to build apps cost-effectively

Hence we offerstandards for:

• what we call things

• format of open data

Defining datasets and linking via URIs

Datasets

Schemas

Data

items

Def

ine

stru

ctu

re o

f

Co

nta

in

Local authorities

Official geographies

Neighbour-hoods

Services grouped by function

Other eg:

• Planning categories

• Entertainment types

UNIFORM RESOURCE IDENTIFIERS (URIS)

Combining data through standards and metrics 6

URIs - Uniform Resource Identifiers

Codes that give precise definitions of things, eg:

• http://id.esd.org.uk/service/860 – premises licence service

• http://opendatacommunities.org/id/unitary-authority/yorkYork City Council

• http://statistics.data.gov.uk/id/statistical-geography/E06000014 York area - from ONS

These normally resolve to descriptions with properties that are human and machine readable

Finding a URI - uris.opendata.esd.org.uk

What we know about each service

Service

Web page for each English council

URIs resolved - service

standards.esd.org.uk

STANDARDS FOR DATA FORMATS - SCHEMAS

Combining data through standards and metrics 11

The inventory schema

• Indexes datasets & their schemas against functions & services

• Automatically harvested by data.gov.uk

• Automatically output by DataShare DKAN and CKAN following

• Can be uploaded to and validated by esd-toolkit

Inventory

Dataset Documents

ODF, PDF, HTML

Data

CSV, XML, ...

Dataset Documents

ODF, PDF , HTML

Data

CSV, XML, ...

inventory.esd.org.uk

Dataset schemas

• Define the structure of data for a service or function (group of services)

• Shared to allow (not mandate) consistency

• Some validated schemas encouraged

• Formats:

– Tabular: DataShare definitions and CSV validation files as used by the ODI’s csvlint.io

– XML

– Linked data profiles

CSV Checker - http://csvchecker.opendata.esd.org.uk/

presentatienaam 14

Inventories of datasets

Combined index of local datasets

Datasets from many municipalities using the same schema

One feed of all aggregated CSV datasets

METRICS – STANDARD MEASURES

Combining data through standards and metrics 18

Metric types

See here

Policy Policy Metrics Services

Increase healthiness / quality of life

ObesityPsychiatric illnessCardiac illness

Dietary advice, School mealsGreen spaces,Recreational facilities

Increase economic activity EmploymentStreet crimeEducational attainment

Careers advicePolicing, CCTVSchooling, Adult education

Safer roads Road accidents Traffic control, Signage

Metrics

Policies Servicesdetermine

Evidence-led policy

Metrics structure

Metric value

Date/Time

Dimensions Area

Organisation

Examples• Miles of roads• Road accidents• Spending on roads

Example• Norfolk County

Council

Example• Norfolk administrative area• East of England• England

At a point or

Over a period

Examples• Age• Gender• Severity

Metric type

2012 6,142.5

183,003.4

2010-12

353

2,47324,027.8

21,534

£75.3M

EoE £573.2M England £208.8M

2011/12

has

has has has

has

gove

rns

Example data analysis – Comparing Municipalities

Full report: Live PDF

Example data analysis – Comparing areas within a municipality

See here

API with RESTful web methods

See here

Further information

• UK local government open data: http://opendata.esd.org.uk/

• Standards: http://standards.esd.org.uk/

• Metric types http://id.esd.org.uk/list/metricTypes

• Dimensions (Circumstances) http://id.esd.org.uk/list/circumstances

• Municipalities http://opendatacommunities.org/data/local-authorities

• Administrative areas http://statistics.data.gov.uk/doc/statistical-geography/

• Natural neighbourhoods: http://neighbourhoods.esd.org.uk/

• Tools http://about.esd.org.uk/

• API: http://api.esd.org.uk/

[email protected]

@MikeThacker