Open data and Collaborative Governance (the UW lecture)

Open Data and Collaborative Governance: Perspectives and Research Challenges

Yannis CharalabidisAssistant Professor, University of the AegeanHead of Research, Greek Interoperability Centre

University of Washington, Seattle, 3rd December 2012

Your speaker for the dayStudied computer engineering, at the National Technical University of Athens. PhD in complex information systems, NTUA

7 years a researcher in RTD projects for businesses and governments

7 years in the software industry (Greece, Netherlands, Germany Poland). Managing director of Baan-Singular ERP company

Already 5 years in University of the Aegean and the Greek Interoperability centre, teaching and researching on eGovernance. The next 7 years ?

My aim for the day: to give you food for thought.

Hold on …

Activities Research in Greece and European Union

(FP7/ICT, CIP/PSP, e-Infrastructures, REGPOT, LIFE, INTERREG, Greek CSF/RTD programmes)

Industry-Academia programmes and projects (Student practice, industry-oriented theses, PhD research, targeted research)

High-level, innovation-oriented consulting for Governments, and Businesses worldwide (typically in partnership with industry and other institutions)

Scientific global-scale events organisation (WeGov Awards, The Samos Summit, Aegean Start-Ups)

Dissemination and Training activities

Areas of Expertise1. Unified Process and Data Modelling methodologies with

emphasis in collaborative process modelling, advanced CCTS-based XML modelling, business process management, simulation methods and tools

2. Interoperability Standardisation and Application Frameworks, including National Standardisation Frameworks for businesses and governments, interoperability testing and demonstration platforms

3. Service-Oriented Information Systems for Businesses and Governments in Local, National and European level, including Electronic Services Portals, eGIS, eSCM, Service Registries and middleware components

4. Web 2.0 technologies for participative services, including mashups, social networking applications, enterprise 2.0 applications

5. Electronic Governance models and systems with the use of ontological representation and federated repositories for policy modelling, argumentation support, knowledge visualisation, legislation management

6. Skill management and educational material development for public / private sector training on eBusiness, eGovernment, SOA and Interoperability

SINTEF

FhG-FOKUS

NCC

TELIN

EPFLBoC

ALBANY Univ, US

USC, US

GIC

Collaborating Centres of Excellence in eGovernment & eBusiness

Countries with user organisations

NIST, US

VUBI-VLAB

GIC International Network

CNR

SyriaIsraelPalestine

UNINOVA

UPV

SAP

An exerciseThere is a photo of the class in

twitter

Can you retrieve it ?(search for #UWopendata or @yannisc)

Then, you can post online questions in twitter using #UWopendata

ON OPEN DATA

Open data is the idea that certain data should be freely available to everyone to use and republish as they wish, without restrictions from copyright,patents or other mechanisms of control. The goals of the open data movement are similar to those of other "Open" movements such as open source,open content, and open access … (wikipedia)

Why is Open Data important ?Organises public knowledgeLeads to better, new servicesFights against corruptionSupports transparency Can motivate citizens Can contribute to better democracy Gives data to other sciences Gives ideas for start-ups

Can you give us some good examples ?

Organises public knowledge : data.gov (UK)Leads to better, new services : data.gov (US)Supports transparency: diavgeia.gr (GR)Can motivate citizens: toronto.ca (CA)Fights against corruption :

ipaidabribe.com (IN)Can contribute to better democracy:

opengov (GR) Gives ideas for start-ups:

Open Data Institute (UK)Provides data to science for solving complex

problems of the society: ENGAGE (EU)

The ENGAGE EU project on Open DataA European e-Infrastructure, for advancing

open data provision across countries and scientific communities, to solve complex societal problems

To provide state of the art methods and tools for data gathering, curation, publication, maintenance

A public-private partnership of research (Greek Interoperability Centre University Aegean, TU Delft, Fraunhofer FOKUS) industry (Microsoft, IBM, Intrasoft intl) and administrations from 5 EU countries

www.engage-project.eu

The ENGAGE “Two-way” Open Data Usage Scenarios

Delivering Public Sector Data to Researchers and Citizens

Delivering Open Data Needs and guidelines to Public Sector Organisations

An Open Data Platform generic architecture

User Interface

Application Interface

(for systems)

Various Apps(PC &

mobile)

Data Curation

(annotation, linking, formats)

Data Visualisation Data Linking

Data Acquisition

UI

Data Acquisition

API

Directories of sourcesAcquisition

Processing

Provision

Open Data Platform architecture

Providing PSI to research communities and citizens in a personalised manner

Curating, Annotating, Harmonising , Visualising

Gathering data from governmental organisations and systems (the Gov Cloud)

Data LinkingData Linking Semantic AnnotationSemantic Annotation AnonymisationAnonymisation HarmonisationHarmonisation

Visualisation - Analytics

Visualisation - Analytics

Search and Navigation tools

Search and Navigation tools

Social sciencesSocial

sciences

Data Service Provision Infrastructure

Data Curation Infrastructure

Public Sector Information Sources

Tailored data services

Research and Industry Research and Industry Governance and

policy makingGovernance and

policy makingCitizens and education

Citizens and education

Data analytics

Data analytics

Knowledge / Data Mining Knowledge / Data Mining Directory services

and direct linking to data archives

Directory services and direct linking to

data archives

ICTICTCitizensCitizens

Natural Sciences and Engineering

Natural Sciences and Engineering GovernanceGovernance

User groups

Collaboration / Communities

Collaboration / Communities

PersonalisationPersonalisation

Single point of Access

Single point of Access

Data QualityData Quality Knowledge MappingKnowledge Mapping

Public Organisations, Repositories, Databases

LawLawPolicy

Modelling Policy

Modelling

Automatic curation algorithmsAutomatic curation algorithms

The Global reach of ENGAGE

Challenges for Open Data PlatformsMetadata schemas “2.0”: automated filling &

self classification, multiple levels of abstraction for different user groups

Develop auto-calculating new, metrics for open datasets: semantic closeness / distance, linking possibility, data quality will allow for automatically linking open data (A-LOD)

Full API and SaaS operation: automated input and publication of open data “from the source”

Novel ways of visualisation for open / linked data

Build ecosystems around open data, for sharing and usage that can make our lives better, for real

ON METADATAThe term metadata is ambiguous, as it is used for two fundamentally different concepts (types). Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at design time the application contains no data. In this case the correct description would be "data about the containers of data". Descriptive metadata, on the other hand, is about individual instances of application data, the data content.

Unlocking the Open Data Vault:The “Key” is Metadata

Metadata provides the means for discovery of relevant datasets

Metadata provides the context for understanding the datasetMetadata provides the restrictions on use of the dataset:

rights, possibly costsMetadata provides the access to the datasetMetadata can assist in the further processing of the dataset(s)

by providing information on data syntax (type, structure) and semantics (meaning)

Metadata can record provenance (what has been done to the dataset)

Metadata can record information for digital preservation to assure the future existence of the dataset

Metadata can record user reaction to datasets: quality, utility

PSI Metadata IssuesConventional metadata for PSI

(data.gov.xx) is:• Flat (lacking structure)• Inadequate for describing the context of

the dataset• Inadequate for software processing of the

dataset• Inadequate for scientific use of open data• Inadequate for automating linking• Inadequate for automating visualisation

• But ... suitable for initial discovery

PSI Metadata• In ENGAGE we shall provide:

• Much more detailed metadata• With formal syntax (structure) and declared

semantics (meaning)• From the world of research information• Congruent with the EC e-infrastructure and

associated projects

• Within an architecture allowing the end-user to• Use conventional PSI browsing and query

• Semantic web / linked open data /Simple metadata• Access to datasets and limited processing / visualisation

• Or use information system query, reporting, analysis, visualisation, simulation• Rich metadata / Full range of relational processing

We will try in the next slides to show you what is the level of expectation

from metadata handling from a 2nd generation open data system

Imagine you are in front of the ENGAGE system, and you have your URI from a dataset, somewhere in the

cloud,(copied as string in the clipboard)

And begin …

Prescreening: User only gives URI of the dataset

Enter (paste) the URI of your dataset

 _

(then for 30 seconds you see this screen, changing)

Progress of ENGAGE Resource Prescreening:

( 45% ) of jobs completed

Managed to :

Identify xls file

Autofill, provisionally: Title

Autofill, provisionally: Creator

Create unique ENGAGE URI

Calculate keywords

Autofill, provisionally: keywords

…

…

(When finishing import, the report)

Report

 ENGAGE managed to automatically, provisionally fill in ( 21 ) of 43 metadata attributes for your dataset.

Your current validity is at ( 45% )

For your dataset to be inserted in the database, you need to continue filling in ( 5 ) mandatory attributes.

Your dataset will then be inserted with validity ( 55% )

If all ( 17 ) non-mandatory attributes are filled in, validity will be maximum, at 70% / limit of the insertion phase.

Please select next action: Continue Park Cancel

After import …

… and then, we enter the metadata insertion page with

pre-filled data, etc.When we finish, we get a similar

final report.

When all metadata fields are filled-in, we can ask all types of queries for open data, at an international

scale

As a conclusion on open data …

Open data, collaborative governance and ICT will be key pillars of the new, value-based administration in this century

Open data and applications can play an important role for entrepreneurship and development

European Union member states, having already adopted a collaborative governance example, can now partner and work together with Gov 2.0 initiatives internationally

In the Greek Interoperability Centre and the University of AEGEAN we can leverage on European experiences and best practices, delivering them worldwide

ON COLLABORATIVE GOVERNANCECollaborative governance is a process and a form of governance in which participants (parties, agencies, stakeholders) representing different interests are collectively empowered to make a policy decision or make recommendations to a final decision-maker who will not substantially change consensus recommendations from the group

Society: increasingly interconnected, flexible, fast-evolving, unpredictable

Governance: often silos-based, linear, obscure, hierarchical, over-simplified

Policies, Disciplines and Actors are isolated

The Problem: Gap between Society and Governance

Policies Health R&D Social

Disciplines Economics Mathematics ICT

Actors Government Citizens Industry

"The problems that we have created cannot be solved at the level of thinking

that created them"

Albert Einstein

So ?

We need three axes to move along:

More people involved (collaborative governance)

More accurate and analytical, modeling and simulation tools

More data available (open data)

2020

2010

We need to formalise ICT research for governance, and pursue major pillars (the Grand Challenges)

Web Technologies

Social Informatics

Systems & Services Technologies

Management Tools

We need multidisciplinarity: the 4 domains that need to

converge

Web 2.0Argument Visualization

Mixed Reality Pattern Recognition

Serious Games

Electronic ParticipationTranslation Systems

Social Networks

Behavioral ModellingSocietal ModellingSocial Simulation

Public Sector Service Systems Workflow Systems

Enterprise Resource ManagementCloud computing

PS Knowledge ManagementLegal Structures Management

Business IntelligenceData & Opinion Mining

SimulationForecasting - Backcasting

OptimizationSystems Dynamics

Adaptive Models

“Hard”

“Soft”

Society Administration

The eGovernance Research Hype Curve

Service Delivery Platforms

Mobile Government

Online Opinion Mining

Instant, proactive Service Delivery for all services

eVoting

(Automated) Argument Visualisation

Federated eID

Gov Cloud (SaaS)

Science Basefor ICT-enabled Governance

Gov Cloud (PaaS)

Social Media in Policy Making

Semantic Interoperability

Agent-based Societal Simulation

eParticipation

Model-Based Decision Making

Visibility

Inflated Expectations Disillusionment Productivity

Linked Data

Visual Analytics

Legal Informatics

Service Co-creation / Web Services for all basic services

Open data

ICT-enabled historiography

Gov Cloud (IaaS)

Technical Interoperability

Organisational Interoperability

Web Services /SOA in core registries

Governance Model Composability & Reuse

(Seamless) Identity management & trust mechanisms

Internal, Static Workflow Mgt

Dynamic, External Workflow Mgt

Available for application

Should be around, soon

Will take many years

Government Service Utility

Serious Games for Governance

Participatory Sensing / IoT

Open Source Software forService Mgt

MunicipalityERP

Readiness, over time

Back to reality: Our current projects on ICT-enabled Governance

PADGETS: Policy Making through Social Media Interoperability www.padgets.eu

ENGAGE: Open, Linked Governmental Data for scientists and citizens www.engage-project.eu

NOMAD: Non-moderated opinion mining (the opinion web) www.nomad-project.eu

CROSSOVER: New horizons in ICT-enabled governance www.crossover-project.eu

As a conclusion

We need a totally different set of tools for evidence-based decision making by governments

Societal Simulation, Data and Opinion Mining, Service Co-creation will be the next “big things” for governments that wish to make a difference

We need to go beyond pure ICT approaches and embark in a multi-disciplinary journey. That’s why we need a science base for ICT-enabled Governance

But most importantly …

Stay tuned at:

Mail: [email protected]

Twitter: @yannisc

Blog: t-government.blogspot.com

Site: charalabidis.gr

eGovernance Research is about our children’s future:

It is not enough to “do the things right” … we have to “do the right things”