IQML overview and Metadata in Codacmos

CES

IQML overviewand

Metadata in Codacmos

Defeng Ma ([email protected])

Centre for Educational Sociology

The University of Edinburgh, UK

mailto:[email protected]

2CES15 October 2003, EuroStat, Luxembourg

Outline

• IQML overview

• IQML QDT (Questionnaire Design Tool)

• IQML SAT (Survey Administration Tool)

• IQML Registry/Repository

• Codacmos project

• Metadata for data collection (WP6 in Codacmos)

CES

Intelligent Questionnaire Markup Language (IQML)Overview


IQML Partners

• University of Edinburgh, UK – coordinator, QDT

• Comfact, Sweden – QPT, DIT, XML specification

• Desan, Netherlands – SAT

• Dimension EDI, UK – standards, repository

• CSO, Ireland – user

• Statistics Norway – user

• National Technical University of Athens, Greece – user

Funding under EuroStat EPROS as part of the EU Funding under EuroStat EPROS as part of the EU 55thth Framework R&D Framework R&D


IQML modules

• QDT(Questionnaire Design Tool): design the questionnaire

• SAT (Survey Administration Tool): manage the survey

• Metadata Repository: storage and retrieval of metadata

• QPT (Questionnaire Presentation Tool): present questionnaire to the user and collection the data

• DIT (Database Interrogation Tool): extraction of data from database to the questionnaire


The IQML Processing/Data Flow

Surveyinfo

Surveyinfo

RespondentRespondent

DatasetDataset

qnaireqnaire

IQML QDT IQML SAT

SAT IQML QPT/DITQPT/DIT

IQML Metadata Repository

MetadataRepository


System Structure

APIAPI Metadata

Repository

Questionnaire Designer

API API

DatabaseIntegration Tool

SurveyAdministration

Questionnaire Presentation

QuestionnaireXML

Common Model

•Questionnaire presentation (QPT)

•Database interrogation (DIT)

•Shareable metadata

•Published questionnaire

Interoperability, Technical Infrastructure and Standards

MetadataRepository

Web Application

•Questionnaire designer (QDT)

•Survey administration (SAT)

•Externalisation (e.g.IQML XML form)

ApplicationLocal Storage

EEG6/WG4 adopted (as basis for XML4DR)

Syntax is XML

Storage

API

Repositpry

API

Web Servlet

C++, Java Script, Active X

Java

Common modelQuestionnaire

id : String

QuestionnaireUML Model in QDT

Set, Tuple, Data, Format

order of Data must be same as order or Domain

Code

codeValue : String

Questionnaire

id : String

Domain

domainId : String

Set

setId : StringsetType : Stringstate : String

Rendering

renderingType

CodeList

codeListId : String

Data

dataId : String

Tuple

tupleId

DataType

type : String

Decimals

decimals : String

Format

formatId : String1..n

1

1

0..n0..n

<<Ordered>>

0..1

1

1..n

1

0..n0..n

1

0..1

0..n1 0..n

0..n

0..n0..n

<<Ordered>>

0..n0..n

0..10..1

1 0..11

0..n

0..1

0..n

0..10..1

0..n

0..10..1

0..10..1

IQML Common Model used in MR


Models mapping forInteroperability

• QDT model -> IQML common model (High level only)» Questionnaire, questionnaire body -> Questionnaire

» Section, Question -> Set

» Response Group -> Domain

» Variable Group (Sub question) -> Tupel

» Variable -> Data

» Member of Response Group -> Code List, Code

• This mapping is used to publish questionnaire from QDT to Metadata repository


Mapping to external standards: DDI (in Cosmos project)

Surveyinfo

Respondent

Dataset

qnaire

IQML QDT

IQML SAT

IQML QPT/DIT

IQML Metadata Repository

FasterNesstar publisher

CosmosRegistry

Catalogue.xml

Catalogue.xml Searching

URL of DDI object

FasterSurvey

FasterNesstarServer

(UKDA)

Publishing

QnaireDDI

UEDINWeb

Server

SurveyDDI

DESANWeb

Server

CES

QDT: Questionnaire Design Tool


Philosophy of QDT

question sectionquestionquestionquestionquestion

sectionsectionsectionsection

NotesGroup

elementsNotesNotesNotesNotes

Group elements

Group elements

Group elements

Group elements

DefineQuestion type

Create basic components

Question bank

Assemble questionnaire

Publish questionnaire


Defining Question Type

Question type defines • data type (text / integer / real / boolean)• response type (simple / multi / choice)• sub-questions (yes / no)


Main QDT components

• Question type

• Question bank

» Element groups

Response Group

Missing value group

Sub-Question group

» Note

» Question

» Section

• Control Elements

» Calculation

» Validation

» Navigation Rule (‘Dynamic Guidance’)

• Questionnaire


QDT – complex question

Sub-question group

Question text Response group

variablevariable group

CES

IQMLSurvey Administration Tool


• Sampling import population database

select sample(s)

• Survey definition questionnaire (QDT -> Reg/Repository -> SAT)

sample(s) (SAT -> Reg/Repository -> SAT)

• Distribute survey• Monitor response• Send reminders

Functions


Sample & Survey Process

Questionnaire(QDT)

txt xsl

Pop

sample Sample Survey

(RR)


Distribution & Monitor Process

Survey

Message

•Html generation•Prefill•Distribution

Response(QPT)

Monitoring

Reminder

CES

Registry/Repository


IQML Reg/Rep

• ebXML standard

• Web servlet Interface with Tomcat web server » Remote accessible

• Administration GUI

• Not directly visible for the user, the QDT and SAT can communicate through the Reg/Rep

• IQML common model for storing objects in the repository


Contact Details for IQML

Defeng Ma ([email protected])

http://www.epros.ed.ac.uk/iqml


IQML -> Codacmos

• IQML is mainly focus on the primary data collection based on the web/email survey (Trails in SSB, CSO and NSSG)

• Codacmos (Cluster of Data Collection Integration & Metadata Systems for Official Statistics ) is a cluster of relevant projects» TELER,

» DATAMED,

» IQML and

» IPIS

CES

Codacmos projectsand WP6 (Metadata)


Codacmos: Goal

• Main Goal is to identify ways » improve the cost-effectiveness of the data collection

process and the quality of official statistics, and» lowering the statistical burden of the respondents


Codacmos: Workpackages

• WP1: Management and Coordination

• WP2: Integration of available results

• WP3: Primary data collection

• WP4: Secondary data collection

• WP5: Integration of primary and secondary data collection

• WP6: Metadata system for data collection

• WP7: Relevant issues for FP6

• WP8: Assessment, Evaluation and Peer review

• WP9: Dissemination and Publication

CES

WP6

Metadata system for Data collection


Data collection

NSI

Market ResearchInstitute

TaxOffice

Respondent

Respondent

Respondent

Respondent


Data sharing

Data Provider

Data Provider

Data Provider

NSI

EuroStat

World Bank


Optimum solution?

Virtual Metadata

Repository

Input OutputData Provider

Data Provider

Data Provider

NSI

EuroStat

World Bank


Scope of the WP6 model

DisseminationProductioncollection

DataDefinition of statistics

Statistical production process*

Focus on Data collection, Thinking of the whole processBut: where to stop?

* Picture from H. Rouhurirta: “On the structing of statistical Information”, First Metanet conference, 2001


Metadata for data collection

Primary Data Collection

Secondary Data Collection

Integration of primary and secondary

Processing metadata

StatisticalData

Content Metadata

• Metadata for definition statistics (survey, questionnaire, statistical unit, etc.)

• Metadata for production (disclosure, quality control, constrains, etc.)

Additional:


Looking back …

• State-of-art review

• Existing systems/models review and analyse

• Study» Similarity/Common elements

» Difference

» Pro/Con

• Archive the common understanding of the requirements/user needs


Top-down approach for Common metadata model

• Develop the Abstract model at top level first to cover the common aspect of the metadata for data collection process (cover the large scope)

• Then go down in the low levels with more details of the metadata models (few examples) as far as we can:» Statistical data (content)

» Processing

» Constraints

» Survey/questionnaire

» …


Working with the top down

• Generalise the data collection process

• Abstract metadata model for data collection

• Define the level of the metadata

• Define the category of the metadata

• Define the exchange model of the metadata

• Define common terminology


The abstract model

• Keep the model» Generic

» Simple

» Clear


Bottom-up approach: Start with what we have…

• Metadata in the view of different aspect» Primary/Secondary data collection

» Organisation/International

» Enterprise/individual

» Economical/Social

» Administrative/statistical data

» Centralised/distributed

» Register/Survey

» Questionnaire/processing/transformation, etc.


Integration/Combination of top down and bottom up

• Mapping existing models to common model

• Identify the tasks for integration

• Outline the working direction/guideline for improve the existing systems

• Refine the common model

• Specify/define the demo experiment field and strategy of the demo


WP6 work plan

• Before 1st TM: » preparation, collection, review of state-of-art

• 1st TM (Sep. 2003): » Get common understanding on the target and tasks

» Detailed working plan

» Draft structure of the common model

» Initial discussion of the Integration analyse

• After 1st TM: » common core model

» integration analyse of the existing system

» Demo definition and specification


Work plan (cont.)

• 2nd TM (Jan. 2004)» Finalize common model and integration of existing system

» Demo preparation

» Workshop preparation

» Deliverables preparation

• After 2nd TM» Workshop

» Demo

» Deliverables


Codacmos website

http://www.codacmos.eu.org

Project coordinator: Alberto [email protected]

Contact Author: [email protected]

IQML overview and Metadata in Codacmos

Documents