DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality Interoperability Approaches, case studies and open issues DL.org Quality Working Group Rome, 28 th.

DL.org All WGs Meetings, Rome, 26-28 May 2010

Quality InteroperabilityApproaches, case studies and open issues

DL.org Quality Working Group

Rome, 28th May 2010


Quality comprehensive models

Fuhr et al., 2001

Digital Libraries: A Generic Classification and Evaluation Scheme


Quality comprehensive modelsGoncalves et al., 2006What is a good digital library? A quality model for digital libraries


Quality comprehensive models

Zhang, 2010

Holistic DL evaluation

model


Quality comprehensive modelsCandela et al., The DELOS RM Quality concept map, 2008Annotations by the DL.org Quality WG


Quality

Quality is subjectiveQuality is dynamic

Quality is vagueQuality needs policies

…


Quality WG meeting results

• Our motivating scenario: consider that representatives of two (or more) DLs have a round table to negotiate a service level agreement (SLA) defining their interoperability requirements and for this establish a quality threshold that each individual DL has to meet or exceed; “Quality” would provide transparent qualitative or quantitative parameters for defining the threshold

• Our approach is practical: Quality Interoperability Survey, Quality scenarios

• The Cookbook TOC• The interoperability scenarios 1&2


Quality Interoperability Survey

• Survey Pilot questionnaire + results• Simplification and improvement• Disambiguation (Glossary)• Collection strategy• Data analysis and interpretation• Results expected by June 2010• Best practices


The Cookbook

• Definitions of quality• Context: how the Quality WG faced the

investigation challenges• Consistent terminology

Introduction


The CookbookLooking at Web solutions at a

technical level

FRONT END: Web Interfaces (usability, accessibility), eg. W3C

BACK END: Web services ontologies (QoS), eg. FIPA, WS-QoS, MOQ, DAML-QoS


The CookbookQuality insurance procedures

for specific DL systemsSemantic and organisational

levels• Document repositories (eg. DINI, DRIVER)• Research data archives (eg. Data Seal of

Approval)• Preservation systems (eg. TRAC,

DRAMBORA)↓

Template for comparison within each class


The Cookbook – Case studiesTemplate

Aspect DINI Certificate

DRIVER Guidelines

Explicit quality policy for protocol and metadata implementation Yes Yes

Explicit policy for operations (personell, support etc.) Yes No

Personal quality check (questionaire, on-site review) Yes No

Intellectual quality check (remote) Yes Yes

Automatic self validation No Yes

Organized through sustainable Organisation DINI COAR

Explicit branding when checked Yes No

Translation in English, Spanish, Portuguese, Japanese No Yes

Green and Gold Yes No

Strictly full-text oriented Yes Yes


The CookbookBest practices from the

professional community

The results of the Survey will be included as best practices from the professional community. We are aware that quality is subjective, that we are dealing also with two “primitive” interoperability challenges

1. researchers vs professionals 2. different disciplines involvedBut we want to know from DLs people!


The CookbookThe Quality WG checklist

Light-weight document – based on the Quality Core Model parameters - with practical

recommendations from the DL.org Quality WG based on the QCM parameters and the

Quality Interoperability Survey results


Interoperability ScenariosScenario no. 1 – Quality and

Functionality issues Metadata Evaluation Formats quality Geographic referencing (Content) and related pinpointing

issues (Functionality) Controlled vocabulary + related Functionalities issues

(search, browsing, etc) Multilingualism Quality of the annotations (User profiling, User

authentication) Sources evaluation, i.e. Provenance


Interoperability ScenariosScenario no. 1 – Policy issues

Copyright issues Reuse issues Content policy issues

Possible solutions: change the copyright policy, use creative commons license

Suggestions: review the scenario and make it consistent (logical incoherencies on map reproduction, restored video, 3D Model regeneration), less chaotic and more realistic


Interoperability ScenariosScenario no. 2 – Quality issues

• Merged query results• Machine translations for supporting two languages• Articles quality (one has editorial reviews the other has not)• Mark-up of content• Metadata evaluation: Accuracy and Completeness• System performance• Technical correctness


Interoperability ScenariosScenario no. 2 - Approaches

• Two localizations, one for each country, each supporting one language for content from both systems (very difficult!)• Functionalities of both source DLs (i.e. union of), including: access control,

search, results views • Metadata quality: solved selecting targets, customizing

search screen based on targets e.g. which common metadata fields they support• Search QoS: solution in the user

interface design to indicate how far along the search is and cluster the results

• Ownership of images: search results must contain author, photographer, agency and rights metadata so the user can see if they need to pay, who they pay, and what rights they get


Some conclusive thoughts

• Quality: dynamic, subjective, not completely definable

• Provenance = the resource story = how to establish quality

• DL systems and their implementation teams will always struggle with quality, but users will have the last word


THANK YOU

DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality Interoperability Approaches, case studies and open issues DL.org Quality Working Group Rome, 28 th.

Documents